From Xugglewiki

Contents 1 Background Research 2 Purpose 3 Architecture 4 Work Required 4.1 Xuggler Java-Web Start version 4.1.1 Advantages 4.1.2 Disadvantages 4.2 Xuggle Screen Capture Product 4.2.1 Advantages 4.2.2 Disadvantages 4.3 Xuggler Windows 64-bit Version 4.3.1 Advantages 4.3.2 Disadvantages 4.4 Xuggler Mac-OSX 32-bit Version 4.4.1 Advantages 4.4.2 Disadvantages 4.5 Add Codec-Specific Option Support to FFmpeg 4.5.1 Advantages 4.5.2 Disadvantages 4.5.3 Alternates 4.6 Add Format specific option support to FFmpeg 4.6.1 Advantages 4.6.2 Disadvantages 4.6.3 Alternates 4.7 JxCapture Integration 4.7.1 Advantages 4.7.2 Disadvantages

Background Research

See Xuggle_Screensharing_Notes here.

Purpose

This wiki page documents my thoughts on what would be required to build a screen-sharing-and-broadcasting product using Xuggler so it can be run from a client machine. It has a grab bag of work that MIGHT need to be done, but not all steps are required (they depend on what we learn with performance, and what the ultimate business requirements end up being). It is meant to facilitate design discussions internally.

Lastly, this is NOT A RECOMMENDATION TO BUILD THIS. It's just a listing of the steps that would be required if we took this approach.

Architecture

The basic approach is:

• Require Java installed on the machine that will broadcast
• Use a Java API (either Robot or JxCapture) to access screen images and create UI (swing) and API to configure
• Enable Xuggler to be usable as a Java Web Start application
• Create separate (closed source) application that uses Xuggler in a web-start app to implement UI controls and broadcasting controls
• Encode images taken from capture API into H264 or Screen or other codecs
• Broadcast/save images using RTMP, RTP, or other supported Xuggler prototols/Containers

The requirements are:

• P1: Work on Windows 32-bit Java VMs
• P1: Work on Mac 10.6 32-bit Java VMs
• P1: Be able to capture at 3 FPS for screens up to 720p resolution
• P2: Work on Windows 64-bit Java VMs
• P2: Work on Mac 10.6 64-bit Java VMs
• P2: Be able to capture at 15 FPS for screens up to 1080i resolution
• P1: Be able to capture the entire desktop or a rectangular (specified) region of the desktop
• P2: Be able to capture a specific OS window
• P1: Be able to encode H264 and broadcast to RTMP
• P1: Be able to encode Screen and broadcast to RTMP
• P2: Be able to encode H264 and save to file on broadcasting machine

The assumptions are:

• Windows 7, Vista or XP only
• Mac OS X 10.6 only
• Java can be installed on client machines
• UI work for controls are underspecified right now.

The big risk areas:

• Today it's not easy to run Xuggler as a web-download
• Today Xuggler does not support 64-bit Windows
• Robot class may not be fast enough at capturing for 1080i; if not it may require integration of JxCapture
• Will require new code for Capture product
• H264 encoding may need additional x264 options (e.g. thread-slicing) enabled to minimize latency; if so we'll need to add Codec-Specific options to FFmpeg/Xuggler. The good news here is we know that x264 CAN encode fast enough -- it's just exposing that functionality that matters.
• RTMP flow may require changes in how FFmpeg specific URL; today FFmpeg assumes the application is always "rtmp://host/appname" and that the rest of the URL specifies the stream. Not all RTMP deployments assume that. If this is an issue, we may need to add format/protocol specific options to FFmpeg.
• UI work not covered in these thoughts -- but how to enable UI developers is.

Things that reduce risk:

• We have already done a POC (in our demo code) for using Robot and encoding to disk (and community has validated)
• We have already proven encoding and broadcasting to RTMP works (and community has validated)
• We have already proven that Xuggler can encode both Screen and H264 (and community has validated).
• We have externally validated that H264 can be made to encode 720p @ 15FPS with <200 m/s latency (will discuss how in person)
• We have externally validated that making a Web Start version of a native library is possible (will discuss how in person)
• A member of the Xuggler community submitted a patch to make Xuggler build on 64-bit windows (we have not integrated their suggestions)
• With the right partner (hint hint) the Flex UI work should not only be doable, but crazy fun!

Work Required

Xuggler Java-Web Start version

• Change JNILoader to enable dynamic libraries to be specified via XML and bundle a specific FFmpeg version; estimate 2-3 days.
• Create separate jar file with native components (will break backwards compatibility, but community knows I'm going to do that for 4.0): estimate 2-3 days.
• Change automated build system to create OS-specific JNLP jars: estimate 2-3 days.

Advantages

• Often requested feature of community to have Xuggler be able to be used as a applet/web-start app
• Can use open-source community for testing
• Open-sourcing this work lets us use x264 encoder
• Will open up leads in a potentially interesting market

Disadvantages

• Have to write code
• Will require Xuggler build system changes to support
• Will require documentation on JNLP setup (can use JxCapture docs for reference)
• Other parties will be able to use this work (the web-start functionality)
• Art will have to do this work personally (makes no sense to transfer this knowledge for the one-time job).

Xuggle Screen Capture Product

• Integrate with Xuggler as a web-start application
• Start with Java Robot class for initial POC and determine if it can scale to 15 FPS
• Consider purchasing JxCapture source code (~$10K) if Robot is not fast enough
• Implement REST-based API so Flex, Web or other apps running on the same machine can connect and control
• Implement a rudimentary Java Swing API for testing of the REST API and for quick-start for those who don't want to implement their own UI
• Test that 3 FPS @ 720P works; if not, create native component to take JxCapture images and pass to Xuggler w/o Java-boundary overlap
• Test whether x264 encoder can work fast enough; if not, add Xuggler X264 Custom Options feature

Work estimate: Hard to tell. POC first is probably about 1-2 weeks and then schedule will become more apparent. My best guess is this is a 2 people on a 4-6 week total effort.

Advantages

• Closed source so competitive advantage
• Encoding risk pushed to Xuggler so they can test
• UI can be completely outsourced to Flex developers
• Will open up leads in a potentially interesting market
• Robert could lead Java component, with Art consulting and other folks leading the UI effort.

Disadvantages

• Have to write code
• Will require build system support

Xuggler Windows 64-bit Version

• Investigate cross-compiler options (same as FFmpeg FATE does today using mingw_64) and build Windows on Linux both 32-bit and 64-bit: estimate 1 week; member of community has already submitted 64-bit windows patches needed. Ramiro & Mike will likely assist with this to reduce risk / answer questions.

Advantages

• Will enable Windows 64-bit support
• Will vastly simplify current build system so that we don't need a windows machine to build -- only to test.

Disadvantages

• Have to write code
• Will require build system support
• Art would need to do this; makes no sense to transfer this knowledge for one-time change.

Xuggler Mac-OSX 32-bit Version

Today Xuggler automatically builds a 64-bit version for Mac, but doesn't automatically build a 32-bit version. We can build a 32-bit version though, and have before.

• Investigate cross-compiler options (same as FFmpeg FATE does today using mingw_64) for mac and then use; estimate 2-3 days; This may not be possible because I'm not sure the FFmpeg does this already, and if not..
• Modify our existing Mac-OS build machine to build a 32-bit and a 64-bit version; estimate 2-3 days; the biggest challenge here is the brain-dead way that Mac's install Java.

Advantages

• Will enable Mac 32-bit support
• May help simplify current build system so that we don't need a windows machine to build -- only to test.

Disadvantages

• Have to write code
• Will require build system support
• Art would need to do this; makes no sense to transfer this knowledge for one-time change.

Add Codec-Specific Option Support to FFmpeg

Today FFmpeg's embedded codecs can only support options that are 'common' to all codecs. That means several x264 options are not exposed (e.g. slice-based multi-threading) that can decrease latency. I have been working with the FFmpeg team on the right approach to fix that, but it would need to be added to Xuggler and FFmpeg

• Patch FFmpeg to allow codec-specific options: 2-3 days; the biggest barrier here is how hard I want to push this for inclusion in mainline FFmpeg
• Push patch into mainline FFmpeg: indefinite time
• Modify our patched FFmpeg if patch accepted WITH CHANGES into mainline: 1 day
• Modify Xuggler to allow setting of codec-specific options: 4 hours

Advantages

• Works the FFmpeg way, so we have a generic framework to constantly add new options
• Will be transparent to the Xuggler API (although internal Xuggler code will need to change)
• The x264 developers are 100% onboard with this approach, and would do most of the work of exposing new options for us once we do this.

Disadvantages

• Pushing stuff into FFmpeg is hard and require patience.
• Art would need to do; He has the FFmpeg relationships and expertise.

Alternates

• We could just hack our FFmpeg version to add a Xuggler specific hack without pushing to mainline FFmpeg, but this means as mainline changes we'll have to maintain the patch.

Add Format specific option support to FFmpeg

Today FFmpeg's formats/protocols can only support options that are 'common' to all format/codec option. That means several RTMP related options are not exposed (e.g. you cannot specify the app name separately from the stream name in RTMP). I have been working with the FFmpeg team on the right approach to fix that, but it would need to be added to Xuggler and FFmpeg

• Patch FFmpeg to allow format and protocol options: 5 days; the biggest barriers are (a) I need to determine best approach to use and (b) here is how hard I want to push this for inclusion in mainline FFmpeg
• Push patch into mainline FFmpeg: indefinite time
• Modify our patched FFmpeg if patch accepted WITH CHANGES into mainline: 1 day
• Modify Xuggler to allow setting of format-specific options: 2 days

Advantages

• Works the FFmpeg way, so we have a generic framework to constantly add new options
• Will be transparent to the Xuggler API (although internal Xuggler code will need to change)
• Other FFmpeg developers would likely run with this and start exposing options that may be useful to us down the road.
• Art would need to do; He has the FFmpeg relationships and expertise.

Disadvantages

• Pushing stuff into FFmpeg is hard and requires patience.

Alternates

• We could just hack our FFmpeg version to add a Xuggler specific hack without pushing to mainline FFmpeg, but this means as mainline changes we'll have to maintain the patch.

JxCapture Integration

It is possible that java.awt.Robot will not be fast enough for our needs. Also java.awt.Robot only allows you to select the screen, or a rectangle on the screen -- you cannot specify capturing a specific window. JxCapture claims to support these features, and directly uses Windows and Mac API calls for performance. We would only undertake this work if requirements push us here, or performance is not fast enough with Robot

Work estimate: unknown, but probably weeks.

Advantages

• We could purchase the source code (~$10K) so get a big jump on this work, plus can bundle into a closed-source product
• Could more easily be optimized for performance than Robot
• If fast, then a real competitive advantage here

Disadvantages

• We have not yet played directly with JxCapture and so would require an evaluation of quality/features.
• Source code may be poor (but unlikely given the architecture and offerings they already have).
• Requires coding.
• Will require build system work to integrate into our build system.
• Will require some native coding work, which means Art will likely have to be deeply involved.