From Xugglewiki

Contents 1 Research 1.1 Background 1.2 Codec Trade-Offs 1.3 Current Vendors / Projects

Research

Background

Screensharing in this context means capturing a portion or all of a users screen, encoding the video live, and broadcasting to a RTMP based server (e.g. FMS, Wowza or Red5). There are several approaches used in the market today.

1. Adobe Connect Pro has a custom download that gets installed, and then broadcasts a RTMP stream to a Flash Media Server using the Flash Screen Codec. This is a simple block-based encoder that sends key-frames occasionally, and then divides the screen into rectangular blocks and only sends changed blocks when the pixels on that area of the screen switches between frames. Flash Media Server only accepts RTMP streams with this codec when send from the Adobe Connect Pro plugin.

2. There are a variety of open-source implementations of the Screen codec used by projects such as Big Blue Button, Open Meetings, and others that implement screen sharing by using (a) The Java Robot class to capture a portion of the screen (b) use a java-based implementation of Screen to encode and (c) use a RTMP library (such as Red5 Client) to talk to a server. This approach works with Wowza and Red5, but does not work with FMS.

3. There are some closed source vendors that implement screen-capture and encoding using the H264 codec, but most of those do not do live broadcasting via RTMP. NOTE: Confirm this with folks in the know.

This feature turns out to be one of the most often requested features in the Red5 communities. Specifically people are looking for:

• A solution that works on Windows and Mac; Linux is less important to most users.
• The ability to specify the whole desktop, an area on the desktop, or a specific application.
• The ability to broadcast to Red5, Wowza or FMS.
• The ability to control bandwidth so as to be able to broadcast with <1mb/s
• The ability to record the screencast, and sync with audio captured via other sources
• The ability to annotate/whiteboard the screencast while broadcasting.

Codec Trade-Offs

The Screen-Based Codec approach has the following advantages:

• Easily implemented and free. Several open-source implementations exist in Java and C.
• Can be decoded in Flash versions since 8.x.
• Can be computationally less demanding than H264.
• Low latency as no frame-buffering is required to encode a given frame.

The Screen-Based Codec approach has the following disadvantages:

• Higher bandwidth required than H264 (e.g. 1080i at 3 FPS w/ key-frames sent on average every 30 seconds can require > 1mb/s upstream).
• Higher bandwidth can be required from viewers especially if video is changing a lot and to send key-frames.
• Does not work with Flash Media Server unless using Adobe's proprietary plugin.

The H264 Codec approach has the following advantages:

• Lower bandwidth required. 1080i at 3FPS can be done in < 1 mb/s.
• Can be optimized to require lower bandwidth key-frames (progressive IDR frames) after the first frame.
• Works with Flash Media Server, Wowza or Red5.
• Codec under active development in the open-source community (libx264, and some people have been able to achieve 720p @ 15FPS with under 200 millisecond latency.
• Codec under active development in commercial space (e.g. vidyo.com).

The H264 Codec approach has the following disadvantages:

• Computationally more expensive on broadcasting machine.
• Computationally more expensive on viewing machine.
• Can only be decoded in Flash version 9.0.124 or later.
• Requires more latency to achieve compression savings (e.g. will usually need to buffer 3 frames).
• Requires H264 patent licenses to be negotiated with the MPEG-LA authority.
• Can be played back by more than just Flash

Current Vendors / Projects

The following list contains vendors who make screen-encoding software, although most cannot also broadcast RTMP: Wikipedia List of Current Vendors

Here is a list of companies/projects that also have RTMP broadcasting support that I know of:

• Adobe Connect Pro -- Using Screen codec with RTMP. Proprietary solution that is the only option supported with FMS today.
• Screentoaster -- although it is not OEMable; They use H264 with RTMP in Java as a plugin. They bundle x264 today but don't GPL their own source as they should (i.e. they have a JNI based interface to x264).
• VH Screen Capture Driver -- Windows-only; uses Screen Codec and RTMP today. Proprietary product, but their Java version embeds x264 (and hence should be GPL).
• Big Blue Button -- Screen codec with RTMP; open-source java implementation (AGPL) but requires commercial license for use if you don't want to AGPL your source. Tested only with Red5.
• Open Meetings -- Screen codec with RTMP; open-source implementation (Eclipse Public License) and tested only with Red5. Uses Red5 client for RTMP broadcasting.
• red5-screenshare -- Screen codec with RTMP; open-source implementation (LGPL) and tested only with Red5. Uses Red5 client for RTMP broadcasting

Here is a list of companies / projects that have the ability to capture the screen (but not encode):

• Java java.awt.Robot Class; Pure-Java interface. Allows you to capture the whole desktop or part of the desktop; does not easily allow grabbing a specific application window.
• JxCapture; Java interface with native (JNLP enabled) plugin for Windows and Mac; commercial but you can purchase source-code for $6k.
• DirectShow; Windows API for getting at video. Can be used in C++/.NET on Windows
• There's a Mac API too, but I don't know what it is.

Here is a list of open-source projects that can encode Screen or H264 given an image:

• Xuggler; Java with native plugin. Not currently JNLP enabled. Can do Screen or H264; can also broadcast RTMP natively.
• libavcodec; C library. This is the FFmpeg library (used by Xuggler). Can do Screen or H264 (if compiled with libx264); can also broadcast RTMP natively

Added 25 jan 2011

• Smaxe Java/JNA RTMP client; Java with Native plugin and also options for choice between SVC1, SVC2 and h264. H264 implementation is the native part, taken from x264.dll compiled by avidemux, hence should be GPL, but again it is not. The native part is basically a JNA wrapper around some win32 libraries (like gdi32, user32, avicap (which does the capturing) ) and the encoding of the captured data is made directly by JNA calls to x264. Again, as they used GPL'ed code, their code should also be GPL, with an argument that as they did not compile the source of the GPL library, they chose to close their source.

Here are some alternate approaches that folks have tried:

• VNC server running on the screen sharing machine with Java VNC connector running on the server