Encoding Video from a sequence of Images

From Xugglewiki

The Easier Way

It is much easier to use the MediaTool to encode video from a sequence of images. See that tutorial here

Requirements

To start, you will need to have Xuggler installed on your machine. If you're new to Xuggler, please go through one of the Audio/Video Decoding tutorials first, to familiarize yourself with Xuggler. This tutorial assumes you have the knowledge covered in those tutorials.

Description

The program in this tutorial takes screenshots of your desktop at regular intervals (3 times a second), then uses Xuggler to encode the images into a file as video. The tutorial focuses on the encoding video part.

Getting Started

Create a new Java Project. Import the necessary Xuggler libraries. For simplicity, all code in this tutorial will be put into one class, called CaptureScreenToFile.java

Setting up the streams

First of all, we're going to need to setup all the necessary objects and their properties. We're going to need an IContainer object, which represents the file to which we want to encode to. Let's call it outContainer.

outContainer = IContainer.make();

int retval = outContainer.open(outFileUrl, IContainer.Type.WRITE, null);
if (retval <0)

throw new RuntimeException("could not open output file");

As you can see, we're creating an empty container using the make() method, and then opening a new file (as specified in outFileUrl) - as we are going to encode our images into a movie. If you wish to encode the images into another format, change the suffix on the filename you use. Also, since Xuggler calls the ffmpeg libraries, which are written in C, many Xuggler methods return integers which signify whether the method return successfully or not, in which case we can throw an error. For brevity, the rest of this tutorial will not include these safety checks, but it is good programming practice to include them in your program.

Next, we need an IStream object, which represents our video stream - the only stream we are going to have in our IContainer. Along with an IStream, we need an IStreamCoder, which does the actual encoding after we have specified the necessary parameters. We will call these two objects outStream and outStreamCoder

outStream = outContainer.addNewStream(0);
outStreamCoder = outStream.getStreamCoder();

We add a new stream to our IContainer. Since the Container only contains one stream and this is it, we give it an index of 0. We then retrieve the IStreamCoder from the newly created stream.

We now need to set the codec with which to encode the video. You can set the codec manually, or you can let Xuggler guess the codec for you, which we will do here:

ICodec codec = ICodec.guessEncodingCodec(null, null, outFileUrl, null, ICodec.Type.CODEC_TYPE_VIDEO);

The guessEncodingCodec() method lets you pass in a bunch of parameters based on which Xuggler makes an educated guess. In this case, I found it was sufficient to pass in a file name with the extention .flv and tell Xuggler that we want to encode Video.

Next come a bunch of attributes we need to set for our video. Since we are creating a new video from scratch, we need to set things such as the frame rate, bit rate, height, width, quality, etc...

outStreamCoder.setNumPicturesInGroupOfPictures(10);
outStreamCoder.setCodec(codec);

outStreamCoder.setBitRate(25000);
outStreamCoder.setBitRateTolerance(9000);
outStreamCoder.setPixelType(IPixelFormat.Type.YUV420P);
outStreamCoder.setHeight(1050);
outStreamCoder.setWidth(1680);
outStreamCoder.setFlag(IStreamCoder.Flags.FLAG_QSCALE, true);
outStreamCoder.setGlobalQuality(0);

IRational frameRate = IRational.make(3,1);
outStreamCoder.setFrameRate(frameRate);
outStreamCoder.setTimeBase(IRational.make(frameRate.getDenominator(), frameRate.getNumerator()));
frameRate = null;

From top to bottom: setNumPicturesInGroupOfPictures() method determines the number of interframes between each keyframe. The setBitRate() and setBitRateTolerance() methods are self explanatory. The relationship between the bit rate and the size of your video is also not quite linear. I've found that setting the bit rate too low actually increases the size of the video. Why this is I don't know, but just as a precaution you might want to play with the bit rate a little bit. setPixelType() sets the color scheme. More about that later. The height and width of the video must be the same as the height and width of the images you are using, otherwise you need to resample the images to the correct dimensions before encoding. setGlobalQuality(0) sets the video encoding to the highest quality. The frameRate is the number of images per second of your end result video. The time base is set to the inverse of the frame rate.

Now that we have everything set up, we open the IStreamCoder, and write the header of the file we are encoding to. It is important to do this before we start encoding the video.

outStreamCoder.open();
outContainer.writeHeader();

Setting up the stream properly was the most complicated part of the whole process. Now we cut to the chase - converting the actual images to video. For this, we make a method, encodeImage(BufferedImage image), which takes a Java BufferedImage and encodes it into our IContainer.

BufferedImage worksWithXugglerBufferedImage = convertToType(originalImage,
BufferedImage.TYPE_3BYTE_BGR);

IPacket packet = IPacket.make();
IConverter converter = ConverterFactory.createConverter(worksWithXugglerBufferedImage, IPixelFormat.Type.YUV420P);

We also need to create a time stamp for this picture. Encoding can take a lot of time (for example if you have a large screen or a CPU-intensive codec), so we'll calculate the new time stamp based on the clock time:

long now = System.currentTimeMillis();
if (firstTimeStamp == -1)
firstTimeStamp = now;
long timeStamp = (now - firstTimeStamp)*1000; // convert to microseconds

IVideoPicture outFrame = converter.toPicture(worksWithXugglerBufferedImage, timeStamp);
outFrame.setQuality(0);
outStreamCoder.encodeVideo(packet, outFrame, 0);

if (packet.isComplete()) outContainer.writePacket(packet);

The IConverter class converts a BufferedImage to an IVideoPicture, which is the class Xuggler uses to represent video frames. Make sure the PixelFormat for the image converter is the same PixelFormat you used earlier for the IStreamCoder, otherwise the encoding will not work.

All that's left to be done is to properly close he streams when you are done encoding the file. Xuggler does most of this for you, so it's not necessary to close all the coders and containers. However you need to write a trailer:

outContainer.writeTrailer();

This tells Xuggler to add the final meta data to a file, and should make your file be scrubbable in a video player (for FLV), or actually valid if a container like MP4.

Now that everything is all peachy, we need to get some images to encode. That is outside of the scope of this tutorial, but I will post the entire program if you need sample code on how to capture the screen in Java.

Possible problems with captured images

Xuggler provides a package called com.xuggle.xuggler.video that can convert from Java image types into Xuggler IVideoPicture types and back.

Unfortunately when the Java Robot class creates a snapshot, the resulting BufferedImage is of a type which the Xuggler ConverterFactory will not be able to create a converter for. An exception is thrown "No converter found for BufferedImage type #1." The following method can be used to convert BufferedImage from any type to any other type:

 /** 
  * Convert a {@link BufferedImage} of any type, to {@link
  * BufferedImage} of a specified type.  If the source image is the
  * same type as the target type, then original image is returned,
  * otherwise new image of the correct type is created and the content
  * of the source image is copied into the new image.
  *
  * @param sourceImage the image to be converted
  * @param targetType the desired BufferedImage type 
  *
  * @return a BufferedImage of the specifed target type.
  *
  * @see BufferedImage
  */

 public static BufferedImage convertToType(BufferedImage sourceImage, 
   int targetType)
 {
   BufferedImage image;

   // if the source image is already the target type, return the source image

   if (sourceImage.getType() == targetType)
     image = sourceImage;

   // otherwise create a new image of the target type and draw the new
   // image 

   else
   {
     image = new BufferedImage(sourceImage.getWidth(), sourceImage.getHeight(),
       targetType);
     image.getGraphics().drawImage(sourceImage, 0, 0, null);
   }
    
   return image;
 }

To convert images into a type Xuggler can work with, call above method like this:

 BufferedImage worksWithXugglerBufferedImage = convertToType(doesNotWorkWithXuggerBufferedImage, BufferedImage.TYPE_3BYTE_BGR);

The folks at Xuggle plan to incorporate this image conversion code into Xuggler library in the near future, but the above code should keep people going.

Source

The full working source code can be had here

The original code this is based off of can be found here. If you are interested in an open source implementation of a webcasting component done using red5 and flash, see the BigBlueButton project

Credits

This tutorial was originally written by Denis Zgonjanin with the image conversion addendum by Robert Harris, and the code to generalize for any format added by Art Clarke