Concepts
From Xugglewiki
Contents |
Basic Internet Video Concepts
We put together a little video that introduces the concepts behind Internet Video. We call it the Overly Simplistic Guide to Internet Video. Start there.
How to Write Code with Xuggler
You probably want to start with the Tutorials and come back here for the more advanced concepts later.
Time Bases
Xuggler (and most media programs) make extensive use of a concept called "Time Bases". What the hell are they? Read on...
What Are Time Bases?
First off let's do some definitions.
Frame Rate for video is the frequency per second at which video pictures appear in a container. For broadcast TV for example, a new picture is sent about 29.97 times per second. Screencasting software will often record video at 15 frames per second. Really shitty webcams from the 1990's would record video at 3 frames per second. A common abbreviation for frame rate is FPS.
Time Base is the units that a stream uses in a container for each 'tick' of it's clock. For example, each packet in an FLV streams contains a 'time stamp', and each time stamp represents one millisecond (or 1/1000 of a second). So a packet with time stamp 500 will display half-a-second into the video, and a packet with time stamp 23,000 will display 23 seconds into a video. For MPEG4 video, it's more complicated, and depends on the variant of the MPEG4 container and the codec used to encode, but often times the time-base is 1/90000. So a packet with time stamp 45,000 appears half a second into a video, and a packet with time stamp 2,070,000 will display 23 seconds into a video.
When you are decoding files with Xuggler you generally don't need to worry about that. Xuggler will convert between stream time-bases, and when you get decoded media (IVideoPicture and IAudioSamples) they will always be in MICROSECONDS (1/1,000,000 of a second).
For encoding it gets more complicated. Some older codecs (most notably MPEG2) did not split the concept of a frame-rate from a time-base, and only supported a fixed frame rate. So if you encoded MPEG2 at 24 FPS, it could only go into a stream in a container with a timebase of 1/24. Newer codecs (H264 for example) sets a time base of around 1/90000 but allows variable frame-rates. In those cases, frame-rate is a value calculated when encoding or decoding which represents the average rate at which frames are encoded -- but it is only an average.
In all cases when encoding video, a StreamCoder needs to know which time-base to use -- if it's using a codec that also requires a fixed frame rate, IStreamCoder will use (1/time-base) as the frame-rate.
In general the easiest way to use this method is if you're recording fixed 24 fps video, then set the time base on your IStreamCoder to 1/24:
coder.setTimeBase(IRational.make(1,24));
This is because, given that you know the exact rate of encoding, when using 1/(fps) for the timebase, timestamps can be generated by the stream coder without doing division. For example, imagine a fps of 3. If you set the timebase to
coder.setTimeBase(IRational.make(1,3));
then the first 5 packets will have the following timestamps:
0, 1, 2, 3, 4
If on the other hand you set the timebase to IRational.make(1,1000), the stream coder will end up rounding timestamps, which over time can introduce significant errors.
0 333 667 1000 1333
(the IStream object will still convert the packet to the correct timebase, but at least it will do the conversion at the latest moment and therefore introduce the least error).
If you're encoding a variable rate video format (i.e. the fps is not constant), then set your time base to highest-resolution time base of the stream you'll decode into. For example, if your ultimate source container is FLV, those streams have a time base of 1/1000, so:
coder.setTimeBase(IRational.make(1,1000));
When you encode a packet using IStreamCoder.encodeVideo(...), we will generate two time stamps (Pts and Dts) and also attach a time-base telling you what units they are in (as of Xuggler 2.1).
Which brings us to one final confusing question. Why are there two methods:
IStreamCoder.getTimeBase(IRational); IStream.getTimeBase(IRational);
The first method gives the time base the stream coder will use to encode packets. The second gives the time base that the given stream in the container should use. When using an IStreamCoder that you retrieved by calling:
IStream.getStreamCoder()
calls to encodeAudio(...) and encodeVideo(...) will automatically encode each packet for the stream they are going into. In general you don't need to ever call IStream.getTimeBase(IRational) in this case.
Advanced Time Bases
But there are two methods for an advanced use case. It is possible to encode video without knowing what type of stream it's going to go into. Let's say you want to encode video as H263 video, but later mux that packet into both a Quicktime MOV container and a Adobe Flash FLV container. MOV uses a time base of 1/90000 for its video streams. FLV uses a time base of 1/1000. What should you do? It'd be a shame if you had to re-encode that video twice well all that is different is the time stamps.
The answer is set IStreamCoder.setTimeBase(IRational) to the highest-resolution timebase possible (in this case 1/90000). Then encode that video and pass the packet to whatever is muxing. The Muxers should take the packet and copy the metadata (see IPacket.make(...) for a way to do that), and then reset the timestamp into the timebase of their stream. Some example pseudo-code:
coder.setTimeBase(IRational.make(1,90000)); ... set other coder settings and open the coder ... coder.encodeVideo(packet, picture, 0);
... in your MOV muxer --- IPacket wrapper = IPacket.make(packet, false); IRational timeBase = stream.getTimeBase(); // will be 1/90000 wrapper.setDts(timeBase.rescale(packet.getDts(),packet.getTimeBase())); wrapper.setPts(timeBase.rescale(packet.getPts(),packet.getTimeBase())); container.writePacket(wrapper);
... in your FLV muxer --- IPacket wrapper = IPacket.make(packet, false); IRational timeBase = stream.getTimeBase(); // will be 1/1000 wrapper.setDts(timeBase.rescale(packet.getDts(),packet.getTimeBase())); wrapper.setPts(timeBase.rescale(packet.getPts(),packet.getTimeBase())); container.writePacket(wrapper);
Rounding and Timestamps
One final point about time-bases and time-stamps. Xuggler uses IRational values internally for time bases in order to avoid doing any division until the latest moment. This is in order to minimize rounding error introduced.
Multi-Threaded Programming
Xuggler can be used in multi-threaded programs. At Xuggle we routinely use Xuggler to open multiple files, demux on one thread, decode on a separate thread, render on a separate thread, encode on a separate thread and mux on a separate thread. However you need to take some care on your part to ensure things work as you might expect.
One Thread At A Time
Each Xuggler object should only be accessed by one thread at a time. You must ensure this occurs. For example if you open a container with a call to IContainer.open you should ensure all further calls on that IContainer either happen on the same thread, or if done on other threads never overlap with another call to IContainer. Basically follow the same rules for multi-threaded programming with Xuggler objects that you would with a java.lang.HashMap object -- synchronize access. That said... there is one other big caveat.
Use Your Own Locks
Using synchronized on Xuggler objects will not work as you expect. For example the following code does not lock correctly and results in two threads accessing the same Container object at the same time (where behaviour is undefined):
IContainer container = IContainer.make();
IContainer container2 = container.copyReference();
...
// on first thread...
synchronized(container) {
container.open(...);
}
// on second thread ...
synchronized(container2) {
container2.readNextPacket(...);
}
This happens because Java Xuggler objects are really just thin Java proxy objects to underlying Native code objects. copyReference() makes a new Java object, but refers to the same underlying Java object. The best way to lock with Xuggler objects is to use and maintain separate locks.
IContainer container = IContainer.make();
IContainer container2 = container.copyReference();
Object containerLock = new Object();
...
// on first thread...
synchronized(containerLock) {
container.open(...);
}
// on second thread ...
synchronized(containerLock) {
container2.readNextPacket(...);
}
get... Always Returns a New Object
One of the side-effects of the Java proxy objects is that any Xuggler get method returning one of the Xuggler I... objects always returns a new proxy object -- even if referencing the same underlying native-code object. Xuggler maps the Java hashCode() and equals() method to make proxy objects appear to be equal when inserted in maps and other Java containers, but they are actually different objects. That means the following code exhibits the following somewhat strange (but correct) behavior:
IContainer container = ...
// Get the first stream in the container
IStream stream1 = container.getStream(0);
// Get the first stream in the container again
IStream stream2 = container.getStream(0);
System.out.println("stream1 == stream2: " + (stream1 == stream2));
System.out.println("stream1.equals(stream2): " + stream1.equals(stream2));
// This is not strictly necessary in the JAVA_STANDARD_HEAP memory model, but
// is a good idea if you're running with the NATIVE_BUFFERS memory model
// Note that each proxy object has a corresponding delete, even if they refer
// to the same underlying Native object.
stream1.delete();
stream2.delete();
container.delete();
// Lastly if you don't call delete, Xuggler will *eventually* call it for you.
// Depending on which memory model you use, *eventually* can be a long time,
// but it will eventually come.
This will print out:
stream1 == stream2: false stream1.equals(stream2): true
Don't Forget to copyReference
If you are passing a Xuggler object from one thread to another, it's probably a good idea to call RefCounted.copyReference() on the object before you pass it. That way, if the first thread decides to call RefCounted.delete(), the second thread has its own reference that can be safely used.
IContainer container = IContainer.make();
// on first thread
final IContainer otherThreadContainer = container.copyReference();
Thread thread = new Thread(new Runnable(){
public void run(){
// on second thread
otherThreadContainer.open(...);
...
// this is safe because the thread got it's own reference passed to it.
otherThreadContainer.delete();
}
}).start();
// on first thread still
// if we didn't do the copyReference() above, this following operation could
// delete the container out from under thread 2
container.delete();
Global Locks
There are a small number of cases where [FFmpeg] and/or Xuggler requires a global lock.
In all of those cases, Xuggler manages the details under the scenes and either uses AtomicLong operations (much faster than attempting a synchronize) or holds a full lock for the shortest allowable duration. A partial list of those moments are:
- When calling IBuffer.getByteBuffer(...) or IMediaData.getByteBuffer(...), a WeakReference needs to be added to a thread safe queue and the queue is briefly locked for the add.
- In Xuggler 3.0 and later you can determine exactly when the WeakReference is deleted by asking the IBuffer to give you back a JNIReference object when you request a ByteBuffer.
- In Xuggler 3.2 and later, getting the ByteBuffer requires only an atomic increment in most cases.
- In earlier versions of Xuggler you cannot control if and when the WeakReference is deleted.
- When calling .make() or .copyReference(), a WeakReference needs to be added to thread-safe queue and the queue is briefly locked for the add.
- In Xuggler 3.0 and later, copyReference() only requires a brief hold on the global lock.
- In Xuggler 3.2 and later, copyReference() requires only an atomic increment in most cases.
- In earlier versions of Xuggler, copyReference() required both a global lock, and a Java->JNI->Java call to occur meaning it could be relatively expensive if done a lot.
- When calling .delete() or when Java collects a WeakReference, the reference needs to be removed from a thread-safe queue and the queue is briefly locked for the remove.
- In Xuggler 3.0 this just requires an AtomicLong.decrementAndGet() call and one global lock.
- In Xuggler 3.2 and later, no global lock is required on delete().
- In earlier versions of Xuggler delete would need an object-level synchronize and a global-lock.
- When calling IStreamCoder.open(...) [FFmpeg] requires a global lock to occur.
- In Xuggler 3.0 and later, Xuggler ensures this happens for the shortest possible time -- even if the open is called deep from within FFmpeg. The lock is never held across a blocking call.
- In earlier versions of Xuggler it was impossible to catch this inside Xuggler, meaning you, the developer, had to globally lock all calls on any
IContainermethod and onIStreamCoder.open(). You have to hold the lock across potentially blocking calls (likeIContainer.readNextPacket(...)andIContainer.getNumStreams()). The reason for this is thatIContainercalls may end up secretly opening and closing codecs behind the scenes in order to parse packets, and those opens could fail without a global lock.
- When adding new codecs, formats, or protocols to [FFmpeg] a global lock must occur.
- In Xuggler 3.0 this happens during class-loading and has no later run-time effect.
- In earlier versions of Xuggler, in very rare situations, codec adding could occur during run-time and potentially register codecs multiple times and/or corrupt memory.
- When allocating a Direct ByteBuffer (e.g.
IBuffer.getByteBuffer(...)) for the first time from JNI code, a global lock needs to be held to work around a [Sun Java 1.5 and 1.6 race-condition].- In Xuggler 3.0 we allocate a 1-byte ByteBuffer at class loading time (where we're guaranteed to be only on one thread) to work around the Sun bug.
- In earlier versions of Xuggler we could fail hard due to this bug when running with many simultaneous threads.
