Framework Documentation

Copyright (C) 2004-2009 Ushodaya Enterprises Limited
Author: Charles Yates <charles.yates@pandora.be>
Last Revision: 2005-05-08


MLT FRAMEWORK
-------------

Preamble:

	MLT is a multimedia framework designed for television broadcasting. As such, 
	it provides a pluggable architecture for the inclusion of new audio/video 
	sources, filters, transitions and playback devices.

	The framework provides the structure and utility functionality on which
	all of the MLT applications and services are defined. 

	On its own, the framework provides little more than 'abstract classes' and
	utilities for managing resources, such as memory, properties, dynamic object
	loading and service instantiation. 

	This document is split roughly into 3 sections. The first section provides a
	basic overview of MLT, the second section shows how it's used and the final
	section shows structure and design, with an emphasis on how the system is 
	extended.


Target Audience:

	This document is provided as a 'road map' for the framework and should be
	considered mandatory reading for anyone wishing to develop code at the MLT
	level. 

	This includes:

	1. framework maintainers;
	2. module developers;
	3. application developers;
	4. anyone interested in MLT.

	The emphasis of the document is in explaining the public interfaces, as
	opposed to the implementation details.

	It is not required reading for the MLT client/server integration - please
	refer to libmvsp.txt and mvsp.txt for more details on this area.


SECTION 1 - BASIC OVERVIEW
--------------------------

Basic Design Information:

	MLT is written in C. 
	
	The framework has no dependencies other than the standard C99 and POSIX 
	libraries. 

	It follows a basic Object Oriented design paradigm, and as such, much of the
	design is loosely based on the Producer/Consumer design pattern. 
	
	It employs Reverse Polish Notation for the application of audio and video FX.

	The framework is designed to be colour space neutral - the currently
	implemented modules, however, are very much 8bit YUV422 oriented. In theory,
	the modules could be entirely replaced.

	A vague understanding of these terms is assumed throughout the remainder of
	this document.


Structure and Flow:

	The general structure of an MLT 'network' is simply the connection of a
	'producer' to a 'consumer':

	+--------+   +--------+
	|Producer|-->|Consumer|
	+--------+   +--------+

	A typical consumer requests MLT Frame objects from the producer, does 
	something with them and when finished with a frame, closes it. 
	
	 /\  A common confusion with the producer/consumer terminology used here is 
	/!!\ that a consumer may 'produce' something. For example, the libdv consumer
	\!!/ produces DV and the libdv producer seems to consume DV. However, the
	 \/  naming conventions refer only to producers and consumers of MLT Frames.

	To put it another way - a producer produces MLT Frame objects and a consumer 
	consumes MLT Frame objects.

	An MLT Frame essentially provides an uncompressed image and its associated
	audio samples.

	Filters may also be placed between the producer and the consumer:

	+--------+   +------+   +--------+
	|Producer|-->|Filter|-->|Consumer|
	+--------+   +------+   +--------+

	A service is the collective name for producers, filters, transitions and
	consumers. 

	The communications between a connected consumer and producer or service are 
	carried out in 3 phases:

	* get the frame
	* get the image
	* get the audio

	MLT employs 'lazy evaluation' - the image and audio need not be extracted
	from the source until the get image and audio methods are invoked. 

	In essence, the consumer pulls from what it's connected to - this means that
	threading is typically in the domain of the consumer implementation and some
	basic functionality is provided on the consumer class to ensure realtime
	throughput.


SECTION 2 - USAGE
-----------------

Hello World:

	Before we go in to the specifics of the framework architecture, a working
	example of usage is provided. 
	
	The following simply provides a media player:

	#include <stdio.h>
	#include <unistd.h>
	#include <framework/mlt.h>

	int main( int argc, char *argv[] )
	{
	    // Initialise the factory
	    if ( mlt_factory_init( NULL ) == 0 )
	    {
	        // Create the default consumer
	        mlt_consumer hello = mlt_factory_consumer( NULL, NULL );

	        // Create via the default producer
	        mlt_producer world = mlt_factory_producer( NULL, argv[ 1 ] );

	        // Connect the producer to the consumer
	        mlt_consumer_connect( hello, mlt_producer_service( world ) );

	        // Start the consumer
	        mlt_consumer_start( hello );

	        // Wait for the consumer to terminate
	        while( !mlt_consumer_is_stopped( hello ) )
	            sleep( 1 );

	        // Close the consumer
	        mlt_consumer_close( hello );

	        // Close the producer
	        mlt_producer_close( world );

	        // Close the factory
	        mlt_factory_close( );
	    }
	    else
	    {
	        // Report an error during initialisation
	        fprintf( stderr, "Unable to locate factory modules\n" );
	    }

	    // End of program
	    return 0;
	}

	This is a simple example - it doesn't provide any seeking capabilities or
	runtime configuration options. 

	The first step of any MLT application is the factory initialisation - this
	ensures that the environment is configured and MLT can function. The factory
	is covered in more detail below.

	All services are instantiated via the factories, as shown by the
	mlt_factory_consumer and mlt_factory_producer calls above. There are similar
	factories for filters and transitions. There are details on all the standard
	services in services.txt. 

	The defaults requested here are a special case - the NULL usage requests
	that we use the default producers and consumers. 
	
	The default producer is "loader". This producer matches file names to 
	locate a service to use and attaches 'normalising filters' (such as scalers,
	deinterlacers, resamplers and field normalisers) to the loaded content -
	these filters ensure that the consumer gets what it asks for.

	The default consumer is "sdl". The combination of loader and sdl will
	provide a media player.

	In this example, we connect the producer and then start the consumer. We
	then wait until the consumer is stopped (in this case, by the action of the
	user closing the SDL window) and finally close the consumer, producer and
	factory before exiting the application.

	Note that the consumer is threaded - waiting for an event of some sort is 
	always required after starting and before stopping or closing the consumer.

	Also note, you can override the defaults as follows:

	$ MLT_CONSUMER=xml ./hello file.avi

	This will create a XML document on stdout.

	$ MLT_CONSUMER=xml MLT_PRODUCER=avformat ./hello file.avi

	This will play the video using the avformat producer directly, thus it will
	bypass the normalising functions.

	$ MLT_CONSUMER=libdv ./hello file.avi > /dev/dv1394

	This might, if you're lucky, do on the fly, realtime conversions of file.avi
	to DV and broadcast it to your DV device.


Factories:

	As shown in the 'Hello World' example, factories create service objects.

	The framework itself provides no services - they are provided in the form of
	a plugin structure. A plugin is organised in the form of a 'module' and a
	module can provide many services of different types. 

	Once the factory is initialised, all the configured services are available
	for use.

	The complete set of methods associated to the factory are as follows:

	int mlt_factory_init( char *prefix );
	const char *mlt_factory_prefix( );
	char *mlt_environment( char *name );
	mlt_producer mlt_factory_producer( char *name, void *input );
	mlt_filter mlt_factory_filter( char *name, void *input );
	mlt_transition mlt_factory_transition( char *name, void *input );
	mlt_consumer mlt_factory_consumer( char *name, void *input );
	void mlt_factory_close( );

	The mlt_factory_prefix returns the path to the location of the installed
	modules directory. This can be specified in the mlt_factory_init call
	itself, or it can be specified via the MLT_REPOSITORY environment variable,
	or in the absence of either of those, it will default to the install
	prefix/shared/mlt/modules. 

	The mlt_environment provides read only access to a collection of name=value
	pairs as shown in the following table:

	+------------------+------------------------------------+------------------+
	|Name              |Description                         |Values            |
	+------------------+------------------------------------+------------------+
	|MLT_NORMALISATION |The normalisation of the system     |PAL or NTSC       |
	+------------------+------------------------------------+------------------+
	|MLT_PRODUCER      |The default producer                |"loader" or other |
	+------------------+------------------------------------+------------------+
	|MLT_CONSUMER      |The default consumer                |"sdl" or other    |
	+------------------+------------------------------------+------------------+
	|MLT_TEST_CARD     |The default test card producer      |any producer      |
	+------------------+------------------------------------+------------------+

	These values are initialised from the environment variables of the same
	name.

	As shown above, a producer can be created using the 'default normalising'
	producer, and they can also be requested by name. Filters and transitions 
	are always requested by name - there is no concept of a 'default' for these.


Service Properties:

	As shown in the services.txt document, all services have their own set of
	properties than can be manipulated to affect their behaviour.

	In order to set properties on a service, we need to retrieve the properties
	object associated to it. For producers, this is done by invoking:

	    mlt_properties properties = mlt_producer_properties( producer );

	All services have a similar method associated to them.

	Once retrieved, setting and getting properties can be done directly on this
	object, for example:

	    mlt_properties_set( properties, "name", "value" );
	
	A more complete description of the properties object is found below.


Playlists:

	So far, we've shown a simple producer/consumer configuration - the next
	phase is to organise producers in playlists.

	Let's assume that we're adapting the Hello World example, and wish to queue
	a number of files for playout, ie:

	    hello *.avi

	Instead of invoking mlt_factory_producer directly, we'll create a new
	function called create_playlist. This function is responsible for creating
	the playlist, creating each producer and appending to the playlist.

	mlt_producer create_playlist( int argc, char **argv )
	{
	    // We're creating a playlist here
	    mlt_playlist playlist = mlt_playlist_init( );

	    // We need the playlist properties to ensure clean up
	    mlt_properties properties = mlt_playlist_properties( playlist );

	    // Loop through each of the arguments
	    int i = 0;
	    for ( i = 1; i < argc; i ++ )
	    {
	        // Create the producer
	        mlt_producer producer = mlt_factory_producer( NULL, argv[ i ] );

	        // Add it to the playlist
	        mlt_playlist_append( playlist, producer );

	        // Close the producer (see below)
	        mlt_producer_close( producer );
	    }

	    // Return the playlist as a producer
	    return mlt_playlist_producer( playlist );
	}

	Notice that we close the producer after the append. Actually, what we're 
	doing is closing our reference to it - the playlist creates its own reference
	to the producer on append and insert, and it will close its reference 
	when the playlist is destroyed[*].

	Note also that if you append multiple instances of the same producer, it 
	will create multiple references to it.

	Now all we need do is to replace these lines in the main function:

	    // Create a normalised producer
	    mlt_producer world = mlt_factory_producer( NULL, argv[ 1 ] );

	with:

	    // Create a playlist
	    mlt_producer world = create_playlist( argc, argv );

	and we have a means to play multiple clips.

	[*] This reference functionality was introduced in mlt 0.1.2 - it is 100%
	compatable with the early mechanism of registering the reference and 
	destructor with the properties of the playlist object.


Filters:

	Inserting filters between the producer and consumer is just a case of
	instantiating the filters, connecting the first to the producer, the next
	to the previous filter and the last filter to the consumer.

	For example:

	    // Create a producer from something
	    mlt_producer producer = mlt_factory_producer( ... );

	    // Create a consumer from something
	    mlt_consumer consumer = mlt_factory_consumer( ... );

	    // Create a greyscale filter
	    mlt_filter filter = mlt_factory_filter( "greyscale", NULL );

	    // Connect the filter to the producer
	    mlt_filter_connect( filter, mlt_producer_service( producer ), 0 );

	    // Connect the consumer to filter
	    mlt_consumer_connect( consumer, mlt_filter_service( filter ) );

	As with producers and consumers, filters can be manipulated via their
	properties object - the mlt_filter_properties method can be invoked and
	properties can be set as needed.

	The additional argument in the filter connection is an important one as it
	dictates the 'track' on which the filter operates. For basic producers and
	playlists, there's only one track (0), and as you will see in the next
	section, even multiple tracks have a single track output.


Attached Filters:

	All services can have attached filters.

	Consider the following example:

	    // Create a producer
	    mlt_producer producer = mlt_factory_producer( NULL, clip );

	    // Get the service object of the producer
	    mlt_producer service = mlt_producer_service( producer );

	    // Create a filter
	    mlt_filter filter = mlt_factory_filter( "greyscale" );

	    // Create a playlist
	    mlt_playlist playlist = mlt_playlist_init( );

	    // Attach the filter to the producer
	    mlt_service_attach( producer, filter );

	    // Construct a playlist with various cuts from the producer
	    mlt_playlist_append_io( producer, 0, 99 );
	    mlt_playlist_append_io( producer, 450, 499 );
	    mlt_playlist_append_io( producer, 200, 399 );
	
	    // We can close the producer and filter now
	    mlt_producer_close( producer );
	    mlt_filter_close( filter );

	When this is played out, the greyscale filter will be executed for each frame 
	in the playlist which comes from that producer.

	Further, each cut can have their own filters attached which are executed after 
	the producer's filters. As an example:

	    // Create a new filter
	    filter = mlt_factory_filter( "invert", NULL );
	
	    // Get the second 'clip' in the playlist
	    producer = mlt_playlist_get_clip( 1 );
	
	    // Get the service object of the clip
	    service = mlt_producer_service( producer );
	
	    // Attach the filter
	    mlt_service_attach( producer, filter );
	
	    // Close the filter
	    mlt_filter_close( filter );
	
	Even the playlist itself can have an attached filter:

	    // Create a new filter
	    filter = mlt_factory_filter( "watermark", "+Hello.txt" );
	
	    // Get the service object of the playlist
	    service = mlt_playlist_service( playlist );
	
	    // Attach the filter
	    mlt_service_attach( service, filter );
	
	    // Close the filter
	    mlt_filter_close( filter );
	
	And, of course, the playlist, being a producer, can be cut up and placed on 
	another playlist, and filters can be attached to those cuts or on the new 
	playlist itself and so on ad nauseum.

	The main advantage of attached filters is that they remain attached and don't 
	suffer from the maintenance problems associated with items being inserted and 
	displacing calculated in/out points - this being a major issue if you 
	exclusively use the connect or insert detached filters in a multitrack field 
	(described below).


Introducing the Mix:

	The mix is the simplest way to introduce transitions between adjacent clips
	on a playlist.

	Consider the following playlist:

	+-+----------------------+----------------------------+-+
	|X|A                     |B                           |X|
	+-+----------------------+----------------------------+-+

	Let's assume that the 'X' is a 'black clip' of 50 frames long.

	When you play this out, you'll get a 50 frames of black, abrupt cut into
	A, followed by an abrupt cut into B, and finally into black again.

	The intention is to convert this playlist into something like:

	+-+---------------------+-+------------------------+-+
	|X|A                    |A|B                       |B|
	|A|                     |B|                        |X|
	+-+---------------------+-+------------------------+-+

	Where the clips which refer to 2 clips represent a transition. Notice that
	the representation of the second playlist is shorter than the first - this is
	to be expected - a single transition of 50 frames between two clips will 
	reduce the playtime of the result by 50 frames. 

	This is done via the use of the mlt_playlist_mix method. So, assuming you get 
	a playlist as shown in the original diagram, to do the first mix, you could do
	something like:

	    // Create a transition
	    mlt_transition transition = mlt_factor_transition( "luma", NULL );

	    // Mix the first and second clips for 50 
	    mlt_playlist_mix( playlist, 0, 50, transition );

	    // Close the transition
	    mlt_transition_close( transition );

	This would give you the first transition, subsequently, you would apply a similar
	technique to mix clips 1 and 2. Note that this would create a new clip on the 
	playlist, so the next mix would be between 3 and 4.

	As a general hint, to simplify the requirement to know the next clip index,
	you might find the following simpler:

	    // Get the number of clips on the playlist
	    int i = mlt_playlist_count( );

	    // Iterate through them in reverse order
	    while ( i -- )
	    {
	        // Create a transition
	        mlt_transition transition = mlt_factor_transition( "luma", NULL );

	        // Mix the first and second clips for 50 
	        mlt_playlist_mix( playlist, i, 50, transition );

	        // Close the transition
	        mlt_transition_close( transition );
	    }
	
	There are other techniques, like using the mlt_playlist_join between the 
	current clip and the newly created one (you can determine if a new clip was 
	created by comparing the playlist length before and after the mix call).

	Internally, the mlt_playlist_mix call generates a tractor and multitrack as 
	described below. Like the attached filters, the mix makes life very simple
	when you're inserting items into the playlist.

	Also note that it allows a simpler user interface - instead of enforcing the
	use of a complex multitrack object, you can do many operations on a single
	track. Thus, additional tracks can be used to introduce audio dubs, mixes
	or composites which are independently positioned and aren't affected by 
	manipulations on other tracks. But hey, if you want a bombastic, confusing
	and ultimately frustrating traditional NLE experience, that functionality
	is provided too ;-).


Practicalities and Optimisations:

	In the previous two sections I've introduced some powerful functionality 
	designed to simplify MLT usage. However, a general issue comes into this -
	what happens when you introduce a transition between two cuts from the same
	bit of video footage?

	Anyone who is familiar with video compression will be aware that seeking 
	isn't always without consequence from a performance point of view. So if
	you happen to require two frames from the same clip for a transition, the
	processing is going to be excessive and the result will undoubtedly be very
	unpleasant, especially if you're rendering in realtime...

	So how do we get round this?

	Actually, it's very simple - you invoke mlt_producer_optimise on the top 
	level object after a modification and MLT will determine how to handle it.
	Internally, it determines the maximum number of overlapping instances 
	throughout the object and creates clones and assigns clone indexes as
	required.

	In the mix example above, you can simply call:

	    // Optimise the playlist
	    mlt_producer_optimise( mlt_playlist_producer( playlist ) );
	
	after the mix calls have be done. Note that this is automatically applied
	to deserialised MLT XML.


Multiple Tracks and Transitions:

	MLT's approach to multiple tracks is governed by two requirements:

	1) The need for a consumer and producer to communicate with one another via
	a single frame;
	2) The desire to be able to serialise and manipulate a 'network' (or filter
	graph if you prefer).

	We can visualise a multitrack in the way that an NLE presents it:

	   +-----------------+                          +-----------------------+
	0: |a1               |                          |a2                     |
	   +---------------+-+--------------------------+-+---------------------+
	1:                 |b1                            |
	                   +------------------------------+

	The overlapping areas of track 0 and 1 would (presumably) have some kind of
	transition - without a transition, the frames from b1 and b2 would be shown 
	during the areas of overlap (ie: by default, the higher numbered track takes 
	precedence over the lower numbered track). 

	MLT has a multitrack object, but it is not a producer in the sense that it
	can be connected directly to a consumer and everything will work correctly.
	A consumer would treat it precisely as it would a normal producer, and, in
	the case of the multitrack above, you would never see anything from track 1
	other than the transitions between the clips - the gap between a1 and a2 
	would show test frames.

	This happens because a consumer pulls one frame from the producer it's 
	connected to while a multitrack will provide one frame per track.
	Something, somewhere, must ensure that all frames are pulled from the
	multitrack and elect the correct frame to pass on.

	Hence, MLT provides a wrapper for the multitrack, which is called a
	'tractor', and its the tractors task to ensure that all tracks are pulled
	evenly, the correct frame is output and that we have 'producer like'
	behaviour.

	Thus, a multitrack is conceptually 'pulled' by a tractor as shown here:

	+----------+
	|multitrack|
	| +------+ |    +-------+
	| |track0|-|--->|tractor|
	| +------+ |    |\      |
	|          |    | \     |
	| +------+ |    |  \    |
	| |track1|-|--->|---o---|--->
	| +------+ |    |  /    |
	|          |    | /     |
	| +------+ |    |/      |
	| |track2|-|--->|       |
	| +------+ |    +-------+
	+----------+

	With a combination of the two, we can now connect multitracks to consumers.
	The last non-test card will be retrieved and passed on. 

	The tracks can be producers, playlists, or even other tractors. 

	Now we wish to insert filters and transitions between the multitrack and the
	tractor. We can do this directly by inserting filters directly between the
	tractor and the multitrack, but this involves a lot of connecting and
	reconnecting left and right producers and consumers, and it seemed only fair
	that we should be able to automate that process. 

	So in keeping with our agricultural theme, the concept of the 'field' was 
	born. We 'plant' filters and transitions in the field and the tractor pulls 
	the multitrack (think of a combine harvester :-)) over the field and 
	produces a 'bail' (sorry - kidding - frame :-)).

	Conceptually, we can see it like this:

	+----------+
	|multitrack|
	| +------+ |    +-------------+    +-------+
	| |track0|-|--->|field        |--->|tractor|
	| +------+ |    |             |    |\      |
	|          |    |   filters   |    | \     |
	| +------+ |    |     and     |    |  \    |
	| |track1|-|--->| transitions |--->|---o---|--->
	| +------+ |    |             |    |  /    |
	|          |    |             |    | /     |
	| +------+ |    |             |    |/      |
	| |track2|-|--->|             |--->|       |
	| +------+ |    +-------------+    +-------+
	+----------+

	So, we need to create the tractor first, and from that we obtain the
	multitrack and field objects. We can populate these and finally 
	connect the tractor to a consumer.

	In essence, this is how it looks to the consumer:

	+-----------------------------------------------+
	|tractor          +--------------------------+  |
	| +----------+    | +-+    +-+    +-+    +-+ |  |
	| |multitrack|    | |f|    |f|    |t|    |t| |  |
	| | +------+ |    | |i|    |i|    |r|    |r| |  |
	| | |track0|-|--->| |l|- ->|l|- ->|a|--->|a|\|  |
	| | +------+ |    | |t|    |t|    |n|    |n| |  |
	| |          |    | |e|    |e|    |s|    |s| |\ |
	| | +------+ |    | |r|    |r|    |i|    |i| | \|
	| | |track1|-|- ->| |0|--->|1|--->|t|--->|t|-|--o--->
	| | +------+ |    | | |    | |    |i|    |i| | /|
	| |          |    | | |    | |    |o|    |o| |/ |
	| | +------+ |    | | |    | |    |n|    |n| |  |
	| | |track2|-|- ->| | |- ->| |--->|0|- ->|1|/|  |
	| | +------+ |    | | |    | |    | |    | | |  |
	| +----------+    | +-+    +-+    +-+    +-+ |  |
	|                 +--------------------------+  |
	+-----------------------------------------------+

	An example will hopefully clarify this. 
	
	Let's assume that we want to provide a 'watermark' to our hello world 
	example. We have already extended the example to play multiple clips,
	and now we will place a text based watermark, reading 'Hello World' in 
	the top left hand corner:

	mlt_producer create_tracks( int argc, char **argv )
	{
	    // Create the tractor
	    mlt_tractor tractor = mlt_tractor_new( );

	    // Obtain the field
	    mlt_field field = mlt_tractor_field( tractor );
	
	    // Obtain the multitrack
	    mlt_multitrack multitrack = mlt_tractor_multitrack( tractor );
	
	    // Create a composite transition
	    mlt_transition transition = mlt_factory_transition( "composite", "10%/10%:15%x15%" );
	
	    // Create track 0
	    mlt_producer track0 = create_playlist( argc, argv );
	
	    // Create the watermark track - note we NEED loader for scaling here
	    mlt_producer track1 = mlt_factory_producer( "loader", "pango" );
	
	    // Get the length of track0
	    mlt_position length = mlt_producer_get_playtime( track0 );
	
	    // Set the properties of track1
	    mlt_properties properties = mlt_producer_properties( track1 );
	    mlt_properties_set( properties, "text", "Hello\nWorld" );
	    mlt_properties_set_position( properties, "in", 0 );
	    mlt_properties_set_position( properties, "out", length - 1 );
	    mlt_properties_set_position( properties, "length", length );
	    mlt_properties_set_int( properties, "a_track", 0 );
	    mlt_properties_set_int( properties, "b_track", 1 );

	    // Now set the properties on the transition
	    properties = mlt_transition_properties( transition );
	    mlt_properties_set_position( properties, "in", 0 );
	    mlt_properties_set_position( properties, "out", length - 1 );
	
	    // Add our tracks to the multitrack
	    mlt_multitrack_connect( multitrack, track0, 0 );
	    mlt_multitrack_connect( multitrack, track1, 1 );
	
	    // Now plant the transition
	    mlt_field_plant_transition( field, transition, 0, 1 );

	    // Close our references
	    mlt_producer_close( track0 );
	    mlt_producer_close( track1 );
	    mlt_transition_close( transition );

	    // Return the tractor
	    return mlt_tractor_producer( tractor );
	}

	Now all we need do is to replace these lines in the main function:

	    // Create a playlist
	    mlt_producer world = create_playlist( argc, argv );

	with:

	    // Create a watermarked playlist
	    mlt_producer world = create_tracks( argc, argv );

	and we have a means to play multiple clips with a horribly obtrusive
	watermark - just what the world needed, right? ;-)

	Incidentally, the same thing could be achieved with the more trivial
	watermark filter inserted between the producer and the consumer.


SECTION 3 - STRUCTURE AND DESIGN
--------------------------------

Class Hierarchy:

	The mlt framework consists of an OO class hierarchy which consists of the
	following public classes and abstractions:

	mlt_properties
	  mlt_frame
	  mlt_service
	    mlt_producer
	      mlt_playlist
	      mlt_tractor
	    mlt_filter
	    mlt_transition
	    mlt_consumer
	mlt_deque
	mlt_pool
	mlt_factory

	Each class defined above can be read as extending the classes above and to
	the left.

	The following sections describe the properties, stacking/queuing and memory 
	pooling functionality provided by the framework - these are key components 
	and a basic understanding of these is required for the remainder of the
	documentation.


mlt_properties:

	The properties class is the base class for the frame and service classes.

	It is designed to provide an efficient lookup table for various types of
	information, such as strings, integers, floating points values and pointers
	to data and data structures. 

	All properties are indexed by a unique string. 
	
	The most basic use of properties is as follows:

	    // 1. Create a new, empty properties set;
	    mlt_properties properties = mlt_properties_new( );

	    // 2. Assign the value "world" to the property "hello";
	    mlt_properties_set( properties, "hello", "world" );

	    // 3. Retrieve and print the value of "hello";
	    printf( "%s\n", mlt_properties_get( properties, "hello" ) );

	    // 4. Reassign "hello" to "world!";
	    mlt_properties_set( properties, "hello", "world!" );

	    // 5. Retrieve and print the value of "hello";
	    printf( "%s\n", mlt_properties_get( properties, "hello" ) );

	    // 6. Assign the value "0" to "int";
	    mlt_properties_set( properties, "int", "0" );

	    // 7. Retrieve and print the integer value of "int";
	    printf( "%d\n", mlt_properties_get_int( properties, "int" ) );

	    // 8. Assign the integer value 50 to "int2";
	    mlt_properties_set_int( properties, "int2", 50 );

	    // 9. Retrieve and print the double value of "int2";
	    printf( "%s\n", mlt_properties_get( properties, "int2" ) );

	Steps 2 through 5 demonstrate that the "name" is unique - set operations on 
	an existing "name" change the value. They also free up memory associated to 
	the previous value. Note that it also possible to change type in this way
	too.

	Steps 6 and 7 demonstrate that the properties object handles deserialisation
	from strings. The string value of "0" is set, the integer value of 0 is
	retrieved.

	Steps 8 and 9 demonstrate that the properties object handles serialisation
	to strings.

	To show all the name/value pairs in a properties, it is possible to iterate
	through them:

	    for ( i = 0; i < mlt_properties_count( properties ); i ++ )
	        printf( "%s = %s\n", mlt_properties_get_name( properties, i ),
	                             mlt_properties_get_value( properties, i ) );

	Note that properties are retrieved in the order in which they are set. 

	Properties are also used to hold pointers to memory. This is done via the
	set_data call:

	uint8_t *image = malloc( size );
	mlt_properties_set_data( properties, "image", image, size, NULL, NULL );

	In this example, we specify that the pointer can be retrieved from
	properties by a subsequent request to get_data:

	    image = mlt_properties_get_data( properties, "image", &size );

	or:

	    image = mlt_properties_get_data( properties, "image", NULL );

	if we don't wish to retrieve the size.

	Two points here:

	1) The allocated memory remains after the properties object is closed unless
	   you specify a destructor. In the case above, this can be done with:

	   mlt_properties_set_data( properties, "image", image, size, free, NULL );

	   When the properties are closed, or the value of "image" is changed, the 
	   destructor is invoked.
	
	2) The string value returned by mlt_properties_get is NULL. Typically, you
	   wouldn't wish to serialise an image as a string, but other structures
	   might need such functionality - you can specify a serialiser as the last
	   argument if required (declaration is char *serialise( void * )).

	Properties also provides some more advanced usage capabilities. 
	
	It has the ability to inherit all serialisable values from another properties 
	object:

	    mlt_properties_inherit( this, that );

	It has the ability to mirror properties set on this on another set of
	properties:

	    mlt_properties_mirror( this, that );

	After this call, all serialisable values set on this are passed on to that.


mlt_deque:

	Stacks and queues are essential components in the MLT framework. Being of a
	lazy disposition, we elected to implement a 'Double Ended Queue' (deque) -
	this encapsulates the functionality of both.

	The API of the deque is defined as follows:

	mlt_deque mlt_deque_init( );
	int mlt_deque_count( mlt_deque this );
	int mlt_deque_push_back( mlt_deque this, void *item );
	void *mlt_deque_pop_back( mlt_deque this );
	int mlt_deque_push_front( mlt_deque this, void *item );
	void *mlt_deque_pop_front( mlt_deque this );
	void *mlt_deque_peek_back( mlt_deque this );
	void *mlt_deque_peek_front( mlt_deque this );
	void mlt_deque_close( mlt_deque this );

	The stacking operations are used in a number of places:

	* Reverse Polish Notation (RPN) image and audio operations
	* memory pooling

	The queuing operations are used in:
	
	* the consumer base class;
	* consumer implementations may require further queues.


mlt_pool:

	The MLT framework provides memory pooling capabilities through the mlt_pool
	API. Once initilialised, these can be seen as a straightforward drop in
	replacement for malloc/realloc/free functionality.

	The background behind this API is that malloc/free operations are
	notoriously inefficient, especially when dealing with large blocks of memory
	(such as an image). On linux, malloc is optimised for memory allocations
	less than 128k - memory blocks allocated of these sizes or less are retained
	in the process heap for subsequent reuse, thus bypassing the kernel calls
	for repeated allocation/frees for small blocks of memory. However, blocks of
	memory larger than that require kernel calls and this has a detrimental
	impact on performance.

	The mlt_pool design is simply to hold a list of stacks - there is one stack
	per 2^n bytes (where n is between 8 and 31). When an alloc is called, the
	requested size is rounded to the next 2^n, the stack is retrieved for that
	size, and an item is popped or created if the stack is empty. 

	Each item has a 'header', situated immediately before the returned address - 
	this holds the 'stack' to which the item belongs.

	When an item is released, we retrieve the header, obtain the stack and push
	it back.

	Thus, from the programmers point of view, the API is the same as the
	traditional malloc/realloc/free calls:

	void *mlt_pool_alloc( int size );
	void *mlt_pool_realloc( void *ptr, int size );
	void mlt_pool_release( void *release );


mlt_frame:

	A frame object is essentially defined as:

	+------------+
	|frame       |
	+------------+
	| properties |
	| image stack|
	| audio stack|
	+------------+

	The life cycle of a frame can be represented as follows:

	+-----+----------------------+-----------------------+---------------------+
	|Stage|Producer              |Filter                 |Consumer             |
	+-----+----------------------+-----------------------+---------------------+
	| 0.0 |                      |                       |Request frame        |
	+-----+----------------------+-----------------------+---------------------+
	| 0.1 |                      |Receives request       |                     |
	|     |                      |Request frame          |                     |
	+-----+----------------------+-----------------------+---------------------+
	| 0.2 |Receives request      |                       |                     |
	|     |Generates frame for   |                       |                     |
	|     |current position      |                       |                     |
	|     |Increments position   |                       |                     |
	+-----+----------------------+-----------------------+---------------------+
	| 0.3 |                      |Receives frame         |                     |
	|     |                      |Updates frame          |                     |
	+-----+----------------------+-----------------------+---------------------+
	| 0.4 |                      |                       |Receives frame       |
	+-----+----------------------+-----------------------+---------------------+

	Note that neither the filter nor the consumer have any conception of
	'position' until they receive a frame. Speed and position are properties of
	the producer, and they are assigned to the frame object when the producer
	creates it.

	Step 0.3 is a critical one here - if the filter determines that the frame is
	of interest to it, then it should manipulate the image and/or audio stacks
	and properties as required.

	Assuming that the filter deals with both image and audio, then it should
	push data and methods on to the stacks which will deal with the processing. 
	This can be done with the mlt_frame_push_image and audio methods. In order for 
	the filter to register interest in the frame, the stacks should hold:

	image stack:
	[ producer_get_image ] [ data1 ] [ data2 ] [ filter_get_image ]

	audio stack:
	[ producer_get_audio ] [ data ] [ filter_get_audio ]

	The filter_get methods are invoked automatically when the consumer invokes a
	get_image on the frame. 

	+-----+----------------------+-----------------------+---------------------+
	|Stage|Producer              |Filter                 |Consumer             |
	+-----+----------------------+-----------------------+---------------------+
	| 1.0 |                      |                       |frame_get_image      |
	+-----+----------------------+-----------------------+---------------------+
	| 1.1 |                      |filter_get_image:      |                     |
	|     |                      | pop data2 and data1   |                     |
	|     |                      | frame_get_image       |                     |
	+-----+----------------------+-----------------------+---------------------+
	| 1.2 |producer_get_image    |                       |                     |
	|     |  Generates image     |                       |                     |
	+-----+----------------------+-----------------------+---------------------+
	| 1.3 |                      |Receives image         |                     |
	|     |                      |Updates image          |                     |
	+-----+----------------------+-----------------------+---------------------+
	| 1.4 |                      |                       |Receives image       |
	+-----+----------------------+-----------------------+---------------------+

	Obviously, if the filter isn't interested in the image, then it should leave
	the stack alone, and then the consumer will retrieve its image directly from
	the producer.

	Similarly, audio is handled as follows:

	+-----+----------------------+-----------------------+---------------------+
	|Stage|Producer              |Filter                 |Consumer             |
	+-----+----------------------+-----------------------+---------------------+
	| 2.0 |                      |                       |frame_get_audio      |
	+-----+----------------------+-----------------------+---------------------+
	| 2.1 |                      |filter_get_audio:      |                     |
	|     |                      | pop data              |                     |
	|     |                      | frame_get_audio       |                     |
	+-----+----------------------+-----------------------+---------------------+
	| 2.2 |producer_get_audio    |                       |                     |
	|     |  Generates audio     |                       |                     |
	+-----+----------------------+-----------------------+---------------------+
	| 2.3 |                      |Receives audio         |                     |
	|     |                      |Updates audio          |                     |
	+-----+----------------------+-----------------------+---------------------+
	| 2.4 |                      |                       |Receives audio       |
	+-----+----------------------+-----------------------+---------------------+

	And finally, when the consumer is done with the frame, it should close it.

	Note that a consumer may not evaluate both image and audio for any given
	frame, especially in a realtime environment. See 'Realtime Considerations'
	below.

	By default, a frame has the following properties:

	+------------------+------------------------------------+------------------+
	|Name              |Description                         |Values            |
	+------------------+------------------------------------+------------------+
	|_position         |The producers frame position        |0 to n            |
	+------------------+------------------------------------+------------------+
	|_speed            |The producers speed                 |double            |
	+------------------+------------------------------------+------------------+
	|image             |The generated image                 |NULL or pointer   |
	+------------------+------------------------------------+------------------+
	|alpha             |The generated alpha mask            |NULL or pointer   |
	+------------------+------------------------------------+------------------+
	|width             |The width of the image              |                  |
	+------------------+------------------------------------+------------------+
	|height            |The height of the image             |                  |
	+------------------+------------------------------------+------------------+
	|normalised_width  |The normalised width of the image   |720               |
	+------------------+------------------------------------+------------------+
	|normalised_height |The normalised height of the image  |576 or 480        |
	+------------------+------------------------------------+------------------+
	|progressive       |Indicates progressive/interlaced    |0 or 1            |
	+------------------+------------------------------------+------------------+
	|top_field_first   |Indicates top field first           |0 or 1            |
	+------------------+------------------------------------+------------------+
	|audio             |The generated audio                 |NULL or pointer   |
	+------------------+------------------------------------+------------------+
	|frequency         |The frequency of the audio          |                  |
	+------------------+------------------------------------+------------------+
	|channels          |The channels of the audio           |                  |
	+------------------+------------------------------------+------------------+
	|samples           |The samples of the audio            |                  |
	+------------------+------------------------------------+------------------+
	|aspect_ratio      |The sample aspect ratio of the image|double            |
	+------------------+------------------------------------+------------------+
	|test_image        |Used to indicate no image available |0 or 1            |
	+------------------+------------------------------------+------------------+
	|test_audio        |Used to indicate no audio available |0 or 1            |
	+------------------+------------------------------------+------------------+

	The consumer can attach the following properties which affect the default
	behaviour of a frame:

	+------------------+------------------------------------+------------------+
	|test_card_producer|Synthesise test images from here    |NULL or pointer   |
	+------------------+------------------------------------+------------------+
	|consumer_aspect_  |Apply this aspect ratio to the test |double            |
	|ratio             |card producer                       |                  |
	+------------------+------------------------------------+------------------+
	|rescale.interp    |Use this scale method for test image|"string"          |
	+------------------+------------------------------------+------------------+

	While most of these are mainly self explanatory, the normalised_width and
	normalised_height values require a little explanation. These are required
	to ensure that effects are consistently handled as PAL or NTSC, regardless 
	of the consumers or producers width/height image request. 

	The test_image and audio flags are used to determine when images and audio
	should be synthesised.

	Additional properties may be provided by the producer implementation, and
	filters, transitions and consumers may add additional properties to
	communicate specific requests. These are documented in modules.txt.

	The complete API for the mlt frame is as follows:

	mlt_frame mlt_frame_init( );
	mlt_properties mlt_frame_properties( mlt_frame this );
	int mlt_frame_is_test_card( mlt_frame this );
	int mlt_frame_is_test_audio( mlt_frame this );
	double mlt_frame_get_aspect_ratio( mlt_frame this );
	int mlt_frame_set_aspect_ratio( mlt_frame this, double value );
	mlt_position mlt_frame_get_position( mlt_frame this );
	int mlt_frame_set_position( mlt_frame this, mlt_position value );
	int mlt_frame_get_image( mlt_frame this, uint8_t **buffer, mlt_image_format *format, int *width, int *height, int writable );
	uint8_t *mlt_frame_get_alpha_mask( mlt_frame this );
	int mlt_frame_get_audio( mlt_frame this, int16_t **buffer, mlt_audio_format *format, int *frequency, int *channels, int *samples );
	int mlt_frame_push_get_image( mlt_frame this, mlt_get_image get_image );
	mlt_get_image mlt_frame_pop_get_image( mlt_frame this );
	int mlt_frame_push_frame( mlt_frame this, mlt_frame that );
	mlt_frame mlt_frame_pop_frame( mlt_frame this );
	int mlt_frame_push_service( mlt_frame this, void *that );
	void *mlt_frame_pop_service( mlt_frame this );
	int mlt_frame_push_audio( mlt_frame this, void *that );
	void *mlt_frame_pop_audio( mlt_frame this );
	void mlt_frame_close( mlt_frame this );


mlt_service:

	The service base class extends properties and allows 0 to m inputs and 0 to
	n outputs and is represented as follows:

	    +-----------+
	- ->|           |- ->
	- ->|  Service  |- ->
	- ->|           |
	    +-----------+
	    | properties|
	    +-----------+

	Descendents of service impose restrictions on how inputs and outputs can be 
	connected and will provide a basic set of properties. Typically, the service 
	instance is encapsulated by the descendent in order for it to ensure that
	its connection rules are followed.

	A service does not define any properties when constructed. It should be
	noted that producers, filters and transitions my be serialised (say, via the
	xml consumer), and care should be taken to distinguish between
	serialisable and transient properties. The convention used is to prefix
	transient properties with an underscore.

	The public interface is defined by the following functions:

	int mlt_service_init( mlt_service this, void *child );
	mlt_properties mlt_service_properties( mlt_service this );
	int mlt_service_connect_producer( mlt_service this, mlt_service producer, int index );
	int mlt_service_get_frame( mlt_service this, mlt_frame_ptr frame, int index );
	void mlt_service_close( mlt_service this );

	Typically, only direct descendents of services need invoke these methods and
	developers are encouraged to use those extensions when defining new services. 


mlt_producer:

	A producer has 0 inputs and 1 output:

	    +-----------+
	    |           |
	    | Producer  |--->
	    |           |
	    +-----------+
	    | service   |
	    +-----------+

	A producer provides an abstraction for file readers, pipes, streams or any
	other image or audio input. 

	When instantiated, a producer has the following properties:

	+------------------+------------------------------------+------------------+
	|Name              |Description                         |Values            |
	+------------------+------------------------------------+------------------+
	|mlt_type          |The producers type                  |mlt_producer      |
	+------------------+------------------------------------+------------------+
	|_position         |The producers frame position        |0 to n            |
	+------------------+------------------------------------+------------------+
	|_speed            |The producers speed                 |double            |
	+------------------+------------------------------------+------------------+
	|fps               |The output frames per second        |25 or 29.97       |
	+------------------+------------------------------------+------------------+
	|in                |The in point in frames              |0 to length - 1   |
	+------------------+------------------------------------+------------------+
	|out               |The out point in frames             |in to length - 1  |
	+------------------+------------------------------------+------------------+
	|length            |The length of the input in frames   |0 to n            |
	+------------------+------------------------------------+------------------+
	|aspect_ratio      |aspect_ratio of the source          |0 to n            |
	+------------------+------------------------------------+------------------+
	|eof               |end of clip behaviour               |"pause" or "loop" |
	+------------------+------------------------------------+------------------+
	|resource          |Constructor argument (ie: file name)|"<resource>"      |
	+------------------+------------------------------------+------------------+

	Additional properties may be provided by the producer implementation.

	The public interface is defined by the following functions:

	mlt_producer mlt_producer_new( );
	int mlt_producer_init( mlt_producer this, void *child );
	mlt_service mlt_producer_service( mlt_producer this );
	mlt_properties mlt_producer_properties( mlt_producer this );
	int mlt_producer_seek( mlt_producer this, mlt_position position );
	mlt_position mlt_producer_position( mlt_producer this );
	mlt_position mlt_producer_frame( mlt_producer this );
	int mlt_producer_set_speed( mlt_producer this, double speed );
	double mlt_producer_get_speed( mlt_producer this );
	double mlt_producer_get_fps( mlt_producer this );
	int mlt_producer_set_in_and_out( mlt_producer this, mlt_position in, mlt_position out );
	mlt_position mlt_producer_get_in( mlt_producer this );
	mlt_position mlt_producer_get_out( mlt_producer this );
	mlt_position mlt_producer_get_playtime( mlt_producer this );
	mlt_position mlt_producer_get_length( mlt_producer this );
	void mlt_producer_prepare_next( mlt_producer this );
	void mlt_producer_close( mlt_producer this );


mlt_filter:

	The public interface is defined by the following functions:

	int mlt_filter_init( mlt_filter this, void *child );
	mlt_filter mlt_filter_new( );
	mlt_service mlt_filter_service( mlt_filter this );
	mlt_properties mlt_filter_properties( mlt_filter this );
	mlt_frame mlt_filter_process( mlt_filter this, mlt_frame that );
	int mlt_filter_connect( mlt_filter this, mlt_service producer, int index );
	void mlt_filter_set_in_and_out( mlt_filter this, mlt_position in, mlt_position out );
	int mlt_filter_get_track( mlt_filter this );
	mlt_position mlt_filter_get_in( mlt_filter this );
	mlt_position mlt_filter_get_out( mlt_filter this );
	void mlt_filter_close( mlt_filter );


mlt_transition:

	The public interface is defined by the following functions:

	int mlt_transition_init( mlt_transition this, void *child );
	mlt_transition mlt_transition_new( );
	mlt_service mlt_transition_service( mlt_transition this );
	mlt_properties mlt_transition_properties( mlt_transition this );
	int mlt_transition_connect( mlt_transition this, mlt_service producer, int a_track, int b_track );
	void mlt_transition_set_in_and_out( mlt_transition this, mlt_position in, mlt_position out );
	int mlt_transition_get_a_track( mlt_transition this );
	int mlt_transition_get_b_track( mlt_transition this );
	mlt_position mlt_transition_get_in( mlt_transition this );
	mlt_position mlt_transition_get_out( mlt_transition this );
	mlt_frame mlt_transition_process( mlt_transition this, mlt_frame a_frame, mlt_frame b_frame );
	void mlt_transition_close( mlt_transition this );


mlt_consumer:

	The public interface is defined by the following functions:

	int mlt_consumer_init( mlt_consumer this, void *child );
	mlt_service mlt_consumer_service( mlt_consumer this );
	mlt_properties mlt_consumer_properties( mlt_consumer this );
	int mlt_consumer_connect( mlt_consumer this, mlt_service producer );
	int mlt_consumer_start( mlt_consumer this );
	mlt_frame mlt_consumer_get_frame( mlt_consumer this );
	mlt_frame mlt_consumer_rt_frame( mlt_consumer this );
	int mlt_consumer_stop( mlt_consumer this );
	int mlt_consumer_is_stopped( mlt_consumer this );
	void mlt_consumer_close( mlt_consumer );


Specialised Producers:

	There are two major types of specialised producers - playlists and tractors.

	The following sections describe these.


mlt_playlist:

	mlt_playlist mlt_playlist_init( );
	mlt_producer mlt_playlist_producer( mlt_playlist this );
	mlt_service mlt_playlist_service( mlt_playlist this );
	mlt_properties mlt_playlist_properties( mlt_playlist this );
	int mlt_playlist_count( mlt_playlist this );
	int mlt_playlist_clear( mlt_playlist this );
	int mlt_playlist_append( mlt_playlist this, mlt_producer producer );
	int mlt_playlist_append_io( mlt_playlist this, mlt_producer producer, mlt_position in, mlt_position out );
	int mlt_playlist_blank( mlt_playlist this, mlt_position length );
	mlt_position mlt_playlist_clip( mlt_playlist this, mlt_whence whence, int index );
	int mlt_playlist_current_clip( mlt_playlist this );
	mlt_producer mlt_playlist_current( mlt_playlist this );
	int mlt_playlist_get_clip_info( mlt_playlist this, mlt_playlist_clip_info *info, int index );
	int mlt_playlist_insert( mlt_playlist this, mlt_producer producer, int where, mlt_position in, mlt_position out );
	int mlt_playlist_remove( mlt_playlist this, int where );
	int mlt_playlist_move( mlt_playlist this, int from, int to );
	int mlt_playlist_resize_clip( mlt_playlist this, int clip, mlt_position in, mlt_position out );
	void mlt_playlist_close( mlt_playlist this );

mlt_tractor: