     [file version.texi not included]

The libextractor Library
************************

This manual is for libextractor (version VERSION, UPDATED), which is GNU's library for meta data extraction.

Copyright © 2007 Christian Grothoff

     Permission is granted to copy, distribute and/or modify this document under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. The text of the license can be found in the section entitled ``Copying''.

Introduction
************

libextractor is GNU's library for extracting meta data from files.  Meta data includes format information (such as mime type, image dimensions, color depth, recording frequency), content descriptions (such as document title or document description) and copyright information (such as license, author and contributors). Meta data extraction is an inherently uncertain business --- a parse error can be a corrupt file, an incompatibility in the file format version, an entirely different file format or a bug in the parser.  As a result of this uncertainty, libextractor deliberately avoids to ever report any errors.  Unexpected file contents simply result in less or possibly no meta data being extracted.

libextractor uses plugins to handle various file formats. Technically a plugin can support multiple file formats; however, most plugins only support one particular format.  libextractor uses a certain set of default plugins.  Certain additional plugins are only used if specifically requested.  Optional plugins exist for computing various cryptographic hash functions (the hash of the file is returned as meta data), thumbnail creation for images and full-text extraction (based on language-specific dictionaries).  A few plugins exist to manipulate results from other plugins, such as splitting meta data into individual words at particular characters and converting the output to lower case letters.

libextractor is distributed with the extract command (Some distributions ship extract in a seperate package.) which is a command-line tool for extracting meta data.  extract is given a list of filenames and prints the resulting meta data to the console.  The extract source code also serves as an advanced example for how to use libextractor.

This manual focuses on providing documentation for writing software with libextractor.  The only relevant parts for end-users are the chapter on compiling and installing libextractor (see Preparation.).  Also, the chapter on existing plugins maybe of interest (see Existing Plugins.).  Additional documentation for end-users can be find in the man page on extract (using man extract).

libextractor is licensed under the GNU Public License.  The developers have frequently received requests to license GNU libextractor under alternative terms.  However, libextractor borrows plenty of GPL-licensed code from various other projects. Hence we cannot change the license (even if we wanted to). (It maybe possible to switch to GPLv3 in the future.  For this, an audit of the license status of our dependencies would be required.  The new code that was developed specifically for libextractor has always been licensed under GPLv2 or any later version.)

Preparation
***********

Compiling libextractor follows the standard GNU autotools build process using configure and make.  For details, read the INSTALL file and query ./configure --help for additional options.

libextractor has various dependencies, some of which are optional.  You can find the current list specified in terms of Debian GNU/Linux packages in README.debian.  Note that transitive dependencies are not listed.

Once you have compiled and installed libextractor, you should have a file include/extractor.h installed in your include directory.  This file should be the starting point for your C and C++ development with libextractor.  The build process also installs the extract binary and man pages for extract and libextractor.  The extract man page documents the extract tool.  The libextractor man page gives a brief summary of the C API for libextractor.

When you install libextractor, various plugins will be installed in the lib/libextractor/ directory.  The main library will be installed as lib/libextractor.so.  Note that libextractor will attempt to find the plugins relative to the path of the main library.  Consequently, a package manager can move the library and its plugins to a different location later --- as long as the relative path between the main library and the plugins is preserved.  As a method of last resort, the user can specify an environment variable LIBEXTRACTOR_PREFIX.  If libextractor cannot locate a plugin, it will look in LIBEXTRACTOR_PREFIX/lib/libextractor/.

Generalities
************

Each public symbol exported by libextractor has the prefix EXTRACTOR_.  All-caps names are used for constants.  For the impatient, the minimal C code for using libextractor (on the executing binary itself) looks like this:

#include <extractor.h>
int main(int argc, char ** argv) {
  EXTRACTOR_ExtractorList * plugins
    = EXTRACTOR_loadDefaultLibraries();
  EXTRACTOR_KeywordList * keywords
    = EXTRACTOR_getKeywords(plugins, argv[0]);
  EXTRACTOR_printKeywords(stdout, keywords);
  EXTRACTOR_freeKeywords(keywords);
  EXTRACTOR_removeAll(plugins);
  return 0;
}

libextractor defines three main data types.  Understanding these types is key for programming with libextractor.

First, EXTRACTOR_KeywordType is a C enum which defines a list of over 100 different types of meta data.  The total number can differ between different libextractor releases; the maximum value for the current release can be obtained using the EXTRACTOR_getHighestKeywordTypeNumber function.

The function EXTRACTOR_getKeywordTypeAsString can be used to obtain an English string s describing the meta data type.  The string maybe translated into other languages using GNU gettext with the domain set to libextractor (dgettext("libextractor", s)).

Second, libextractor uses a linked list EXTRACTOR_KeywordList to store extracted meta data.  The structure is extremely simple, consisting of a field keyword which holds the meta data as a zero terminated UTF-8 string, a field keywordType of type EXTRACTOR_KeywordType which describes the type of the meta data and a next pointer to the next meta data record (or NULL for the last entry).

Clients are allowed to directly access the keyword list.  After returning the list to the client, libextractor makes no assumptions about how the client may process the list.  The client is responsible for freeing both the linked list and the keywords stored in the linked list.  The EXTRACTOR_freeKeywords function provides a convenient way to do so.

Finally, the third data type used by libextractor is the EXTRACTOR_ExtractorList.  This linked list is used to store information about the plugins that libextractor is using to obtain meta data.  The struct contains information identifying the plugin, a pointer to the meta data extraction function provided by the plugin and options for the plugin. Most clients should never be concerned with the internals of this type.

Clients should use the EXTRACTOR_loadDefaultLibraries function to create an extractor list with the default set of plugins. Additional functions are provided for loading optional plugins.  Once the meta data extraction is complete, clients should call EXTRACTOR_removeAll to unload the plugins and free the extractor list.  Note that loading and unloading the plugins is a relatively costly operation.  If possible, programmers should avoid needlessly loading and unloading the plugins.

All of the functions for loading and unloading plugins, including EXTRACTOR_loadDefaultLibraries and EXTRACTOR_removeAll, are thread-safe, but not reentrant. The functions use operating system specific calls to dynamically load and unload objects, and in many cases, these calls are not thread-safe. In libextractor code, locking is used to protect access to these operating system specific calls.

Generally, libextractor is fully thread-safe and mostly reentrant. All plugin code is required to be reentrant, so the same set of plugins can be safely used concurrently by multiple threads. Reentrant functions, naturally, do not protect their arguments (plugin or keyword lists) against concurrent access. For further information on libextractor's thread-safety and reentrancy, also see the following chapters on libextractor usage.

Extracting meta data
********************

The main functions for extracting keywords with libextractor are EXTRACTOR_getKeywords and EXTRACTOR_getKeywords2.  The first argument to both functions is an extractor list which specifies the set of plugins that should be used for extracting meta data.   Both functions return a linked list of keywords (of type EXTRACTOR_KeywordList). Also, both functions are reentrant.

The difference between the two functions is in how the file from which the meta data should be obtained is specified. The EXTRACTOR_getKeywords function simply takes the name of the file as its second argument.  The EXTRACTOR_getKeywords2 function takes a (read-only) buffer and its size (in bytes) as arguments, allowing meta data extraction from memory.

Meta data extraction should never really fail -- at worst, libextractor should return a NULL pointer indicating that no meta data was found.  By design, libextractor should never crash or leak memory, even given corrupt files as input.  Note however, that running libextractor on a corrupt file system (or incorrectly mmaped files) can result in the operating system sending a SIGBUS (bus error) to the process.  libextractor will not attempt to catch this signal and your application is likely to crash.  Note again that this should only happen if the file system is corrupt (not if individual files are corrupt).  If this is not acceptable, you might want to consider running libextractor out-of-process (as done, for example, by doodle (http://gnunet.org/doodle/)).

Customizing the Plugins
=======================

The functions described in this section can be used to control which plugins are loaded and in which order they are executed.  This provides more control than simply sticking to the default list.  It is recommended that clients start by loading the default plugins and then allow the user to perform further customizations using command-line options or a configuration file.

Like EXTRACTOR_loadDefaultLibraries and EXTRACTOR_removeAll, these functions are thread-safe, but not reentrant.

The EXTRACTOR_addLibrary function can be used to add a single plugin to the front of the list of already loaded plugins.

Use EXTRACTOR_addLibraryLast to add a plugin to the end of the list of plugins.  Plugins at the end can benefit from analyses performed by plugins earlier (such as mime-type detection).  This can be useful to avoid more expensive processing.  Some plugins need to be loaded last since they transform the keyword list.  For example, the split plugin tokenizes the meta data returned by other plugins.  Since this plugin needs the keywords generated by the other plugins as its' input, it must be run last in order to ensure that it is applied to all keywords.

The EXTRACTOR_removeLibrary function can be used to unload a specific plugin.

The EXTRACTOR_loadConfigLibraries provides a simple interface for highly customized loading of plugins.  The provided configuration string can specify multiple plugins (seperated by colons) with optional arguments to the plugins (in parenteses after the plugin name).  By default the plugins are pre-pended to the plugin list in the order that the plugins are processed; however, by prefixing the plugin name with a minus-sign that particular library will be appended to the list.

Language bindings
*****************

libextractor works immediately with C and C++ code. Bindings for Java, Ruby, Perl, PHP and Python are available for download from the main libextractor website.  Documentation for these bindings (if available) is part of the downloads for the respective binding.  In all cases, a full installation of the C library is required before the binding can be installed.

Utility functions
*****************

This chapter describes various utility functions for libextractor usage. All of the functions are reentrant.

Utility Constants
=================

The constant EXTRACTOR_VERSION is a hexadecimal representation of the version number of the installed libextractor header.  The hexadecimal format is 0xAABBCCDD where AA is the major version (so far always 0), BB is the minor version, CC is the revision and DD the patch number.  For example, for version 0.5.18, we would have AA=0, BB=5, CC=18 and DD=0.  Minor releases such as 0.5.18a or significant changes in unreleased versions would be marked with DD=1 or higher.

If your code needs to find out what the set of default plugins that would be loaded with EXTRACTOR_loadDefaultLibraries is, the function EXTRACTOR_getDefaultLibraries returns a constant string which follows the format of EXTRACTOR_loadConfigLibraries and specifies the set of default plugins for this version of libextractor.

Misc Keyword List
=================

This section describes a couple of convenience functions for manipulating the keyword list.

The EXTRACTOR_printKeywords is a simple function which prints the meta data found with libextractor to a file.  The function is mostly useful for debugging and as an example for how to manipulate the keyword list.

The EXTRACTOR_countKeywords returns the number of keywords found.  It simply traverses the linked list and determines its length.

Cleaning up the Keyword List
============================

The EXTRACTOR_removeDuplicateKeywords function can be used to remove duplicate keywords.  This is useful since the same keyword can be extracted by multiple plugins.  In addition to the keyword list, the function requires an argument options which is used to specify what should be considered a duplicate.  The value zero should be used if only exact matches (same keyword string and same meta data type) should be removed.  Use EXTRACTOR_DUPLICATES_TYPELESS in order to remove duplicate keywords even if the meta data types are different.  Use EXTRACTOR_DUPLICATES_REMOVE_UNKNOWN to remove entries where the keywords match exactly and one of the keyword types is EXTRACTOR_UNKNOWN.  Given the choice, EXTRACTOR_removeDuplicateKeywords will always prefer to remove the duplicate keyword with the EXTRACTOR_UNKNOWN and EXTRACTOR_SPLIT types.

The EXTRACTOR_removeEmptyKeywords simply removes meta data entries that are empty strings.

The EXTRACTOR_removeKeywordsOfType function can be used to remove all keywords of a particular meta data type.

Finding Keywords
================

The EXTRACTOR_extractLast function returns the last keyword value in the linked list that has the given meta data type.

The EXTRACTOR_extractLastByString works exactly like EXTRACTOR_extractLast except that the meta data type is specified using the corresponding string value.

Binary Meta data
================

libextractor supports binary meta data.  At this point, binary meta data is rarely encountered -- the default meta data plugins do not generate binary meta data.  However, you can load one of the thumbnail-plugins to generate thumbnails for various images. These thumbnails are binary meta data.  In the future it is conceivable that additional binary meta data extractors will be implemented.

Since the simple linked list of keywords that is used by the libextractor API only contains 0-terminated strings, binary meta data cannot be directly returned as-is by the plugins. Also, if binary meta data is returned, the keyword string does not point to the usual UTF-8 encoded string.  Applications must be aware of this in order to avoid printing binary meta data by accident as if it was UTF-8.

The EXTRACTOR_isBinaryType function is used to test if a particular meta data type contains binary meta data.  If this function returns 0, the client can be certain that the keyword is UTF-8 and not binary.

In order to handle zero-bytes in binary data, libextractor encodes binary data in a particular format.  The function EXTRACTOR_binaryDecode can be used to convert the 0-terminated string back to the standard binary format.

The EXTRACTOR_binaryEncode function converts binary data into libextractor's internal 0-terminated binary format.  This function is mostly useful for writing plugins that need to return binary meta data.

Existing Plugins
****************

HTML, PDF, PS, OLE2 (DOC, XLS, PPT), OpenOffice (sxw), StarOffice (sdw), DVI, MAN, MP3 (ID3v1 and ID3v2), NSF (NES Sound Format), SID, OGG, WAV, EXIV2, JPEG, GIF, PNG, TIFF, DEB, RPM, TAR(.GZ), ZIP, ELF, REAL, RIFF (AVI), MPEG, QT and ASF.

Writing new Plugins
*******************

Writing a new plugin for libextractor usually requires writing of or interfacing with an actual parser for a specific format.  How this is can be accomplished depends on the format and cannot be specified in general.  However, care should be taken for the code to be reentrant and highly fault-tolerant, especially with respect to malformed inputs.

Plugins should start by verifying that the header of the data matches the specific format and immediately return if that is not the case. Even if the header matches the expected file format, plugins must not assume that the remainder of the file is well formed.

The plugin library must be called libextractor_XXX.so, where XXX denotes the file format of the plugin. The library must export a method libextractor_XXX_extract, with the following signature:
struct EXTRACTOR_Keywords *
libextractor_XXX_extract
   (const char * filename,
    const char * data,
    size_t size,
    struct EXTRACTOR_Keywords * prev,
    const char * options);

The argument filename specifies the name of the file being processed and maybe NULL.  data is a pointer to the typically memory mapped contents of the file.  Note that plugins cannot ignore the const annotation since the memory mapping may have been done read-only (and thus writes to this page will result in an error).  The size argument specifies the size of the data buffer in bytes.

prev is the list of keywords extracted so far by other plugins for the file.  The function is expected to return an updated list of keywords.  If the format does not match the expectations of the plugin, prev is returned.  Most plugins use a function such as addKeyword to extend the list:

static void addKeyword
   (struct EXTRACTOR_Keywords ** list,
    const char * keyword,
    EXTRACTOR_KeywordType type)
{
  EXTRACTOR_KeywordList * next;
  next = malloc(sizeof(EXTRACTOR_KeywordList));
  next->next = *list;
  next->keyword = strdup(keyword);
  next->keywordType = type;
  *list = next;
}

A typical use of addKeyword is to add the MIME type once the file format has been established (beyond reasonable doubt).

Internal utility functions
**************************

Various libextractor plugins make use of the internal convert.h header which defines a function EXTRACTOR_common_convert_to_utf8 which can be used to easily convert text from any character set to UTF-8.  This conversion is important since the linked list of keywords that is returned by libextractor is expected to contain only UTF-8 strings.  Naturally, proper conversion may not always be possible since some file formats fail to specify the character set.  In that case, it is often better to not convert at all.

The arguments to EXTRACTOR_common_convert_to_utf8 are the input string (which does not have to be zero-terminated), the length of the input string, and the character set (which must be zero-terminated). Which character sets are supported depends on the platform, a list can generally be obtained using the iconv -l command.  The return value from EXTRACTOR_common_convert_to_utf8 is a zero-terminated string in UTF-8 format.  The responsibility to free the string is with the caller, so storing the string in the keyword list is acceptable.

Reporting bugs
**************

libextractor uses the Mantis bugtracking system (http://gnunet.org/mantis/).  If possible, please report bugs there.  You can also e-mail the libextractor mailinglist at libextractor@gnu.org.

     [file gpl.texi not included]