File: source-highlight.texinfo

package info (click to toggle)
source-highlight 3.1.7-1
links: PTS
area: main
in suites: jessie, jessie-kfreebsd
size: 10,332 kB
ctags: 5,233
sloc: sh: 11,270; cpp: 10,206; ansic: 9,515; makefile: 1,865; lex: 1,200; yacc: 1,021; php: 213; perl: 211; awk: 98; erlang: 94; lisp: 90; java: 75; ruby: 69; python: 61; asm: 43; ml: 38; ada: 36; haskell: 27; xml: 23; cs: 11; sql: 8; tcl: 6; sed: 4
file content (4568 lines) | stat: -rw-r--r-- 169,600 bytes
\input texinfo @c -*-texinfo-*-
@c This file uses the @command command introduced in Texinfo 4.0.
@c %**start of header
@setfilename source-highlight.info
@include version.texi
@settitle GNU Source-highlight @value{VERSION}
@finalout
@c @setchapternewpage odd
@c %**end of header

@set myhomepage @uref{http://www.lorenzobettini.it}
@set langfilesurl @uref{http://www.gnu.org/software/src-highlite/lang_files/}
@set outlangfilesurl @uref{http://www.gnu.org/software/src-highlite/outlang_files/}
@set srchilite Source-highlight
@set srchilitelib Source-highlight library

@copying
This manual is for GNU Source-highlight 
(version @value{VERSION}, @value{UPDATED}),
which given a source file, produces a document with syntax highlighting.

Copyright @copyright{} 2005-2008 Lorenzo Bettini, @value{myhomepage}.

@quotation
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.1 or
any later version published by the Free Software Foundation; with no
Invariant Sections, with no Front-Cover Texts, and no Back-Cover
Texts. A copy of the license is included in the section entitled ``GNU
Free Documentation License.''
@end quotation
@end copying

@dircategory Utilities
@direntry
* Source-highlight: (source-highlight).  Highlights contents
@end direntry

@titlepage
@title GNU Source-highlight
@subtitle given a source file, produces a document with syntax highlighting.
@subtitle for GNU Source-highlight Version @value{VERSION}
@subtitle updated on @value{UPDATED-MONTH}

@author by Lorenzo Bettini, @value{myhomepage}

@page
@vskip 0pt plus 1filll
@insertcopying
@end titlepage

@contents

@ifnottex
@node Top, Introduction, (dir), (dir)
@top GNU Source-highlight

GNU Source-highlight, given a source file, produces a document with
syntax highlighting.

This is Edition @value{EDITION} of the Source-highlight manual.

This file documents GNU Source-highlight version @value{VERSION}.

@insertcopying
@end ifnottex

@c All the nodes can be updated using the EMACS command
@c texinfo-every-node-update, which is normally bound to C-c C-u C-e.
@c @node Top, Introduction, (dir), (dir)

@c All the menus can be updated with the EMACS command
@c texinfo-all-menus-update, which is normally bound to C-c C-u C-a.
@menu
* Introduction::                What's it for?
* Installation::                Download and installation           
* Copying::                     Licence issues
* Simple Usage::                Very basic usage
* Configuration files::         Files needed for execution
* Invoking source-highlight::   How to run @command{source-highlight}.
* Language Definitions::        How to define an input language
* Output Language Definitions::  How to define an output format
* Generating References::       Anchors and cross references
* Examples::                    Some output examples
* Problems::                    Reporting bugs.
* Mailing Lists::               
* Concept Index::               Index of concepts.
@end menu

@node Introduction, Installation, Top, Top
@chapter Introduction

@cindex introduction
@cindex features

GNU Source-highlight, given a source file, produces a document with
syntax highlighting.  The colors and the styles can be specified (bold,
italics, underline) by means of a configuration file, and some other
options can be specified at the command line.   

The program already recognizes many programming languages (e.g., C++,
Java, Perl, etc.) and file formats (e.g., log files, ChangeLog, etc.),
and some output formats (e.g., HTML, ANSI color escape sequences,
@LaTeX{}, etc.).  Since version 2.0, it allows you to specify your own
input source language via a simple syntax described later in this manual
(@ref{Language Definitions}).  Since version 2.1,  it allows you to
specify your own output format language via a simple syntax described
later in this manual (@ref{Output Language Definitions}).  Since version
2.2, it is able to generate cross references (e.g., to variable names,
field names, etc.) by relying on the program @emph{ctags},
@url{http://ctags.sourceforge.net} (@ref{Generating References}).

@cindex library
Since version 3.0, GNU Source-highlight also provides a C++ library
(which is used by the main program itself), that can be used
by C++ programmers to add highlighting functionalities to their
programs.  @inforef{Introduction,,source-highlight-info}.

@menu
* Supported languages::         
* The program source-highlight-settings::  
* Notes on some languages::     
* Using source-highlight as a simple formatter::  
* Related Software and Links::  
@end menu

@node Supported languages, The program source-highlight-settings, Introduction, Introduction
@section Supported languages

The complete list of languages (indeed, file extensions) natively
supported by this version of Source-highlight (@value{VERSION}), as
reported by @code{--lang-list}, is the following:

@example
@include lang-list.texinfo
@end example

The complete list of output formats natively supported by this version
of Source-highlight (@value{VERSION}), as reported by
@code{--outlang-list}, is the following:

@example
@include outlang-list.texinfo
@end example

@noindent
The meaning of the suffix @code{-css} is explained in @ref{Output
Language map}@footnote{Up to version 2.9, there were also the suffixes
@code{-doc} and @code{-css-doc}, but this mechanism was quite confusing
and complex; hopefully, this new one should be better.}.

Please, keep in mind, that I haven't tested personally all these
language definitions: I actually checked that the definition files are
syntactically correct (with the command line option @code{--check-lang}
and @code{--check-outlang}, @ref{Invoking source-highlight}), but I'm
not sure their definition actually respects that language syntax (e.g.,
I've put up together some language definitions by searching for
information in the Internet, but I've never programmed in that
language).  So, if you find that a language definition is not precise,
please let me know.  Moreover, if you have a program example in a
language that's not included in the @file{tests} directory, please send
it to me so that I can include it in the test suite.

@node The program source-highlight-settings, Notes on some languages, Supported languages, Introduction
@section The program @code{source-highlight-settings}

@cindex source-highlight-settings
Since version 3.0, GNU Source-highlight includes also the program
@code{source-highlight-settings}, which can be used to check whether
source-highlight will be able find its language definition files, and
other configuration files, and in case, to store the correct settings in
a configuration file, in the user home directory.

@cindex source-highlight.conf
In particular, the stored configuration file will be called
@file{source-highlight.conf} and stored in
@file{$HOME/.source-highlight/}.

@cindex @code{--data-dir}
For the moment, this file only stores the default value for
the @code{--data-dir} option.

The user can always override the contents of this configuration file,
and the default hardcoded value, by using the environment variable
@cindex @code{SOURCE_HIGHLIGHT_DATADIR}
@code{SOURCE_HIGHLIGHT_DATADIR}.

@node Notes on some languages, Using source-highlight as a simple formatter, The program source-highlight-settings, Introduction
@section Notes on some languages

In this section I'd like to go into details on the highlighting of some
specific programming languages.  These notes might be useful when the
highlighted language has some ``dialects'' that might require some
further specification at the command line (e.g., to select a specific
dialect).

@menu
* Fortran::                     
* Perl::                        
@end menu

@node Fortran, Perl, Notes on some languages, Notes on some languages
@subsection Fortran

@cindex Fortran
As Toby White explained to me, Fortran comes into different ``flavors'':
a fixed-format, where some characters have a different semantics
depending on their column position in the source file, and a free-format
where this is not true.  For instance, in the former, @code{*} and
@code{c} start a command line, but only if they are specified in the
first column (while this is not true in the free-format).

By default, the free-format is assumed for Fortran files; if you want to
use the fixed-format, you need to specify @code{fortran-fixed} at
the @code{--src-lang} command line option.

@node Perl,  , Fortran, Notes on some languages
@subsection Perl

@cindex Perl
Perl syntax forms, especially its regular expression specifications, are
quite a nightmare ;-)  I tried to specify as much as possible in the
@file{perl.lang} but some particular regular expressions might not be
highlighted correctly.  Actually, I never programmed in Perl, so, if you
see that some parts of your Perl programs are not highlighted correctly,
please do not hesitate to contact me, so that I can improve Perl
highlighting.

@cindex @code{--infer-lang}
Moreover, although the standard extension for Perl files is @code{.pl},
since the Prolog language definition was implemented in source-highlight
before Perl, this extension is assigned, by default, to Prolog files.
However, you can use @code{--infer-lang} command line option, so that
source-highlight can try to detect the language by inspecting the first
lines of the input file (@ref{How the input language is discovered});
you can also use @code{--src-lang=perl} command line specification to
explicitly require Perl highlighting.

@node Using source-highlight as a simple formatter, Related Software and Links, Notes on some languages, Introduction
@section Using source-highlight as a simple formatter

You can also use source-highlight as a simple formatter of input file,
i.e., without performing any highlighting@footnote{Although this might
have been achieved with previous version, it is an official supported
feature since version 2.5.}.

@cindex nohilite.lang
You can achieve this by using, as the language definition file for input
sources the file @file{nohilite.lang}, using the command line option
@code{--lang-def} (@ref{Invoking source-highlight}).  Since that
language definition is empty, no highlighting will be performed;
however, source-highlight will transform the input file in the output
format.  Note, in the input language associations in @ref{Supported
languages}, that @file{nohilite.lang} is also associated to txt files.

This, for instance, makes source-highlight useful in cases you want to
transform a text file into HTML or @LaTeX{}.  During the output, in
fact, source-highlight will correctly generate characters that have a
specific meanings in the output format.

@include txt2texinfo.txt

This is the Texinfo source of the above sentence:

@include txt2texinfo.txt.texinfo

@noindent
This was processed by source-highlight as a simple text file, without no
highlighting; however since it was formatted in Texinfo, all the
necessary escaping was automatically performed.  This way, it is very
easy to insert, in the same document, a code, and its result (as in this
example).

This is actually the formatting performed by source-highlight; except
for the comment, this is basically what you should have written yourself
to do all the escaping stuff manually:

@include txt2texinfo.txt.texinfo.texinfo

@cindex failsafe
In case source-highlight does not handle a specific input language, you
can still use the option @code{--failsafe} (@ref{Invoking
source-highlight}) and also in that case no highlighting will be
performed, but source-highlight will transform the input file in the
output format.

@cindex default.lang
Note, however, that if the input language cannot be established, the
@file{default.lang} will be used: an empty language definition file
which you might want to customize.

@node Related Software and Links,  , Using source-highlight as a simple formatter, Introduction
@section Related Software and Links

Here we list some software related to source-highlight in the sense that
it uses it as a backend (i.e., provides an interface to
source-highlight) or it uses some of its features (e.g., definition
files):

@itemize

@item
@cindex Source-Highlight-Qt
@cindex Qt
Source-highlight-qt is a library for performing syntax highlighting in
Qt documents by relying on GNU Source-Highlight library.  This library
provides an implementation of the qt abstract class QSyntaxHighlighter
class, and it deals both with Qt3 and Qt4.

@uref{http://srchiliteqt.sourceforge.net}.

@item
@cindex QSource-Highlight
@cindex Qt
QSource-Highlight is a Qt4 front-end for GNU Source-Highlight (it relies
on the library Source-Highlight-Qt).  You can highlight your code on the
fly, and have the highlighted output in all the formats supported by
source-highlight (e.g., HTML, LaTeX, Texinfo, etc.). You can then copy
the formatted output and paste it (e.g., in your blog), or save it to a
file.  A preview of the highlighted output is available for some output
formats (e.g., HTML, XHTML, etc.).

@uref{http://qsrchilite.sourceforge.net}.

@item
@cindex SourceHighlightIDE
@cindex KDE
SourceHighlightIDE is a small IDE (based on Qt4 and
Source-highlight-qt) I wrote for developing
and debugging new language definitions for source-highlight:

@uref{http://srchighliteide.sourceforge.net}.

@item 
@cindex Ksrc2highlight
@cindex KDE
Martin Gebert implemented a KDE interface to source-highlight programs
(and he did a wonderful job!), and it is called @emph{Ksrc2highlight};
if you want to test it: 

@uref{http://www.mgebert.de/Ksrc2highlight}.

@item
@cindex java2html
There's also a Java version of java2html, you can find it at

@uref{http://www.generationjava.com/projects/Java2Html.shtml}.

@item
This web site provides a web interface to source-highlight
so that you can highlight your code on-line:

@uref{http://www.alaide.com/outils_colorsyntaxe.php}

@item
@cindex SHJS
SHJS is a JavaScript program that highlights source code passages in
HTML documents. Documents using SHJS are highlighted on the client side
by the web browser.  SHJS uses language definitions from
Source-highlight.

@uref{http://shjs.sourceforge.net}

@item
@cindex code2blog
Code2blog is a pyGTK front-end to source-highlight for easy conversion
from source code to HTML.

@uref{http://code.google.com/p/code2blog}

@item
@cindex Apache
Andy Buckley wrote a wrapper around source-highlight, which can be used
as an Apache filter to highlight source code in Web pages on the fly.

@uref{http://www.insectnation.org/projects/filter-src-highlight}

@item
@cindex RapidWeaver
Roger Nilsson wrote a frontend for source-highlight that is used in a
popular webdesign app for OSX called RapidWeaver. The frontend is called
High-Light and allows users to easily add syntax-colored code inside
RapidWeaver.

@uref{http://nilrogsplace.se/webdesign/rapidweaver/plugins/high-light/index_en.html}

@item
@cindex Firefox
Mauricio Zepeda published in his blog an article with a script
to automatically highlight a file and show it in Firefox:

@uref{http://chillorb.com/?p=122}

@item
@cindex Wiki
@cindex Ikiwiki
Jason Blevins made a plugin for Ikiwiki that enables syntax highlighting
of source code fragments and whole files via source-highlight.

@uref{http://jblevins.org/projects/ikiwiki/code}

@item
@cindex Wiki
@cindex Php
Pascal Bleser created a PHP extension that uses the GNU source-highlight
library directly from PHP, instead of relying on spawning a process or
using the source-highlight CGI.

@uref{http://code.google.com/p/php-source-highlight/}

@item
@cindex SIP
@cindex Python
@cindex PyQt
Roberto Alsina made a partial python binding using SIP so
that you can use Source-Highlight-Qt in PyQt programs.

@uref{http://marave.googlecode.com/svn/trunk/marave/highlight/}

@item
@cindex Perl
A perl binding for source-highlight is available at CPAN:

@uref{http://search.cpan.org/perldoc?Syntax::SourceHighlight}

@item
@cindex Pastebin
Danijel Tasov wrote a pastebin service based on
perl source-highlight binding:

@uref{http://pb.rbfh.de}

@end itemize


@node Installation, Copying, Introduction, Top
@chapter Installation
@cindex installation

@cindex compilation
See the file @file{INSTALL} for detailed building and  installation
instructions; anyway if you're used to compiling Linux software that
comes with sources you may simply follow the usual procedure, i.e., untar
the file you downloaded in a directory and then:

@example
cd <source code main directory>
./configure
make
make install
@end example

@cindex shadow build
We strongly suggest to use shadow builds, thus, create a build
directory, say @file{build} and run configuration and make in that
directory:

@example
cd <source code main directory>
mkdir build
cd build
../configure
make
make install
@end example

However, before you do this, please check that you have everything that
is needed to build source-highlight, @ref{What you need to build
source-highlight}.

Note: unless you specify a different install directory by 
@code{--prefix} option of
configure (e.g. @code{./configure --prefix=<your home>}), 
you must be root to run @code{make install}.

You may want to run @code{./configure --help} to see all the possible
options that can be passed to the configuration script.

@cindex directories
Files will be installed in the following directories:

@table @code
@item Executables
        @code{prefix/bin}
@item docs and output examples
        @code{prefix/share/doc/source-highlight}
@item library examples
        @code{prefix/share/doc/source-highlight/examples}
@item library API documentation
        @code{prefix/share/doc/source-highlight/api}
@item conf files           
        @code{prefix/share/source-highlight}
@end table

Default value for prefix is @code{/usr/local} 
but you may change it with @code{--prefix}
option to configure.  For further @code{configure} options, you
can run @code{configure --help}.

@cindex bash completion
Tiziano Muller wrote a bash completion configuration file for
source-highlight; this will be installed by default in the directory
@code{sysconfdir/bash_completion.d}, where @code{sysconfdir} defaults to
@code{prefix/etc}; however, typically, the directory where the bash
completion script searches for configuration file is
@code{/etc/bash_completion.d}.  Thus, we suggest you explicitly specify
this directory with the configuration script command line option
@code{--with-bash-completion}.

@cindex library
@cindex @code{--with-doxygen}
@cindex doxygen
If you want to build and install the API documentation of
Source-highlight library, you need to run @code{configure} with the
option @code{--with-doxygen}, but you need the program @emph{Doxygen},
@url{http://www.doxygen.org}, to build the documentation.
The documentation will be installed in the following directory:

@table @code
@item Library API documentation
        @code{prefix/share/doc/source-highlight/api}
@end table


@cindex java2html
@cindex cpp2html
NOTE: Originally, instead of Source-highlight, there were  two
separate programs, namely @emph{GNU java2html} and @emph{GNU cpp2html}.
There are two shell scripts with the same name that will be installed
together with Source-highlight in order to facilitate the migration
(however their use is not advised and it is deprecated).

@menu
* Building with qmake::         
* Download::                    
* Anonymous Git Checkout::      
* What you need to build source-highlight::  
* Tips on installing Boost Regex library::  
* Patching from a previous version::  
* Using source-highlight with less::  
* Using source-highlight as a CGI::  
* Building .rpm::               
@end menu

@node Building with qmake, Download, Installation, Installation
@section Building with qmake
@cindex qmake

Since version 3.1.2, @value{srchilite} can be built also using
@code{qmake}, the build tool from Qt libraries (@uref{http://qt.nokia.com}).
This was made available to build @value{srchilite} on Windows based systems
without using a Unix shell, and in particular to build @value{srchilite}
@cindex MSVC
with Microsoft MSVC compiler.  You should use this method only if you
don't have a Unix shell or if you really need to use the MSVC compiler
(e.g., if you want to build @value{srchilitelib} to be used in MSVC
based programs).
@cindex boost
You still need the boost regex library, and if you
use MSVC, you can find installation packages for this library at
@url{http://www.boostpro.com}.

This build mechanism is still experimental, and, when using MSVC, only
a static version of @value{srchilitelib} can be built (not a .dll).
@cindex MinGW
You can also use this method if you have the MinGW compiler,
@uref{http://www.mingw.org}, (e.g.,
the one that comes with Qt Windows distribution) and you don't have
@cindex msys
Msys (@uref{http://www.mingw.org/wiki/MSYS}).  Otherwise, you should
still use the @code{configure} based mechanims.

Using @code{qmake}, only a few options can be specified during the building
(besides the ones you usually use with qmake), and these options can
be specified only using environment variables:

@table @code
@item BOOST_REGEX
By default, @code{boost_regex} will be used to link the boost library
(i.e., @code{-lboost_regex}); if your boost regex library has a different
name you must specify this name using this environment variable; e.g.,
if the library file is called @code{libboost_regex-mt.lib} or
@code{boost_regex-mt.dll} you must set this variable to
@code{boost_regex-mt}.
@item INCPATH
Specify the path of the boost header files.
@item LIBS
Specify the path of the boost lib files.
@end table

Please, take into consideration that specifying the boost library
include and library paths is completely up to you, using
@code{INCPATH} and @code{LIBS}, if they're not in the system
path directories.

Also remember to always use the option @code{-recursive} when running
qmake.

If you then want to run @code{make install}, you can use the
variable @code{INSTALL_ROOT} to prefix the installation path, which,
otherwise, is the root directory.

@node Download, Anonymous Git Checkout, Building with qmake, Installation
@section Download
@cindex download

You can download it from GNU's ftp site:
@uref{ftp://ftp.gnu.org/gnu/src-highlite} or from one of its mirrors (see
@uref{http://www.gnu.org/prep/ftp.html}).

I do not distribute Windows binaries anymore; since, they can be built
by using Cygnus C/C++ compiler, available at
@uref{http://www.cygwin.com}.  However, if you don't feel like
downloading such compiler or you experience problems with the Boost
Regex library (see also @ref{Tips on installing Boost Regex library};
please also keep in mind that if you don't have these libraries
installed, and your C/C++ compiler distribution does not provide a
prebuilt package, it might take some time, even hours, to build the
Boost libraries from sources), you can request such binaries directly to
me, by e-mail (find my e-mail at my home page) and I'll be happy to send
them to you.  An MS-Windows port of Source-highlight is available from
@uref{http://gnuwin32.sourceforge.net}; however, I don't maintain those
binaries personally, and they might be out of date.

Archives are digitally signed by me (Lorenzo Bettini) with GNU gpg
(@uref{http://www.gnupg.org}). My GPG public key can be found at my home
page (@value{myhomepage}).

You can also get the patches, if they are available for a particular
release (see below for patching from a previous version).

@node Anonymous Git Checkout, What you need to build source-highlight, Download, Installation
@section Anonymous Git Checkout

@cindex Git
This project's git repository can be checked out through the following
clone instruction@footnote{Since version 3.1.2 of Source-highlight the
CVS repository was dismissed in favor of Git
(@uref{http://git-scm.com/}).}:

@example
git clone git://git.savannah.gnu.org/src-highlite.git
@end example

Further instructions can be found at the address:

@uref{http://savannah.gnu.org/projects/src-highlite}.

And the git repository can also browsed on-line at

@uref{http://git.savannah.gnu.org/cgit/src-highlite.git}.

Please note that this way you will get the latest development sources
of Source-highlight, which may also be unstable.  This solution is the
best if you intend to correct/extend this program: you should send me
patches against the latest git repository sources.

If, on the contrary, you want to get the sources of a given release,
through git, say, e.g., version X.Y.Z, you must specify the tag
@code{rel_X_Y_Z}.

When you compile the sources that you get from the git repository,
before running the @code{configure} and @code{make} commands, for the
first time, you must run the command:

@example
autoreconf -i
@end example

@noindent
This will run the autotools commands in the correct order, and also copy
possibly missing files.  You should have installed recent versions of
@cindex automake
@cindex autoconf
@cindex libtool
@code{automake}, @code{autoconf} and @code{libtool} in order for this to
succeed.

@cindex shadow build
We strongly suggest to use shadow builds, thus, create a build
directory, say @file{build} and run configuration and make in that
directory:

@example
cd <source code main directory>
mkdir build
cd build
../configure
make
make install
@end example

To summarize, the steps to get the sources from git and make the first build
are:

@example
git clone git://git.savannah.gnu.org/src-highlite.git
cd src-highlite
autoreconf -i
mkdir build
cd build
../configure
make
@end example

@node What you need to build source-highlight, Tips on installing Boost Regex library, Anonymous Git Checkout, Installation
@section What you need to build source-highlight

@cindex compilation requirements
@cindex building requirements

@cindex boost
Since version 2.0 Source-highlight relies on regular expressions as
provided by boost (@uref{http://www.boost.org}), so you need to install at
least the regex library from boost.  

Most GNU/Linux distributions provide this library already in a compiled
form.  If you use your distribution packages, please be sure to install
also the development package of the boost libraries.

If you experience problems in installing Boost Regex library, or in
compiling source-highlight because of this library, please take a look
at @ref{Tips on installing Boost Regex library}.

If you want to use a specific version of the Boost regex library
(because you have many versions of it), you can use the configure option
@code{--with-boost-regex} to specify a particular suffix.  For instance,

@example
./configure --with-boost-regex=boost_regex-gcc-1_31
@end example

Source-highlight has been developed under GNU/Linux, using gcc (C++),
and bison (yacc) and flex (lex), and ported under Win32 with Cygwin
C/C++compiler, available at @uref{http://www.cygwin.com}.  

I use the excellent 
@cindex automake
@cindex autoconf
@cindex libtool
GNU Autoconf@footnote{@uref{http://www.gnu.org/software/autoconf}},
GNU Automake@footnote{@uref{http://www.gnu.org/software/automake}} and
GNU Libtool@footnote{@uref{http://www.gnu.org/software/libtool}}.
@cindex gnulib
Since version 2.6 I also started to use Gnulib - The GNU Portability
Library@footnote{@uref{http://www.gnu.org/software/gnulib}}, ``a central
location for common GNU code, intended to be shared among GNU packages''
(for instance, I rely on Gnulib for checking for the presence and
correctness of @code{getopt_long} function).

Finally I used @emph{GNU gengetopt}
(@uref{http://www.gnu.org/software/gengetopt}), for command line parsing.

I started to use also @emph{doublecpp}
(@uref{http://doublecpp.sourceforge.net}) that permits achieving dynamic
overloading.

Actually, apart from the boost regex library, you don't need the other
tools above to build source-highlight (indeed I provide the output
sources generated by the above mentioned tools), unless you want to
develop source-highlight.

However, if you obtained sources through Git, you need some other tools,
see @ref{Anonymous Git Checkout}.

@node Tips on installing Boost Regex library, Patching from a previous version, What you need to build source-highlight, Installation
@section Tips on installing Boost Regex library

@cindex Boost regex

If you experience no problem in compiling source-highlight, you can
happily skip this section@footnote{Since version 2.11, the @code{configure} script should be able
to correctly find the boost regex library if it is in the compiler
default path.} :-)

I created this section because many users reported some problems after
installing Boost Regex library from sources; other users had problems in
compiling source-highlight even if this library was already correctly
installed (especially windows users, using cygwin).  I hope this section
sheds some light in installing/using the Boost Regex library.  Please,
note that this section does not explain how to compile the Boost
libraries (the documentation you'll find on @uref{http://www.boost.org}
is well done); it explains how to tweak things if you have problems in
compiling source-highlight even after a successful installation of Boost
libraries.

First of all, if your distribution provides packages for the Boost regex
library, please be sure to install also the development package of the
boost libraries, i.e., those providing also the header files needed to
compile a program using these libraries.  For instance, on my Debian
system I had to install the package @code{libboost-regex-dev}, besides
the package @code{libboost-regex}.

If your distribution does not provide these packages then you have to
download the sources of Boost libraries from @uref{http://www.boost.org}
and follow the instructions for compilation and installation.  However,
I suggest you specify @file{/usr} as prefix for installation, instead of
relying on the default prefix @file{/usr/local} (unless
@file{/usr/local/include} is already in the inclusion path of your C++
compiler), since this will make things easier when compiling
source-highlight.  I suggest this, since @file{/usr/include} is usually
the place where C++ searches for header files during compilation.

If you successfully compiled and installed the Boost Regex library, or
you installed the package from your distribution, but you STILL
experience problems in compiling source-highlight, then you simply have
to adjust some things as described in the following.

If the @code{./configure} command of source-highlight reports this
error:

@example
ERROR! Boost::regex library not installed.
@end example

@noindent
then, the compiler cannot find the header files for this library.  In
this case, check that the directory @file{/usr/include/boost} actually
exists; if it does not, then probably you'll find a similar directory,
e.g., @file{/usr/include/boost-1_33/boost}, depending on the version of
the library you have installed.  Then, all you have to do is to create a
symbolic link as follows:

@example
ln -s /usr/include/boost-1_33/boost /usr/include/boost
@end example

@noindent
@cindex @code{CXXFLAGS}
Alternatively, you might run source-highlight's configure as follows:

@example
./configure CXXFLAGS=-I/usr/include/boost-1_33/
@end example

If you install (or build) the Boost Regex library in a non standard
path, e.g., somewhere in your home directory, say
@file{/home/myhome/boost-1_33}, you'll have to update the
@code{CXXFLAGS} variable accordingly on the @code{configure} command
line; in this particular case, you might also have to specify the path
of actual library files (@code{CXXFLAGS} will only specify the path of
header files).  In particular, you'll have to know where the lib files
are within the boost installation (or build directory); for instance,
if they are in @file{/home/myhome/boost-1_33/stage/lib}, while the
header files (i.e., the @file{boost} header files directory) are in
@file{/home/myhome/boost-1_33}, the complete @code{configure} command
should be

@cindex @code{LDFLAGS}
@example
./configure CXXFLAGS=-I/home/myhome/boost-1_33 \
            LDFLAGS=-L/home/myhome/boost-1_33/stage/lib
@end example

If then @code{./configure} command of source-highlight reports this
other error:

@example
ERROR! Boost::regex library is installed, but you
must specify the suffix with --with-boost-regex at configure
for instance, --with-boost-regex=boost_regex-gcc-1_31
@end example

@noindent
then, there's still another thing to fix: you must find out the exact
names of the files of your installed Boost Regex libraries; you can do
this by using the command:

@example
$ ls -l /usr/lib/libboost_regex*
@end example

@noindent
that, for instance, on one of my cygwin installation reports:

@example
-rwxr-x---+ Nov  9 23:29 /usr/lib/libboost_regex-gcc-mt-s-1_33.a
-rwxr-x---+ Nov 22 09:22 /usr/lib/libboost_regex-gcc-mt-s.a 
-rwxr-x---+ Nov  9 23:29 /usr/lib/libboost_regex-gcc-mt-s-1_33.so
-rwxr-x---+ Nov 22 09:22 /usr/lib/libboost_regex-gcc-mt-s.so
@end example

@noindent
Now, you have all the information to correctly run the
source-highlight's configure command:

@example
./configure --with-boost-regex=boost_regex-gcc-mt-s-1_33
@end example

@noindent
or, if you solved the first problem in the second way@footnote{Command
lines that are too long are split into multiple indented lines separated
by a @code{\}.  Of course these commands are to be given in one line
only, anyway.},

@example
./configure CXXFLAGS=-I/usr/include/boost-1_33/ \
            --with-boost-regex=boost_regex-gcc-mt-s-1_33
@end example

Of course, you have to modify this command according to the names of
your Boost Regex library installed files.

These instructions managed to let many users, who were experiencing
problems, to compile source-highlight If you still have problems, please
send me an e-mail.

@node Patching from a previous version, Using source-highlight with less, Tips on installing Boost Regex library, Installation
@section Patching from a previous version
@cindex patching

If you downloaded a patch, say
@file{source-highlight-1.3-1.3.1-patch.gz} (i.e., the patch to go from version
1.3 to version 1.3.1), cd to the directory with sources from the
previous version (source-highlight-1.3) and type:  

@example
gunzip -cd ../source-highlight-1.3-1.3.1.patch.gz | patch -p1 
@end example

and restart the compilation process (if you had already run configure a
simple make should do).

@node Using source-highlight with less, Using source-highlight as a CGI, Patching from a previous version, Installation
@section Using source-highlight with less

@cindex src-hilite-lesspipe.sh
This was suggested by Konstantine Serebriany.   The script
@file{src-hilite-lesspipe.sh} will be installed together with
source-highlight.  You can use the following environment variables:

@example 
export LESSOPEN="| /path/to/src-hilite-lesspipe.sh %s"
export LESS=' -R '
@end example

This way, when you use less to browse a file, if it is a source file
handled by source-highlight, it will be automatically highlighted.

@cindex source-highlight-esc.sh
Xavier-Emmanuel Vincent recently provided an alternative version of
ANSI color scheme, @file{esc256.style}: some terminals can handle 256
colors.  Xavier also provided a script which checks how many colors
your terminal can handle, and in case, uses the 256 variant.  The
script is called @file{source-highlight-esc.sh} and it will be
installed together with the other binaries.

@node Using source-highlight as a CGI, Building .rpm, Using source-highlight with less, Installation
@section Using source-highlight as a CGI

@cindex CGI
CGI support was enabled thanks to Robert Wetzel; I haven't tested it
personally.  If you want to use source-highlight as a CGI program, you
have to use the executable source-highlight-cgi.  You can build such
executable by issuing

@example
make source-highlight-cgi
@end example

@noindent
in the @file{src} directory.



@node Building .rpm,  , Using source-highlight as a CGI, Installation
@section Building .rpm
@cindex rpm

Christian W. Zuckschwerdt added support for building an .rpm and an
.rpm.src.  You can issue the following command

@example 
rpmbuild -tb source-highlight-@value{VERSION}.tar.gz
@end example

for building an .rpm with binaries and 

@example
rpmbuild -ts source-highlight-@value{VERSION}.tar.gz
@end example

for building an .rpm.src with sources. 


@node Copying, Simple Usage, Installation, Top
@chapter Copying Conditions
@cindex Copying conditions

GNU Source-highlight is free software; you are free to use, share and modify it
under the terms of the GNU General Public License that accompanies this software
(see @file{COPYING}).

GNU @command{source-highlight} was written and
maintained by Lorenzo Bettini @value{myhomepage}.

@node Simple Usage, Configuration files, Copying, Top
@chapter Simple Usage
@cindex sample

Here are some realistic examples of running
@command{source-highlight}@footnote{Command lines that are too long are
split into multiple indented lines separated by a @code{\}.  Of course
these commands are to be given in one line only, anyway.}.

Source-highlight only does a lexical analysis of the source code, so the
program source is assumed to be correct!

Here's how to run source-highlight (for this example we will use
C/C++ input files, but this is valid also for other source-highlight
input languages):

@example
source-highlight --src-lang cpp --out-format html \
    --input @var{<C++ file>} \
    --output @var{<html file>} \
    --style-file @var{<style file>} \
    @var{options}
@end example

For input files, apart from the @code{-i (--input)} option and the
standard input redirection, you can simply specify some files at the
command line and also use regular expressions (for instance
@file{*.java}).  In this case the name for the output files will be
formed using the name of the source file with a .<ext> appended, where
<ext> is the extension chosen according to the output format specified
(in this example it would be .html).  The style file 
(@ref{Output format style}) 
contains information on how to format specific language parts
(e.g., keywords in blue and boldface, etc.).

IMPORTANT: you must choose one of the above two invocation modes: either
you use @code{-i (--input)}, @code{-o (--output)} (possibly replacing
them with standard input/output redirection), or you specify one or many
files without @code{-i (--input)}; if you try to mix them you'll get an
error:

@example
source-highlight -o main.html main.cpp
Please, use one of the two syntaxes for invocation:
source-highlight [OPTIONS]... -i input_file -o output_file
source-highlight [OPTIONS]... [FILES]...
@end example

If @code{STDOUT} string is passed as @code{-o (--output)} option, then
the output is forced to the standard output anyway.

If @code{-s (--src-lang)} is not specified, the source language is
inferred by the extension of the input file or from the file name itself
(possibly using also lower case versions); this, of course, does not
work with standard input redirection.  For further details, see @ref{How
the input language is discovered}.

If @code{-f (--out-format)} is not specified, the output will be
produced in HTML.

If @code{--style-file} is not specified, the @file{default.style}, which
is included in the distribution, will be used (see @ref{Output format style} 
for further information).

@menu
* HTML and XHTML output::       
* LaTeX output::                
* Texinfo output::              
* DocBook output::              
* ANSI color escape sequences::  
* Odf output::                  
@end menu

@node HTML and XHTML output, LaTeX output, Simple Usage, Simple Usage
@section HTML and XHTML output
@cindex HTML
@cindex XHTML

The default output format for HTML and XHTML uses fixed width fonts by
inserting all the formatted output between @code{<tt>} and @code{</tt>}.
Thus, for instance, specification for fixed width and not fixed width
(see @ref{Output format style}) will have no effect: every character
will have fixed width.  If you don't like this default behavior and
would like to have not fixed fonts by default (as it happens, e.g., with
@LaTeX{} output) you can use the file @file{html_notfixed.outlang} with
the command line argument @code{--outlang-def}.  For XHTML output, the
corresponding file is @file{xhtml_notfixed.outlang}

Furthermore, the file @file{htmltable.outlang} can be used to generate
HTML output enclosed in an HTML table (which will use also a background
color if specified in the style file).   The file
@file{xhtmltable.outlang} does the same but for XHTML output.

@node LaTeX output, Texinfo output, HTML and XHTML output, Simple Usage
@section @LaTeX{} output
@cindex @LaTeX{}

When using @LaTeX{} output format you can choose between monochromatic
output (by using @code{-f latex}) or colored output (by using @code{-f
latexcolor}).  When using colored output, you need the
@code{color} package (again this should be present in your system).
Of course, you are free to define your own @LaTeX{} output format,
see @ref{Output Language Definitions}.

@node Texinfo output, DocBook output, LaTeX output, Simple Usage
@section Texinfo output
@cindex Texinfo

When using the Texinfo output format, you may want to use a dedicated
style file, @file{texinfo.style}, which comes with the source-highlight
distribution, with the option @code{--style-file}.  For instance, the
example in @ref{Examples} is formatted with this style file.

@node DocBook output, ANSI color escape sequences, Texinfo output, Simple Usage
@section DocBook output
@cindex DocBook

DocBook output is generated using the @code{<programlisting>} tag.  If
the @code{--doc} command line option is given, an @code{<article>}
document is generated.

@node ANSI color escape sequences, Odf output, DocBook output, Simple Usage
@section ANSI color escape sequences
@cindex ANSI color

If you're using this output format, for instance together with
@code{less} (see @ref{Using source-highlight with less}), you may want
to use the @file{esc.style} (or @file{esc256.style} if your terminal
can handle 256 colors), which comes with the source-highlight
distribution, with the option @code{--style-file}.  This should result
in a more pleasant coloring output.

@node Odf output,  , ANSI color escape sequences, Simple Usage
@section Odf output

The ODF language output for GNU source-highlight enables the user to
generate color-highlighted ODF output of source code files. Or to
generate ODF color-highlighted snippets to be used by ODF back-ends
(like asciidoc-odf).  We create an @code{.fodt} file, which is a Text
document that newer versions of LibreOffice can open.

@node Configuration files, Invoking source-highlight, Simple Usage, Top
@chapter Configuration files
@cindex configuration files

During execution, source-highlight needs some files where it finds
directives on how to recognize the source language (if not  specified
explicitly with @code{--src-lang} or @code{--lang-def}), on which output
format to use (if not specified explicitly with @code{--out-format} or
@code{--outlang-def}), on how to format specific source elements (e.g.,
keywords, comments, etc.), and source and output language definitions.
These files will be explained in the next sections.

@cindex @code{--data-dir}
If the directory for such files is not explicitly specified with the
command line option @code{--data-dir}, these files are searched for in
the following order:

@itemize
@item 
the current directory;
@item
the installation directory for conf files, see @ref{Installation}
(please keep in mind that this directory is hard-coded into
source-highlight during compilation).
@item
if the source-highlight command is specified with an
explicit path name, the installation directory name is
still used, but relative to the explicit path name.
@end itemize

In particular, the user can set the value also with the environment
variable
@cindex @code{SOURCE_HIGHLIGHT_DATADIR}
@code{SOURCE_HIGHLIGHT_DATADIR} (see also @ref{The program
source-highlight-settings}).

If you want to be sure about which file is used during the
execution, you can use the command line option @code{--verbose}.

@menu
* Output format style::         
* Output format style using CSS::  
* Default Styles::              
* Language map::                
* Language definition files::   
* Output Language map::         
* Output Language definition files::  
* Developing your own definition files::  
@end menu

@node Output format style, Output format style using CSS, Configuration files, Configuration files
@section Output format style
@cindex output style
@cindex default.style

You must specify your options for syntax highlighting in the file
@file{default.style}@footnote{Before version 2.1, this file was called
@file{tags.j2h} which used to be a very obscure name.  I hope this name
convention is a better one :-).}.  
You can specify formatting options for each element defined
by a language definition file (you can get the list of such elements,
@cindex @code{--show-lang-elements}
by using @code{--show-lang-elements}, see @ref{Listing Language
Elements}).

@cindex @code{bgcolor}
@cindex background color
Since version 2.6, you can also specify the background color for the
output document, using the keyword @code{bgcolor} (this might be visible
only when the @code{--doc} command line option is used).

If many elements share the same formatting options, you can specify
these elements in the same line, separated by a comma@footnote{Since
version 2.6.}.

Here's the @file{default.style} that comes with this distribution (this
is formatted by using the @file{style.lang} that is shown in
@ref{Tutorials on Language Definitions}):

@include default.style.texinfo

This file tries to define a style for most elements defined in the
language definition files that comes with Source-highlight
distribution.

@cindex @code{--style-file}
You can specify your own file (it doesn't have to be named
@file{default.style}) with the command line option
@code{--style-file}@footnote{Before version 2.1, this command line
option was called @code{--tags-file} which used to be a very obscure
name.  I hope this name convention is a better one :-).}, see
@ref{Invoking source-highlight}.

You can also specify the color of normal text by adding this line 

@example
normal darkblue ;
@end example

@cindex color
@cindex background color
As you might see the syntax of this file is quite straightforward: after
the element (or elements, separated by commas) you can specify the
color, and the background color@footnote{Since version 2.6.} by using
the prefix @code{bg:} (for instance, in the @file{default.style} above
the background color is specified for the @code{todo} element).

Note that the background color might not be available for all output
formats: it is available for XHTML and @LaTeX{} but not for
HTML@footnote{Of course, if you use HTML and an external CSS file you
will achieve the same result.}.

@cindex bold
@cindex italics
@cindex underline
@cindex fixed
@cindex notfixed
Then, you can specify further formatting options such as bold, italics,
etc.; these are the keywords that can be used:

@example
b = bold
i = italics
u = underline
f = fixed
nf = not fixed
noref = no reference information is generated for these elements
@end example

@cindex color
Since version 2.2, the color specification is not required.  For
instance, the @file{texinfo.style} is as follows (we avoid colors for
Texinfo outputs):

@include texinfo.style.texinfo

You may also specify more than on of these options separated by commas,
e.g.

@example
keyword blue u, b ;
@end example

@noindent
Please keep in mind that in this case the order of these specified
options is kept during the generation of the output; for instance,
depending on the specific output format, the sequences @code{u, b} and
@code{b, u} may lead to different results.  In particular, the style
that comes first is used after the ones that follow.  For instance, in
the case of HTML, the sequence @code{u, b} will lead to the following
formatting: @code{<u><b>...</b></u>}.

@cindex noref
The @code{noref} option specifies that for this element reference
information are not generated (see @ref{Generating References}).  For
instance, this is used for the @code{comment} element, since we do not
want that elements in a comment are searched for cross-references.

These are all possible color logical names handled by
source-highlight@footnote{You can see these colors in HTML in the file
@file{colors.html}.}:

@example
black 
red 
darkred 
brown 
yellow 
cyan 
blue 
pink 
purple 
orange 
brightorange 
green 
brightgreen 
darkgreen 
teal 
gray 
darkblue
white
@end example

@cindex direct color scheme
You can also use the direct color scheme for the specific output format,
by using double quotes, such as, e.g., @code{"#00FF00"} in
HTML@footnote{Note that, since version 2.2, you must use double
quotes.} or even string colors in double quotes@footnote{Since version
2.6.}, such as @code{"lightblue"}.  Of course, the double quotes will be
discarded during the generation.

For instance, this is the @file{syslog.style} used in the @file{tests}
directory.  This uses direct color schemes.

@include syslog.style.texinfo

Note that, if you use direct color schemes, source-highlight will
perform no transformation, and will output exactly the color scheme you
specified.  For instance, the specification @code{"brown"} is different
from @code{brown}: the former will be output as it is, while the latter
will be translated in the corresponding color of the output format (for
HTML the visible result is likely to be the same).

It is up to you to specify a color scheme string that is handled by the
specific output format.  Thus, direct color schemes might not be
portable in different output formats; for instance, @code{"#00FF00"} is
valid in HTML but not in @LaTeX{}.

@node Output format style using CSS, Default Styles, Output format style, Configuration files
@section Output format style using CSS

@cindex CSS
Since version 2.6 you can specify the output format style also using
a limited CSS syntax.  Please, note that this has nothing to do
with output produced by source-highlight using the @code{--css} option.

@cindex @code{--style-css-file}
By using a CSS file as the style file (i.e., passing it to the
@code{--style-css-file} command line option) you will only specify the
output format style using the same syntax of CSS.  This means that you
can use a css syntax for specifying the output format style
independently from the actual output (this is what the output format
style is for).  Thus, you can use a css file as the output format style
also for @LaTeX{} output (just like you would do with a source-highlight
output format style, @ref{Output format style}).

This feature is provided basically for code re-use: you can specify the
output format style using a css file, and then re-use the same css file
as the actual style sheet of other HTML pages (or even output files
produced by source-highlight using the @code{--css} option).

Note that this feature is quite primordial, so only a limited subset
of CSS syntax is recognized.  In particular, selectors are always
intended as CSS class selectors, so they must start with a dot.
@code{/* */} comments are handled.  Properties (and their values) not
handled by source-highlight are simply (and silently) discarded).

This is an example of CSS specification handled correctly by
source-highlight as a style format specification:

@include css_example.css.texinfo

Finally, this is the @file{default.css} that corresponds to
@file{default.style} presented in @ref{Output format style}:

@include default.css.texinfo

If you pass this file to the @code{--style-css-file} command line option
and you produce an output file, you will get the same result of using
@file{default.style}.

Source-highlight comes with a lot of CSS files that can be used either
as standard CSS files for HTML documents, or as style files to pass to
@code{--style-css-file}.  In the documentation installation directory
(see @ref{Installation}) you will find the file
@file{style_examples.html} which shows many output examples, each one
with a different CSS style.

@node Default Styles, Language map, Output format style using CSS, Configuration files
@section Default Styles

This file@footnote{Since version 2.9.} (the default file is
@file{style.defaults}) lists the default style for a language element
whose output style is not specified in the style file; in particular the
following line (comment lines start with @code{#}):

@example
elem1 = elem2
@end example

@noindent
tells that, if the style for an element, say elem1, is not specified in
the style file, then elem1 will have the same style of elem2.

@cindex style.defaults
For instance, this is the @file{style.defaults} that comes with
Source-highlight:

@include style.defaults.texinfo

@noindent
In this case the style for the element @code{preproc} will default to
the style of the element @code{keyword}.

This file is useful when you want to create your own style file and you
don't want to specify styles for all the elements that will have the
same output style in your style (e.g., the default style formats
@code{preproc} elements differently from keywords, but if in your style
you don't specify a style for it, a @code{preproc} element will still be
formatted as a @code{keyword}).

@node Language map, Language definition files, Default Styles, Configuration files
@section Language map
@cindex language map

This configuration file associates a file extension to a specific
language definition file.  You can also use such file extension to
specify the @code{--src-lang} option (see @ref{Simple Usage}).
Source-highlight comes with such a file, called @file{lang.map}.

Of course, you can override the settings of this file by writing your
own language map file and specify such file with the command line option
@code{--lang-map}).  Moreover, as explained above, if a file
@file{lang.map} is present in the current directory, such version will
be used.  The format of such file is quite simple (comment lines start
with @code{#}):

@example
extension = language definition file
@end example

The default language definition file is shown in @ref{Introduction}.

@node Language definition files, Output Language map, Language map, Configuration files
@section Language definition files

These files are crucial for source-highlight since they specify the
source elements that have to be highlighted.   These files also allow to
specify your own language definitions in order to deal with a language
that is not handled by source-highlight@footnote{This is the main
difference introduced in version 2.0 with respect the previous
version.}.  The syntax for these files is explained in @ref{Language
Definitions}.

@node  Output Language map, Output Language definition files, Language definition files, Configuration files
@section Output Language map
@cindex output language map

This configuration file associates an output format to a specific output
language definition file.  You can use the name of that output format to
specify the @code{--out-format} option (see @ref{Simple Usage}).
Source-highlight comes with such a file, called @file{outlang.map}.

Of course, you can override the settings of this file by
writing your own output language map file and specify such file
with the command line option @code{--outlang-map}).
Moreover, as explained above, if a file @file{outlang.map}
is present in the current directory, such version will be used.
The format of such file is quite simple:

@example
output format name = language definition file
@end example

The default language definition file is shown in @ref{Introduction}.

In particular, there is a convention for the output format name in the
output language map: the one with @code{-css} suffix is the one used
when @code{--css} command line option is given


@node Output Language definition files, Developing your own definition files, Output Language map, Configuration files
@section Output Language definition files

These files are crucial for source-highlight since they specify how the
source elements are highlighted.   These files also allow to specify
your own output format definitions in order to deal with an output
format that is not handled by source-highlight@footnote{This is the main
difference introduced in version 2.1 with respect the the previous
version.}.  The syntax for these files is explained in @ref{Output
Language Definitions}.

These files are part of source-highlight distribution, but they can
also be downloaded, independently, from here:

@value{outlangfilesurl}

@node  Developing your own definition files,  , Output Language definition files, Configuration files
@section Developing your own definition files

I encourage those who write new language definitions or correct/modify
existing language definitions to send them to me so that they can be
added to the source-highlight distribution!

Since these files require more explanations (that, however, are not
necessary to the standard usage of source-highlight), they are carefully
explained in separate parts: @ref{Language Definitions} and @ref{Output
Language Definitions}.

These files are part of source-highlight distribution, but they can
also be downloaded, independently, from here:

@value{langfilesurl}


@node Invoking source-highlight, Language Definitions, Configuration files, Top
@chapter Invoking @command{source-highlight}
@cindex invoking
@cindex version
@cindex options
@cindex usage
@cindex help
@cindex getting help

The format for running the @command{source-highlight} program is:

@example
source-highlight @var{option} @dots{}
@end example

@code{source-highlight} supports the following options, shown by
the output of @code{source-highlight --detailed-help}:

@smallexample
@include help-output.texinfo
@end smallexample

Let us explain some options in details (apart from those that should be
clear from the @code{--help} output itself, and those already explained
in @ref{Simple Usage}).

@c Formatting copied from the Texinfo 4.0 manual.
@table @code

@item --data-dir
@cindex @code{--data-dir}
Source-highlight, during the execution, will need some files, such as,
e.g., language definition files, output format definition files, etc.
These files are installed in @code{prefix/share/source-highlight} where
@code{prefix} is chosen at compilation time (see @xref{Installation}).
Thus, source-highlight should be able to find all the files it needs
independently.  However, if you want to override this setting, e.g.,
because you have your own language definition files, or simply because
you installed a possible source-highlight binary in a different
directory from the one used during the compilation, you can use the
command line option @code{--data-dir}.

@item --doc
@itemx -d
If you want a stand alone output document (i.e., an output file that is
not thought to be included in another document), specify this option
(otherwise you just get some text that you can paste into another
document).  If you choose this option and do not provide a
@code{--title}, the your source file name will be used as the title.

@item --no-doc
The @code{--doc} option above is actually implied by other command line
options (e.g., @code{--css}).  If you do not want this (e.g., you want
to include the output in an existing document containing the global
style sheet), you can disable this by using @code{--no-doc}.

@item --css
@itemx -c
Specify the style sheet file (e.g., a @file{.css} for HTML@footnote{As
explained before, originally Source-highlight was thought mainly for
generating HTML output, this is why the term @emph{css} is used for
style sheets.}) for the output document.  Note that source-highlight
will not use this file: it will simply use this file name when
generating the output file, so to specify that the output file uses this
file as the style sheet (e.g., if the generated HTML relies on this file
as the CSS file).

@item --tab
@itemx -t
With this options, tab characters will be converted into specified
number of space characters (tabulation points will be preserved).  This
option is automatically selected when generating line numbers.

@item --style-file
@itemx --style-css-file
Specify the file that source-highlight will use to produce (i.e.,
format) the output (e.g., colors and other styles for each  language
element).  The formats of these files are detailed in @ref{Output format
style} and in @ref{Output format style using CSS}, respectively.

@item --style-defaults
Specify the file that contains the default styles for elements
whose styles are not found in the style file (see @ref{Default Styles} for
further details).

@item --output-dir
You can pass to source-highlight more than one input file (see
@ref{Simple Usage}).  In this case you cannot specify the output file
name.  In such cases the output files will be automatically generated
into the directory where you invoked the command from; if you want the
output files to be generated into a different directory you can use this
option.

@item --infer-lang
@cindex language inference
@cindex @code{--infer-lang}
Force the inference mechanism for detecting the input language.  This is
detailed in @ref{How the input language is discovered}.

@item --line-number
Line numbers will be generated in the output, using the (optional)
specified padding character@footnote{Padding character can be specified
since version 2.8.} (the default padding character is @code{0}).

@item --line-number-ref
As @code{--line-number}, this option numbers all the output lines, and,
additionally, generates an anchor for each line.  The anchor consists of
the specified prefix (default is @code{line}) and the line number (e.g.,
@code{line25}).  For instance, as prefix, if you deal with many files,
you can use the file name.  Note that some output languages might not
support this feature (e.g., @code{esc}, since it makes no sense in such
case).  See @ref{Anchors and References} for defining how to generate an
anchor in a specific output language.

@cindex line ranges
@cindex range context
@cindex range separator
@item --line-range
@itemx --range-context
@itemx --range-separator
Since version 2.11, you can specify multiple line ranges: only the lines
in the source that are in these ranges will be output.  For instance, by
specifying

@example
--line-range="-5","10","20-25","50-"
@end example

Only the following lines will be output: the first 5 lines, line 10,
lines 20 to 25 and from line 50 to the end of input.  (See also the
examples in @ref{Line ranges}).

Together with @code{--line-range}, you can also specify
@code{--range-context}: this is the number of lines that will be printed
before and after the lines of a range (i.e., the surrounding
``context'').  These lines will not be highlighted: they will be printed
according to the style @code{context}.  For instance, extending the
previous example,

@example
--line-range="-5","10","20-25","50-" --range-context=1
@end example

Also the following lines will be output: 6, 9, 11, 19, 26, 49.  (See
also the examples in @ref{Line ranges (with context)}).

Finally, you can specify a range separator line string with
@code{--range-separator} that will be printed between ranges (See also
the examples in @ref{Line ranges (with context)}).  The separator string
is preformatted automatically, so, e.g., you don't have to escape
special output characters, such as the @{ @} in texinfo output.

@cindex regex ranges
@item --regex-range

Ranges can be expressed also using regular expressions, with the command
line option @code{--regex-range}.  In this case the beginning of the
range will be detected by a line containing (in any point) a string
matching the specified regular expression; the end will be detected by a
line containing a string matching the same regular expression that
started the range.  This feature is very useful when we want to document
some code (e.g., in this very manual) by showing only specific parts,
that are delimited in a ad-hoc way in the source code (e.g., with
specific comment patterns).  You can see some usage examples
in @xref{Regex ranges}.

The specified strings (this option accepts multiple occurrences) must be
valid regular expressions (thus you must escape special characters
accordingly), otherwise you will get an error.

Furthermore, @code{--line-range} and @code{--regex-range} cannot coexist
in the same command line.

@item --failsafe
@cindex failsafe
If no language specification is found, an error will be printed and the
program exits.  With this option, instead, in such situations, the input
is simply formatted in the output format.  This is useful when
@command{source-highlight} is used with many input files, and it is also
used in the @file{src-hilite-lesspipe.sh} script.  Actually I failed to
find a good reason why one should not always use this option.  So my
suggestion is to always use it when you run source-highlight (and
indeed, in the future, this option might become the default one).  See
also @ref{Using source-highlight with less}, @ref{Using source-highlight
as a simple formatter}.

@cindex default.lang
When using @code{--failsafe}, if no input language can be established,
source-highlight will use the input language definition file
@file{default.lang}, which is an empty file.  You might want to
customize such file, though.

@item --debug-lang
@item --show-regex
Allows to debug a language definition file, @ref{Debugging}.

@end table

The other command line options dealing with references are explained in
more details in @ref{Generating References}.

@menu
* How the input language is discovered::  
@end menu

@node How the input language is discovered,  , Invoking source-highlight, Invoking source-highlight
@section How the input language is discovered

As already explained, @ref{Simple Usage}, source-highlight uses a
language definition file according the language specified with the
option @code{--src-lang}, or @code{--lang-def}, or by using the input
file extension.

Since version 2.5, source-highlight can use an inference mechanism to
deduce the input language.  For the moment, it can detect script
languages based on the ``sha-bang'' mechanism, i.e., when the first line
of a script contains a line such as, e.g.,

@example
#!/bin/sh
@end example

It detects script languages specified by using the @code{env}
program@footnote{Since version 2.7.}:

@example
#!/usr/bin/env perl
@end example

It recognizes the Emacs convention, of declaring the Emacs
major mode using the format @code{-*- lang -*-}.

For instance, a script starting as the following one:

@example
#!/bin/bash
# -*- Tcl -*-
@end example

@noindent
will be interpreted as a Tcl script, and not as bash script.

Finally, it recognizes @code{<?} specifications (e.g.,
@code{<?php} and @code{<?xml}) and @code{<!doctype} (in that case,
it infers it is an xml file)@footnote{Since version 3.1.2.}.

This inference mechanism is performed, by default, in case the input
language is neither explicitly specified nor found in the language map
file by using the input file extension or the filename itself, possibly
also the lowercase version (the input file may also have no extension at
all, but, for instance, a @file{ChangeLog} input file will be
highlighted using @file{changelog.lang}).

@cindex @code{--infer-lang}
Furthermore, this mechanism can be given priority with the command line
option @code{--infer-lang}.  For instance, this is used in the script
@file{src-hilite-lesspipe.sh} (@ref{Using source-highlight with less})
when running source-highlight, in order to avoid the problem of
formatting a Perl script as a Prolog program (since the extension
@file{.pl} is associated to Prolog programs in the language map file,
@ref{Perl}).

@node Language Definitions, Output Language Definitions, Invoking source-highlight, Top
@chapter Language Definitions
@cindex language definition

Since version 2.0 source-highlight uses a specific syntax
to specify source language elements (e.g., keywords, strings,
comments, etc.).  Before version 2.0, language elements were scanned
through Flex.  This had the drawback of writing a new flex file
to deal with a new language; even worse, a new language could
not be added ``dynamically'': you had to recompile the whole
source-highlight program.

Instead, now, language elements are specified in a file, loaded
dynamically, through a (hopefully) simple syntax.  Then, these
definitions are used internally to create, on-the-fly, regular
expressions that are used to highlight the elements (see also @ref{How
source-highlight works}).  In particular, we use the regular expressions
provided by the Boost library (see @ref{Installation}).   Thus, when
writing a language definition file you will surely have to deal with
regular expressions.  Don't be scared: for most of the languages you may
never have to deal with difficult regular expressions, and you can also
specify language keywords (such as, e.g., ``if'', ``while'', etc., see
@ref{Simple definitions}); moreover, for defining delimited language
elements you will not have to write a regular expression, but just the
delimiters (see @ref{Delimited definitions}).  However, there might be
some language definitions that may require heavy use of more involved
regular expressions (e.g., Perl, just to mention one).

Of course, we use the Boost regex library regular expression syntax.  We
refer to Boost documentation for such syntax,
@uref{http://www.boost.org/libs/regex/doc/syntax.html}, however, in
@ref{Notes on regular expressions}, we provide some notes on regular
expressions that might be helpful for those who never dealt with them.
By default, Boost regex library uses Perl regular expression syntax,
and, at the moment, this is the only syntax supported by
source-highlight.

Here, we see such syntax in details, by relying on many examples.  This
allows a user to easily modify an existing language definition and
create a new one.  These files have, typically, extension @file{.lang}.

Each definition basically associates a regular expression to a language
element and defines a name for the language element.  Such name will be
used to associate a particular style (e.g., bold face, color, etc.) when
highlighting such elements.  You cannot use names that are the same
of keywords used in the language definition syntax (e.g., @code{start},
as shown later, is a reserved word).

Comments can be given by using @code{#}; the rest of the line is
considered as a comment.

Source-highlight will scan each line of the input file separately.  So a
regular expression that tries to match new line characters is destined
to fail.  However, the language definition syntax provides means to deal
with multiple lines (see @ref{Delimited definitions} and
@ref{State/Environment Definitions}).

@menu
* Ways of specifying regular expressions::  
* Simple definitions::          
* Line wide definitions::       
* Order of definitions::        
* Delimited definitions::       
* Variable definitions::        
* Dynamic Backreferences::      
* File inclusion::              Include the contents of another file
* State/Environment Definitions::  
* Explicit subexpressions with names::  
* Redefinitions and Substitutions::  
* How source-highlight works::  
* Notes on regular expressions::  
* The program check-regexp::    
* Listing Language Elements::   
* Concluding Remarks::          
* Debugging::                   Debug a language definition file
* Tutorials on Language Definitions::  
@end menu

@node Ways of specifying regular expressions, Simple definitions, Language Definitions, Language Definitions
@section Ways of specifying regular expressions

Before getting into details of language definition syntax, it is crucial
to describe the 3 possible ways of specifying a regular expression
string.  These 3 different ways, basically differ in the way they handle
regular expression special characters, such, e.g., parenthesis.  For
this reason, one mechanism can be more powerful than another one, but it
could also require more attention; furthermore, there can be situations
where you're forced to use only one mechanism, since the other ones
cannot accomplish the required goal.

@table @code
@cindex @code{"expression"}
@item "expression"

If you use double quotes (note, @code{"} and not @code{``} or
@code{''}) to specify a regular expression, then basically all the
characters, but the alternation symbol, i.e., the pipe symbol @code{|},
are considered literally, and thus will be automatically escaped (e.g.,
a dot @code{.} is interpreted as the character @code{.} not as the
regular expression wild card).  Thus, for instance, if you specify

@example
"my(regular)ex.pre$$ion@{*@}"
@end example

@noindent
source-highlight will automatically transform it into

@example
my\(regular\)ex\.pre\$\$ion\@{\*\@}
@end example

The special character @code{|}, unless it is meant to separate two
alternatives (@ref{Simple definitions}), must be escaped with the
character @code{\}, e.g., @code{\|}.  Also the character @code{\},
if it is intended literally, must be escaped, e.g., @code{\\}.

@cindex @code{'expression'}
@item 'expression'

If you want to enjoy (almost) the full power of regular expressions, you
need to use single quoted strings (@code{'}), instead of double quoted
strings.  This way, you can specify special characters with their
intended meaning.

However, marked subexpressions are automatically transformed in non
marked subexpressions, i.e., the parts in the expression of the shape
@code{(...)} will be transformed into @code{(?:...)}  (as explained in
@ref{Notes on regular expressions}, @code{(?:...)}  lexically groups
part of a regular expression, without generating a marked
sub-expression).

Thus, for instance, if you specify

@example
'my(regular)ex.pre$ion*'
@end example

@noindent
source-highlight will automatically transform it into

@example
my(?:regular)ex.pre$ion*
@end example

Since marked subexpressions cannot be specified with this syntax, then
@emph{backreferences} (see @ref{Notes on regular expressions}) are not
allowed.

@cindex @code{`expression`}
@item `expression`

@cindex marked subexpressions
@cindex backreference
@cindex backtick
This syntax@footnote{Since version 2.7.} (note the difference, this
one uses the @emph{backtick} @code{`} while the previous one uses
@code{'}) for specifying a regular expression was introduced to overcome
the limitations of the other two syntaxes.  With this syntax, the marked
subexpressions are not transformed, and so you can use regular
expressions mechanisms that rely on marked subexpressions, such as
@emph{backreferences} and @emph{conditionals} (see @ref{Notes on regular
expressions}).

This syntax is also crucial for highlighting specific program parts of
some programming languages, such as, e.g., Perl regular expressions
(e.g., in substitution expressions) that can be expressed in many forms,
in particular, separators for the part to be replaced and the part to
replace which can be any non alphanumerical characters@footnote{This
issue concerning Perl regular expression syntax was raised by Elias
Pipping, and this also pushed me to deal with this more powerful syntax
that permits using backreferences, for instance.  Although we're still
far from highlighting Perl syntax completely (@ref{Perl}), I definitely
must thank Elias for his precious information about this matter :-)},
for instance,

@example
s/foo/bar/g
s|foo|bar|g
s#foo#bar#g
s@@foo@@bar@@g
@end example

Using this syntax, and backreferences, we can easily define a single
language element to deal with these expressions (without specifying all
the cases for each possible non alphanumerical character):

@example
regexp = `s([^[:alnum:][:blank:]]).*\1.*\1[ixsmogce]*`
@end example

@end table

Since version 2.11, in all kinds of regular expression specification, you
can insert newline characters, which will simply be ignored.  Thus,
e.g., the file:

@include test_newlines.lang.texinfo

@noindent
and the file:

@include test_nonewlines.lang.texinfo

@noindent
are equivalent.  However, the former is surely more readable.

Note however, that space characters are NOT ignored in regular expression
definitions.

@node Simple definitions, Line wide definitions, Ways of specifying regular expressions, Language Definitions
@section Simple definitions
@cindex simple language definition

The simplest way to specify language elements is to list the possible
alternatives.  This is the case, for instance, for keywords.  For
instance, in @file{java.lang} you have:

@example
keyword = "abstract|assert|break|case|catch|class|const",
          "continue|default|do|else|extends|false|final",
          "finally|for|goto|if|implements|instanceof|interface"
keyword = "native|new|null|private|protected|public|return",
          "static|strictfp|super|switch|synchronized|throw",
          "throws|true|this|transient|try|volatile|while"
@end example

You can separate quoted definitions with commas.  Alternatively, within
a quoted definition, alternatives can be separated with the pipe symbol
@code{|}.  The above definition defines the language element
@code{keyword}.  Each time an element is found in the source file, it is
highlighted with the style for the element with the same name in the
output format style file (note that all elements shown in the example
are taken from the language definition files that come with
source-highlight and there is a style for each of such elements, see
@ref{Configuration files}).  If such an element is not specified in the
output format style file, it is simply not highlighted (actually, it is
highlighted with style @code{normal}, @ref{Configuration files}) (so pay
attention to typos :-).

From the above example you may have noted that language element
definitions are cumulative, so the second @code{keyword} definition does
not replace the first one.   (Indeed, in some cases you may want to
actually redefine a language element; this is possible as explained in
@ref{Redefinitions and Substitutions}).

Note that words specified in double quotes have to match exactly in a
source file, and they must be isolated (not surrounded by anything but
spaces).  Thus for instance @code{class} is matched as a keyword, but in
@code{my_class} the substring @code{class} is not matched as keyword.
From the point of view of regular expressions a string such as
@code{class} in a double quote simple definition is intended as
@code{\<(class)\>}.

Special characters have to be escaped with the character @code{\}.  So
for instance if you want to specify the character @code{|}, which is
normally used to separate alternatives in double quoted strings, you
have to specify @code{\|}.

As explained in @ref{Ways of specifying regular expressions},
definitions in double quotes are interpreted literally (thus, e.g., a
dot @code{.} is interpreted as the character @code{.} not as the regular
expression wild card).   If you want to enjoy the full power of regular
expressions to specify a language alternative, you have to use single
quoted strings (@code{'}), instead of double quoted strings, or strings
quoted with backticks (@code{`}).

For instance, the following is the definition for a preprocessor
directive in C/C++:

@example
preproc = '^[[:blank:]]*#([[:blank:]]*[[:word:]]*)'
@end example

Note that the definition @code{'class'} is different from
@code{"class"}, as explained above.  Thus, for instance @code{'class'}
matches also the sub-expression @code{class} inside @code{my_class}.

Furthermore, you are not allowed to specify, in the same list, double
quoted strings and single quoted strings: you need to split such list
definitions.  Thus, for instance, the following definition is wrong:

@example
preproc = "#define",'^[[:blank:]]*#([[:blank:]]*[[:word:]]*)'
@end example

while the following one is correct:

@example
preproc = "#define"
preproc = '^[[:blank:]]*#([[:blank:]]*[[:word:]]*)'
@end example

@cindex nonsensitive
Finally, at the end of a list of definitions, one can specify the
keyword @code{nonsensitive}; in that case, the specified strings will be
interpreted in a non case sensitive way.  For instance, we use this
feature in Pascal language definition, @file{pascal.lang} where keywords
are parsed in a non sensitive way:

@example
keyword = "alfa|and|array|begin|case|const|div",
      "do|downto|else|end|false|file|for|function|get|goto|if|in",
      "label|mod|new|not|of|or|pack|packed|page|program",
      "put|procedure|read|readln|record|repeat|reset|rewrite|set",
      "text|then|to|true|type|unpack|until|var|while|with|writeln|write"
  nonsensitive
@end example

@node Line wide definitions, Order of definitions, Simple definitions, Language Definitions
@section Line wide definitions
@cindex lines

It is often useful to define a language element that affects all the
remaining characters up to the end of the line.   For such definitions,
instead of the @code{=} you must use the keyword @code{start}.   For
instance, the following is the definition of a single line comment in
C++:

@example
comment start "//"
@end example

This means that when the two characters @code{//} are encountered in the
source file, everything from these characters on, up to the end of
the line, will be highlighted according to the style @code{comment}.

@node Order of definitions, Delimited definitions, Line wide definitions, Language Definitions
@section Order of definitions
@cindex definition order

It is important to observe that the order of language definitions is
important since it will be used during regular expression matching (this
will be detailed in @ref{How source-highlight works}).  You then have to
make sure that, if there are definitions that start with same
characters, the longest expression is specified first in the file.  For
instance if you write

@example
symbol = "/"
comment start "//"
@end example

The first expression will always be matched first, and the
second expression will never be matched.  The right order is

@example
comment start "//"
symbol = "/"
@end example

@node Delimited definitions, Variable definitions, Order of definitions, Language Definitions
@section Delimited definitions
@cindex delimited definitions

Many elements are delimited by specific character sequences.
For instance, strings and multiline comments.  The syntax for
such an element definition is

@example
<name> delim <left delimited> <right delimiter> \
        @{escape <escape character>@} \
        @{multiline@} @{nested@}
@end example

The @code{escape} statement specifies the escape character that may
precede one of the delimiters inside the element.  This is optional.

For instance, this is the definition of C-like strings:

@example
string delim "\"" "\"" escape "\\"
@end example

Note that @code{\} is a special characters in definitions so it has to
be escaped.  If the @code{escape} specification was omitted, the C
string @code{"write \"hello\" string"} would have been highlight
incorrectly (it would have been highlighted as the string
@code{"write \"}, the normal character sequence @code{hello\} and
the string @code{" string"}).

The option @code{multiline} specifies that the element can spawn
multiple lines.  For instance, PHP strings are defined as follows:

@example
string delim "\"" "\"" escape "\\" multiline
@end example

The option @code{nested} instructs to count possible multiple
occurrences of delimited characters and to match relative multiple
occurrences (using a stack).  For instance, if we wanted to highlight
C-like multiline comments in a nested way@footnote{As Ed Kelly correctly
pointed out, C-style comments are NOT nested; it's a big shame I've been
using C++ and Java for years and have always thought they were nested
:-)... Thus, in previous versions of source-highlight distributions,
C-style comments were (uncorrectly) defined as nested.  Thank you Ed,
for your feedback!}, we could use the following definition:

@example
comment delim "/*" "*/" multiline nested 
@end example

If @code{nested} was not used, then the closing @code{*/} of the
following nested comment would conclude the comment (and the second
@code{*/} would not be highlighted as a comment):

@example
/*
   This is a /* nested comment */
*/
@end example

Note that, in order for a delimited language element to be nested, its
starting and ending elements must be different; thus, for instance, the
following definition is not correct:

@example
string delim "\"" "\"" nested # WRONG!
@end example

As said above, definitions are cumulative, and they are also cumulative
even when using different syntactic forms.  Thus, for instance, the
complete definition for C++-style comments are the following (actually,
the definition of C-style comment is more involved, see the file
@file{c_comment.lang}):

@example
comment start "//"
comment delim "/*" "*/" multiline
@end example

@node Variable definitions, Dynamic Backreferences, Delimited definitions, Language Definitions
@section Variable definitions
@cindex variables

It is possible to define variables to be re-used in
many parts in a language definition file.
A variable is defined by using 

@code{vardef} <name of the variable> @code{=} <list of definitions>

Once defined, a variable can be used by prepending the
symbol @code{$} to its name.  For instance,

@example
vardef FUNCTION = '(?:[[:alpha:]]|_)[[:word:]]*(?=[[:blank:]]*\()'
function = $FUNCTION
@end example

The capital letters are used only for readability.

It is also possible to concatenate variables and expressions, and reuse
variables inside further variable definitions:

@example
vardef basic_time = '[[:digit:]]@{2@}:[[:digit:]]@{2@}:[[:digit:]]@{2@}'
vardef time = '\<' + $basic_time + '\>'
@end example

@node Dynamic Backreferences, File inclusion, Variable definitions, Language Definitions
@section Dynamic Backreferences
@cindex dynamic backreference

With @emph{dynamic backreferences} you can refer to a string matched by
the regular expression of the first element of a @code{delim}
specification@footnote{Since version 2.8}.  I called these
backreferences dynamic in order to distinguish them by the
backreferences of regular expression syntax, @ref{Ways of specifying
regular expressions}.  This is crucial in cases when the right delimiter
depends on a subexpression matched by the left delimiter; for instance,
Lua comments can be of the shape @code{--[[ comment ]]} or @code{--[=[
comment ]=]}, but not @code{--[=[ comment ]]} neither @code{--[[ comment
]=]} (furthermore, they can be nested)@footnote{I'm grateful to Jurgen
Hotzel for rising this issue about Lua comments; this led me to
introduce dynamic backreferences.}.  Thus, the regular expression of
the right element depends on the one of the left element.

A dynamic backreference is similar to a variable (@ref{Variable definitions}),
but there's no declaration, and have the shape of

@example
@@@{number@}
@end example

@noindent
where @code{number} is the number of the marked subexpression in the
left delimiter (source-highlight will actually check that such a marked
subexpression exists in the left delimiter).

For instance, this is the definition of Lua comments (see also
@file{lua.lang}):

@example
environment comment delim `--\[(=*)\[` "]" + @@@{1@} + "]" 
            multiline nested begin
  include "url.lang"
  ...
end
@end example

@noindent
Note how the left delimiter can match an optional @code{=}, as a
marked subexpression, and the right delimiter refers to that with @@@{1@}.

Source-highlight will take care of escaping possible special characters
during dynamic backreference substitutions.  For instance, suppose that
you must substitute @code{|} for @@@{1@}, because we matched @code{|}
with the subexpression @code{[^[:alnum:]]} in a delim element like the
following one:

@example
comment delim `([^[:alnum:]])` @@@{1@} 
@end example

@noindent
Since @code{|} is a special character in regular expression syntax
source-highlight will actually replace @code{@@@{1@}} with @code{\|}.

IMPORTANT: the right delimiter can only refer to subexpressions of its
left delimiter; thus, in case of nested delim element definitions (e.g.,
in states or environment, @ref{State/Environment Definitions}), the left
delimiter acts as a binder and hides possible subexpressions defined in
outer delim elements.

This is crucial to correctly match nested delimited elements with
backreferences: source-highlight will correctly recognize this nested
(and syntactically correct) Lua comment:

@example
--[[
  first level comment
  --[=[
    second level
     --[[
       third level
    ]]
  ]=]
]]
@end example

@node File inclusion, State/Environment Definitions, Dynamic Backreferences, Language Definitions
@section File inclusion
@cindex file inclusion

It is possible to include other language definition files into another
file.  This is inclusion actually physically includes the contents of
the included file into the current file during parsing, at the exact
point of inclusion (just like the @code{#include} in C/C++).   This is
useful for re-using definitions in many files.   For instance, C++
comment definitions are given in a file @file{c_comment.lang}, and this
file is included in the Java and C++ definition files.  The same happens
for number and functions.  For instance, the file @file{java.lang}
contains the following include instructions:

@example
include "c_comment.lang"

include "number.lang"

keywords ...

include "function.lang"
@end example

Note that the order of inclusion is crucial since the order of
definition is crucial.  If function definition was included before
keyword definitions, then the sentence @code{if (exp)} would be
highlighted as a function invocation (see @ref{Order of definitions} and
@ref{How source-highlight works}).

@node State/Environment Definitions, Explicit subexpressions with names, File inclusion, Language Definitions
@section State/Environment Definitions
@cindex states
@cindex environments

Sometimes you want some source element to be highlighted only if they
are surrounded by other elements.  Source-highlight language definitions
provides also this feature.

@example
state|environment <standard definition> begin
  <other definitions>
end
@end example

This structure is recursive (so other state/environment definitions can
be given within a state/environment).   The meaning of a
state/environment is that the definitions within the @code{begin
... end} are matched only if the definitions that define the
state/environment have been matched.  When entering a state/environment,
however, the definitions given outside the state/environment are not
matched.  The difference between @code{state} and @code{environment} is
that in the latter, normal parts of the source language (i.e., those
that do not match any definition) are highlighted according to the style
of the definition that defines the environment.

As an example, the following defines the multiline  nested C comment,
and highlights URL and e-mail addresses only when they appear inside a
comment (note that this uses file inclusion):

@example
environment comment delim "/*" "*/" multiline nested begin
      include "url.lang"
end
@end example

Note that we used @code{environment} because everything else inside a
comment has to be formatted according to the comment style.

While for programming language definitions states/environments can be
avoided (although they allow to highlight some parts only if inside a
specific environment, e.g., URLs inside comments, or documentation tags
in Javadoc comments), they are pretty important for highlighting files
such as logs and ChangeLog files, since elements have to be highlighted
when they appear in a specific position.   For instance, for ChangeLog
(see @file{changelog.lang}), we use a state for highlighting the date,
name, e-mail or URL (taken from @file{url.lang}):

@example
state date start '[[:digit:]]@{2,4@}-?[[:digit:]]@{2@}-?[[:digit:]]@{2@}' begin
  include "url.lang"
  name = '([[:word:]]|[[:punct:]])+'
end
@end example

Note that definitions that appear inside a state/environment have the
same scope of the expressions that define the environment.  While this
makes sense for @code{start} and @code{delim} definitions, it may make
less sense for simple definitions (i.e., those that simply lists all
possible expressions): in fact, in this case, such expressions do not
define a scope.  For such definitions, the semantics of
state/environment is that the state/environment starts after matching one
of the alternatives.  And where will it end?   In this case you must
explicitly exit the environment.   For instance, you can say that, when
inside a state/environment, a specific language definition, when
encountered also exits the environment, with the keyword @code{exit}
(you can also specify the number of states to exit).
You can even exit all the environments with @code{exitall}.   For
instance, the following definition, highlights a non empty string
following a web method:

@example
vardef non_empty = '[^[:blank:]]+'

state webmethod = "OPTIONS|GET|HEAD|POST|PUT|DELETE",
          "TRACE|CONNECT|PROPFIND|MKCOL|COPY|MOVE|LOCK|UNLOCK" begin
  string = $non_empty exit
end
@end example

If you ever need such advanced features, you may want to take a look at
the @file{log.lang} definition file that defines highlighting for
several log files (access logs, Apache logs, etc.).  Moreover, there
might be cases, and the above one is one of such cases, explicit
subexpressions with names will be enough (see @ref{Explicit
subexpressions with names}).

We conclude this section with an interesting example: comments in
M4 files can start with the @code{dnl} keyword (up to the end of line),
e.g.,

@example
dnl @@synopsis AC_CTAGS_FLAGS
@end example

Now if we want to highlight the @code{dnl} as a keyword, and the rest of
line as a comment, we cannot simply rely on an environment, since this
would highlight all the line with the same style.  Moreover, we want
to highlight elements starting with @code{@@} differently, so we actually
need a state (this would allow us also to highlight urls inside a comment
just like in C++ comments in the example above).  Thus, we need
to simulate an environment with a state, and we do this for M4 as follows
(see the file @file{m4.lang}):

@example
state keyword start "dnl" begin
  # avoid spaces in front of urls or @@[[:alpha:]]+ be captured as prefixes
  comment = '[[:blank:]]+'
  include "url.lang"
  include "html.lang"
  type = '@@[[:alpha:]]+'
  # everything else is a comment
  comment = '.+'
end
@end example

Once entered the state, every isolated space character is highlighted as
a comment; then we have rules for URLs and @@ elements; then everything
else (@code{.+}) is highlighted as a comment.

One might think that a smarter way would be to have simply the following
definition (after all, why bothering highlighting spaces as comments):

@example
state keyword start "dnl" begin
  include "url.lang"
  include "html.lang"
  type = '@@[[:alpha:]]+'
  comment = '.+'
end
@end example

Well, with this definition spaces in front of matched URLs or @@
elements would be highlighted as normal, being considered as prefixes.
This is due to how source-highlight searches for matching rules; we refer
to @ref{How source-highlight works} for further details.

@node Explicit subexpressions with names, Redefinitions and Substitutions, State/Environment Definitions, Language Definitions
@section Explicit subexpressions with names

Often, you need to specify two program elements in the same regular
expressions, because they are tightly related, but you also need to
highlight them differently.

For instance, you might want to highlight the name of a class (or
interface) in a class (or interface)
definition (e.g., in Java).  Thus, you can rely on the preceding
@code{class} keyword which will then be followed by an identifier.

A definition such as

@example
keyword = '(\<(?:class|interface))([[:blank:]]+)([$[:alnum:]]+)'
@end example

@noindent
will not produce a good final result, since the name of the class will
be highlighted as a keyword, which is not what you might have wanted:
for instance, the class name should be highlighted as a @code{type}.

Up to version 2.6, the only way to do this was to use state or
environments (@ref{State/Environment Definitions}) but this tended to be
quite difficult to write.

Since version 2.7, you can specify a regular expression with marked
subexpressions and bind each of them to a specific language element (the
regular expression must be enclosed in @code{`}, see @ref{Ways of
specifying regular expressions}):

@example
(elem1,...,elemn) = `(subexp1)(...)(subexpn)`
@end example

Now, with this syntax, we can accomplish our previous goal:

@example
(keyword,normal,type) = 
  `(\<(?:class|interface))([[:blank:]]+)([$[:alnum:]]+)`
@end example

@noindent
This way, the @code{class} (or @code{interface}) will be highlighted as
a keyword, the separating blank characters are formatted as
@code{normal}, and the name of the class as a @code{type}.

Note that the number of element names must be equal to the number of
subexpressions in the expression; furthermore, at least in the current
version, the expression can contain only marked subexpressions (no
character outside is allowed) and no nested subexpressions are allowed.

Thus, the following specifications are NOT correct:

@example
(keyword,symbol) = `(...)(...)(...)` # number of elements doesn't match
(keyword,symbol) = `(...(...)...)(...)` # contains nested subexpressions
(keyword,symbol) = `...(...)...(...)` # outside characters
@end example

This mechanism permits expressing regular expressions for some situation
in a much more compact and probably more readable way.  For instance,
for highlighting ChangeLog parts (the optional @code{*} as a symbol, the
optional file name and the element specified in parenthesis as a
@code{file} element, and the rest as @code{normal}) such as

@example
  * src/Makefile.am (source_highlight_SOURCES): correctly include
  changelog_scanner.ll

  * this is a comment without a file name
@end example

@noindent
before version 2.6, we used to use these two language definitions:

@example
state symbol start '^(?:[[:blank:]]+)\*[[:blank:]]+' begin
  state file start '[^:]+\:' begin
    normal start '.'
  end
end

state normal start '^(?:[[:blank:]]+)' begin
  state file start '[^:]+\:' begin
    normal start '.'
  end
end
@end example

@noindent
which can be hard to read after having written them.  Now, we can write
them more easily (see @file{changelog.lang}):

@example
(normal,symbol,normal,file)= 
  `(^[[:blank:]]+)(\*)([[:blank:]]+)((?:[^:]+\:)?)` 
(normal,file)= `(^[[:blank:]]+)((?:[^:]+\:)?)` 
@end example

Since a language element definition using explicit subexpressions with
names consists of more than one element, and thus of more than one
formatting style, it cannot be used to start an environment (what would
the default element be?); while, as seen above, they can be used to
start a state.

@node Redefinitions and Substitutions, How source-highlight works, Explicit subexpressions with names, Language Definitions
@section Redefinitions and Substitutions

These two features are useful when you want to define
a language by re-using an existing language definition
with some changes.  Typically you @code{include} another
language definition file and you redefine/substitute some
elements.

@cindex redef
When you use @code{redef} you erase all the previous
definitions of that language elements with the new one.
The new language element definition will be placed exactly
in the point of the new definition.
We use this feature, for instance, when we define the
@code{sml} language by re-using the @code{caml} one:
they differ only for the keywords@footnote{At least, to the best
of my knowledge :-)}.  In fact, the contents of
@file{sml.lang} is summarized as follows:

@example
include "caml.lang"

redef keyword = "abstraction|abstype|and|andalso..."

redef type = "int|byte|boolean|char|long|float|double|short|void"
@end example

Since the new language element definition appears in the
exact point of the redefinition, this means that
such a regular expression will be matched only if all
the previous ones (the ones of the included file) cannot
be matched.  This may lead to unwanted results in some
cases (not in the @code{sml} case though).
In other words the following code

@example
keyword = "foo"
keyword = "bar"
type = "int"
redef keyword = "myfoo"
@end example

@noindent
is equivalent to the following one

@example
type = "int"
keyword = "myfoo"
@end example

@cindex subst
If this is not what you want, you can use @code{subst},
which is similar to @code{redef} apart from that it
replaces the previous first definition of that language
element in the exact point of that first definition
(all other possible definitions are simply erased).
That is to say that the following code

@example
keyword = "foo"
keyword = "bar"
type = "int"
subst keyword = "myfoo"
@end example

@noindent
is equivalent to the following one

@example
keyword = "myfoo"
type = "int"
@end example

It is up to you to decide which one fits best your needs.
We could use this feature to define @code{javascript} in terms
of @code{java}, e.g.:

@example
include "java.lang"

subst keyword = "abstract|break|case|catch|class|if..."
@end example

@noindent
Here using @code{redef} would have led to the unwanted behavior that
@code{if (exp)} would have been highlighted as a function call, since
the function element definition would have come first (and then
matched first) than the redefinition of @code{if} as a keyword.
Another example is the language definition for C# by reusing the one
for C/C++, @ref{Highlighting C/C++ and C#}.

@node How source-highlight works, Notes on regular expressions, Redefinitions and Substitutions, Language Definitions
@section How source-highlight works

As hinted at the beginning of @ref{Language Definitions},
source-highlight uses the definitions in the language definition file to
internally create, on-the-fly, regular expressions that are used to
highlight the tokens of an input file.  Here we provide some internal
details that are crucial to understand how to write language definition
files correctly@footnote{The strategy used by source-highlight for
matching regular expressions changed since version 2.11 (and in version
2.10 the strategy used was not completely conceptually correct and
it had a lot of overhead).}.

First of all, for each element definition an highlighting rule is
created by source-highlight (even if they correspond to the same
language element); thus, each language definition file will correspond
to a list of highlighting rules.  For each line of the input file,
source-highlight will try to match all these rules against the whole
line (more formally, against the part of the line that has not been
highlighted yet).  It will not stop as soon as an highlighting rule
matched, since there might be another rule that matches ``better''.
Now, everything basically reduces to the semantics of that @emph{better
match}.

@cindex matching strategy
The strategy used by source-highlight is to select the first matching rule

@itemize

@item
with empty prefix (or prefix containing only space characters, i.e.,
spaces or tabs) or

@item
with the smallest prefix,

@end itemize

@cindex prefix
where the @emph{prefix} of a matched rule is the part of the examined
string that did not match@footnote{according to the terminology of
regular expressions.}.  Thus, for instance, if we try to match
the simple regular expression @code{=} against the string

@example
i = 10;
@end example

@cindex suffix
then the prefix is @code{i }, including the space.  Following the
terminology of regular expression, the remaining part that did not
match, i.e., @code{ 10;}, is the @emph{suffix}.  When source-highlight
finds a matching rule, according to the above strategy, it formats the
matched part (and the prefix as @code{normal}), and then it starts again
searching for a matching rule on the suffix, until it processed the
whole line.


Let us explain this strategy a little bit further with an example.
Consider the following language definition file:

@include strategy_example.lang.texinfo

@noindent
and the following line to be highlighted:

@example
int i = null
@end example

Then source-highlight performs these steps:

@enumerate

@item
The first matching rule is the one for @code{type}; since it has
an empty prefix, there's no need to look any further: it highlights
@code{int} as @code{type}; the remaining part to be processed is now
@code{ i = null};

@item
the first matching rule is the one for @code{keyword}, with the prefix
@code{ i = }; since the prefix is not empty (nor it contain only
spaces), we inspect other rules;

@item
the next matching rule is the one for @code{symbol}, with prefix @code{
i }, which is smaller than the one for @code{keyword}, and since there
are no other matching rules, the one for @code{symbol} is better, and we
highlight @code{=} as symbol; the remaining part to be processed is now
@code{ null};

@item
the first matching rule is the one for @code{keyword}, and, since it has
a prefix with only spaces, we look no further, and we highlight
@code{null} as @code{keyword}.

@end enumerate

We conclude this section by showing the following language definition,
which summarizes what we said about the highlighting strategy:

@include test_string_stop.lang.texinfo


@node Notes on regular expressions, The program check-regexp, How source-highlight works, Language Definitions
@section Notes on regular expressions
@cindex regular expressions

Although we refer to Boost documentation for such
syntax@footnote{@uref{http://www.boost.org/libs/regex/doc/syntax.html}},
we want to provide here some explanations of some forms of regular
expressions that might be unknown but that are pretty useful in language
definitions.

@cindex non-marking parenthesis
Typically, when you need to group sub-expressions with parenthesis,  but
you don't want the parenthesis to spit out another marked
sub-expression,  you can use a @emph{non-marking parenthesis}
@code{(?:expression)}.  This is not necessary in the language definition
syntax: even though you use standard parenthesis, source-highlight will
transform it into a non-marking parenthesis.

@cindex marked subexpressions
Source-highlight translates possible @emph{marked subexpressions}, i.e.,
those enclosed in @code{(} and @code{)}, into non-marked subexpressions
(i.e., those explained above).  Since version 2.7, if you specify the
expression inside @code{`} the marked subexpressions are left as such
(see also @ref{Ways of specifying regular expressions}).  This is useful
for @emph{backreferences} and @emph{conditionals}.

@cindex backreference
An escape character followed by a digit n, where n is in the range 1-9,
is a @emph{backreference} matches the same string that was matched by
sub-expression n.  For example the expression @code{^(a*).*\1$} will
match the string: @code{aaabbaaa} but not the string @code{aaabba}.
Backreferences are useful to write compact language elements, such
as in the case of Perl's substitution modifiers; thus

@example
regexp = `s([^[:alnum:][:blank:]]).*\1.*\1[ixsmogce]*`
@end example

will match all these forms

@example
s/foo/bar/g
s|foo|bar|g
s#foo#bar#g
s@@foo@@bar@@g
@end example

@cindex lookahead asserts
A useful regular expression form is the @emph{Forward Lookahead Asserts}
that come in two forms, one for positive forward lookahead asserts, and
one for negative lookahead asserts:

@table @code
@item (?=abc) 
matches zero characters only if they are followed by the expression
``abc''.
@item (?!abc) 
matches zero characters only if they are not followed by the expression
``abc''.
@end table

For instance, in the definition of a function (@file{function.lang}) we
use the following regular expression:

@example
([[:alpha:]]|_)[[:word:]]*(?=[[:blank:]]*\()
@end example

@noindent
Thus after the name of a function we test, with the regular expression
@code{(?=\()} whether an open parenthesis @code{(} can be matched.  If
it can be matched, however, we leave that part in the input, so that the
parenthesis will not be formatted the same way of a function name (see
also @ref{How source-highlight works} to understand better this language
element definition).

Please, be careful when using such regular expression forms: since part
of the input is not actually removed you may end up always scanning the
same input part (thus looping) if you do not write the regular
expressions well.  For instance, consider this language definition

@example
state foo = '(?=foo)' begin
  foo = '(?=foo)'
end
@end example

@noindent
and the following input file:

@example
hello
foo
bar
@end example

@noindent
As soon as we match the word @code{foo} we leave it in the input and we
enter a state where we try to match the word @code{foo} still leaving it
in the input.  As you might have guess this will make source-highlight
loop forever.  Probably one might have wanted to write this
language definition:

@example
state foo = '(?=foo)' begin
  foo = 'foo'
end
@end example

@noindent
but a cut-and-paste error had its way ;-)

@cindex lookbehind asserts
You can also use @emph{Lookbehind Asserts}:

@table @code
@item (?<=pattern)
 consumes zero characters, only if pattern could be matched against the
 characters preceding the current position (pattern must be of fixed
 length).
@item (?<!pattern) 
 consumes zero characters, only if pattern could not be matched against the
 characters preceding the current position (pattern must be of fixed
 length).
@end table

@cindex conditional expressions
Another advanced regular expression mechanism is the one of
@emph{conditional expressions}

@table @code
@item (?(condition)yes-pattern|no-pattern) 
attempts to match yes-pattern if the condition is true, otherwise
attempts to match no-pattern.
@item (?(condition)yes-pattern) 
attempts to match yes-pattern if the condition is true, otherwise fails.
@end table

Condition may be either a forward lookahead assert, or the
index@footnote{the index only, without the escape character.} of a
marked sub-expression (the condition becomes true if the sub-expression
has been matched).  For instance, the following expression@footnote{This
expression was provided by John Maddock, the author of the Boost regex
library, as a solution of a problem I posted on the boost list,

@uref{http://thread.gmane.org/gmane.comp.lib.boost.devel/158237/focus=158276}},
that we wrote on more lines to try to make it more readable

@example
(?:
  (\()
  |(\[)
  |(\@{)
)
[[:alpha:]]*
(?:
  (?(1)
    \)
    |(?:(?(2)
      \]
      |(?:\@}
)))))
@end example

will match @code{(foo)}, @code{[foo]} and @code{@{foo@}} but not
@code{(foo]}, @code{@{foo]} or @code{@{foo)}.

@node The program check-regexp, Listing Language Elements, Notes on regular expressions, Language Definitions
@section The program @command{check-regexp}

@cindex @command{check-regexp}
Since version 2.7, the source-highlight package comes with a
small additional program, @command{check-regexp}, that permits
testing regular expressions on the command line.

You simply pass as the first command line argument the regular
expression and then the strings you want to try to match (actually, the
program searches the string for the given regular expression, so it is
not required to match the whole string).  It is crucial, in order to
avoid shell substitutions, to enclose both the expression and the
strings in single quotes.

The program then prints some information about the (possibly successful
matching).  The @code{what[0]} part represents the whole match, and
@code{what[i]} part represents the i-th marked subexpression that
matched.  The program also prints possible prefix and suffix.

Here's an example of output of the program:

@example
check-regexp '(a+)(.*)\1' 'aabcdaa' 'babbbacc'

searching      : aabcdaa
for the regexp : (a+)(.*)\1
what[0]: aabcdaa
  what[1]: aa
  length: 2
  what[2]: bcd
  length: 3
total number of matches: 1

searching      : babbbacc
for the regexp : (a+)(.*)\1
prefix: b
what[0]: abbba
  what[1]: a
  length: 1
  what[2]: bbb
  length: 3
suffix: cc
total number of matches: 1
@end example

And here's the example of matching parenthesis we saw in @ref{Notes on
regular expressions}:

@smallexample
check-regexp \
   '(?:(\()|(\[)|(\@{))[[:alnum:]]*(?:(?(1)\)|(?:(?(2)\]|(?:\@})))))' \
   '@{ciao@}' '(foo]' '[hithere]'

searching      : @{ciao@}
for the regexp : (?:(\()|(\[)|(\@{))[[:alnum:]]*(?:(?(1)\)|(?:(?(2)\]|(?:\@})))))
what[0]: @{ciao@}
  what[3]: @{
  length: 1
total number of matches: 1

searching      : (foo]
for the regexp : (?:(\()|(\[)|(\@{))[[:alnum:]]*(?:(?(1)\)|(?:(?(2)\]|(?:\@})))))
total number of matches: 0

searching      : [hithere]
for the regexp : (?:(\()|(\[)|(\@{))[[:alnum:]]*(?:(?(1)\)|(?:(?(2)\]|(?:\@})))))
what[0]: [hithere]
  what[2]: [
  length: 1
total number of matches: 1
@end smallexample

@node Listing Language Elements, Concluding Remarks, The program check-regexp, Language Definitions
@section Listing Language Elements

In order for language definitions to be really useful they must be
used in proper combination with formatting styles (see @ref{Output
format style}).  However, these different files might not be developed
by the same person, or simply some one may want to customize one of
these.  In order to define good output formatting style files you should
be aware of each language element defined by a language definition file.
Instead of having to look inside the language definition file itself
(and recursively in each included file) you can use the command line
@cindex @code{--show-lang-elements}
option @code{--show-lang-elements}@footnote{Since version 2.4.}, that
simply prints to the standard output all the language elements that
can be highlighted with a specific language definition file.

For instance, for @code{cpp.lang} you get:

@example
@include cpp.langelems.texinfo
@end example

while for @code{log.lang} you get:

@example
@include log.langelems.texinfo
@end example

@node Concluding Remarks, Debugging, Listing Language Elements, Language Definitions
@section Concluding Remarks

By mixing all these features you can unleash your imagination and define
highlighting for complex source languages such as Flex and Bison by
writing few lines of code and re-use existing ones.  For instance, Flex
and Bison have their own syntax and lets you write C/C++ code in
specific parts of the source language, e.g., the code between  the
outmost brackets, in the following example, is C++ code, and should be
highlighted following C++ language definitions (apart from variables
that are prefixed with @code{$}):

@example
globaltags : options @{ if (...) @{ setTags( $1 ); @} @}
@end example

This is easy to do (taken from @file{flex.lang}):

@example
state cbracket delim "@{" "@}" multiline nested begin
  variable = '\$.'
  include "cpp.lang"
end
@end example

Note that, since we used @code{nested} we can be sure
that the C++ language definitions are not considered
anymore when we matched the last closing @code{@}}.

@node Debugging, Tutorials on Language Definitions, Concluding Remarks, Language Definitions
@section Debugging
@cindex debug

When writing a language definition file, it is quite useful to be able
to debug it (by using complex regular expressions one may experience
unwanted behaviors).  Since version 2.1 the command line option
@code{--debug-lang} is available.  When using this option, some
additional information are printed to the standard output.  

Since version 2.5 this option also accepts the a sub specification (see
@ref{Invoking source-highlight}).  When using @code{dump} (the default)
all the additional information explained below will be dumped without
interaction with the user.  When using @code{interactive}, for each
formatted string the program will stop waiting for a command from the
user.  In this very primordial version of interactive debug, the user
will only have to press @code{ENTER} to make the program continue until
the next formatted string.  This way, the programmer will have the
chance to step the highlighting of each part of the input file.
Moreover, when debugging is enabled, no buffering will be performed by
the program, thus each formatted element will be immediately available
in the output.  For instance, you can use the command @command{tail -f}
to see the modifications on the output file on-the-fly.

When using this command line option the additional information produced
has the following format:

@example
<.lang filename>:<line number>
expression: <matched subexpression>
formatting: <source file string to be formatted>
entering: <next state's id>
exiting state, level: <number of states> 
@end example

The lines starting with @code{entering}, @code{exiting} and
@code{exitingall} are related to entering a new state/environment and
exiting one and all states/environments (@code{current state}, if shown,
comes after @code{entering} and prints the same state's regular
expression but after the substitution of dynamic backreferences,
@ref{Dynamic Backreferences}).  The first line shows a link to the
@file{.lang} definition file and the line number, i.e., and the
sub-expression that matched and the line starting with @code{formatting}
shows the source file string that matched with that expression.  If a
line starting with @code{formatting} is not preceded by a line with the
link to the sub-expression, it means that no particular regular
expression has matched, and thus the style @code{normal} will be used to
format that string.

Consider the following (simplified) Java source file:

@include debug.java.texinfo

Now you can debug the @file{java.lang} file by using the
@code{--debug-lang} command line option.  And the output is as follows:

@example
c_comment.lang:24
expression: "/\*"
formatting "/*" as comment
entering state: 23
formatting "  This is to demonstrate --debug-lang" as default
formatting "  " as default
url.lang:3
expression: "(?:(?:<?)[[:word:]]+://[[:word:]\./\-_]+(?:>?))"
formatting "http://www.lorenzobettini.it" as url
c_comment.lang:24
expression: "\*/"
formatting "*/" as comment
exiting state, level: 1
java.lang:1
expression: "\<(?:import|package)\>"
formatting "package" as preproc
formatting " hello" as default
symbols.lang:1
expression: "(?:~|!|%|\^|\*|\(|\)|-|\+|=|\[|\]|\\|:|;|,|\.|/|\?|&|<|>|\|)"
formatting ";" as symbol
... omissis ...
c_comment.lang:13
expression: "//"
formatting "//" as comment
entering state: 12
formatting " just some greetings ;-)  /*" as default
c_comment.lang:13
expression: "\z"
formatting "" as comment
exiting state, level: 1
... omissis ...
@end example

This should provide enough information to understand how the regular
expressions are used and how the states/environments are entered and
exited.  Please note that the sub-expressions that are shown may
differ from the original ones specified in the @file{.lang} file.  This
is due to the preprocessing that is performed by Source-highlight.
Moreover, some sub-expressions are not defined at all in the
@file{.lang} file: for instance, this is the case for line wide
definitions, i.e., those that are defined with the keyword @code{start},
@ref{Line wide definitions}.  The last lines above, showing
@code{expression: "\z"}, means that we matched the end of a line.

Another useful feature in debugging is the option @code{--show-regex}
that shows, on the standard output, the regular expression automaton
that source-highlight creates.

For instance, consider this language definition
(@file{comment-show.lang}):

@include comment-show.lang.texinfo

@noindent
If you now execute the following command:

@example
source-highlight --show-regex=comment-show.lang
@end example

@noindent
you will get, on the standard output, the following output@footnote{Up
to version 2.9 the output of @code{--show-regex} was a little bit more
complex to read; hopefully this output is better.}:

@include comment-show.show.texinfo

@noindent
This shows the states and highlight rules of the regular expression
automaton that source-highlight creates and will use to format an input
source.

Each state is associated a unique number in order to identify it;
moreover, the default element of the state is shown (i.e., if none of
the state's rule match, then that part is highlighted with the default
element style). For instance, in the initial state the default style is
normal.  Then for each state it shows the rules for that state.  
For each rule you can see the corresponding element of the rule, 
the regular expression for the rule and some other information, that we
explain in the following.

We can see that if we match a @code{/**} (it is shown as a string with
escaped special characters, @code{/\*\*}) we enter a new state, in this
case the state 2 (@code{next: 2}).  This corresponds to the delimited
element defining a new environment (@ref{State/Environment
Definitions}).  The fact that it is actually an environment and not a
state@footnote{Please note that this concept of state is different
from the concept of ``state'' of an automaton.} can be seen by the fact
that the default element is the same of the environment itself.  If we
match a @code{*/}, i.e., the end of the delimited element, we exit one
level (@code{exit level: 1}) meaning that we go back to state 1.  Then
we have the state for @code{cbracket}, which is not an environment, in
fact its default state is normal.  The second rule of this state,
@code{\\.} represents the @code{escape} string of the state definition.
Since the delimited element is defined as nested, we have a third rule
@code{@{} which has the @code{nested} information; thus, if we match it,
we simply enter a new instance of state 3 itself.

The @code{string} and @code{string2} show the difference implied by the
@code{multiline} option: since source-highlight handles a line of input
separately, the first delimited definition can be handled with a single
regular expression while the multiline version cannot.

Note that the states/environments are indented so that it's
easier to understand the outer and the inner states.

Let us now consider a variation of the previous example:

@include comment-show2.lang.texinfo

and let us see the output of @code{--show-regex}

@include comment-show2.show.texinfo

Since in the rule @code{regexp} we used the @code{`} regular expression
(see @ref{Ways of specifying regular expressions}), then, the marked
subexpressions are not translated in order to make backreferences work
correctly.

The last rule uses explicit subexpressions with names (see @ref{Explicit
subexpressions with names}); although that expression is made up of
different elements, the expression is matched as a whole.

@node Tutorials on Language Definitions,  , Debugging, Language Definitions
@section Tutorials on Language Definitions

Now we provide some examples of language definitions.   In the previous
sections we have already provided some code snippets, while here we
provide complete examples of language definitions that are included in
the source-highlight distribution itself.

In particular we will first show the language definition for the
language definition syntax itself (file @file{langdef.lang}).  This will
be used to highlight the examples of language definitions that we will
show in this section (the highlighting will not be visible if you are
viewing this manual with the @code{info} command).  Of course, this
example is highlighted itself.

@include langdef.lang.texinfo

The style that is used to highlight these examples in Texinfo is
@file{texinfo.style} that is shown in @ref{Output format style}.  The
language definition for the style syntax (file @file{style.lang}) is
even simpler:

@include style.lang.texinfo

Note that this definition is pretty simple since the language
definition syntax is simple.  In the next examples we will see how to
use more complex features to highlight more complex language syntaxes.

@menu
* Highlighting C/C++ and C#::   
* Highlighting Diff files::     
* Pseudo semantic analysis::    
@end menu

@node Highlighting C/C++ and C#, Highlighting Diff files, Tutorials on Language Definitions, Tutorials on Language Definitions
@subsection Highlighting C/C++ and C#

This is the language definition for C, included in the file
@file{c.lang}:

@include c.lang.texinfo

@noindent
Note that this makes use of lots of @code{include}s since these parts
are reused in other language definitions (e.g., Java has lots of parts
that are in common with C/C++ so we wrote these parts in separate
files).  In particular the comments definitions:

@include c_comment.lang.texinfo

@noindent
Here we have the definitions for line-wide comments (@code{//}) and
for multi line comments where we highlight also URL addresses and
e-mail addresses (defined in the file @file{url.lang} not shown here).
Moreover, for comments that are used in automatic documentation
generation tools (such as Doxygen or Javadoc), i.e., those that start
with @code{/**} or @code{///}) we also highlight the complete HTML
syntax (defined in the file @file{html.lang} not shown here).

Going back to @file{c.lang} we see that we use subexpressions with names
(see @ref{Explicit subexpressions with names}) for highlighting the
struct name (when preceded by @code{struct}, highlighted as a keyword).

For preprocessor directives @code{#include} we use a state definition
since in this case the file included with the @code{<file>} syntax must
be formatted as strings (and only in this context the @code{<>} must be
considered as strings, anywhere else they are operators).  Since a state
erases definitions defined outside the state we must include
@file{c_comment.lang} again in order to highlight comments also in this
context@footnote{As a future extension we might think of providing a
way, in the language definition syntax, to define a state/environment
that extends the outer contexts instead of overriding them.}.  Then we
have a definition of @code{preproc} that catches all the other
preprocessor directives.

The included file @file{number.lang} defines the regular expression that
catches number constants (not shown here), then we include the file
@file{c_string.lang} that define strings (again shared by Java):

@include c_string.lang.texinfo

@noindent
inside a string we want to highlight in a different way the special
characters (such as, e.g., @code{\n}, @code{\t}, etc.)  and in general
escaped characters, matched by the regular expression `@code{\\.}'.

The included file @file{symbols.lang} defines all the symbols (shared
also by other languages):

@include symbols.lang.texinfo

@noindent
This has nothing interesting but the fact that it shows that
the character @code{\} and @code{|} have to be escaped.

The included file @file{function.lang} defines the regular expression to
match a function definition or invocation:

@include function.lang.texinfo

@noindent
that shows an example of forward lookahead assert for the opening
parenthesis (see @ref{Notes on regular expressions}).  As noted in
@ref{File inclusion}, it is crucial that this file is included after the
keyword definition.

Finally, @file{c.lang} includes the file
@file{clike_vardeclaration.lang}:

@include clike_vardeclaration.lang.texinfo

This definition, using subexpressions with names (see @ref{Explicit
subexpressions with names}), tries@footnote{This was not tested
extensively and might not catch all the correct situations.} to match
user types (e.g., struct names) in function parameter and variable
declarations.  It basically tries to match a type identifier, then a
possible template specification@footnote{OK, there are no templates in
C, and they are only in C++, but we think it should no harm when
highlighting C files.} and then we have a complete lookahead assert
(@ref{Notes on regular expressions}) that tries to match the variable
identifier, possibly with @code{&} and @code{*} reference and pointer
specification, followed by an assignment @code{=} or a @code{;}, more
generally a @code{[:punct:]} or @code{[]} (for array specifications).
This should catch the user types in the correct contexts, as in the
following (where we intentionally highlighted @code{usertype} in
italics):

@include test_vardecl.cc.texinfo

Note that since for the third group we use a lookahead assert, what is
matched is not actually formatted but it is put back in the input stream
so that it can be formatted using other rules (e.g., @code{symbol} for
@code{*} and @code{=}).

Since, at least syntactically, C++ is an extension of C, the language
definition for C++, included in the file @file{cpp.lang}, relies on
@file{c.lang}@footnote{Before version 2.9, there was only
@file{cpp.lang} which was used both for C and C++; however, this way, if
you had a C program where you were using a C++ keyword as a variable
name---which of course is correct in C---that variable was actually
highlighted as a keyword and this was not correct.}:

@include cpp.lang.texinfo

In particular, it extends the set of keywords.  Moreover, note that we
use subexpressions with names (see @ref{Explicit subexpressions with
names}) for highlighting the class (or struct) name (when preceded by
@code{class}, @code{struct} or @code{typename}, highlighted as a
keyword).  A similar rule was also present in @file{c.lang}, but it
concerned only @code{struct}.

Now that we wrote the language definition for C/C++, writing the one
for C# is straightforward, since we only need to add the keyword
@code{using} as a preprocessor element, and redefine  (or better,
``substitute'', @ref{Redefinitions and Substitutions}) the keywords
and types:

@include csharp.lang.texinfo

@node Highlighting Diff files, Pseudo semantic analysis, Highlighting C/C++ and C#, Tutorials on Language Definitions
@subsection Highlighting Diff files

Now we want to highlight files that are generated by @code{diff}
(typically used to create patches).  This program can generate outputs
in three different formats (at least at best of my knowledge).

With the option @code{-u|--unified} the differences among files
are shown in the same context, for instance (the examples of the
diff files shown here are manually modified so that they can
fit in the page width):

@include example-u.diff.texinfo

With the option @code{-c--context} the differences are shown into
two different parts:

@include example-c.diff.texinfo

Without options it generates only the essential difference
information without any addition context lines:

@include example-n.diff.texinfo

Summarizing, we would like to be able to handle all these three
different syntaxes; note that the first format and the second format
have something conflicting: the first one uses the @code{---} to
indicate the new version of a file while the second format uses it to
indicate the old version of a file.  Since we want to highlight
differently the old parts and the new parts (this is not visible in the
Texinfo highlighting due to the lack of enhanced formatting features,
but it is visible for instance in HTML output where we use two
different colors), this behavior adds some difficulties.  Of course, we
could define three different language definitions, one for each diff
output format.  However, we prefer to handle them all in the same
file!

This is the language definition for diff files:

@include diff.lang.texinfo

Since we can safely assume that when we process a diff file it contains
only information created with the same diff command line switch, we
define three different states that correspond to the three diff output
formats.  Note that these states are entered with a simple definition;
as noted in @ref{State/Environment Definitions}, this means that no
automatic exit means are provided, and since no explicit exit condition
is specified, this means that once one of this state is entered it will
never be exited.  This is consistent with our goal.  Of course, the
expression that makes us enter a state must be defined correctly, and in
particular we first search for an initial @code{---} sequence since this
is used as the first difference specification by the @code{-u|--unified}
option, so this is a distinguishing feature to be used to
infer which diff format file we are processing.

Another interesting thing, is that we use the forward lookahead assert
for the opening parenthesis (see @ref{Notes on regular expressions}),
since we only want to see which file format we are processing.  Once we
entered the right state we can define the regular expressions for the
elements of the specific diff file format.

For the files created with the option @code{-c|--context} we define two
inner environments, one for the new file part and one for the old file
part (these are delimited by a @code{---} or @code{***} and line number
information).  Note that these are environments, so anything that is
not matched by any expression is formatted according to the style of the
element that defines the environment.  Thus, we provide an expression
for text that must be formatted as normal.  For diff files this
corresponds to a line that start with a space or with @code{diff} (take
a look at the examples above).  In particular the latter case can take
place only during the new file part.  In both environments we must
define the exit conditions.  In both cases these correspond to the
beginning of the complementary part; also in this case we use forward
lookahead assertions, since we use it only to exit the environment.  The
outer definitions for @code{oldfile} and @code{newfile} are used to
match the lines with source file information information.

The third state, corresponding to the normal diff output format, should
be straightforward by now.

@node Pseudo semantic analysis,  , Highlighting Diff files, Tutorials on Language Definitions
@subsection Pseudo semantic analysis

Source-highlight, by means of regular expressions can only perform
lexical analysis of the input source.  In particular, it is based on the
assumption that the input source is syntactically correct with respect
to the input language.  However, by using the language definition syntax
and by writing the right regular expression it is  possible to simulate
some sort of semantic analysis of the input source.

For instance, consider the following C (or C++) source file:

@include test_extreme_comment.cpp.texinfo

@noindent
It is easy to verify that the code between @code{#if 0} and 
@code{#else} will be never executed (indeed it will not even
be compiled).  Thus, we might want to format it as a comment.

We then write another language definition file, based on the file
@file{cpp.lang}:

@include extreme_comment.lang.texinfo

@noindent
We intentionally included an error in this first version:
we used the @code{start} element to start the environment,
but such element has the scope of a single line, thus,
it does not have the desired behavior:

@include test_extreme_comment_wrong.texinfo

A better solution is the following one:

@include extreme_comment2.lang.texinfo

@noindent
here we enter the @code{comment} environment by not using a delimited
element, but simply the regular expression to match @code{#ifdef 0}.
Then we exit the environment either when we match an @code{#else} or a
@code{#endif}.  This seems to work:

@include test_extreme_comment_1.texinfo

However, it does not work if we consider nested @code{#if...#else}; for
instance consider the following code, formatted with the previous
language definition:

@include test_extreme_comment_wrong2.texinfo

@noindent
The problem is that the previous language definition does not consider
nested @code{#if} and thus, the first time it matches a @code{#else} or
an @code{#endif} it exits the @code{comment} environment.

We must then take into account possible nested occurrences.  This can be
done by using a delimited element with the @code{nested} option
(@ref{Delimited definitions}):

@include extreme_comment3.lang.texinfo

@noindent
This time the right block of code is correctly formatted as a comment:

@include test_extreme_comment_2.texinfo

Note that it is crucial to exit the environment even when we match an
@code{#else} (not only an @code{#endif}, since, this way, we can match
again another @code{#ifdef 0}; consider, for instance, the following
code:

@include test_extreme_comment_3.texinfo

@node  Output Language Definitions, Generating References, Language Definitions, Top
@chapter Output Language Definitions
@cindex output language definition

Since version 2.1 source-highlight uses a specific syntax to specify
output formats (e.g., how to format in HTML, @LaTeX{}, etc.).  Before
version 2.1, in order to add a new output format, many C++ classes had
to be written.  This had the drawback that a new output format could not
be added ``dynamically'': you had to recompile the whole
source-highlight program.

Instead, now, an output format is specified in a file, loaded
dynamically, through a (hopefully) simple syntax.  Then, these
definitions are used internally to create, on-the-fly,  text formatters.

Here, we see such syntax in details, by relying on many examples.  This
allows a user to easily modify an existing output format definition and
create a new one.  These files have, typically, extension
@file{.outlang}.

Each definition basically associates a text style (such as, e.g., bold,
italics, colors, etc.)  to the representation of that style into the
output format (such as, e.g., @code{<b>$text</b>} in HTML).  The
representation is given in @code{"} and you can use the classic escape
character @code{\} to use the @code{"} inside the definition.  If you
want to specify the ASCII code for a character you can do so by
specifying the numeric code in hexadecimal notation preceded by
@code{\x}, for an example, see @ref{Style template}.

If no definition is given for a specific style, e.g., bold, then when
that style is requested during formatting, the text will be formatted as
it is, i.e., the style without the definition is simply ignored.

Comments can be given by using @code{#}; the rest of the line is
considered as a comment.

Files can be included in the same way as for language definitions,
@ref{File inclusion}.

In any case, if a definition for a style is given more than once, the
last definition replaces all the others.

@menu
* File extension::              Specify the output file extension
* Text styles::                 Bold, Italics, Underline, etc.
* Colors::                      Style and definitions for colors
* Anchors and References::      
* One style::                   
* Style template::              
* Line prefix::                 
* String translation::          
* Document template::           
* Generating HTML output::      
@end menu

@node File extension, Text styles, Output Language Definitions, Output Language Definitions
@section File extension

With the line:

@example
extension "<file extension>"
@end example

@noindent
you define the default file extension (without the @code{.}) used to
generate files formatted according to this output format.  This is used
when no output file name is specified; if the file extension is not
included in the @code{.outlang} is not defined, and no output file name
is specified, an error will occur.

For instance, this is used in @file{html_common.outlang}:

@example
extension "html"
@end example

@node Text styles, Colors, File extension, Output Language Definitions
@section Text styles
@cindex bold
@cindex italics
@cindex underline
@cindex fixed
@cindex notfixed

These are the text styles that one can define:

@example
bold
italics
underline
notfixed
fixed
@end example

@noindent
These, of course, correspond to the ones used to specify the output
format style, @ref{Output format style}.

These definitions, for instance, are from the HTML format definition:

@example
bold "<b>$text</b>"
italics "<i>$text</i>"
underline "<u>$text</u>"
@end example

@cindex $text
@noindent
Inside a definition you use the special variable @code{$text} to specify
where the actual text to be formatted has to be inserted.  For instance,
the definition of @code{bold} above says that if you need to format the
keyword @code{class} in bold in HTML, the following text will be
generated: @code{<b>class</b>}.  This variable is used also when mixing
more than one styles recursively, in particular if you want to format in
bold and italics (i.e, first bold and then italics, or, in other words,
the sequence @code{i, b} is used in the the output format style file, see
@ref{Output format style}), then first the text
@code{class} is substituted for @code{$text} into @code{<b>$text</b>}
and then the text @code{<b>class</b>} will be substituted for
@code{$text} into @code{<i>$text</i>}, thus obtaining
@code{<i><b>class</b></i>}.

@node  Colors, Anchors and References, Text styles, Output Language Definitions
@section Colors
@cindex colors
@cindex $style

The definition for using colors during formatting requires
the definition for the @code{color} style

@example
color "..."
@end example

@cindex background color
@noindent
and for the @code{bgcolor} style@footnote{Since version 2.6.}:

@example
bgcolor "..."
@end example

This definition concerns only the background color for a specific
highlighted element, i.e., the color specified in the style file with
the prefix @code{bg:} (see @ref{Output format style}) or the property
@code{background-color} specified in a CSS file passed to
@code{--style-css-file} (see @ref{Output format style using CSS}).
Thus it should not be confused with the background color of the entire
output (i.e., the one specified using @code{bgcolor} in a style file or
the property @code{background-color} of the @code{body} selector in a
CSS).  The background color for the entire document is explained in
@ref{Document template}.

Note that the background color might not be available for all output
formats.   For instance, for HTML we only have:

@example
color "<font color=\"$style\">$text</font>"
@end example

@noindent
while for XHTML we have:

@example
color "<span style=\"color: $style\">$text</span>"
bgcolor "<span style=\"background-color: $style\">$text</span>"
@end example

Apart from the variable @code{$text} that we already saw, we
have also the variable @code{$style}, that will be replaced
with the actual color.

Source-highlight recognizes a number of color constants,
see @ref{Output format style}.

You then must associate a color constant to the color definition in the
output format, through the @code{colormap} definition:

@example
colormap
"color constant" "color representation"
"color constant" "color representation"
...
default "default color representation"
end
@end example

The @code{default} row (note the absence of @code{"}) defines the
color to be used in case a color constant is used during formatting, but
it is not defined in the output format.

For instance, for HTML we have:

@example
colormap
"green" "#33CC00"
"red" "#FF0000"
"darkred" "#990000"
"blue" "#0000FF"
"brown" "#9A1900"
"pink" "#CC33CC"
"yellow" "#FFCC00"
"cyan" "#66FFFF"
"purple" "#993399"
"orange" "#FF6600"
"brightorange" "#FF9900"
"brightgreen" "#33FF33"
"darkgreen" "#009900"
"black" "#000000"
"teal" "#008080"
"gray" "#808080"
"darkblue" "#000080"
default "#000000"
end
@end example

If your output format does not handle colors you can simply avoid the
definitions of @code{color} and @code{colormap} and Source-highlight
will simply ignore colors.

The color is applied after applying the other styles,
e.g., bold, italics, etc.

Thus, by continuing the example of the previous section, suppose you defined
the following output style for keywords:

@example
keyword blue i, b;
@end example

@noindent
then the @code{class} text will be replaced to @code{$text} variable and
the value @code{#0000FF} to @code{$style} inside the color definition
@code{<font color="$style">$text</font>} obtaining @code{<font
color="#0000FF">class</font>} which will then be replaced to
@code{$text} in @code{<b>$text</b>} and so on for italics, finally
obtaining 

@code{<i><b><font color="#0000FF">class</font></b></i>}.

@node   Anchors and References, One style, Colors, Output Language Definitions
@section Anchors and References
@cindex $linenum

When using the command line option @code{--line-number-ref}
(@ref{Invoking source-highlight}) an anchor is generated in the output
file for each line numbering.  The style of the anchor is defined by the
definition @code{anchor}.  If this is not defined, the option
@code{--line-number-ref} has no effect.  The @code{$linenum} variable will
be replaced with the line number, and the @code{$text} variable
with the actual text.

For instance, for HTML we have

@example
anchor "<a name=\"$linenum\">$text</a>"
@end example

@cindex $infile
@cindex $infilename
@cindex $outfile
Since version 2.2 source-highlight can also generate references to
several elements (e.g., variables, class definitions, etc.),
@ref{Generating References}.  Also in this case the definition
@code{anchor} is used; furthermore, the definition of @code{reference}
is required.   In the definition of @code{anchor} and @code{reference},
apart from the variable @code{$linenum}, we also have the variables
@code{$infile} (the name of the original input file) and
@code{$infilename} (the name of the original input file without the
path) and in the definition of @code{reference} we also have the
variable @code{$outfile} (the name of the file where the anchor is).
One can decide how to define an anchor and a reference by using these
two variables.  For instance, for HTML we have

@example
reference "<a href=\"$outfile#$linenum\">$text</a>"
@end example

@noindent
Note, that in this case we use the @code{$outfile} since we actually
generate a link to another (or possibly the same) output file.

On the contrary, for @LaTeX{}, since we do not generate a ``clickable''
reference, we refer to the original input file (we use both
@code{$infilename} and @code{$linenum} in both definitions of @code{anchor}
and @code{reference}):

@example
anchor "\label@{$infilename:$linenum@}$text"
reference "@{\hfill $text $\rightarrow$ $infile:$linenum, \
           page~\pageref@{$infilename:$linenum@}@}"
@end example

@noindent
In particular, we use @code{$infilename} for generating the
@code{\label} and not @code{$infile} because the path symbol would
``disturb'' @LaTeX{} (while we use the complete file path in the textual
information of the reference).

This will generate a right aligned reference.  Note that it is assumed
that when generating references in @LaTeX{} one uses
@code{--gen-references=postline} or @code{--gen-references=postdoc} and
not @code{--gen-references=inline} (@ref{Generating References}), since
it makes no sense to generate an inline reference (or at least I would
not know how to generate a nice looking one :-).

Furthermore, for Texinfo:

@example
anchor "@@anchor@{$infilename:$linenum@}$text"
reference "@@flushright
@@xref@{$infilename:$linenum,$text,$text $infile:$linenum@}.
@@end flushright"
@end example

@noindent
Note that using both @code{$infilename} (and not @code{$infile} for
the same reasons) and @code{$linenum} also in the definition of
@code{anchor} somehow ensures that there are no duplicate anchors; this
is done for @LaTeX{} and Texinfo but not for HTML because it is assumed
that the generated @file{.tex} and @file{.texinfo} file is included
directly in a master file, as it is done in this manual (while, for
instance, it is assumed that a separate HTML file is generated for each
source and kept separate).  If this is not your case you can change the
definitions of @code{anchor} and @code{reference} as you see fit.  Some
examples of outputs with references in Texinfo are shown in
@ref{Examples}.

@cindex inline_reference
@cindex postline_reference
@cindex postdoc_reference
Indeed, one can use three more definitions for @code{reference} that
corresponds to the three arguments that can be passed to
@code{--gen-references} command line option (@ref{Generating
References}): @code{inline_reference}, @code{postline_reference} and
@code{postdoc_reference}.  If one of this not defined, then the same
definition of @code{reference} is used.  Having the possibility of
specifying different definitions is useful for instance in the case of
HTML: the same style for an inline reference is pretty ugly when used
also for a postline or postdoc reference:

@smallexample
postline_reference "<a href=\"$outfile#$linenum\">$text -> $infile:$linenum</a>"
postdoc_reference "<a href=\"$outfile#$linenum\">$text -> $infile:$linenum</a>"
reference "<a href=\"$outfile#$linenum\">$text</a>"
@end smallexample

@node One style, Style template, Anchors and References, Output Language Definitions
@section One style
@cindex one style

If the output format you are defining does not have a specific style
for bold, italics, ... and for colors you can simply use the definition
@code{onestyle}, where you can use both @code{$style} and @code{$text}.
This will be used for any style (indeed any other definition such as
bold, italics, color will be ignored).  Indeed, in this case, it is
assumed that the style of each source element is defined in a file with
its own syntax, i.e., not with a syntax defined by Source-highlight.
(This is the case, for instance, of HTML using CSS style sheets.)
Moreover, since the output format style is not used, during formatting
the variable @code{$style} will be replaced with the name of the element
to highlight (e.g., @code{keyword}, @code{comment}, etc.).

For instance, for HTML CSS, we simply have:

@example
onestyle "<span class=\"$style\">$text</span>"
@end example

@noindent
In fact, HTML CSS relies on style definitions provided in a separate
file (the @file{.css} file indeed).  Thus, when formatting a
@code{keyword}, e.g., @code{abstract}, we will obtain:

@example
<span class="keyword">abstract</span>
@end example

@noindent
Of course, the style for @code{keyword} must be defined in the
@file{.css} file.

@node Style template, Line prefix, One style, Output Language Definitions
@section Style template
@cindex style template
@cindex style separator

Some output formats are based on a unique template that where the other
styles are composed; during composition the styles can be separated with
a specific separator:

@example
styletemplate "..."
styleseparator "..."
@end example

This is used, for instance, for the ANSI color escape sequence
output format (@file{esc.outlang}):

@example
styletemplate "\x1b[$stylem$text\x1b[m"
styleseparator ";"

bold "01$style"
underline "04$style"
italics "$style"
color "$style"
@end example

@noindent
Note that, since more than one style can be mixed into the style
template, @code{bold}, @code{underline}, ... explicitly use the variable
@code{$style}.

@node Line prefix, String translation, Style template, Output Language Definitions
@section Line prefix

This feature allows you to generate a string as the prefix of each
generated line that corresponds to an input line (i.e., this prefix is
not generated for other generated output elements, e.g., the lines in
the header, footer, etc.).

We use this feature in the @LaTeX{} output (@ref{LaTeX output}):

@example
lineprefix "\mbox@{@}"
@end example

@noindent
This way each line in the @LaTeX{} output is prefixed with
@code{\mbox@{@}}@footnote{This is a sort of trick to insert spaces at
the beginning of a line without using a tabular environment; without the
leading @code{\mbox@{@}} these spaces would be ignored.  This is the
only way I found to achieve this, if you have suggestions, please let me
know!}.

Another interesting example that uses @code{lineprefix} is the javadoc
output, see @ref{Generating HTML output}.

@node String translation, Document template, Line prefix, Output Language Definitions
@section String translation

Some character sequences that are in the source file may have a
special meaning in an output format, so they need some preprocessing
(e.g., escaping them).  You can specify the translation table with:

@example
translations
"original sequence" "transformed sequence"
'regex' "transformed sequence"
...
end
@end example

@noindent
The difference between @code{"original sequence"} and
@code{'regex'}@footnote{Since version 2.4.} is that with the former
you specify a character sequence that will be matched literally, apart
from special characters such as @code{\} (which, if needed to be
inserted, must be escaped), @code{\n} (new line) and @code{\t} (tab
character).  Instead, with the latter, you can specify a regular
expression (this is basically the same difference between @code{"} and
@code{'} in language definitions, see @ref{Simple definitions}).

For instance, for HTML, we have the following translation table:

@example
translations
"&" "&amp;"
"<" "&lt;"
">" "&gt;"
end
@end example

For @LaTeX{}, the translation table is a little bit bigger; here we
show only a little part, that shows how to escape special characters
(such as @code{\}), to translate a new line character and tab
character:

@example
translations
"<" "$<$"
">" "$>$"
"&" "\\&"
"\\" "\\textbackslash@{@}"
"\n" " \\\\\n"
" " "\\ "
"\t" "\\ \\ \\ \\ \\ \\ \\ \\ "
end
@end example

@noindent
Note that, since a new character must be translated in @LaTeX{} with
@code{\\}, we have to escape two @code{\} (i.e., @code{\\\\}) and then
we want to actually insert a new line in the output file @code{\n}.

For HTML with not fixed font by default, @file{html_notfixed.outlang}
(see @ref{HTML and XHTML output}), we need two translate two space sequence
(i.e., two adjacent spaces, since in HTML more adjacent spaces are
rendered as only one space@footnote{Unless they are inside a
@code{<tt>...</tt>}.}, while we want them as they are), and we also
need to translate a space starting a new line in the source (thus we
use the regular expression @code{^ }, enclosed in @code{'}); thus we
have:

@example
translations
"\n" "<br>\n"
"  " "&nbsp; "
'^ ' "&nbsp;" # a space at the beginning of a line
"\t" "&nbsp; &nbsp; &nbsp; &nbsp; "
end
@end example

@node Document template, Generating HTML output, String translation, Output Language Definitions
@section Document template

You can define the beginning and the end of an output file, with

@cindex doctemplate
@example
doctemplate
"...beginning..."
"...end..."
end
@end example

@cindex nodoctemplate
@example
nodoctemplate
"...beginning..."
"...end..."
end
@end example

The first one is used when the @code{--doc} command line option is
specified, while the second one is used in the other case@footnote{Up
to version 2.9, there was only @code{doctemplate} and for @code{--doc}
there was a separate @code{.outlang} file; I think the present solution
is better and reduces the number of files.}.

For instance, for HTML we have

@example
nodoctemplate
"<!-- Generator: $additional -->
$header<pre><tt>"
"</tt></pre>$footer
"
end
@end example

@noindent
Note that in the end part there is an explicit new line.

In the definition of the @code{doctemplate} and @code{nodoctemplate} the
following variables can be used and will be replaced during the output
generation:

@table @code
@item $title
the value of the title for the output file (e.g., the one passed with
the @code{--title} command line option;
@item $header
the contents of the file specified with the command line option
@code{--header};
@item $footer
the contents of the file specified with the command line option
@code{--footer};
@item $css
the value passed with the command line option @code{--css};
@item $additional
other additional information.  Source-highlight replaces this with its
name and its version.
@item $docbgcolor@footnote{Since version 2.6.}
the background color for the output document.  Source-highlight replaces
this with the value specified in the @code{bgcolor} of the @file{.style}
file (see @ref{Output format style}) or in the @code{body} selector of
the CSS file passed with @code{--style-css-file} (see @ref{Output format
style using CSS}).
@end table

For instance, for an HTML document with css, (file
@file{htmlcss.outlang}) we have:

@example
doctemplate
"<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.0//EN\"
    \"http://www.w3.org/TR/REC-html40/strict.dtd\">
<html>
<head>
<meta http-equiv=\"Content-Type\"
content=\"text/html; charset=iso-8859-1\">
<meta name=\"GENERATOR\" content=\"$additional\">
<title>$title</title>
<link rel=\"stylesheet\" href=\"$css\" type=\"text/css\">
</head>
<body>
$header<pre><tt>"
"</tt></pre>
$footer</body>
</html>
"
end
@end example

For an HTML document with header and footer, (file
@file{html.outlang}) we have (note the use of @code{$docbgcolor}):

@example
doctemplate
"<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML//EN\">
<html>
<head>
<meta http-equiv=\"Content-Type\" content=\"text/html; charset=iso-8859-1\">
<meta name=\"GENERATOR\" content=\"$additional\">
<title>$title</title>
</head>
<body bgcolor=\"$docbgcolor\">
$header<pre><tt>"
"</tt></pre>
$footer</body>
</html>
"
end
@end example

And for an HTML table output (file @file{htmltable.outlang}):

@example
doctemplate
"<table  BGCOLOR=\"$docbgcolor\" NOSAVE >
<tr NOSAVE>
<td NOSAVE>
<pre><tt>"
"</tt></pre>
</td>
</tr>
</table>
"
end
@end example 

@node Generating HTML output,  , Document template, Output Language Definitions
@section Generating HTML output

As a complete example we show the file @file{html_common.outlang} which
contains the common definitions for the various HTML output formats
(@file{html.outlang}, @file{htmltable.outlang}, etc.):

@include html_common.outlang.texinfo

Moreover, this file is also used for generating javadoc output:

@include javadoc.outlang.texinfo

The javadoc output format is useful to format code snippets that have to
be included inside a javadoc comment of another Java
file@footnote{Although I haven't tested it, I think this will work also
for Doxygen comments.}.  Apart from being formatted nicely in the
generated HTML documentation, this also releases the programmer from
escaping specific characters in the code snippet (i.e., @code{&},
@code{<} and @code{>}).  Note also that it also avoids the sequence
@code{*/} to be interpreted as the closing of the (javadoc) comment.
For instance, if you write this code:

@example
/**
 * This is an example of usage
 *
 * <pre><tt>
 * System.out.println("*/");
 * </tt></pre>
 */
@end example

@noindent
The resulting Java code contains a syntax error.  If you use
source-highlight to format the code to insert in a javadoc comment you
will avoid these problems.

An example of a javadoc generated HTML page containing a code snippet
formatted with source-highlight can be found in the file
@file{SimpleClass-doc.html} in the documentation directory.

@node Generating References, Examples, Output Language Definitions, Top
@chapter Generating References
@cindex reference
@cindex anchor
@cindex ctags

Since version 2.2 Source-highlight also produces references to fields,
variables, etc.  In order to do this it relies on the program
@emph{Exuberant Ctags}, by Darren Hiebert, available at
@url{http://ctags.sourceforge.net}.  Thus, you must install this program
if you want Source-highlight to provide this feature.

The @code{ctags} program generates an index (or ``tag'') file for a
variety of language objects found in file(s).  This allows these items
to be quickly and easily located by a text editor or other utility (as
in this case for Source-highlight).  A ``tag'' signifies a language
object for which an index entry is available (or, alternatively, the
index entry created for that object)@footnote{This description is taken
from the ctags man page}.

This means that Source-highlight is able to generate references for a
specific source language if and only if @code{ctags} handles such
language.  We refer to the command line options of @code{ctags}:
@code{--list-maps} and @code{--list-languages} to find out the
associations of file extensions and supported languages.

Reference generation is enable by using the command line option
@code{--gen-references} (@ref{Invoking source-highlight}).  This option
takes an argument that rules how references will be generated:

@table @code
@item inline
a reference pointer will be generated exactly in the same place of the
specific element.  This is useful in output formats that naturally
supports links, such as HTML, while it is useless for output formats
that do not support inline links, such as @LaTeX{}.
@item postline
if a line of the input source contains elements for which we found
references, the list of references will be generated right after the
line (see the examples, @ref{Examples}).
@item postdoc
All the references will be generated after the whole input file has been
generated.
@end table

There is an exception: when an element has more than one reference
(because a variable is defined in many sources or because a method is
overloaded) then if @code{inline} is specified, the generation switches
to @code{postline} for that occurrence.

When @code{--gen-references} is specified, Source-highlight first
invokes @code{ctags}.  The use can customize this call by using the
command line option @code{--ctags} (@ref{Invoking source-highlight}).
In particular, if one does not want @code{ctags} to be invoked by
Source-highlight (e.g., because the tags file has already been
generated) then @code{--ctags} must be passed an empty string,
@code{""}.  In this case or when the specified @code{ctags} command line
generates an alternative output tag file (the default generated file is
@file{tags}), one must specify the exact tag file with the command line
option @code{--ctags-file}.

Once the tag file is generated, Source-highlight relies on the library
@file{readtags} provided by the @code{ctags} distribution, and included
in the Source-highlight sources.

Note that if a program element is formatted according to a style that
has the option @code{noref} (see @ref{Output format style}) then this
element is not considered a tag, and no reference is generated.  This is
the case, for instance, for a @code{comment} element: each string that
is generated with the @code{comment} style, since this is declared with
the option @code{noref}, it is not considered a tag (see @ref{Examples}).

@node  Examples, Problems, Generating References, Top
@chapter Examples

Here we provide some examples of sources formatted with
Source-highlight using the @code{-f texinfo}
command line option.  Please keep in mind that the highlighting
will not be visible in the Info file, but only in the
printed manual and in the HTML output (well, at least line
numbers are visible everywhere :-).

@menu
* Simple example::              
* References::                  
* Line ranges::                 
* Line ranges (with context)::  
* Regex ranges::                
@end menu

@node Simple example, References, Examples, Examples
@section Simple example

The first example is produced by using the command:

@example
source-highlight -f texinfo -i test.java -o test.java.texinfo -n
@end example

and here's the result

@include test.java.texinfo

@node References, Line ranges, Simple example, Examples
@section References

This example shows the use of @code{--gen-references}
functionality.  In particular, the following output is generated with
the command:

@example
source-highlight -f texinfo -i test.h -o test_ref.h.texinfo -n \
     --gen-references=postline
@end example

and here's the result (note how the comment line containing the string
@code{mysum} does not contain references, since it is a @code{comment}
element, and this element has the option @code{noref} in the
@file{texinfo.style}, see @ref{Output format style}.  The same holds for
the @code{_TEXTGEN_H} comment in the last comment line).

@include test_ref.h.texinfo

@node Line ranges, Line ranges (with context), References, Examples
@section Line ranges

@cindex line ranges
This is an example that uses @code{--line-range} command line
option on the input file shown in @xref{Simple example}:

@example
source-highlight -f texinfo -i test.java -n \
     --line-range="12-18","29-34"
@end example

This generates the following output

@include test_lineranges1.java.texinfo

Note that, although the specified line ranges span comment environments,
the highlighting is respected: the starting of the comment is not
printed, but the remaining parts of the comment are correctly
highlighted as comment.

@node Line ranges (with context), Regex ranges, Line ranges, Examples
@section Line ranges (with context)

@cindex line ranges
@cindex range context
@cindex range separator
This is an example that uses the command line option @code{--line-range}
together with the @code{--range-context} and @code{--range-separator}:

@example
source-highlight -f texinfo -i test.java -n \
     --line-range="12-18","29-34" \
     --range-context=2 \
     --range-separator="@{... not in range ...@}"
@end example

This generates the following output

@include test_lineranges2.java.texinfo

Note the two additional 2 lines before and after the ranges (compare it
with the output in @ref{Line ranges}).  Note that the (elements of the)
context lines are not highlighted.  Moreover, the range separator line
@code{"@{... not in range ...@}"} is printed between ranges (the
separator string is preformatted automatically, so, e.g., you don't have
to escape special output characters, such as the @{ @} in texinfo
output).

@node Regex ranges,  , Line ranges (with context), Examples
@section Regex ranges

@cindex regex ranges
Ranges can be expressed also using regular expressions, with the command
line option @code{--regex-range}.  In this case the beginning of the
range will be detected by a line containing (in any point) a string
matching the specified regular expression; the end will be detected by a
line containing a string matching the same regular expression that
started the range.  This feature is very useful when we want to document
some code (e.g., in this very manual) by showing only specific parts,
that are delimited in a ad-hoc way in the source code (e.g., with
specific comment patterns).

For instance, the following output was produced, starting from the
source file shown in @xref{Simple example}, by specifying:

@example
--regex-range="/// [[:alpha:]]+"
@end example

@noindent
Note that the lines containing @code{/// class}, which determine the
range, are not shown in the output:

@include test_regexrange1.java.texinfo

Furthermore, the line numbers are consistent with the lines of the
original file.

If we want to output only what is included between @code{/* main */}, we
specify (note that we must escape the special regular expression
character @code{*}):

@example
--regex-range="/\* main \*/"
@end example

@noindent
and we get:

@include test_regexrange_main.java.texinfo

If we want to show only the methods, which in the source file
are delimited by comment lines containing the method's name, we can specify:

@example
--regex-range="/\* [[:alpha:]]+ \*/"
@end example

@include test_regexrange_methods.java.texinfo

In this case, we might have also specified:

@example
--regex-range="/\* main \*/","/\* mymethod \*/"
@end example

@noindent
since @code{--regex-range} accepts multiple regular expressions.

IMPORTANT: the order of regular expression specification is crucial,
since they are tested in the same order they are specified at the
command line.

@node Problems, Mailing Lists, Examples, Top
@chapter Reporting Bugs
@cindex bugs
@cindex problems

If you find a bug in @command{source-highlight}, please send electronic
mail to 

@code{bug-source-highlight at gnu dot org}

Include the version
number, which you can find by running @w{@samp{source-highlight
--version}}.  Also include in your message the output that the program
produced and the output you expected.@refill

Even better, please file a bug report at Savannah site:

@uref{https://savannah.gnu.org/bugs/?group=src-highlite}

If you have other questions, comments or suggestions about
@command{source-highlight}, contact the author via electronic mail
(find the address at @value{myhomepage}).  The author will try to help
you out, although he may not have time to fix your problems.

@node Mailing Lists, Concept Index, Problems, Top
@chapter Mailing Lists
@cindex mailing list

The following mailing lists are available: 

@code{help-source-highlight at gnu dot org}

for generic discussions about the program and for asking for help about
it (open mailing list),
@uref{http://mail.gnu.org/mailman/listinfo/help-source-highlight}
 
@code{info-source-highlight at gnu dot org}

for receiving information about new releases and features (read-only
mailing list),
@uref{http://mail.gnu.org/mailman/listinfo/info-source-highlight}.
 
If you want to subscribe to a mailing list just go to the URL and follow
the instructions, or send me an e-mail and I'll subscribe you.

I'll describe new features in new releases also in my blog, at
this URL:

@uref{http://tronprog.blogspot.com/search/label/source-highlight}

@node Concept Index,  , Mailing Lists, Top
@unnumbered Concept Index

@cindex tail recursion
@printindex cp

@shortcontents


@bye