[comment {-*- tcl -*- doctools manpage}]
[manpage_begin docidx_fmt n 1.0]
[copyright {2003-2004 Andreas Kupries <andreas_kupries@users.sourceforge.net>}]
[moddesc   {Documentation tools}]
[titledesc {Specification of a simple Tcl Markup Language for Keyword Indices}]
[description]
[para]

This document specifies version 1 of a text format for keyword
indices. The name of this format is [term docidx] and it provides all
the necessary commands to write a keyword index for a group of
documents.

It is intended to complement both the [term doctools] format for
writing documentation and the [term doctoc] format for writing tables
of contents.

The formal specifications for these two formats can be found in the
companion documents [term doctools_fmt] and [term doctoc_fmt].

A third companion document describes the package

[package doctools::idx], which provides commands for the processing of
text in [term docidx] format.

[para]

Like for the formats [term doctools] and [term doctoc] a generic
framework for the conversion of [term docidx] to any number of
different output formats is provided.

This framework is provided by the package [package doctools::idx].

[para]

Anyone who wishes to write an index formatting engine which plugs into
this framework has to read the document [term docidx_api]. This is the
formal specification of the API between the framework and its engines.


[section {FORMAT SPECIFICATION}]


[subsection OVERVIEW]

While [term doctools] is similar to LaTeX [term docidx] is only
superficially so. Input written in this format consists of a series of
markup commands, which may be separated by whitespace. Other text is
not allowed. The best comparison would be to imagine a LaTeX document
with all regular text removed.

[para]

The format used to mark commands is different from LaTeX however. All
text between matching pairs of [lb] and [rb] is a command, possibly
with arguments. Note that both brackets have to be on the same line
for a command to be recognized.

[subsection {GRAMMAR}]

The overall syntax of a keyword index is best captured in a formal
context-free grammar. Our notation for the grammar is EBNF. Strings
will stand for markup commands, however their arguments (if they have
any) are not part of the grammar. Our grammar contains lexical
elements as well.

[para]

First we specify the whitespace at the lexical level, which also
includes comments.

[para]
[example {
    COMMENT  ::= "comment"
    WHITE    ::= { '\n' | '\t' | ' ' | '\r' | COMMENT }
}]
[para]

Then we define rules for all the keywords. Here we introduce our
knowledge that all commands can be separated by whitespace, and also
that the inclusion of other files may happen essentially everywhere,
like the definition of document variables. The content of any included
file has to fit into the including file according to the location in
the grammar the inclusion is at.

[para]
[example {
    BEGIN    ::= "index_begin" WHITE DEFUN
    END      ::= "index_end"   WHITE
    KEY      ::= "key"         WHITE DEFUN
    MANPAGE  ::= "manpage"     WHITE DEFUN
    URL      ::= "url"         WHITE DEFUN

    INCLUDE  ::= "include"     WHITE
    VSET     ::= "vset"        WHITE

    DEFUN    ::= { INCLUDE | VSET }
}]
[para]

At last we can specify the whole index.

[para]
[example {
    INDEX    ::= DEFUN BEGIN CONTENT END

    CONTENT  ::= KEYWORD { KEYWORD }
    KEYWORD  ::= KEY REF { REF }
    REF      ::= MANPAGE | URL
}]


[subsection COMMANDS]

Here we specify the commands used in the grammar. Some commands
specified here were not used in the grammar at all. The usage of these
commands is confined to the arguments of other commands.

[list_begin definitions]

[call [cmd comment] [arg text]]

This command declares that the argument [arg text] is a comment.


[call [cmd include] [arg filename]]

This command loads the contents of the file [arg filename] for
processing at its own place.


[call [cmd vset] [arg varname] [arg value] ]

This form of the command sets the document variable [arg varname] to
the specified [arg value]. It does not generate output. It is this
form the grammar is refering to.


[call [cmd vset] [arg varname]]

This form of the command returns the value associated with the
document variable [arg varname]. This form is not used by the grammar
and restricted to usage in the arguments of other commands.


[call [cmd lb]]

This command adds a left bracket to the output. Its usage is
restricted to the arguments of other commands.


[call [cmd rb]]

This command adds a right bracket to the output. Its usage is
restricted to the arguments of other commands.


[call [cmd index_begin] [arg text] [arg title]]

This is the command to start a keyword index.

The [arg text] argument provides a label for the whole group of
documents the index refers to. Often this is the name of the package
(or extension) the documents belong to.

The [arg title] argument provides the overall title text for the index.


[call [cmd index_end]]

This is the command to close a keyword index.


[call [cmd key] [arg text]]

This command adds the keyword [arg text] to the index.


[call [cmd manpage] [arg file] [arg text]]

This command adds an element to the index which refers to a
document.

The document is specified through the symbolic name [arg file]. The
[arg text] argument is used to label the reference.

[nl]

The symbolic names are used to preserve the convertibility of this
format to any output format. The actual name of the file will be
inserted by the chosen formatting engine when converting the
input. This will be based on a mapping from symbolic to actual names
given to the engine.


[call [cmd url] [arg url] [arg label]]

This is the second command to add an element to the index. To refer to
a document it is not using a symbolic name however, but a (possibly
format-specific) url describing the exact location of the document
indexed here.

[list_end]


[section NOTES]

It is possible to generate a keyword index in [term docidx] format
automatically from a collection of documents in [term doctools]
format. All is needed is a special doctools formatting engine which
extracts the relevant metadata and some glue code to convert this data
to [term docidx].

[para]

The application [cmd dtplite] is an example for this.


[section EXAMPLE]

The example is a keyword index for all manpages in the module 
[term base64].

[para]
[example {
[index_begin tcllib/base64 {De- & Encoding}]
    [key base64]
        [manpage base64]
    [key encoding]
        [manpage base64]
        [manpage uuencode]
        [manpage yencode]
    [key uuencode]
        [manpage uuencode]
    [key yEnc]
        [manpage yencode]
    [key ydecode]
        [manpage yencode]
    [key yencode]
        [manpage yencode]
[index_end]
}]


[see_also doctools::idx docidx_api doctools_fmt doctoc_fmt]
[keywords docidx index {keyword index} keywords documentation]
[keywords markup {generic markup} TMML HTML nroff LaTeX]
[manpage_end]