% Copyright 2008 Taco Hoekwater.
% You may freely use, modify and/or distribute this file.

\enablemode[realfonts]
\enablemode[newstyle]

\usemodule[map-se]

\def\TEXMF{{\sc texmf}}

%\setupwhitespace[big]
%\setupindenting[no]

\setuplayout[grid=yes]

\starttext

\startArticle[Title={Integrating the pool file},
              SubTitle={},
              Year=2007,
              Number=35,
              Page=9,
              Period=VOORJAAR,
              Author=Taco Hoekwater,
              Email=taco(a)elvenkind.com]

\startAbstract
This short article discusses the method that is used in \METAPOST\ and
\LUATEX\ to integrate the string pool file into the program.

This method allows the redistribution of a single updated executable
in place of both a program and a data file, and this makes
updating those programs easier on both the user and the developer (me).
\stopAbstract

\subject{How a pool file is created}

The readers who regularly update their (pdf)\TeX\ or \METAPOST\
executables will probably be familiar with the concept of pool files
already, but I will explain the mechanics in some detail.

Programs written in the WEB language normally do not contain the
strings inside the executable proper, but in a separate file, called
the `pool file'.

The most important reason for the existence of this file is that back
when Don Knuth was creating \TeX\ and \METAFONT, there was not yet a
standardized way to handle strings inside the Pascal language, so he
had to invent his own solution for printing messages and warnings.  

In order to illustrate what is in a pool file, I~will show you the
required steps. First, here is a bit of WEB source from \METAPOST:

\starttyping
...
if minx_val(h)>maxx_val(h) then 
  print("0 0 0 0")
else begin
  ps_pair_out(minx_val(h),miny_val(h));
  ps_pair_out(maxx_val(h),maxy_val(h));
  end;
print_nl("%%Creator: MetaPost ");
print(metapost_version);
print_nl("%%CreationDate: ");
\stoptyping

this excerpt is from one of the PostScript output routines. Here,
there are still recognizable strings that are used as function
arguments (as well as the symbolic value \type{metapost_version},
that is actually a macro resolving to a string).

The processor \type{tangle} converts this input into a proper Pascal
source file. While doing so, it resolves all of the many WEB macros
that are present in the code. \type{metapost_version} is one of those,
but also the constructs like \type{minx_val(h)} and
\type{maxx_val(h)}.  It also removes the underscores from function
names, because traditional Pascal compilers did not allow \type{_} to
appear in identifiers.

\blank

What we are focusing on now, is that it also collects all of the
double||quoted strings in the input. It puts all of the unique
multi||character strings into an internal array, and replaces the
actual string in its output with the index number it has given the string
inside that array. Of course, functions like \type{print()} are
written in such a way that they expect numbers as arguments instead 
of string values.

The Pascal output file looks like this:

\kern3pt

\starttyping
...
if mem[h+2].int>mem[h+4].int then print(1279)
else begin pspairout(mem[h+2].int,mem[h+3].int);
pspairout(mem[h+4].int,mem[h+5].int);end;
printnl(1281);print(256);printnl(1282);
\stoptyping

\kern3pt

As you can see, this file is clearly intended for a compiler only.
The complete lack of indentation makes it near impossible for a human
to read the generated code, but of course a Pascal compiler has no
problem with it.  

\blank

Nowadays, creating an executable program from the
WEB source file happens in a few extra steps, and one of these steps
is a conversion from Pascal to C source code, by means of the
\type{web2c} system. You may find the output of \type{web2c} easier to
read, because it re-indents the code for human reading:

\starttyping
...
if ( mem [h + 2 ].cint > mem [h + 4 ].cint ) 
  print ( 1279 ) ;
else {
  pspairout(mem [h + 2].cint,mem [h + 3].cint);
  pspairout(mem [h + 4].cint,mem [h + 5].cint);
} 
printnl ( 1281 ) ;
print ( 256 ) ;
printnl ( 1282 ) ;
\stoptyping

\page

So, where did the strings go? \type{tangle} put the multi||character
strings into a separate file, in this case named \type{mp.pool}. Each
line of that file contains two digits indicating the length of the
string, followed by the string itself. Around line 1000, you will find
this:

\snaptogrid\vbox{
\starttyping
...
070 0 0 0
20%%HiResBoundingBox: 
20%%Creator: MetaPost 
16%%CreationDate: 
...
\stoptyping
}

\type{07} is the length in bytes of `\type{0 0 0 0}', \type{20} is the
length of `\type{%%HiResBoundingBox: }', including the trailing space
character, etcetera. Single character strings are not written to the
pool file, because there is no need: all single||character strings
simply have an assumed index value matching their contents, and the
first string in the \type{pool} file receives index number 256.

\blank

The Pascal source code (or C source code) is now converted into an
executable, and you end up with \type{mpost.exe} as well as
\type{mp.pool}. The pool file is stored somewhere in the \TEXMF\ tree,
and one of the very first things that the \type{--ini} version of
\METAPOST\ does, is that it reads \type{mp.pool} to initialize its
internal arrays. When the user executes the \type{dump} command,
\METAPOST\ writes all of the string items to the \type{.mem} file,
from where it will be retrieved by production runs of \METAPOST.

There is nothing wrong with this system as such. In fact, it has
worked well for nearly 30~years. But it does make updating executables
a bit harder than desired: users not only have to copy the actual
program to a folder in the path, but they also have to figure out
where to place the new and improved \type{mp.pool} file.

As the maintainer of \METAPOST\ and \LUATEX, both programs that are
updated frequently, I was getting annoyed with having to explain to
almost each updating user what a pool file was, why it was important,
and where it should go in their \TEXMF\ tree.

\snaptogrid\vbox{
\subject{How a pool file disappears again}
}

So I decided to do something about it, and that was how the
\type{makecpool} program was born. The concept is very simple: it
converts the \type{mp.pool} into a C source file named
\type{loadpool.c}.  In fact, it is so obvious that the idea has been
proposed a number of times already, for instance by Fabrice Popineau.
But somehow it has never made it to the core \TeX\ distribution yet.

The structure of the created file is straightforward: there is one big
static array, and a fairly simple C function that replaces the Pascal
procedure for pool file reading. In abbreviated form, \type{loadpool.c}
looks like this:


\snaptogrid\vbox{\starttyping
/* This file is auto-generated by makecpool */

#include <stdio.h>
#include "mpdir/mplib.h"

static char *poolfilearr[] = {
  "1.000",

  ...

  "0 0 0 0",
  "%%HiResBoundingBox: ",
  "%%Creator: MetaPost ",
  "%%CreationDate: ",

  ...

  NULL };

int loadpoolstrings (integer spare_size) {
  char *s;
  strnumber g=0;
  int i=0,j=0;
  while ((s = poolfilearr[j++])) {
    int l = strlen (s);
    i += l;
    if (i>=spare_size) return 0;
    while (l-- > 0) strpool[poolptr++] = *s++;
    g = makestring();
    strref[g]= 127;
  }
  return g;
}
\stoptyping
}

In the stage where the various C files are compiled into
\type{mpost.exe}, this file is included in the list, and in that way
the strings will be embedded in the program. At run-time, the C
function is called to put the strings for the C array into the
internal storage area instead of the original file reader.

The result: there is only one single executable file that can be freely
distributed to the users. The source code for \type{makecpool} is part
of the \METAPOST\ and \LUATEX\ distribution package.

Post-publication update: thanks to Akira Kakuto and others, this same
method has now been adapted to all the \TeX\ family programs which used
pool files, and \TeX\ Live 2008 contains no external pool files.  Knuth
has approved the change for the baseline \type{tex} program.

\stopArticle

\stoptext