* Summary

Vnlog (pronounced "vanillog") is a trivially-simple log format:

- A whitespace-separated table of ASCII human-readable text
- Lines beginning with =#= are comments
- The first line that begins with a single =#= (not =##= or =#!=) is a /legend/,
  naming each column

Example:

#+BEGIN_EXAMPLE
#!/usr/bin/whatever
# a b c
1 2 3
## another comment
4 5 6
#+END_EXAMPLE

Such data works very nicely with normal UNIX tools (=awk=, =sort=, =join=), can
be easily read by fancier tools (=numpy=, =matlab=, =excel=, etc), and can be
plotted with =feedgnuplot=. This tookit provides some tools to manipulate
=vnlog= data and a few libraries to read/write it. The core philosophy is to
avoid creating new knowledge as much as possible. Consequently, the toolkit 
relies /heavily/ on existing (and familiar!) tools and workflows. As such, 
the toolkit is small, light, and has a /very/ friendly learning curve.

* Synopsis

I have two sets of historical stock data, from the start of 2018 until now
(2018/11):

#+BEGIN_EXAMPLE
$ < dji.vnl head -n 4
# Date Open High Low Close AdjClose Volume
2018-11-15 25061.48 25354.56 24787.79 25289.27 25289.27 383292840
2018-11-14 25388.08 25501.29 24935.82 25080.50 25080.50 384240000
2018-11-13 25321.21 25511.03 25193.78 25286.49 25286.49 339690000

$ < tsla.vnl head -n 4
# Date Open High Low Close AdjClose Volume
2018-11-15 342.33 348.58 339.04 348.44 348.44 4486339
2018-11-14 342.70 347.11 337.15 344.00 344.00 5036300
2018-11-13 333.16 344.70 332.20 338.73 338.73 5448600
#+END_EXAMPLE

I can pull out the closing prices:

#+BEGIN_EXAMPLE
$ < dji.vnl vnl-filter -p Close | head -n4
# Close
25289.27
25080.50
25286.49
#+END_EXAMPLE

And I can plot them:

#+BEGIN_EXAMPLE
$ < dji.vnl vnl-filter -p Close |
    feedgnuplot --lines --terminal 'dumb 80,30' --unset grid

  27000 +-------------------------------------------------------------------+
        |        *    +            +             +            +             |
        |        * *                                                        |
        |        ***                                            *           |
  26500 |-+     * ***                                           *         +-|
        |       * * *                                           *           |
        | *     *   **                                         * *          |
  26000 |-*     *   *** *                                      * **       +-|
        | *     *    ****                                      *  *         |
        | *    **       **                                     *  *         |
        | *   ***        ** *                              *   *   *        |
  25500 |*+*  ***         *** *                            *   *   *      +-|
        |* * ****         ** **       **                * **   *   *        |
        |* *** *          ** * **     **                * ***  *    *       |
        |*  ** *          *    **     * *               * ***  *    *       |
  25000 |-+ **                 ***   *  *  *           ** ***  *    *     +-|
        |   **                   **  *  * ***    *     ****** *      *      |
        |   **                   **  *  ** **    *    **  *  **             |
        |   **                    * **  **   *   *    **  *  **             |
  24500 |-+ *                     * **   *   *  * **  *      **           +-|
        |                         ***    *   **** *** *      **             |
        |                         **          *** *****      **             |
  24000 |-+                                   ***  ****       *           +-|
        |                                     *    ****       *             |
        |                                           ***                     |
        |             +            +             +  **        +             |
  23500 +-------------------------------------------------------------------+
        0             50          100           150          200           250
#+END_EXAMPLE

For this documentation I'm plotting in ascii ("dumb" terminal). Normally you'd
use the default terminal (by omitting =--terminal ...=), and you'd get
interactive, graphical plots. Here I kept /only/ the closing price column, so
the x-axis is just the row index. The data was in reverse chronological order,
so this plot is also in reverse chronological order. Let's fix that:

#+BEGIN_EXAMPLE
$ < dji.vnl vnl-sort -k Date |
    vnl-filter -p Close |
    feedgnuplot --lines --terminal 'dumb 80,30' --unset grid

  27000 +-------------------------------------------------------------------+
        |             +            +             +          * +             |
        |                                                 * *               |
        |    *                                            * *               |
  26500 |-+  *                                            ** *            +-|
        |    *                                            ** *              |
        |   ***                                         **   *     *        |
  26000 |-+** *                                      *****    *    **     +-|
        |  *  *                                      ****     *    **       |
        |  *   *                                    **        **   **       |
        |  *   *   *                             *  **        **   **       |
  25500 |-*    *   *                           * ***          **  * *     +-|
        | *    *  ** *                **       *****          *** * *       |
        |**    * *** *                **      *** **          * * * **      |
        |*     * *** *                ***    ***   *          * * * *       |
  25000 |*+    * *** **            *  * *   ***                 ***       +-|
        |*     * ******     *    * * *  *   *                   **          |
        |      * * ** **    **   ** **  *   *                   **          |
        |      **  **  *    **  *   **   *  *                   **          |
  24500 |-+    **      *  ****  *   **   * *                     *        +-|
        |      **       * ***** *   *    ***                                |
        |       *       * ** ****         **                                |
  24000 |-+     *       ***  ***                                          +-|
        |       *       ***    *                                            |
        |               **                                                  |
        |             + **         +             +            +             |
  23500 +-------------------------------------------------------------------+
        0             50          100           150          200           250
#+END_EXAMPLE

This is in the correct order, but it'd be nice to see the actual dates on the
x-axis:

#+BEGIN_EXAMPLE
$ < dji.vnl vnl-filter -p Date,Close | head -n4
# Date Close
2018-11-15 25289.27
2018-11-14 25080.50
2018-11-13 25286.49

$ < dji.vnl vnl-sort -k Date |
    vnl-filter -p Date,Close |
    feedgnuplot --lines --unset grid --terminal 'dumb 80,30'
                --timefmt %Y-%m-%d --domain

  27000 +-------------------------------------------------------------------+
        |     +     +     +      +     +     +     +     +      *     +     |
        |                                                    *  *           |
        |    *                                               ****           |
  26500 |-+  **                                              *****        +-|
        |    **                                              **  *          |
        |   ***                                             **   *     *    |
  26000 |-+** *                                         *** **   *     *  +-|
        |  *  *                                         ****     *     *    |
        | **  *                                        **        **    *    |
        | *   *    *                                * * *        ***   **   |
  25500 |-*   *    **                             *****          ***   ** +-|
        | *   *    ** *                  *        ** **          **** ***   |
        |*    *  * ** *                 **      ** * **          ** * * **  |
        |*    *  **** *                 ***     **    *          *  * * *   |
  25000 |-+   *  *******            *   * *    * *                  * *   +-|
        |      **** ** *      *   **** *  *    *                    **      |
        |      ***  ** **     *   ******  **  **                    **      |
        |      ***  *   *    **   *  ***   *  *                     **      |
  24500 |-+    ***      *  *****  *   *    ** *                      *    +-|
        |      **       *  *** ****   *     ***                             |
        |       *       ** **  ***          **                              |
  24000 |-+     *       *****  ***                                        +-|
        |       *       ****     *                                          |
        |               * *                                                 |
        |     +     +   * *      +     +     +     +     +      +     +     |
  23500 +-------------------------------------------------------------------+
      01/01 02/01 03/01 04/01  05/01 06/01 07/01 08/01 09/01  10/01 11/01 12/01
#+END_EXAMPLE

What was the highest value of the Dow-Jones index, and when did it happen?

#+BEGIN_EXAMPLE
$ < dji.vnl vnl-sort -rk Close |
  head -n2 |
  vnl-align

#  Date      Open     High      Low     Close  AdjClose   Volume
2018-10-03 26833.47 26951.81 26789.08 26828.39 26828.39 280130000
#+END_EXAMPLE

Alrighty. Looks like the high was in October. Let's zoom in on that month:

#+BEGIN_EXAMPLE
$ < dji.vnl vnl-sort -k Date |
    vnl-filter 'Date ~ /2018-10/' -p Date,Close |
    feedgnuplot --lines --unset grid --terminal 'dumb 80,30'
                --timefmt %Y-%m-%d --domain

  27000 +-------------------------------------------------------------------+
        |          ** +            +             +            +             |
        |        **  *                                                      |
        |       *     *                                                     |
  26500 |-+            *   ****                                           +-|
        |               ***    *                                            |
        |                      *                                            |
        |                       *                                           |
        |                       *                                           |
  26000 |-+                     *                                         +-|
        |                       *            **                             |
        |                        *           * *                            |
        |                        *          *   *                           |
  25500 |-+                      *          *   * ******                  +-|
        |                         *  *******     *      *                   |
        |                         * *      *             **                 |
        |                          **                      *              * |
  25000 |-+                        *                       *  *          *+-|
        |                                                  *  **        *   |
        |                                                   ** *        *   |
        |                                                   **  *****  *    |
        |                                                   *        * *    |
  24500 |-+                                                           *   +-|
        |                                                                   |
        |                                                                   |
        |             +            +             +            +             |
  24000 +-------------------------------------------------------------------+
      09/27         10/04        10/11         10/18        10/25         11/01
#+END_EXAMPLE

OK. Is this thing volatile? What was the largest single-day gain?

#+BEGIN_EXAMPLE
$ < dji.vnl | vnl-filter -p '.,d=diff(Close)' |
    head -n4 |
    vnl-align

#  Date      Open     High      Low     Close  AdjClose   Volume     d
2018-11-15 25061.48 25354.56 24787.79 25289.27 25289.27 383292840    0
2018-11-14 25388.08 25501.29 24935.82 25080.50 25080.50 384240000 -208.77
2018-11-13 25321.21 25511.03 25193.78 25286.49 25286.49 339690000  205.99


$ < dji.vnl | vnl-filter -p '.,d=diff(Close)' |
  vnl-sort -rk d |
  head -n2 |
  vnl-align

#  Date      Open     High      Low     Close  AdjClose   Volume     d
2018-02-02 26061.79 26061.79 25490.66 25520.96 25520.96 522880000 1175.21
#+END_EXAMPLE

Whoa. So the best single-gain day was 2018-02-02: the dow gained 1175.21 points
between closing on Feb 1 and Feb 2. But it actually lost ground that day! What
if I looked at the difference between the opening and closing in a single day?

#+BEGIN_EXAMPLE
< dji.vnl | vnl-filter -p '.,d=Close-Open' |
  vnl-sort -rgk d |
  head -n2 |
  vnl-align

#  Date      Open     High      Low     Close  AdjClose   Volume    d
2018-02-06 24085.17 24946.23 23778.74 24912.77 24912.77 823940000 827.6
#+END_EXAMPLE

I guess by that metric 2018-02-06 was better. Let's join the Dow-jones index
data and the TSLA data, and let's look at them together:

#+BEGIN_EXAMPLE
$ vnl-join --vnl-autosuffix dji.vnl tsla.vnl -j Date |
  head -n4 |
  vnl-align

#  Date    Open_dji High_dji  Low_dji Close_dji AdjClose_dji Volume_dji Open_tsla High_tsla Low_tsla Close_tsla AdjClose_tsla Volume_tsla
2018-11-15 25061.48 25354.56 24787.79 25289.27  25289.27     383292840  342.33    348.58    339.04   348.44     348.44        4486339
2018-11-14 25388.08 25501.29 24935.82 25080.50  25080.50     384240000  342.70    347.11    337.15   344.00     344.00        5036300
2018-11-13 25321.21 25511.03 25193.78 25286.49  25286.49     339690000  333.16    344.70    332.20   338.73     338.73        5448600


$ vnl-join --vnl-autosuffix dji.vnl tsla.vnl -j Date |
  vnl-filter -p '^Close' |
  head -n4 |
  vnl-align

# Close_dji Close_tsla
25289.27    348.44
25080.50    344.00
25286.49    338.73


$ vnl-join --vnl-autosuffix dji.vnl tsla.vnl -j Date |
  vnl-filter -p '^Close' |
  feedgnuplot --domain --points --unset grid --terminal 'dumb 80,30'

  380 +---------------------------------------------------------------------+
      |         +         +         +         +         +         +         |
      |                             A           A                           |
      |                      A                                              |
  360 |-+                             A A         A                       +-|
      |                       A           A A A         A  AA               |
      |             A    A        AA    AAAA A           AA AA   A          |
  340 |-+         A  AA             AAA A A AAA  A  A   A           A     +-|
      |               A  A A       A  AAA AA   A    A           A           |
      |              A A    A A AAA     A A A                               |
      |                      A A AA     A  A                                |
  320 |-+                     A A AAAAAA         A AA    A                +-|
      |      A              A A   A AAA  AA                                 |
      |        A   AA  A A     A     A      A      A     A      AA   A      |
      |             A  AA AA   A     A         A A      A A                 |
  300 |A+     AA  A A  A     A  AA       A AAA         A        A AA A A  +-|
      |            A AA   AA A A  A     A                A A             A  |
      |         A    A   A  A  AA         A            AA  A                |
      |        AA A   AAAA    A     A                A AA    A       A      |
  280 |-+    A                AAAAA                 A  A                  +-|
      |                                           A                         |
      |          AA                                              A          |
  260 |-+                                AA AA        A          A        +-|
      |      A                             A    A                           |
      |  A                           A                            A         |
      |         +         +         +         +         +         +         |
  240 +---------------------------------------------------------------------+
    23500     24000     24500     25000     25500     26000     26500     27000
#+END_EXAMPLE

Huh. Apparently there's no obvious, strong correlation between TSLA and
Dow-Jones closing prices; I would have expected a clear trend here, but the data
doesn't lie.

* Description

Vnlog tools are designed to be very simple and light. There exist a number of
other tools that manipulate data on the commandline. Some of these:

- https://csvkit.readthedocs.io/
- https://github.com/johnkerl/miller
- https://github.com/eBay/tsv-utils-dlang
- http://harelba.github.io/q/
- https://github.com/BatchLabs/charlatan
- https://github.com/dinedal/textql
- https://github.com/BurntSushi/xsv
- https://github.com/dbohdan/sqawk

These all provide facilities to run various analyses, and are neither simple nor
light. Vnlog by contrast doesn't analyze anything, but makes it easy to write
simple bits of awk or perl to process stuff to your heart's content. The main
envisioned use case is one-liners, and the tools are geared for that purpose.
The above mentioned tools are much more powerful than vnlog, so they could be a
better fit for some use cases. I claim that

- 90% of the time you want to do simple things, and vnlog is a great fit for the
  task
- If you really do need to do something complex, you really shouldn't be in the
  shell writing oneliners anymore, and a fully-fledged analysis system (numpy,
  etc) is more appropriate

In the spirit of doing as little as possible, the provided tools are wrappers
around tools you already have and are familiar with. The provided tools are:

- =vnl-filter= is a tool to select a subset of the rows/columns in a vnlog
  and/or to manipulate the contents. This is effectively an =awk= wrapper where
  the fields can be referenced by name instead of index. 20-second tutorial:

#+BEGIN_EXAMPLE
vnl-filter -p col1,col2,colx=col3+col4 'col5 > 10' --has col6
#+END_EXAMPLE
  will read the input, and produce a vnlog with 3 columns: =col1= and =col2=
  from the input, and a column =colx= that's the sum of =col3= and =col4= in the
  input. Only those rows for which =col5 > 10= is true will be output. Finally,
  only those rows that have a non-null value for =col6= will be selected. A null
  entry is signified by a single =-= character.

#+BEGIN_EXAMPLE
vnl-filter --eval '{s += x} END {print s}'
#+END_EXAMPLE
  will evaluate the given awk program on the input, but the column names work as
  you would hope they do: if the input has a column named =x=, this would
  produce the sum of all values in this column.

- =vnl-sort=, =vnl-join=, =vnl-tail=, =vnl-ts= are wrappers around the
  corresponding commandline tools. These work exactly as you would expect also:
  the columns can be referenced by name, and the legend comment is handled
  properly. These are wrappers, so all the commandline options those tools have
  "just work" (except options that don't make sense in the context of vnlog). As
  an example, =vnl-tail -f= will follow a log: data will be read by =vnl-tail=
  as it is written into the log (just like =tail -f=, but handling the legend
  properly). And you already know how to use these tools without even reading
  the manpages! Note: these were written for and have been tested with the Linux
  kernel and GNU Coreutils =sort=, =join= and =tail=. Other kernels and tools
  probably don't (yet) work. Send me patches.

- =vnl-align= aligns vnlog columns for easy interpretation by humans. The
  meaning is unaffected

- =Vnlog::Parser= is a simple perl library to read a vnlog

- =vnlog= is a simple python library to read a vnlog. Both python2 and python3
  are supported

- =libvnlog= is a C library to simplify writing a vnlog. Clearly all you
  /really/ need is =printf()=, but this is useful if we have lots of columns,
  many containing null values in any given row, and/or if we have parallel
  threads writing to a log. In my usage I have hundreds of columns, so this is
  handy

- =vnl-make-matrix= converts a one-point-per-line vnlog to a matrix of data.
  I.e.

#+BEGIN_EXAMPLE
$ cat dat.vnl
# i j x
0 0 1
0 1 2
0 2 3
1 0 4
1 1 5
1 2 6
2 0 7
2 1 8
2 2 9
3 0 10
3 1 11
3 2 12

$ < dat.vnl vnl-filter -p i,x | vnl-make-matrix --outdir /tmp
Writing to '/tmp/x.matrix'

$ cat /tmp/x.matrix
1 2 3
4 5 6
7 8 9
10 11 12
#+END_EXAMPLE

All the tools have manpages that contain more detail. And more tools will
probably be added with time.

* Manpages
** vnl-filter
#+BEGIN_EXAMPLE
xxx-manpage-vnl-filter-xxx
#+END_EXAMPLE

** vnl-align
#+BEGIN_EXAMPLE
xxx-manpage-vnl-align-xxx
#+END_EXAMPLE

** vnl-sort
#+BEGIN_EXAMPLE
xxx-manpage-vnl-sort-xxx
#+END_EXAMPLE

** vnl-join
#+BEGIN_EXAMPLE
xxx-manpage-vnl-join-xxx
#+END_EXAMPLE

** vnl-tail
#+BEGIN_EXAMPLE
xxx-manpage-vnl-tail-xxx
#+END_EXAMPLE

** vnl-ts
#+BEGIN_EXAMPLE
xxx-manpage-vnl-ts-xxx
#+END_EXAMPLE

* Installation

** On Debian-based boxes
At this time vnlog is a part of Debian/sid, and is a part of Ubuntu cosmic or
later. On those boxes you can simply

#+BEGIN_EXAMPLE
$ sudo apt install vnlog libvnlog-dev libvnlog-perl python-vnlog
#+END_EXAMPLE

to get the binary tools, the C API, the perl and python2 interfaces
respectively.

On a Debian (or Ubuntu) machine that's too old to have the packages already
available, you can build and install them:

#+BEGIN_EXAMPLE
$ git clone git@github.com:dkogan/vnlog.git
$ cd vnlog
$ cp -r packaging/debian .
$ dpkg-buildpackage -us -uc -b
$ sudo dpkg -i ../vnlog*.deb ../libvnlog-dev*.deb ../libvnlog-perl*.deb ../python-vnlog*.deb
#+END_EXAMPLE

** On non-Debian-based boxes

With the exception of the C API, every part of the toolkit is written in an
interpreted language, and there's nothing to "install". You can run everything
directly from the source tree:

#+BEGIN_EXAMPLE
$ git clone git@github.com:dkogan/vnlog.git
$ cd vnlog
$ ./vnl-filter .....
#+END_EXAMPLE

If you /do/ want to install to some location, do this:

#+BEGIN_EXAMPLE
$ make
$ PREFIX=/usr/local make install
#+END_EXAMPLE

* C interface
** Basic usage

For most uses, these logfiles are simple enough to be generated with plain
prints. But then each print statement has to know which numeric column we're
populating, which becomes effortful with many columns. In my usage it's common
to have a large parallelized C program that's writing logs with hundreds of
columns where any one record would contain only a subset of the columns. In such
a case, it's helpful to have a library that can output the log files. This is
available. Basic usage looks like this:

In a shell:

#+BEGIN_EXAMPLE
$ vnl-gen-header 'int w' 'uint8_t x' 'char* y' 'double z' 'void* binary' > vnlog_fields_generated.h
#+END_EXAMPLE

In a C program test.c:

#+BEGIN_SRC C
#include "vnlog_fields_generated.h"

int main()
{
    vnlog_emit_legend();

    vnlog_set_field_value__w(-10);
    vnlog_set_field_value__x(40);
    vnlog_set_field_value__y("asdf");
    vnlog_emit_record();

    vnlog_set_field_value__z(0.3);
    vnlog_set_field_value__x(50);
    vnlog_set_field_value__w(-20);
    vnlog_set_field_value__binary("\x01\x02\x03", 3);
    vnlog_emit_record();

    vnlog_set_field_value__w(-30);
    vnlog_set_field_value__x(10);
    vnlog_set_field_value__y("whoa");
    vnlog_set_field_value__z(0.5);
    vnlog_emit_record();

    return 0;
}
#+END_SRC

Then we build and run, and we get

#+BEGIN_EXAMPLE
$ cc -o test test.c -lvnlog

$ ./test

# w x y z binary
-10 40 asdf - -
-20 50 - 0.2999999999999999889 AQID
-30 10 whoa 0.5 -
#+END_EXAMPLE

The binary field in base64-encoded. This is a rarely-used feature, but sometimes
you really need to log binary data for later processing, and this makes it
possible.

So you

1. Generate the header to define your columns

2. Call =vnlog_emit_legend()=

3. Call =vnlog_set_field_value__...()= for each field you want to set in that
   row.

4. Call =vnlog_emit_record()= to write the row and to reset all fields for the
   next row. Any fields unset with a =vnlog_set_field_value__...()= call are
   written as null: =-=

This is enough for 99% of the use cases. Things get a bit more complex if we
have have threading or if we have multiple vnlog ouput streams in the same
program. For both of these we use vnlog /contexts/.

** Contexts

To support independent writing into the same vnlog (possibly by multiple
threads; this is reentrant), each log-writer should create a context, and use it
when talking to vnlog. The context functions will make sure that the fields in
each context are independent and that the output records won't clobber each
other:

#+BEGIN_SRC C
void child_writer( // the parent context also writes to this vnlog. Pass NULL to
                   // use the global one
                   struct vnlog_context_t* ctx_parent )
{
    struct vnlog_context_t ctx;
    vnlog_init_child_ctx(&ctx, ctx_parent);

    while(records)
    {
        vnlog_set_field_value_ctx__xxx(&ctx, ...);
        vnlog_set_field_value_ctx__yyy(&ctx, ...);
        vnlog_set_field_value_ctx__zzz(&ctx, ...);
        vnlog_emit_record_ctx(&ctx);
    }

    vnlog_free_ctx(&ctx); // required only if we have any binary fields
}
#+END_SRC

If we want to have multiple independent vnlog writers to /different/ streams
(with different columns and legends), we do this instead:

=file1.c=:
#+BEGIN_SRC C
#include "vnlog_fields_generated1.h"

void f(void)
{
    // Write some data out to the default context and default output (STDOUT)
    vnlog_emit_legend();
    ...
    vnlog_set_field_value__xxx(...);
    vnlog_set_field_value__yyy(...);
    ...
    vnlog_emit_record();
}
#+END_SRC

=file2.c=:
#+BEGIN_SRC C
#include "vnlog_fields_generated2.h"

void g(void)
{
    // Make a new session context, send output to a different file, write
    // out legend, and send out the data
    struct vnlog_context_t ctx;
    vnlog_init_session_ctx(&ctx);
    FILE* fp = fopen(...);
    vnlog_set_output_FILE(&ctx, fp);
    vnlog_emit_legend_ctx(&ctx);
    ...
    vnlog_set_field_value__a(...);
    vnlog_set_field_value__b(...);
    ...
    vnlog_free_ctx(&ctx); // required only if we have any binary fields
    vnlog_emit_record();
}
#+END_SRC

Note that it's the user's responsibility to make sure the new sessions go to a
different =FILE= by invoking =vnlog_set_output_FILE()=. Furthermore, note that
the included =vnlog_fields_....h= file defines the fields we're writing to; and
if we have multiple different vnlog field definitions in the same program (as in
this example), then the different writers /must/ live in different source files.
The compiler will barf if you try to =#include= two different
=vnlog_fields_....h= files in the same source.

** Remaining APIs

- =vnlog_printf(...)= and =vnlog_printf_ctx(ctx, ...)= write to a pipe like
=printf()= does. This exists primarily for comments.

- =vnlog_clear_fields_ctx(ctx, do_free_binary)= clears out the data in a context
and makes it ready to be used for the next record. It is rare for the user to
have to call this manually. The most common case is handled automatically
(clearing out a context after emitting a record). One area where this is useful
is when making a copy of a context:

#+BEGIN_SRC C
struct vnlog_context_t ctx1;
// .... do stuff with ctx1 ... add data to it ...

struct vnlog_context_t ctx2 = ctx1;
// ctx1 and ctx2 now both have the same data, and the same pointers to
// binary data. I need to get rid of the pointer references in ctx1

vnlog_clear_fields_ctx(&ctx1, false);
#+END_SRC

- =vnlog_free_ctx(ctx)= frees memory for an vnlog context. Do this before
throwing the context away. Currently this is only needed for context that have
binary fields, but this should be called for all contexts anyway, in case this
changes in a later revision

* numpy interface

The built-in =numpy.loadtxt= =numpy.savetxt= functions work well to read and
write these files. For example to write to standard output a vnlog with fields
=a=, =b= and =c=:

#+BEGIN_SRC python
numpy.savetxt(sys.stdout, array, fmt="%g", header="a b c")
#+END_SRC

Note that numpy automatically adds the =#= to the header. To read a vnlog from a
file on disk, do something like

#+BEGIN_SRC python
array = numpy.loadtxt('data.vnl')
#+END_SRC

These functions know that =#= lines are comments, but don't interpret anything
as field headers. That's easy to do, so I'm not providing any helper libraries.
I might do that at some point, but in the meantime, patches are welcome.

* Caveats and bugs

The tools that wrap standard commandline utilities (=vnl-sort=, =vnl-join=,
=vnl-tail=, =vnl-ts=) are written specifically to work with the Linux kernel and
the GNU coreutils. None of these have been tested with BSD tools or with
non-Linux kernels, and I'm sure things don't just work. It's probably not too
effortful to get that running, but somebody needs to at least bug me for that.
Or better yet, send me nice patches :)

These tools are meant to be simple, so some things are hard requirements. A big
one is that columns are whitespace-separated. There is /no/ mechanism for
escaping or quoting whitespace into a single field. I think supporting something
like that is more trouble than it's worth.

* Repository

https://github.com/dkogan/vnlog/

* Authors

Dima Kogan (=dima@secretsauce.net=) wrote this toolkit for his work at the Jet
Propulsion Laboratory, and is delighted to have been able to release it
publically

Chris Venter (=chris.venter@gmail.com=) wrote the base64 encoder

* License and copyright

This library is free software; you can redistribute it and/or modify it under
the terms of the GNU Lesser General Public License as published by the Free
Software Foundation; either version 2.1 of the License, or (at your option) any
later version.

Copyright 2016-2017 California Institute of Technology

Copyright 2017-2018 Dima Kogan (=dima@secretsauce.net=)

=b64_cencode.c= comes from =cencode.c= in the =libb64= project. It is written by
Chris Venter (=chris.venter@gmail.com=) who placed it in the public domain. The
full text of the license is in that file.