.\" gd_include.3.  The gd_include man page.
.\"
.\" Copyright (C) 2008, 2009, 2010, 2011, 2012, 2014, 2016 D. V. Wiebe
.\"
.\""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
.\"
.\" This file is part of the GetData project.
.\"
.\" Permission is granted to copy, distribute and/or modify this document
.\" under the terms of the GNU Free Documentation License, Version 1.2 or
.\" any later version published by the Free Software Foundation; with no
.\" Invariant Sections, with no Front-Cover Texts, and with no Back-Cover
.\" Texts.  A copy of the license is included in the `COPYING.DOC' file
.\" as part of this distribution.
.\"
.TH gd_include 3 "25 December 2016" "Version 0.10.0" "GETDATA"

.SH NAME
gd_include, gd_include_affix, gd_include_ns \(em add a format specification
fragment to a Dirfile

.SH SYNOPSIS
.SC
.B #include <getdata.h>
.HP
.BI "int gd_include(DIRFILE *" dirfile ", const char *" include_file ,
.BI "int " parent_fragment ", unsigned long " flags );
.HP
.BI "int gd_include_affix(DIRFILE *" dirfile ", const char *" include_file ,
.BI "int " parent_fragment ", const char *" prefix ", const char *" suffix ,
.BI "unsigned long " flags );
.HP
.BI "int gd_include_ns(DIRFILE *" dirfile ", const char *" include_file ,
.BI "int " parent_fragment ", const char *" namespace , 
.BI "unsigned long " flags );
.EC

.SH DESCRIPTION
The
.FN gd_include_affix
function adds the format specification fragment given by the path
.ARG include_file
to the specified dirfile, possibly creating the fragment, using the affixes
specified.  This occurs as if, in the existing fragment indexed
by
.ARG parent_fragment ,
the following directive were present:
.IP
.B /INCLUDE
.I <include_file> <prefix> <suffix>
.PP
(see
dirfile-format(5)).
The
.ARG prefix
may include a namespace, separated from the rest of the prefix, which may be
the empty string, by a dot
.RI ( . ).
If a parser callback function had been specified when the dirfile was opened
using
.F3 gd_cbopen ,
or added later with
.F3 gd_parser_callback ,
this callback function will be called if a syntax error is encountered while
parsing the included fragment.

Passing NULL as
.ARG prefix
or
.ARG suffix
is the same as using the empty string (ie. the corresponding affix is empty).

The function
.FN gd_include
is equivalent to calling
.FN gd_include_affix
with both
.ARG prefix
and
.ARG suffix
equal to NULL.

The function
.FN gd_include_ns
is equivalent to calling
.FN gd_include_affix
with
.ARG suffix
equal to NULL and
.ARG prefix
equal to
.ARG namespace
concatenated with a trailing dot.

The 
.ARG flags
argument should be a bitwise-or'd collection of zero or more of the following
flags:
.PP
.DD GD_ARM_ENDIAN GD_NOT_ARM_ENDIAN
Specifies that double precision floating point raw data on disk are, or are not,
stored in the middle-endian format used by older ARM processors.

These flag only set the default endianness, and will be overridden when an
.B /ENDIAN
directive specifies the byte sex of
.B RAW
fields, unless
.B GD_FORCE_ENDIAN
is also specified.

On every platform, one of these flags
.RB ( GD_NOT_ARM_ENDIAN
on all but middle-ended ARM systems)
indicates the native behaviour of the platform.  That symbol will equal zero,
and may be omitted.
.DD GD_BIG_ENDIAN GD_LITTLE_ENDIAN
Specifies the default byte sex of raw data stored on disk to be either
big-endian (most significant byte first) or little-endian (least significant
byte first).  Omitting both flags indicates the default should be the native
endianness of the platform.

Unlike the ARM endianness flags above, neither of these symbols is ever zero.
Specifying both these flags together will cause the library to assume that the
endianness of the data is opposite to that of the native architecture, whatever
that might be.

These flag only set the default endianness, and will be overridden when an
.B /ENDIAN
directive specifies the byte sex of
.B RAW
fields, unless
.B GD_FORCE_ENDIAN
is also specified.
.DD GD_CREAT
An empty fragment will be created, if one does not already exist.  The fragment
will have mode
.BR S_IRUSR " | " S_IWUSR " | "  S_IRGRP " | "  S_IWGRP " | " S_IROTH " | " S_IWOTH
(0666), modified by the caller's umask value (see
.BR umask (2)).
.DD GD_EXCL
Ensure that this call creates a new fragment: when specified along with
.BR GD_CREAT ,
the call will fail if the file specified by
.ARG include_file
already exists.  If
.B GD_CREAT
is not specified, this flag is ignord.  This flag suffers from all the
limitations of the
.B O_EXCL
flag as indicated in
.BR open (2).
.DD GD_FORCE_ENCODING
Specifies that
.B /ENCODING
directives (see
dirfile-format(5))
found in the fragment should be ignored.  The encoding scheme
specified in
.ARG flags
will be used instead (see below).
.DD GD_FORCE_ENDIAN
Specifies that
.B /ENDIAN
directives (see
dirfile-format(5))
found in the fragment should be ignored.  When specified with one of
.BR GD_BIG_ENDIAN " or " GD_LITTLE_ENDIAN ,
the indicated endianness will be assumed.  If this flag is specified with
neither of those flags, the fragment will be assumed to have the endianness of
the native architecture.
.DD GD_IGNORE_DUPS
If the fragment specifies more than one field with the same name, or a field 
with the same name as an existing field, all but one of them will be ignored by
the parser.  Without this flag, parsing would fail with the
.B GD_E_FORMAT 
error, possibly resulting in invocation of the registered callback function.
Which of the duplicate fields is kept is not specified, nor whether an existing
field takes precedence over a new one or not.  As a result, this flag is
typically only useful in the case where identical copies of a field
specification line are present.

No indication is provided to indicate whether a duplicate field has been
discarded.  If finer grained control is required, the caller should handle
.B GD_E_FORMAT_DUPLICATE
suberrors itself with an appropriate callback function.
.DD GD_IGNORE_REFS
If the dirfile currently has a reference field (either because one was specified
explicitly, or else because the first
.B RAW
field was used),
.B /REFERENCE
directives in the included fragment will be ignored.  Otherwise, a
.B /REFERENCE
directive in the included fragment will replace the current reference field in
the dirfile.
.DD GD_PEDANTIC
Specifies that unrecognised lines found during the parsing of the fragment
should always cause a fatal error.  Without this flag, if a
.B VERSION
directive (see
dirfile-format(5))
indicates that the fragment being opened conforms Standards Version newer than
the version understood by the library, unrecognised lines will be silently
ignored.
.DD GD_TRUNC
If
.ARG include_file
already exists, it will be truncated before opening.  If the file does not
exist, this flag is ignored.

.PP
The
.ARG flags
argument may also be bitwise or'd with one of the following symbols indicating
the default encoding scheme of the fragment.  Like the endianness flags, the
choice of encoding here is ignored if the encoding is specified in the fragment
itself, unless
.B GD_FORCE_ENCODED
is also specified.  If none of these symbols is present,
.B GD_AUTO_ENCODED
is assumed, unless this call results in creation or truncation of the fragment. 
In that case,
.B GD_UNENCODED
is assumed.  See
dirfile-encoding(5)
for details on dirfile encoding schemes.
.DD GD_AUTO_ENCODED
Specifies that the encoding type is not known in advance, but should be detected
by the GetData library.  Detection is accomplished by searching for raw data
files with extensions appropriate to the encoding scheme.  This method will
notably fail if the the library is called via
.F3 gd_putdata
to create a previously non-existent raw field unless a read is first
successfully performed on the dirfile.  Once the library has determined the
encoding scheme for the first time, it remembers it for subsequent calls.
.DD GD_BZIP2_ENCODED
Specifies that raw data files are compressed using the Burrows-Wheeler block
sorting text compression algorithm and Huffman coding, as implemented in the
bzip2 format.
.DD GD_FLAC_ENCODED
Specifies that raw data files are compressed using the Free Lossless Audio
Coded (FLAC).
.DD GD_GZIP_ENCODED
Specifies that raw data files are compressed using Lempel-Ziv coding (LZ77)
as implemented in the gzip format.
.DD GD_LZMA_ENCODED
Specifies that raw data files are compressed using the Lempel-Ziv Markov Chain
Algorithm (LZMA) as implemented in the xz container format.
.DD GD_SIE_ENCODED
Specified that raw data files are sample-index encoded, similar to run-length
encoding, suitable for data that change rarely.
.DD GD_SLIM_ENCODED
Specifies that raw data files are compressed using the slimlib library.
.DD GD_TEXT_ENCODED
Specifies that raw data files are encoded as text files containing one data
sample per line.  
.DD GD_UNENCODED
Specifies that raw data files are not encoded, but written verbatim to disk.
.DD GD_ZZIP_ENCODED
Specifies that raw data files are compressed using the DEFLATE algorithm.  All
raw data files for a given fragment are collected together and stored in a PKZIP
archive called raw.zip.
.DD GD_ZZSLIM_ENCODED
Specifies that raw data files are compressed using a combinations of compression
schemes: first files are slim-compressed, as with the
.B GD_SLIM_ENCODED
scheme, and then they are collected together and compressed (again) into a PKZIP
archive called raw.zip, as in the
.B GD_ZZIP_ENCODED
scheme.

.SH RETURN VALUE
On success, these functions return the format specification index of the newly
added fragment.  On error, they return a negative-valued error code.  Possible
error codes are:
.DD GD_E_ACCMODE
The supplied dirfile was opened in read-only mode.
.DD GD_E_ALLOC
The library was unable to allocate memory.
.DD GD_E_BAD_DIRFILE
The supplied dirfile was invalid.
.DD GD_E_BAD_INDEX
The supplied parent fragment index was out of range.
.DD GD_E_BAD_REFERENCE
The reference field specified by a
.B /REFERENCE
directive in the fragment (see
dirfile-format(5))
was not found, or was not a
.B RAW
field.  In this case, the included fragment will still be added to the dirfile,
but the
.B /REFERENCE
directive will be ignored.
.DD GD_E_CALLBACK
The registered callback function returned an unrecognised response.
.DD GD_E_FORMAT
A syntax error occurred in the fragment.
.DD GD_E_LINE_TOO_LONG
The parser encountered a line in the format specification longer than it was
able to deal with.  Lines are limited by the storage size of
.BR ssize_t .
On 32-bit systems, this limits format specification lines to 2**31 characters.
The limit is larger on 64-bit systems.
.DD GD_E_IO
An I/O error occured while trying to read or create the fragment.
.DD GD_E_PROTECTED
The metadata of the parent fragment was protected from change.
.PP
The error code is also stored in the
.B DIRFILE
object and may be retrieved after this function returns by calling
.F3 gd_error .
A descriptive error string for the error may be obtained by calling
.F3 gd_error_string .

.SH HISTORY
The
.FN dirfile_include
function appeared in GetData-0.4.0.

In GetData-0.7.0, this function was renamed to
.FN gd_include .

The
.FN gd_include_affix
function appeared in GetData-0.8.0.

In GetData-0.10.0, the error return from these functions changed from -1 to a
negative-valued error code.  The
.FN gd_include_ns
function also appeared in this release.

See
.F3 gd_open
for history of the flags.

.SH SEE ALSO
.F3 gd_alter_affixes ,
.F3 gd_error ,
.F3 gd_error_string ,
.F3 gd_fragmentname ,
.F3 gd_nfragments ,
.F3 gd_open ,
.F3 gd_parser_callback ,
.F3 gd_reference ,
.F3 gd_uninclude ,
dirfile(5), dirfile-encoding(5), dirfile-format(5)