.\" gd_open.3.  The gd_open man page.
.\"
.\" Copyright (C) 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2016 D. V. Wiebe
.\"
.\""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
.\"
.\" This file is part of the GetData project.
.\"
.\" Permission is granted to copy, distribute and/or modify this document
.\" under the terms of the GNU Free Documentation License, Version 1.2 or
.\" any later version published by the Free Software Foundation; with no
.\" Invariant Sections, with no Front-Cover Texts, and with no Back-Cover
.\" Texts.  A copy of the license is included in the `COPYING.DOC' file
.\" as part of this distribution.
.\"
.TH gd_open 3 "25 December 2016" "Version 0.10.0" "GETDATA"

.SH NAME
gd_open, gd_cbopen \(em open or create a Dirfile

.SH SYNOPSIS
.SC
.B #include <getdata.h>
.HP
.BI "DIRFILE* gd_open(const char *" dirfilename ", unsigned long " flags );
.HP
.BI "DIRFILE* gd_cbopen(const char *" dirfilename ", unsigned long"
.IB flags ", gd_parser_callback_t " sehandler ", void *" extra );
.EC

.SH DESCRIPTION
The
.FN gd_cbopen
function opens or creates the dirfile specified by
.ARG dirfilename ,
returning a DIRFILE object associated with it.  Opening a dirfile will cause the
library to read and parse the dirfile's format specification (see
dirfile-format(5)).

If not NULL,
.ARG sehandler
should be a pointer to a function which will be called whenever a syntax error
is encountered during parsing the format specification.  Specify NULL for this
parameter if no callback function is to be used.  The caller may use this
function to correct the error or modify the error handling of the format
specification parser.  See
.B The Callback Function
section below for details on this function.  The
.ARG extra
argument allows the caller to pass data to the callback function.  The pointer
will be passed to the callback function verbatim.

The
.FN gd_open
function is equivalent to
.FN gd_cbopen ,
with
.ARG sehandler
and
.ARG extra
set to NULL.

The 
.ARG flags
argument should include one of the
.IR "access modes" :
.B GD_RDONLY
(read-only) or 
.BR GD_RDWR
(read-write), and may also contain zero or more of the following flags,
bitwise-or'd together:
.PP
.DD GD_ARM_ENDIAN GD_NOT_ARM_ENDIAN
Specifies that double precision floating point raw data on disk are, or are not,
stored in the middle-endian format used by older ARM processors.

These flag only set the default endianness, and will be overridden when an
.B /ENDIAN
directive specifies the byte sex of
.B RAW
fields, unless
.B GD_FORCE_ENDIAN
is also specified.

On every platform, one of these flags
.RB ( GD_NOT_ARM_ENDIAN
on all but middle-ended ARM systems)
indicates the native behaviour of the platform.  That symbol will equal zero,
and may be omitted.
.DD GD_BIG_ENDIAN GD_LITTLE_ENDIAN
Specifies the default byte sex of raw data stored on disk to be either
big-endian (most significant byte first) or little-endian (least significant
byte first).  Omitting both flags indicates the default should be the native
endianness of the platform.

Unlike the ARM endianness flags above, neither of these symbols is ever zero.
Specifying both these flags together will cause the library to assume that the
endianness of the data is opposite to that of the native architecture, whatever
that might be.

These flag only set the default endianness, and will be overridden when an
.B /ENDIAN
directive specifies the byte sex of
.B RAW
fields, unless
.B GD_FORCE_ENDIAN
is also specified.
.DD GD_CREAT
An empty dirfile will be created, if one does not already exist.  This will
create both the dirfile directory and an empty format specification file called
.ARG format .
If the call creates a dirfile, then the specified access mode is ignored: a
newly-created DIRFILE is always opened with access mode
.BR GD_RDWR ,
even if
.B GD_RDONLY
had been specified.

The directory will have have mode
.BR S_IRWXU " | " S_IRWXG " | "  S_IRWXO 
(0777), modified by the caller's umask value (see
.BR umask (2)).
The
.ARG format
file will have mode
.BR S_IRUSR " | " S_IWUSR " | "  S_IRGRP " | "  S_IWGRP " | " S_IROTH " | " S_IWOTH
(0666), also modified by the caller's umask.
The owner of the dirfile directory and
.ARG format
file will be the effective user ID of the caller.  Group ownership follows the
rules outlined in
.BR mkdir (2).
.DD GD_EXCL
Ensure that this call creates a dirfile: when specified along with
.BR GD_CREAT ,
the call will fail if the dirfile specified by
.ARG dirfilename
already exists.  If
.B GD_CREAT
is not specified, this flag is ignored.  This flag suffers from all the
limitations of the
.B O_EXCL
flag as indicated in
.BR open (2).
.DD GD_FORCE_ENCODING
Specifies that
.B /ENCODING
directives (see
dirfile-format(5))
found in the dirfile format specification should be ignored.  The encoding
scheme specified in
.ARG flags
will be used instead (see below).
.DD GD_FORCE_ENDIAN
Specifies that
.B /ENDIAN
directives (see
dirfile-format(5))
found in the dirfile format specification should be ignored.  All raw data will
be assumed to have the byte sex indicated through the presence or absence of the
.BR GD_ARM_ENDIAN ", " GD_BIG_ENDIAN ", " GD_LITTLE_ENDIAN ,
and
.B GD_NOT_ARM_ENDIAN
flags.
.DD GD_IGNORE_DUPS
If the dirfile format metadata specifies more than one field with the same name,
all but one of them will be ignored by the parser.  Without this flag, parsing
would fail with the
.B GD_E_FORMAT 
error, possibly resulting in invocation of the registered callback function.
Which of the duplicate fields is kept is not specified.  As a result,
this flag is typically only useful in the case where identical copies of a
field specification line are present.

No indication is provided to indicate whether a duplicate field has been
discarded.  If finer grained control is required, the caller should handle
.B GD_E_FORMAT_DUPLICATE
suberrors itself with an appropriate callback function.
.DD GD_PEDANTIC
Reject dirfiles which don't conform to the Dirfile Standards.  See the
.B Standards Compliance
section below for full details.
.DD GD_PERMISSIVE
Allow non-compliant format specification syntax, even when given along with a
conflicting
.B /VERSION
directive.  See the
.B Standards Compliance
section below for full details.
.DD GD_PRETTY_PRINT
When dirfile metadata are flushed to disk (either explicitly via
.BR gd_metaflush "(3), " gd_rewrite_fragment (3),
or 
.F3 gd_flush
or implicitly by closing the dirfile), an attempt will be made to create a
nicer looking format specification (from a human-readable standpoint).  What
this explicitly means is not part of the API, and any particular behaviour
should not be relied on.  If the dirfile is opened read-only, this flag is
ignored.
.DD GD_TRUNC
If
.ARG dirfilename
specifies an already existing dirfile, it will be truncated before opening.
Since
.FN gd_cbopen
decides whether
.ARG dirfilename
specifies an existing dirfile before attempting to parse the dirfile,
.ARG dirfilename
is considered to specify an existing dirfile if it refers to a directory
containing a regular file called
.BR format ,
regardless of the content or form of that file.

Truncation occurs by deleting every regular file and symlink in the specified
directory, whether the files were referred to by the dirfile before truncation
or not.  Accordingly, this flag should be used with caution.  Unless
.B GD_TRUNCSUB
is also specified, subdirectories are left untouched.  Notably, this operation
does not consider directories used in
.B /INCLUDE
directives.  If the dirfile does not exist, this flag is ignored.
.DD GD_TRUNCSUB
If specified along with
.BR GD_TRUNC ,
truncation will descend into subdirectories, deleting all regular files and
symlinks recursively.  It does not descend into directories pointed to by
symbolic links: in these cases, just the symlink itself is deleted.  If
specified without an accompanying
.BR GD_TRUNC ,
this flag is ignored.
.DD GD_VERBOSE
Specifies that whenever an error is triggered by the library when working
on this dirfile, the corresponding error string, which can be retrieved by
calling 
.F3 gd_error_string ,
should be written on the caller's standard error stream
.RB ( stderr (3))
by GetData.  The error string may be prefixed by a string specified by the
caller; see
.F3 gd_verbose_prefix .
Without this flag, GetData writes nothing to standard error.  (GetData never
writes to standard output.)
.PP
Those flags which affect the operation of the library beyond this call itself
may be modified later using the
.F3 gd_flags
function.
.PP
The
.ARG flags
argument may also be bitwise or'd with one of the following symbols indicating
the default encoding scheme of the dirfile.  Like the endianness flags, the
choice of encoding here is ignored if the encoding is specified in the dirfile
itself, unless
.B GD_FORCE_ENCODED
is also specified.  If none of these symbols is present,
.B GD_AUTO_ENCODED
is assumed, unless the
.FN gd_cbopen
call results in creation or truncation of the dirfile.  In that case,
.B GD_UNENCODED
is assumed.  See
dirfile-encoding(5)
for details on dirfile encoding schemes.
.DD GD_AUTO_ENCODED
Specifies that the encoding type is not known in advance, but should be detected
by the GetData library.  Detection is accomplished by searching for raw data
files with extensions appropriate to the encoding scheme.  This method will
notably fail if the the library is called via
.F3 putdata
to create a previously non-existent raw field unless a read is first
successfully performed on the dirfile.  Once the library has determined the
encoding scheme for the first time, it remembers it for subsequent calls.
.DD GD_BZIP2_ENCODED
Specifies that raw data files are compressed using the Burrows-Wheeler block
sorting text compression algorithm and Huffman coding, as implemented in the
bzip2 format.
.DD GD_FLAC_ENCODED
Specifies that raw data files are compressed using the Free Lossless Audio
Coded (FLAC).
.DD GD_GZIP_ENCODED
Specifies that raw data files are compressed using Lempel-Ziv coding (LZ77)
as implemented in the gzip format.
.DD GD_LZMA_ENCODED
Specifies that raw data files are compressed using the Lempel-Ziv Markov Chain
Algorithm (LZMA) as implemented in the xz container format.
.DD GD_SLIM_ENCODED
Specifies that raw data files are compressed using the slimlib library.
.DD GD_SIE_ENCODED
Specified that raw data files are sample-index encoded, similar to run-length
encoding, suitable for data that change rarely.
.DD GD_TEXT_ENCODED
Specifies that raw data files are encoded as text files containing one data
sample per line.  
.DD GD_UNENCODED
Specifies that raw data files are not encoded, but written as simply binary data
to disk.
.DD GD_ZZIP_ENCODED
Specifies that raw data files are compressed using the DEFLATE algorithm.  All
raw data files for a given fragment are collected together and stored in a PKZIP
archive called raw.zip.
.DD GD_ZZSLIM_ENCODED
Specifies that raw data files are compressed using a combinations of compression
schemes: first files are slim-compressed, as with the
.B GD_SLIM_ENCODED
scheme, and then they are collected together and compressed (again) into a PKZIP
archive called raw.zip, as in the
.B GD_ZZIP_ENCODED
scheme.

.SS Standards Compliance
The latest Dirfile Standards Version which this release of GetData understands
is provided in the preprocessor macro
.B GD_DIRFILE_STANDARDS_VERSION
defined in getdata.h.  GetData is able to open and parse any dirfile which
conforms to this Standards Version, or to any earlier Version.  The
dirfile-format(5)
manual page lists the changes between Standards Versions.

The GetData parser can operate in two modes: a
.I permissive
mode, in which much
non-Standards-compliant syntax is allowed, and a
.I pedantic
mode, in which the parser adheres strictly to the Standards.  The mode made
change during the parsing of a dirfile.  If
.B GD_PEDANTIC
is passed to
.FN gd_cbopen ,
the parser will start parsing the format specification in
.I pedantic
mode, otherwise it will start in
.I permissive
mode.

.I Permissive
mode is provided primarily to allow GetData to be used on dirfiles which
conform to no single Standard, but which were accepted by the GetData parser
in previous versions.  It is notably lax regarding reserved field names, and
field name characters, the mixing of old and new data type specifiers, and
generally ignores the presence of
.B /VERSION
directives.
In read-write mode,
.I permissive
mode should be used with caution, as it can cause unintentional corruption of
dirfile metadata on write, if the heuristics in the parser incorrectly guessed
the intention of non-compliant syntax.  In
.I permissive
mode, actual syntax errors are still reported as such.

In
.I pedantic
mode, the parser conforms to one specific Standards Version. This target
version may change any number of times in the course of scanning a single
format specification.  If invoked using the
.B GD_PEDANTIC
flag, the parser will start in
.I pedantic
mode with a target version equal to
.BR GD_DIRFILE_STANDARDS_VERSION .
Whenever a
.B /VERSION
directive is encountered in the format specification, the target version is
changed to the Standards Version specified.  When encountering a
.B /VERSION
directive in
.I permissive
mode, the parser will switch to
.I pedantic
mode, unless the
.B GD_PERMISSIVE
flag was passed to
.FN gd_cbopen ,
in which case no mode switch will take place.

Independent of the mode of the parser when parsing the format specification,
GetData will calculate a list of Standards Versions to which the parsed
metadata conform to.  The
.F3 gd_dirfile_standards
function can provide this information, and also specify the desired
Standards Version for writing format metadata back to disk.

.SS The Callback Function
The caller-supplied
.ARG sehandler
function is called whenever the format specification parser encounters a syntax
error
.RI ( i.e.
whenever it would return the
.B GD_E_FORMAT
error).  This callback may be used to correct the error, or to tell the parser
how to recover from it.

This function should take two pointers as arguments, and return an
.BR int :
.RS
.HP
.SC
.BI "int " sehandler "(gd_parser_data_t *" pdata ", void *" extra );
.EC
.RE
.PP
The
.ARG extra
parameter is the pointer supplied to
.FN gd_cbopen ,
passed verbatim to this function.  It can be used to pass caller data to the
callback.  GetData does not inspect this pointer, not even to check its
validity.  If the caller needs to pass no data to the callback, it may be NULL.

The
.B gd_parser_data_t
type is a structure with at least the following members:

.in +4n
.fam C
.nf
typedef struct {
  const DIRFILE* dirfile;
  int suberror;
  int linenum;
  const char* filename;
  char* line;
  size_t buflen;

  ...
} gd_parser_data_t;
.fi
.fam
.in
.PP
The
.SPM pdata dirfile
member will be a pointer to a DIRFILE object suitable only for passing to
.FN gd_error_string .
Notably, the caller should not assume this pointer will be the same as the
pointer eventually returned by
.FN gd_cbopen ,
nor that it will be valid after the callback function returns.

The
.SPM pdata suberror
parameter will be one of the following symbols indicating the type of syntax
error encountered:
.DD GD_E_FORMAT_ALIAS
The parent specified for a meta field was an alias.
.DD GD_E_FORMAT_BAD_LINE
The line was indecipherable.  Typically this means that the line contained
neither a reserved word, nor a field type.
.DD GD_E_FORMAT_BAD_NAME
The specified field name was invalid.
.DD GD_E_FORMAT_BAD_SPF
The samples-per-frame of a RAW field was out-of-range.
.DD GD_E_FORMAT_BAD_TYPE
The data type of a RAW field was unrecognised.
.DD GD_E_FORMAT_BITNUM
The first bit of a BIT field was out-of-range.
.DD GD_E_FORMAT_BITSIZE
The last bit of a BIT field was out-of-range.
.DD GD_E_FORMAT_CHARACTER
An invalid character was found in the line, or a character escape sequence was
malformed.
.DD GD_E_FORMAT_DUPLICATE
The specified field name already exists.
.DD GD_E_FORMAT_ENDIAN
The byte sex specified by an
.B /ENDIAN
directive was unrecognised.
.DD GD_E_FORMAT_LITERAL
An unexpected character was encountered in a complex literal. 
.DD GD_E_FORMAT_LOCATION
The parent of a metafield was defined in another fragment.
.DD GD_E_FORMAT_META_META
An attempt was made to use a metafield as the parent to a new metafield.
.DD GD_E_FORMAT_METARAW
An attempt was made to add a RAW metafield.
.DD GD_E_FORMAT_MPLEXVAL
A MPLEX specification has a negative period.
.DD GD_E_FORMAT_N_FIELDS
The number of fields of a LINCOM field was out-of-range.
.DD GD_E_FORMAT_N_TOK
An insufficient number of tokens was found on the line.
.DD GD_E_FORMAT_NO_FIELD
The parent of a metafield was not found.
.DD GD_E_FORMAT_NUMBITS
The number of bits of a BIT field was out-of-range.
.DD GD_E_FORMAT_PROTECT
The protection level specified by a
.B /PROTECT
directive was unrecognised.
.DD GD_E_FORMAT_RES_NAME
A field was specified with the reserved name
.IR INDEX
(or with the reserved name
.IR FILEFRAM
in a dirfile conforming to Standards Version 5 or earlier).
.DD GD_E_FORMAT_UNTERM
The last token of the line was unterminated.
.DD GD_E_FORMAT_WINDOP
The operation in a WINDOW field was not recognised.
.PP
.SPM pdata filename
and
.SPM pdata linenum
members contains the pathname of the fragment and line number where the syntax
error was encountered.  The first line in a fragment is line one.
.PP
The
.SPM pdata line
member contains a copy of the line containing the syntax error.  This line may
be freely modified by the callback function.  It will then be reparsed if the
callback function returns the symbol
.B GD_SYNTAX_RESCAN
(see below).  The size of the memory buffer, which may be greater than the
length of the actual string, is provided in
.SPM pdata buflen\fR ,
and space is available for at least
.B GD_MAX_LINE_LENGTH
bytes.

If the callback function returns
.BR GD_SYNTAX_RESCAN ,
then a different buffer, which may be larger, may be used to hold the new
string, by assigning a pointer to the new buffer to
.SPM pdata line\fR .
This buffer will be deallocated by the library using the
.ARG free
function specified through
.F3 gd_alloc_funcs ,
or else
.F3 free
by default.  Do not deallocate the original buffer passed to the callback
through
.SPM pdata line\fR :
it, too, will be deallocated by the library.

The callback function should return one of the following symbols, which
tells the parser how to subsequently handle the error:
.DD GD_SYNTAX_ABORT
The parser should immediately abort parsing the format specification and fail
with the error
.BR GD_E_FORMAT .
This is the default behaviour, if no callback function is provided (or if
the parser is invoked by calling
.FN gd_open ).
.DD GD_SYNTAX_CONTINUE
The parser should continue parsing the format specification.  However, once
parsing has finished, the parser will fail with the error
.BR GD_E_FORMAT ,
even if no further syntax errors are encountered.  This behaviour may be used by
the caller to identify all lines containing syntax errors in the format
specification, instead of just the first one.
.DD GD_SYNTAX_IGNORE
The parser should ignore the line containing the syntax error completely, and
carry on parsing the format specification.  If no further errors are
encountered, the dirfile will be successfully opened.
.DD GD_SYNTAX_RESCAN
The parser should rescan the
.I line
argument, which replaces the line which originally contained the syntax error.
The line is assumed to have been corrected by the callback function.  If the
line still contains a syntax error, the callback function will be called again.

Note: the line is not corrected on disk; however, the caller may subsequently
correct the fragment on disk by calling
.F3 gd_rewrite_fragment .
.PP
The callback function handles only syntax errors.  The parser may still abort
early, if a different kind of library error is encountered.  Furthermore,
although a line may contain more than one syntax error, the parser will only
ever report one syntax error per line, even if the callback function returns
.BR GD_SYNTAX_CONTINUE .

.SH RETURN VALUE
A call to
.FN gd_cbopen
or
.FN gd_open
always returns a pointer to a newly allocated DIRFILE object, except in
instances when it is unable to allocate memory for the DIRFILE object itself,
in which case it will return NULL.  The DIRFILE object is an opaque structure
containing the parsed dirfile metadata.

If an error occurred, these functions will store a negative-valued error
code in the returned DIRFILE, which may be retrieved by a subsequent call to
.F3 gd_error .
Possible error codes are:
.DD GD_E_ACCMODE
The library was asked to truncate a dirfile opened read-only (i.e.
.B GD_TRUNC
was specified in
.ARG flags
along with
.BR GD_RDONLY ).
.DD GD_E_ALLOC
The library was unable to allocate memory.
.DD GD_E_BAD_REFERENCE
The reference field specified by a
.B /REFERENCE
directive in the format specification (see
dirfile-format(5))
was not found, or was not a
.B RAW
field.
.DD GD_E_CALLBACK
The registered callback function,
.ARG sehandler ,
returned an unrecognised response.
.DD GD_E_CREAT
The library was unable to create the dirfile.
.DD GD_E_EXISTS
The dirfile already exists and both
.BR GD_CREAT " and " GD_EXCL
were specified.
.DD GD_E_FORMAT
A syntax error occurred in the format specification.  See also
.B The Callback Function
section above.
.DD GD_E_IO
The dirfile format file, or another file that it includes, could not be read,
or
.ARG dirfilename
does not specify a valid dirfile.
.DD GD_E_LINE_TOO_LONG
The parser encountered a line in the format specification longer than it was
able to deal with.  Lines are limited by the storage size of
.BR ssize_t .
On 32-bit systems, this limits format specification lines to 2**31 bytes.  The
limit is larger on 64-bit systems.
.PP
A DIRFILE which is returned from a failed open is flagged as invalid, meaning
most functions it is passed to will faill with the error
.BR GD_E_BAD_DIRFILE .
A descriptive error string for the error may be obtained by calling
.F3 gd_error_string .

When no longer needed, the caller should de-allocate any returned DIRFILE object
by calling
.F3 gd_close ,
or
.F3 gd_discard ,
even if the open failed.

.SH BUGS
When working with dirfiles conforming to Standards Versions 4 and earlier
(before the introduction of the
.B /ENDIAN
directive), GetData assumes the dirfile has native byte sex, even though,
officially, these early Standards stipulated data to be little-endian.  This is
necessary since, in the absence of an explicit
.B /VERSION
directive, it is often impossible to determine the intended Standards Version of
a dirfile, and the current behaviour is to assume native byte sex for modern
dirfiles lacking
.BR /ENDIAN .
To read an old, little-ended dirfile on a big-ended platform, an
.B /ENDIAN
directive should be added to the format specification, or else
.B GD_LITTLE_ENDIAN
should be specified by the caller.

GetData's parser assumes it is running on an ASCII-compatible platform.  Format
specification parsing will fail gloriously on an EBCDIC platform.

.SH HISTORY
The
.FN dirfile_open
function appeared in GetData-0.3.0.  The only supported flags were
.BR GD_BIG_ENDIAN ,
.BR GD_CREAT ,
.BR GD_EXCL ,
.BR GD_FORCE_ENDIAN ,
.BR GD_LITTLE_ENDIAN ,
.BR GD_PEDANTIC ,
.BR GD_RDONLY ,
.BR GD_RDWR ,
and
.BR GD_TRUNC .

The
.BR GD_AUTO_ENCODED ,
.BR GD_FORCE_ENCODING ,
.BR GD_SLIM_ENCODED ,
.BR GD_TEXT_ENCODED ,
.BR GD_UNECODED ,
and
.B GD_VERBOSE
flags appeared in GetData-0.4.0.

The
.FN dirfile_cbopen
function and the
.BR GD_BZIP2_ENCODED ,
.BR GD_GZIP_ENCODED ,
and
.BR GD_IGNORE_DUPS
flags appeared in GetData-0.5.0.

The
.B GD_PRETTY_PRINT
and
.B GD_LZMA_ENCODED
flags appeared in GetData-0.6.0.

In GetData-0.7.0 these functions were renamed to
.FN gd_open
and
.FN gd_cbopen .
The
.BR GD_ARM_ENDIAN ,
.BR GD_NOT_ARM_ENDIAN ,
and
.B GD_PERMISSIVE
flags also appeared in this release.

The
.BR GD_SIE_ENCODED ,
.BR GD_TRUNCSUB ,
.BR GD_ZZIP_ENCODED ,
and
.B GD_ZZSLIM_ENCODED
flags appeared in GetData-0.8.0.

The
.B GD_FLAC_ENCODED
flag appeared in GetData-0.9.0.

.SH SEE ALSO
.F3 gd_alloc_funcs ,
.F3 gd_close ,
.F3 gd_dirfile_standards ,
.F3 gd_discard ,
.F3 gd_error ,
.F3 gd_error_string ,
.F3 gd_flags ,
.F3 gd_getdata ,
.F3 gd_include ,
.F3 gd_parser_callback ,
.F3 gd_verbose_prefix ,
dirfile(5), dirfile-encoding(5), dirfile-format(5)