Proposed revised
                        DRAFT CBF/imgCIF DEFINITION

                                  Revisions
                                     by
                            Herbert J. Bernstein
           Bernstein + Sons, P.O. Box 177, Bellport, NY 11713-0177
                        yaya@bernstein-plus-sons.com

                                  based on

                            DRAFT CBF DEFINITION

                                     by
                               Andy Hammersley
  European Synchrotron Radiation Facility, BP 200, Grenoble, 38043, CEDEX,
                                   France
                             hammersley@esrf.fr

  ------------------------------------------------------------------------

   This document and the CBF definitions are still subject to change. This
                document is a draft proposal for discussion.

This is an version of the CBF draft proposal, revised to include a
coordinated pure ASCII ImgCIF definition, based on the Draft CBF Definition
by Andy Hammersley, the work done at the Brookhaven imgCIF workshop, and the
work on "CBFLIB: An ANSI-C API for Crystallographic Binary File" by Paul
Ellis, ellis@SSRL.SLAC.STANFORD.EDU. For the binary CBF format, a
"binary-string" approach, as proposed by Paul Ellis, is used, while for the
ASCII ImgCIF format, binary information is encoded using a variant on MIME
(Multipurpose Internet Mail Extensions) format, which makes the CBF and
ImgCIF formats very similar.

We have included an updated version of John Westbrook's DDL2-compliant CBF
Extensions Dictionary, of Paul Ellis's CBFLIB manual, and examples of
CBF/imgCIF files.

This is just a proposal. My apologies in advance, especially to Andy, John
and especially to Paul for whatever I may have muddled here. Please be
careful about basing any code on this until and unless there has been a
general agreement.
  ------------------------------------------------------------------------
  ------------------------------------------------------------------------

                                   Notices

 Please read the NOTICES, which are part of this package, before making use
                             of this software.
  ------------------------------------------------------------------------
  ------------------------------------------------------------------------

Most of this document is adapted from Andy's, so we follow his convention by
"...[separating] the definition from comments on discussion items by using
round brackets to refer to notes kept separate from the main text e.g. (1)
refers to point 1 in the notes section.". Major differences from Andy's
draft are noted by comments bracketed by <<< >>> pairs.

  ------------------------------------------------------------------------

                              A Draft Proposal
                                     for
                                 A Combined
                     Crystallographic Binary File (CBF)
                                     and
         Image-supporting Crystallographic Information File (ImgCIF)
                                   Format

                                  ABSTRACT

This document describes a proposal for a combined Crystallographic Binary
File (CBF) and Image-supporting Crystallographic Information File (ImgCIF)
format; a simple self-describing binary format for efficient transport and
archiving of experimental data for the crystallographic community, and well
as for the presentation of other image data, such as PICT, GIFs and JPEG,
within publication CIFs. With minor differences, both the binary CBF format
and the ASCII ImgCIF have a similar, CIF-like structure. All the information
other than actual binary data is presented as ASCII strings in both formats.
The formats differ only in the handling of line termination and the actual
presentation of the binary data of an image. The CBF format, presents binary
information as a raw string of octets, while the ImgCIF format presents the
binary information as ASCII-encoded lines. The format of the binary file,
and the new CIF data- items are defined.

NOTE:

   o All numbers are decimal unless otherwise stated.
   o The terms octet and byte refer to a group of eight bits.

1.0 INTRODUCTION

The Crystallographic Binary File (CBF) format is a complementary format to
the Crystallographic Information File (CIF) [1], supporting efficient
storage of large quantities of experimental data in a self-describing binary
format (1). <<<The Image-supporting Crystallographic Information File
(ImgCIF) format is a proposed extension to CIF to assist in ASCII debugging
and archiving of CBF files and to allow for convenient and standardized
inclusion of images, such as maps, diagrams and molecular drawing into
publication CIFs.>>> It is our expectation that, for large images, the raw
binary CBF format will be used both with in laboratories and for interchange
among collaborating groups. For smaller chunks of binary data, either format
should be be suitable, with the ASCII ImgCIF format being more appropriate
for interchange and archiving.

The initial aim is to support efficient storage of raw experimental data
from area-detectors (images) with no loss of information compared to
existing formats. The format should be both efficient in terms of writing
and reading speeds, and in terms of stored file sizes, and should be simple
enough to be easily coded, or ported to new computer systems.

Flexibility and extensibility are required, and later the storage of other
forms of data may be added without affecting the present definitions.

The aims are achieved by a simple file format, consisting of lines of ASCII
information defining information about the binary data as CIF tag-value
pairs and tables, and either raw octets of binary data in delimited
sections, or ASCII-based presentations of the same binary information in
similarly delimited sections.

The present version of the format only tries to deal with simple Cartesian
data. This is essentially the "raw" data from detectors that is typically
stored in commercial formats or individual formats internal to particular
institutes, but could be other forms of data. It is hoped that CBF can
replace individual laboratory or institute formats for "home" built detector
systems, be used as a inter-program data exchange format, and may be offered
as an output choice by a number of commercial detector manufacturers
specialising in X-ray and other detector systems.

This format does not imply any particular demands on processing software nor
on the manner in which such software should work. Definitions of units,
coordinate systems, etc. may quite different. The clear precise definitions
within CIF, and hence CBF, help, when necessary, to convert from one system
to another. Whilst no strict demands are made, it is clearly to be hoped
that software will make as much use as is reasonable of information relevant
to the processing which is stored within the file. It is expected that
processing software will give clear and informative error messages when they
encounter problems within CBF's or do not support necessary mechanisms for
inputting a file.

1.1 CBF and "imgCIF"

CBF and "imgCIF" are two aspects of the same format. Since CIF's are pure
ASCII text files, a separate binary format is needed to be defined to allow
the combination of pseudo-ASCII sections and binary data sections. The
binary file format is the Crystallographic Binary File (CBF). The ASCII
sections are very close to the CIF standard, but must use operating system
independent "line separators". In describing the ASCII sections, we use the
notation "\r\n" (for the pair of characters carriage return, line-feed) for
a line terminator would allow the ASCII sections to viewed with standard
system utilities on a very wide range of operating systems. However, an API
to read the binary format must accept any of the following three alternative
line terminators as the end of an ascii line: "\r", "\n" or "\r\n". An API
to write CBF should write "\r\n" as the line terminator, if at all possible.

imgCIF is also the name of the CIF dictionary which contains the terms
specific to describing the binary data (the orginal, designed by John
Westbrook, without the modifications in this proposal is avaliable from
http://ndbserver.rutgers.edu/NDB/mmcif. Thus a CBF or ImgCIF files uses data
names from the imgCIF dictionary and other CIF dictionaries.

2.0 A SIMPLE EXAMPLE

Before fully describing the format we start by showing a simple, but
important and complete usage of the format; that of storing a single
detector image in a file together with a small amount of useful auxiliary
information. It is intened to be a useful example for people who like
working from examples, as opposed to full definitions. It should also serve
as an introduction or overview of the format defintion. This example uses
CIF DDL2 based dictionary items.

The example is an image of 768 by 512 pixels stored as 16 bit unsigned
integers, in little endian byte order. (This is the native byte ordering on
a PC.) The pixel sizes are 100.5 by 99.5 microns. Comment lines starting
with a hash sign (#) are used to explain the contents of the header. Only
the ASCII part of the file is shown, but comments are used to describe the
start of the binary section.

First the file is shown with the minimum of comments that a typical
outputting program might add. Then it is repeated, but with "over-
commenting" to explain the format.

Here is how a file might appear if listed on a PC or on a Unix system using
"more":

###CBF: VERSION 0.3
# Data block for image 1
data_image_1

_entry.id 'image_1'


# Sample details
_chemical.entry_id                           'image_1'
_chemical.name_common                        'Protein X'

# Experimental details
_exptl_crystal.id                            'CX-1A'
_exptl_crystal.colour                        'pale yellow'

_diffrn.id                                    DS1
_diffrn.crystal_id                            'CX-1A'

_diffrn_measurement.diffrn_id                 DS1
_diffrn_measurement.method                    Oscillation
_diffrn_measurement.sample_detector_distance  0.15

_diffrn_radiation_wavelength.id               L1
_diffrn_radiation_wavelength.wavelength       0.7653
_diffrn_radiation_wavelength.wt               1.0

_diffrn_radiation.diffrn_id                   DS1
_diffrn_radiation.wavelength_id               L1

_diffrn_source.diffrn_id                      DS1
_diffrn_source.source                         synchrotron
_diffrn_source.type                          'ESRF BM-14'

_diffrn_detector.diffrn_id                    DS1
_diffrn_detector.id                           ESRFCCD1
_diffrn_detector.detector                     CCD
_diffrn_detector.type                        'ESRF Be XRII/CCD'

_diffrn_detector_element.id                   1
_diffrn_detector_element.detector_id          ESRFCCD1

_diffrn_frame_data.id                         F1
_diffrn_frame_data.detector_element_id        1
_diffrn_frame_data.array_id                   'image_1'
_diffrn_frame_data.binary_id                  1

# Define image storage mechanism

###<<<Note:  Deleted _array_structure.binary_id >>>

#

loop_
_array_structure.array_id
_array_structure.encoding_type
_array_structure.compression_type
_array_structure.byte_order
image_1       unsigned_16_bit_integer  none  little_endian

loop_
_array_intensities.array_id
_array_intensities.binary_id
_array_intensities.linearity
_array_intensities.undefined_value
_array_intensities.overload_value
image_1     1    linear     0      65535

# Define dimensionality and element rastering
loop_
_array_structure_list.array_id
_array_structure_list.index
_array_structure_list.dimension
_array_structure_list.precedence
_array_structure_list.direction
image_1    1      768    1    increasing
image_1    2      512    2    decreasing

loop_
_array_element_size.array_id
_array_element_size.index
_array_element_size.size
image_1  1  100.5e-6
image_1  2  99.5e-6

###<<<Note:  This is the new part >>>

loop_
_array_data.id
_array_data.binary_id
_array_data.data

image_1 1
;
--CIF-BINARY-FORMAT-SECTION--
Content-Type: application/octet-stream;
     conversions="x-CBF_PACKED"
Content-Transfer-Encoding: BINARY
X-Binary-ID: 1
Content-MD5: jGmkxiOetd9T/Np4NufAmA==

START_OF_BIN
*************'9*****`********* ...
[This is where the raw binary data would be -- we can't print it here]

--CIF-BINARY-FORMAT-SECTION----
;

<<<Andy's prior draft had

###_CRYSTALLOGRAPHIC_BINARY_FILE: VERSION 1.0

###_START_OF_HEADER

before the CIF tags describing a CBF and

###_END_OF_HEADER

###_START_OF_BIN

between the CIF tags and the binary data. At the end of the binary data,

###_END_OF_BINARY

###_END_OF_CBF

appeared to mark the end of the data. Since these are all valid CIF
comments, they may still be properly used, but in this proposal they are not
required. The beginning and end of binary data is now marked by a
combination of CIF text field markers (\n;) and a MIME-style header. The
first line has been changed to "###CBF: VERSION 0.3" >>>

Here the file is shown again, but this time with many comment lines added to
explain the format:

###CBF: VERSION 0.3

# This line starting with a "#" is a CIF and CBF comment line,
# but the first line with the three "#"s is a CBF identifier.
# (a "magic number")  The text "###_CBF: VERSION" identifies
# the file as a CBF and must be present as the very first line of
# every CBF file. Following "VERSION" is the version number of the
# file, which is the corresponding version of the CBF/imgCIF
# extensions dictionary and supporting documentation.   A version
# 0.3 CIF should be readable by any program which
# fully supports the version 1.0 CBF definitions.

# Comment lines and white space (blanks and new lines) may appear
# anywhere outside the binary sections.

# In a CIF, the descriptive tags and values may be presented in
# any convenient order, e.g. the data could come first, and the
# parameters necessary to interpret the data could come later.
# This order-independent convention holds for an imgCIF file, but
# for a CBF, all the tags and values describing binary data (i.e.
# all the tags other than those in the ARRAY_DATA category) should
# be presented before the binary data, in the form of a header.
# This does not mean that there cannot be more useful information
# after the binary data.  There could be another full header and
# more blocks of binary data.  All we are saying is that, in
# the interest of efficiency in processing a CBF, the parameters
# that relate to a particular block of binary data must appear
# earlier in the CBF than the block itself.

# The header begins with "data_", which is the CIF token to identify
# a data block.  Within a data block, any given tag may be presented
# only once, either directly with an immediately following value,
# or as one of the column headings for the rows of a table.  If you will
# need to resuse the same tag, you will have to start a new data block.

# Data block for image 1
data_image_1

# We've chosen to call this data block 'image_1', but this was an
# arbitary choice. Within a data block a data item may only be used
# once.

_entry.id 'image_1'

# Sample details
_chemical.entry_id                           'image_1'
_chemical.name_common                        'Protein X'

# The apostrophes enclose the string which contains a space.
# A double quote (") could have been used, just as well.
# There is also a third way to quote string, with the string
# "\n;", i.e. with a semicolon at the beginning of a line
# which allows multi-lined strings to be presented.  We'll
# use that form of text quotation for the binary data.

# Experimental details
_exptl_crystal.id                            'CX-1A'
_exptl_crystal.colour                        'pale yellow'

_diffrn.id                                    DS1
_diffrn.crystal_id                            'CX-1A'

_diffrn_measurement.diffrn_id                 DS1
_diffrn_measurement.method                    Oscillation
_diffrn_measurement.sample_detector_distance  0.15

_diffrn_radiation_wavelength.id               L1
_diffrn_radiation_wavelength.wavelength       0.7653
_diffrn_radiation_wavelength.wt               1.0

_diffrn_radiation.diffrn_id                   DS1
_diffrn_radiation.wavelength_id               L1

_diffrn_source.diffrn_id                      DS1
_diffrn_source.source                         synchrotron
_diffrn_source.type                          'ESRF BM-14'

_diffrn_detector.diffrn_id                    DS1
_diffrn_detector.id                           ESRFCCD1
_diffrn_detector.detector                     CCD
_diffrn_detector.type                        'ESRF Be XRII/CCD'

_diffrn_detector_element.id                   1
_diffrn_detector_element.detector_id          ESRFCCD1

_diffrn_frame_data.id                         F1
_diffrn_frame_data.detector_element_id        1
_diffrn_frame_data.array_id                   'image_1'
_diffrn_frame_data.binary_id                  1

# Many more data items can be defined, but the above gives the idea
# of a useful minimum set (but not minimum in the sense of compulsory,
# the above data items are optional in a CIF or CBF).

# Define image storage mechanism
#
# Notice that we did not include a binary ID here.  The idea of
# the ARRAY_STRUCTURE category is to present parameters which
# could be common to multiple blocks of binary data, which
# would all have the same array ID, but would have distinct
# binary ID's

loop_
_array_structure.array_id
_array_structure.encoding_type
_array_structure.compression_type
_array_structure.byte_order
image_1       unsigned_16_bit_integer  none  little_endian

# On the other hand, we do provide a binary ID for
# ARRAY INTENSITIES, since there might be different
# paremeters for each binary block.  We could have
# left it out here, since there is only one block and
# the default binary ID happens to be 1

loop_
_array_intensities.array_id
_array_intensities.binary_id
_array_intensities.linearity
_array_intensities.undefined_value
_array_intensities.overload_value
image_1     1   linear     0      65535

# Define dimensionality and element rastering

# Here the size of the image and the ordering (rastering) of the
# data elements is defined. The CIF "loop_" structure is used to
# define different dimensions. (It can be used for defining multiple
# images.)

loop_
_array_structure_list.array_id
_array_structure_list.index
_array_structure_list.dimension
_array_structure_list.precedence
_array_structure_list.direction
image_1    1      768    1    increasing
image_1    2      512    2    decreasing

loop_
_array_element_size.array_id
_array_element_size.index
_array_element_size.size
image_1  1  100.5e-6
image_1  2  99.5e-6

# The "array_id" identifies data items belong to the same array. Here
# we have chosen the name "image_1", but another name could have been
# used, so long as it's used consistently. The "index" component refers
# to the dimension being defined, and the "dimension" component defines
# the number of elements in that dimension. The "precedence" component
# defines which precedence of rastering of the data. In this case the
# first dimension is the faster changing dimension. The "direction"
# component tells us the direction in which the data rasters within a
# dimension. Here the data rasters faster from minimum elements towards
# the maximum element ("increasing") in the first dimension, and more
# slowly from the maximum element towards the minimum element in the
# second dimension. (This is the default rastering order.)

# The storage of the binary data is now fully defined.

# Further data items could be defined, but  we are ready to
# present the data.  That is done with the ARRAY_DATA category.
# The start of this category marks the end of the header
# (Well, almost the end, there is a bit more header information
# below).

loop_
_array_data.id
_array_data.binary_id
_array_data.data

image_1 1

# The binary data itself will come just a little further down,
# as the essential part of the value of _array_data.data, which begins
# as semicolon-quoted text.  The line immediately after the
# line with the semicolon is a MIME boundary marker.  As for
# all MIME boundary markers, it begins with "--".  The next
# few lines are MIME headers, describing some useful information
# we will need to process the binary section.  MIME headers can
# appear in different orders, and can be very confusing (look
# at the raw contents of a email message with attachments), but
# there is only one header which is has to be understood to
# process a CBF; "Content-Transfer-Encoding".  If the value given
# on this header is "BINARY", this is a CBF and the data will
# be presented as raw binary, containing a count (in yet another
# header we did not tell you about yet) so we'll know when to
# start looking for more information.
#

;
--CIF-BINARY-FORMAT-SECTION--
Content-Type: application/octet-stream;
     conversions="x-CBF_PACKED"
Content-Transfer-Encoding: BINARY
X-Binary-ID: 1
X-Binary-Size: 2432840
Content-MD5: jGmkxiOetd9T/Np4NufAmA==

START_OF_BIN
*************'9*****`********* ...
[This is where the raw binary data would be -- we can't print it here]

--CIF-BINARY-FORMAT-SECTION----
;

# After the last octet (i.e. byte) of the binary data, there is a
# special trailer "\n--CIF-BINARY-FORMAT-SECTION----\n;"
# which repeats the initial bounday marker with an extra "--"
# at the end (a MIME convention for the last boundary marker), and
# then the closing semicolon quote for a text section.  This
# is essential in an imgCIF, and we include it in a CBF for consistency.
#

OVERVIEW OF THE FORMAT

This section describes the major "components" of the CBF format.

  1. CBF is a binary file, containing self-describing array data e.g. one or
     more images, and auxiliary data e.g. describing the experiment.

  2. Except for the handling of line terminators, the way binary data is
     presented, and more liberal rules in ordinger information, an ASCII
     imgCIF file is the same as a CBF binary file.

  3. A CBF consists of pseudo-ASCII text header sections, which are "lines"
     of no more tha 80 ASCII characters separated by "line separators" which
     are the pair of ASCII characters carriage return and line-feed (ASCII
     13, ASCII 10), followed by zero, one, or more binary sections presented
     as "binary strings". This structure may be repeated.

  4. An imgCIF consists of ASCII lines of no more than 80 characters using
     the the normal line termination conventions of the current system (e.g.
     ASCII 10 in UNIX) with MIME-encoded binary strings at any appropriate
     point in the file.

  5. The very start of the file has an identification item (magic number)
     (2). This item also describes the CBF version or level. The identifier
     is:

     ###CBF: VERSION

     which must always be present so that a program can easily identify
     whether or not a file is a CBF, by simply inputting the first 15
     characters. (The space is a blank (ASCII 32) and not a tab. All
     identifier characters are uppercase only.)

     The first hash means that this line within a CIF would be a comment
     line, but the three hashes mean that this is a line describing the
     binary file layout for CBF. (All CBF internal identifiers start with
     the three hashes, and all other must immediately follow a "line
     separator".) No whitespace may precede the first hash sign.

     Following the file identifier is the version number of the file. e.g.
     the full line might appear as:

     ###CBF: VERSION 0.3

     The version number must be separated from the file identifier
     characters by whitespace e.g. a blank (ASCII 32).

     The version number is defined as a major version number and minor
     version number separated by the decimal point. A change in the major
     version may well mean that a program for the previous version cannot
     input the new version as some major change has occurred to CBF (3). A
     change in the minor version may also mean incompatibility, if the CBF
     has been written using some new feature. e.g. a new form of linearity
     scaling may be specified and this would be considered a minor version
     change. A file containing the new feature would not be readable by a
     program supporting only an older version of the format.

     <<< Until we reach major version 1 (the first official release), the
     rules are a little more relaxed. While there will be some effort at
     upwards compatability, in order to ensure a reasonable agreed
     specification without too many strange artifacts, changes between minor
     versions may, unfortunately, introduce incompatabilities which require
     program changes to still read CBFs compliant with an earlier draft,
     e.g. the change in the "magic number" and from binary sections to
     binary strings in going to version 0.3. Naturally, such changes should
     be sufficiently well documented to allow for conversions.>>>

  6. Header Information:

       a. The start of an header section is delimited by the usual CIF
          "data_" token. Optionally, the formerly specified header
          identifier,

          ###_START_OF_HEADER

          may be used before the "data_" taken, followed by the carriage
          return, line-feed pair, as an aid in debugging, but it is no
          longer required. (Naturally, another carriage return, line-feed
          pair should immediately precedes this and all other CBF
          identifiers, with the exception of the CBF file identifier which
          is at the very start of the file.)

       b. A header section, including the identification items which delimit
          it, uses only ASCII characters, and is divided into "lines". The
          "line separator" symbols, "\r\n" (carriage return, line-feed) are
          the same regardless of the operating system on which the file is
          written. (This is an importance difference with CIF, but must be
          so, as the file contains binary data, so cannot be translated from
          one O.S. to another, which is the case for ASCII text files.)
          While a properly functioning CBF API should write the full "\r\n"
          line separator, it should recognize any of three sequences "\r",
          "\n", "\r\n" as valid line separators, so that hand-edited headers
          will not be rejected.

       c. The header section within the delimiting identification items
          obeys all CIF rules [1] with the exception of the line separators.

          e.g.

             o "Lines" are a maximum of 80 characters long. (For CBF it is
               probably best to allow for this maximum to be larger.)

             o The tokens "data_" and "loop_" have special meaning to CIF,
               and should not be used except in their indicated places. The
               tokens "save_", "stop_" and "global_" also have special
               meaning to CIF's parent language, STAR, and also should not
               be used.

             o All data names (tags) start with an underscore character.

             o The hash symbol (#) (outside a character string) means that
               all text up to the line separator is a comment.

             o Whitespace outside of character strings is not significant.

             o Data names are case insensitive.

             o The data item follows the data name separator, and may be of
               one of two types: character string (char) or number (numb).
               (The type is specified for each data name.)

             o Character strings may be delimited with single of double
               quotes, or blocks of text may be delimited by semi-colons
               occurring as the first character on a line.

             o The "loop_" mechanism allows a data name to have multiple
               values. Immediately following the "loop_", one or more data
               names are listed without their values, as column headings.
               Then one or more rows of values are given.

          Any CIF data name may occur within the header section.

       d. A single header section may contain one or more data blocks (CIF
          terminology).

       e. The end of the header information is marked by comin the tags from
          the "ARRAY_DATA" category. The formerly specifier special
          identifier:

          ###_END_OF_HEADER

          followed by carriage return, line-feed, may be used as well as an
          aid to debugging, but it is not required.

  7. The header information must contain sufficient data names to fully
     describe the binary data section(s) which follow(s).

  8. Binary Information:

     <<<Under CBFlib "binary sections" have been replaced by "binary
     strings" values within a data name/value pair. The structure of the
     proposed "binary string" is similar to the former binary sections, but
     there are significant differences.>>>

       a. Before getting to the binary data, itself, there are some
          preliminaries to allow a smooth transition from the conventions of
          CIF to those of raw streams of "octets" (8-bit bytes). The binary
          data is given as the essential part of a specially formatted
          semicolon-delimited CIF multi-line text string. This text string
          is the value associated with the tag "_array_data.data&.

       b. Within that text string, the conventions developed for
          transmitting email messages including binary attachments are
          followed. There is secondary ASCII header information, formatted
          as Multipurpose Internet Mail Extensions (MIME) headers (see RFCs
          2045-49 by Freed, et. al). The bounday marker for the beginning of
          all this is the special string

          --CIF-BINARY-FORMAT-SECTION--

          at the beginning of a line. The initial "--" says that this is a
          MIME boundary. We cannot put "###" in front of it and conform to
          MIME conventions. Immediately after the boundary marker are MIME
          headers, describing some useful information we will need to
          process the binary section. MIME headers can appear in different
          orders, and can be very confusing (look at the raw contents of a
          email message with attachments), but there is only one header
          which is has to be understood to process a CBF;
          "Content-Transfer-Encoding". If the value given on this header is
          "BINARY", this is a CBF and the data will be presented as raw
          binary, containing a count (in yet another header we did not tell
          you about yet) so we'll know when to start looking for more
          information.

          If the value given for "Content-Tranfer-Encoding" is one of the
          real encodings: "BASE64", "QUOTED-PRINTABLE", "X-BASE8",
          "X-BASE10" or "X-BASE16", this file is an imgCIF, and we'll need
          some other the other headers to process the encoded binary data
          properly. It is a good practice to give headers in all cases

          The "Content-Type" header tells us what sort of data we have
          (almost always "application/octet-stream" for a miscellaneous
          stream of binary data) and, optionally, the conversions that were
          applied to the original data. In this case we have compressed the
          data with the "CBF-PACKED" algorithm.

          The optional "X-Binary-ID" header should contain the same value as
          was given for _array_data.binary-id above.

          The "X-Binary-Size" header gives the expected size of the binary
          data. This is the size after any compressions, but before any
          ascii encodings. This is useful in making a simple check for a
          missing portion of this file.

          The optional "Content-MD5" header provides a much more
          sophisticated check on the integrity of the binary data.

          In a CBF, the raw binary data begins after an empty line
          terminating the MIME headers and after the START_OF_BIN
          identifier. "START_OF_BIN" contains bytes to separate the "ASCII"
          lines from the binary data, bytes to try to stop the listing of
          the header, bytes which define the binary identifier which should
          match the "binary_id" defined in the header, and bytes which
          define the length of the binary section.

             Octet   Hex  Decimal              Purpose
           1         1A   26       (ctrl-Z) Stop listings in MS-DOS
           2         04   04       (Ctrl-D) Stop listings in UNIX
           3         D5   213      Binary section begins
           4..11                   Binary Section Identifier
                                   (See _array_data.binary_id)
                                   64-bit, little endian
           12..19                  8+ the size (n) of the
                                   binary section in octets
                                   (i.e. the offset from octet
                                   20 to the first byte following
                                   the data)
           20..27                  Compression type:
                                    CBF_NONE 0x0040 (64)

                                    CBF_CANONICAL 0x0050 (80)
                                    CBF_PACKED 0x0060 (96)
                                    ... &NBSP;

           28..28+n-1              Binary data (n octets)

          Only bytes 28..28+n-1 are encoded for an imgCIF file using the
          indicated Content-Transfer-Encoding.

          The binary characters serve specific purposes:

             o The Control-Z will stop the listing of the file on MS-DOS
               type operating systems.

             o The Control-D will stop the listing of the file on Unix type
               operating systems.

             o The unsigned byte value 213 (decimal) is binary 11010101.
               (Octal 325, and hexadecimal D5). This has the eighth bit set
               so can be used for error checking on 7-bit transmission. It
               is also asymmetric, but with the first bit also set in the
               case that the bit order could be reversed (which is not a
               known concern).

             o (The carriage return, line-feed pair before the START_OF_BIN
               and other lines can also be used to check that the file has
               not been corrupted e.g. by being sent by ftp in ASCII mode.)

             o Bytes 4-11 define the binary id of the binary data. This id
               is also used within the header sections, so that binary data
               definitions can be matched to the binary data sections.
               64-bits allows many many more binary data sections to be
               addressed than can conceivably be needed.

             o Bytes 12-19 define the length in bytes of the binary data
               plus the flag word for the compression type (8 bytes). This
               information is critical to recovering alignment with the CIF
               world, since the binary data could easily include bytes which
               look like "\n;" or the boundary marker. The use of 64 bits
               provides for enormous expansion from present images sizes,
               but volume and higher dimensional data may need more than
               32-bit sizes in the future.

               It is tempting to set this value to zero if this is the last
               binary information or header information in the file, but you
               could cause unpleasant warnings in code that expects to be
               able to find the rest of a CIF. This allows a program
               writing, for example, a single compressed image to avoid
               having to rewind the file to write the size of the compressed
               data. (For small files compression within memory may be
               practical, and this may not be an issue. However very large
               files exist where writing the compressed data "on the fly"
               may be the only realistic method.) This should only be done
               for internal use within a group, and a cleanup utility should
               be used to restore the missing data before exporting it to
               groups which may have difficulty processing this truncated
               file. In any case, it is recommended that this value be set,
               as it permits concatenation of files, and a file with a zero
               for this field is not a valid CBF.

               Since the data may have been compressed, knowing the numbers
               of elements and size of each element does not necessarily
               tell a program how many bytes to jump over, so here it is
               stored explicitly. This also means that the reading program
               does not have to decode information in the header section to
               move through the file.

               Bytes 20-27 hold the flag for the compression type. At
               present three are defined: CBF_NONE (for no compression),
               CBF_CANONICAL (for and entropy-coding scheme based on the
               canonical-code algorithm described by Moffat, et al.
               (International Journal of High Speed Electronics and Systems,
               Vol 8, No 1 (1997) 179-231)) and CBF_PACKED for a CCP4-style
               packing scheme. Flags for other compression schemes, such as
               the two in this document will be added to this list in the
               future.

       c. The "line separator" immediately precedes the "start of binary
          identifier", but blank spaces may be added prior to the preceding
          "line separator" if desired (e.g. to force word or block
          alignment).

       d. The binary data does not have to completely fill the bytes defined
          by the byte length value, but clearly cannot be greater than this
          value (except when the value zero has been stored, which means
          that the size is unknown, and no other headers follow). The values
          of any unused bytes are undefined.

       e. At exactly the byte following the full binary section as defined
          by the length value is the end of binary section identifier. This
          consists of the carriage return / line feed pair followed by:

          --CIF-BINARY-FORMAT-SECTION--
          ;

          with each of these lines followed by the carriage return / line
          feed pair. This brings us back into a normal CIF environment

          The first "line separator" separates the binary data from the
          pseudo-ASCII line.

          This identifier is in a sense redundant since the binary data
          length value tells the a program how many bytes to jump over to
          the end of the binary data. However, this redundancy has been
          deliberately added for error checking, and for possible file
          recovery in the case of a corrupted file.

          This identifier must be present at the end of every block of
          binary data.

  9. Whitespace may be used within the pseudo-ASCII sections prior to the
     "start of binary section" identifier to align the start binary data
     sections to word or block boundaries. Similar use may be made of unused
     bytes in binary sections. However, no blank lines should be introduced
     among the MIME headers, since that would terminate processing of those
     headers and start the scan for binary data.

     However, in general no guarantee is made of block nor word alignment in
     a CBF of unknown origin.

 10. The end of the file need not be not explicitly indicated, but including
     a comment of the form:

     ###_END_OF_CBF

     (including the carriage return, line-feed pair) can help in debugging.

 11. All binary data described in a single data block must follow the header
     section prior to another data block, or the end of the file.

     The binary identifier values used within a given data block section,
     and hence the binary data must be unique for any given array_id, and,
     it would be best to make them truly unique.

     A different data block may reuse binary identifier values.

     (This allows concatenation of files without re-numbering the binary
     identifiers, and provides a certain level of localization of data
     within the file, to avoid programs having to search potentially huge
     files for missing binary sections.)

 12. The recommended file extension for a CBF is: cbf
     This allows users to recognise file types easily, and gives programs a
     chance to "know" the file type without having to prompt the user.
     Although they should check for at least the file identifier to ensure
     that the file type is indeed a CBF.

 13. The recommended file extensions for imgCIF are: icf or cif
     (use of "cif" subject to IUCr approval).

 14. CBF format files are binary files and when ftp is used to transfer
     files between different computer systems "binary" or "image" mode
     transfer should be selected.

 15. imgCIF files are ASCII files and when ftp is used to transfer files
     between different computer systems "ascii" transfer should be selected.

3.1 SIMPLE EXAMPLE OF THE ORDERING OF IDENTIFIERS

Here only the ASCII part of the file structuring identifiers is shown. The
CIF data items are not shown, apart from the "data_" identifier which
indicates the beginning of a data block.

This shows the structuring of a simple example e.g. one header section
followed by one binary section. Such as could be used to store a single
image.

###CBF: VERSION 0.3

data_

### ... various CIF tags and values here

loop_
array_data.id
array_data.binary_id
array_data.data

image_1 1
;
--CIF-BINARY-FORMAT-SECTION--
Content-Type: application/octet-stream;
     conversions="x-CBF_PACKED"
Content-Transfer-Encoding: BINARY
X-Binary-ID: 1
Content-MD5: jGmkxiOetd9T/Np4NufAmA==

START_OF_BIN
*************'9*****`********* ...
[This is where the raw binary data would be -- we can't print it here]

--CIF-BINARY-FORMAT-SECTION----
;
###_END_OF_CBF

3.2 MORE COMPLICATED EXAMPLE OF THE ORDERING OF IDENTIFIERS

Here only the ASCII part of the file structuring identifiers is shown. The
CIF data items are not shown, apart from the "data_" identifier which
indicates the beginning of a data block.

This shows the a possible structuring of a more complicated example. Two
header sections, the first contains two data blocks and defines three binary
sections. CIF comment lines, starting with a hash (#) are used to example
the structure.

###CBF: VERSION 0.3

# A comment cannot appear before the file identifier, but can appear
# anywhere else, except within the binary sections.

# Here the first data block starts
data_

### ... various CIF tags and values here
###     but none that define array data items

# The "data_" identifier finishes the first data block and starts the
# second
data_

### ... various CIF tags and values here
###     including ones that define array data items

loop_
array_data.id
array_data.binary_id
array_data.data

image_1 1
;
--CIF-BINARY-FORMAT-SECTION--
Content-Type: application/octet-stream;
     conversions="x-CBF_PACKED"
Content-Transfer-Encoding: BINARY
X-Binary-ID: 1
Content-MD5: jGmkxiOetd9T/Np4NufAmA==

START_OF_BIN
*************'9*****`********* ...
[This is where the raw binary data would be -- we can't print it here]

--CIF-BINARY-FORMAT-SECTION----
;

# Following the "end of binary" identifier the file is pseudo-ASCII
# again, so comments are valid up to the next "start of binary"
# identifier.  Note that we have bumped the binary ID.

image_1 2
;
--CIF-BINARY-FORMAT-SECTION--
Content-Type: application/octet-stream;
     conversions="x-CBF_PACKED"
Content-Transfer-Encoding: BINARY
X-Binary-ID: 2
Content-MD5: xR5kxiOetd9T/Nr5vMfAmA==

START_OF_BIN
*************'9*****`********* ...
[This is where the raw binary data would be -- we can't print it here]

--CIF-BINARY-FORMAT-SECTION----
;

# Third binary section, note that we have a new array id.

image_2 3
;
--CIF-BINARY-FORMAT-SECTION--
Content-Type: application/octet-stream;
     conversions="x-CBF_PACKED"
Content-Transfer-Encoding: BINARY
X-Binary-ID: 3
Content-MD5: yS5kxiOetd9T/NrqTLfAmA==

START_OF_BIN
*************'9*****`********* ...
[This is where the raw binary data would be -- we can't print it here]

--CIF-BINARY-FORMAT-SECTION----
;

# Second Header section

data_

### ... various CIF tags and values here
###     including ones that define array data items

# Since we only have one block left, we won't use a loop

array_data.id         image
array_data.binary_id  1
array_data.data

# Note that I can put a comment here
;
--CIF-BINARY-FORMAT-SECTION--
Content-Type: application/octet-stream;
     conversions="x-CBF_PACKED"
Content-Transfer-Encoding: BINARY
X-Binary-ID: 1
Content-MD5: fooxiOetd9T/serNufAmA==

START_OF_BIN
*************'9*****`********* ...
[This is where the raw binary data would be -- we can't print it here]

--CIF-BINARY-FORMAT-SECTION----
;

###_END_OF_CBF

DATA NAME CATEGORIES

John Westbrook has proposed a number of data name categories as part of his
DDL2 based "imgCIF" dictionary. This category list may be expanded to cover
a structuring of the often multiple data-sets which might be used in a
structurial investigation. Here we only consider the categories concerned
with storing an image (or other N-dimensional topographically regular
cartesian grid).

The _array_* categories cover all data names concerned with the storage of
images or regular array data.

Data names from any of the existing categories may be relevant as auxiliary
information in the header section, but data names from the _diffrn_
category, are likely to be the most relevant, and a number of new data names
in this category are necessary.

The "array" Class of Binary Data

The "array" class is used to store regular arrays of data values, such as
1-D histograms, area-detector data, series of area-detector data, and volume
data. Normally such data is regularly spaced in space or time, however
spatial distorted data could nevertheless be stored in such a format. There
is only one data "value" stored per lattice position, although that value
may be of type complex.

The "array" class is defined by data names from the ARRAY_STRUCTURE and
ARRAY_STRUCTURE_LIST categories.

Here is a short summary of the data names and their purposes.

   * _array_structure.array_id: Alpha numeric identifier for the array
     structure
   * _array_structure.compression_type: Type of data compression used
   * _array_structure.byte_order: Order of bytes for multi-byte integer or
     reals
   * _array_structure.encoding_type: Native data type used to store
     elements.

     e.g. "unsigned_16_bit_integer" is used if the stored image was 16 bit
     unsigned integer values, regardless of any compression scheme used.

"Array" Dimensions and Element Rastering and Orientation

The array dimension sizes, i.e. the number of elements in each dimension are
defined by _array_structure_list.dimension. Which takes an integer value.
This is used in a loop together with the _array_structure_list.index item to
define the different dimensions for one or more arrays.

Fundamental to treating a long line of data values as a 2-D image or an
N-dimensional volume or hyper-volume is the knowledge of the manner in which
the values need to be wrapped. For the raster orientation to be meaningful
we define the sense of the view:

For a detector image the sense of the view is defined as that looking from
the crystal towards the detector.

(For the present we consider only an equatorial plane geometry, with 2-theta
= 0; the detector as being vertically mounted.)

The rastering is defined by the three data names
_array_structure_list.index, _array_structure_list.precedence, and
_array_structure_list.direction data names.

index refers to the dimension index i.e. In an image 1 refers to the
X-direction (horizontal), 2 refers to the Y-direction (vertical).

precedence refers to the order in which the data in wrapped.

direction refers the direction of the rastering for that index.

We define a preferred rastering orientation, which is the default if the
keyword is not defined. This is with the start in the upper-left-hand corner
and the fastest changing direction for the rastering horizontally, and the
slower change from top to bottom.

(Note: With off-line scanners the rastering type depending on which way
round the imaging plate or film is entered into the scanner. Care may need
to be taken to make this consistent.)

"Array_Structure" Examples

To define an image array of 1300 times 1200 elements, with the raster faster
in the first dimension, from left to right, and slower in the second
dimension from top to bottom, the following header section might be used:

# Define image size and rastering
loop_
_array_structure_list.array_id
_array_structure_list.index
_array_structure_list.dimension
_array_structure_list.precedence
_array_structure_list.direction
image_1    1      1300    1    increasing
image_1    2      1200    2    decreasing

To define two arrays, the first a volume of 100 times 100 times 50 elements,
fastest changing in the first dimension, from left to right, changing from
bottom to top in the second dimension, and slowest changing in the third
dimension from front to back; the second an image of 1024 times 1280 pixels,
with the second dimension changing fastest from top to bottom, and the first
dimension changing slower from left to right; the following header section
might be used:

# Define array sizes and rasterings
loop_
_ARRAY_STRUCTURE_LIST.ARRAY_ID
_ARRAY_STRUCTURE_LIST.INDEX
_ARRAY_STRUCTURE_LIST.DIMENSION
_array_structure.precedence
_array_structure.direction
volume_a    1      100    1    increasing
volume_a    2      100    2    increasing
volume_a    3       50    3    increasing
slice_1     1      1024   2    increasing
slice_1     2      1280   1    decreasing

"Array" Element Intensity Scaling

Existing data storage formats use a wide variety of methods for storing
physical intensities as element values. The simplest is a linear
relationship, but square root and logarithm scaling methods have attractions
and are used. Additionally some formats use a lower dynamic range to store
the vast majority of element values, and use some other mechanism to store
the elements which over-flow this limited dynamic range. The problem of
limited dynamic range storage is solved by the data compression methods
byte_offsets and predictor_huffman (see next Section), but the possibility
of defining non-linear scaling must also be provided.

The _array_intensities.linearity data item specifies how the intensity
scaling is defined. Apart from linear scaling, which is specified by the
value linear, two other methods are available to specify the scaling.

One is to refer to the detector system, and then knowledge of the
manufacturers method will either be known or not by a program. This has the
advantage that any system can be easily accommodated, but requires external
knowledge of the scaling system.

The recommended alternative is to define a number of standard intensity
linearity scaling methods, with additional data items when needed. A number
of standard methods are defined by _array_intensities.linearity values:
offset, scaling_offset, sqrt_scaled, and logarithmic_scaled. The "offset"
methods require the data item _array_intensities.offset to be defined, and
the "scaling" methods require the data item _array_intensities.scaling to be
defined. The above scaling methods allow the element values to be converted
to a linear scale, but do not necessarily relate the linear intensities to
physical units. When appropriate the data item _array_intensities.gain can
be defined. Dividing the linearized intensities by the value of
_array_intensities.gain should produce counts. Two special optional data
flag values may be defined which both refer to the values of the "raw"
stored intensities in the file (after decompression if necessary), and not
to the linearized scaled values. _array_intensities.undefined_value
specifies a value which indicates that the element value is not known. This
may be due to data missing e.g. a circular image stored in a square array,
or where the data values are flagged as missing e.g. behind a beam-stop.
_array_intensities.overload_value indicates the intensity value at which and
above, values are considered unreliable. This is usually due to saturation.

"Array_intensities" Example

To define the characteristics of image_1 as linear with a gain of 1.2, and
an undefined value of 0, and a saturated (overloaded) value of 65535, the
following header section might be used:

# Define image intensity scaling
loop_
_array_intensities.array_id
_array_intensities.binary_id
_array_intensities.linearity
_array_intensities.gain
_array_intensities.undefined_value
_array_intensities.overload_value
image_1    1    linear   1.2    0   65535

DATA COMPRESSION

One of the primary aims of imgCIF / CBF is to allow efficient storage, and
efficient reading and writing of data, so data compression is of great
interest. Despite the extra CPU over-heads it can very often be faster to
compress data prior to storage, as much smaller amounts of data need to be
written to disk, and disk I/O is relatively slow. However, optimum data
compression can result in complicated algorithms, and be highly data
specific.

In CBFlib version 0.1, Paul Ellis has coded two lossless compression
algorithms: canonical and packed.

Canonical-code compression

The canonical-code compression scheme encodes errors in two ways: directly
or indirectly. Errors are coded directly using a symbol corresponding to the
error value. Errors are coded indirectly using a symbol for the number of
bits in the (signed) error, followed by the error iteslf.

At the start of the compression, CBFLIB constructs a table containing a set
of symbols, one for each of the 2^n direct codes from -(2^(n-1)) .. 2^(n-1)
-1, one for a stop code, and one for each of the maxbits -n indirect codes,
where n is chosen at compress time and maxbits is the maximum number of bits
in an error. CBFLIB then assigns to each symbol a bit-code, using a shorter
bit code for the more common symbols and a longer bit code for the less
common symbols. The bit-code lengths are calculated using a Huffman-type
algorithm, and the actual bit-codes are constructed using the canonical-code
algorithm described by Moffat, et al. (International Journal of High Speed
Electronics and Systems, Vol 8, No 1 (1997) 179-231).

The structure of the compressed data is:

 Byte                                             Value

 1 .. 8                       Number of elements (64-bit little-endian
                              number)
 9 .. 16                      Minimum element
 17 .. 24                     Maximum element
 25 .. 32                     Repeat length (currently unused)
 33                           Number of bits directly coded, n
 34                           Maximum number of bits encoded, maxbits
 35 .. 35+2^n-1               Number of bits in each direct code

 35+2^n                       Number of bits in the stop code

 35+2^n+1 .. 35+2^n+maxbits-n Number of bits in each indirect code

 35+2^n+maxbits-n+1 ..        Coded data

CCP4-style compression

The CCP4-style compression writes the errors in blocks . Each block begins
with a 6-bit code. The number of errors in the block is 2^n, where n is the
value in bits 0 .. 2. Bits 3 .. 5 encode the number of bits in each error:
                          Value in  Number of bits
                         bits 3 .. 5 in each error
                              0            0
                              1            4
                              2            5
                              3            6
                              4            7
                              5            8
                              6            16
                              7            65

The structure of the compressed data is:

            Byte                        Value
          1 .. 8  Number of elements (64-bit little-endian number)
          9 .. 16 Minumum element (currently unused)
          17 .. 24Maximum element (currently unused)
          25 .. 32Repeat length (used, starting with version 0.2)
          33 ..   Coded data

Additional Compression Schemes

In addition, Andy Hammersley has proposed two types of lossless data
compression algorithms for CBF version 1.0. In later versions other types
including lossy algorithms may be added.

The first algorithm is referred to as byte_offsets and has been chosen for
the following characteristics: it is very simple, may be easily implemented,
and can easily lead to faster reading and writing to hard disk as the
arithmetic complication is very small. This algorithm can never achieve
better than a factor of two compression relative to 16-bit raw data, but for
most diffraction data the compression will indeed be very close to a factor
2.

The second algorithm is referred to as predictor_huffman and has been chosen
as it can achieve close to optimum compression on typical diffraction
patterns, with a relatively fast algorithm, whilst avoiding patent problems
and licensing fees. This will typically provide a compression ratio between
2.5 and 3 on well exposed diffraction images, and will achieve greater
ratios on more weakly exposed data e.g. 4 - 5 on "thin phi-slicing" images.
Normally, this would be a two pass algorithm; 1st pass to define symbol
probabilities; second pass to entropy encode the data symbols. However, the
Huffman algorithm makes it possible to use a fixed table of symbol codes, so
faster single pass compression may be implemented with a small loss in
compression ratio. With very fast cpus this approach may provide faster hard
disk reading and writing than the "byte_offsets" algorithm owing to the
smaller amounts of data to be stored.

There are practical disadvantages to data compression: the value of a
particular element cannot be obtained without calculating the values of all
previous elements, and there is no simple relationship between element
position and stored bytes. If generally the whole array is required this
disadvantage does not apply. These disadvantages can be reduced by
compressing separately different regions of the arrays, which is an approach
available in TIFF, but this adds to the complexity reading and writing
images.

For simple predictor algorithms such as the byte_offsets algorithm a simple
alternative is an optional data item, which defines a look-up table of
element addresses, values, and byte positions within the compressed data,
and it is suggested that this approach is followed.

THE "BYTE_OFFSETS" ALGORITHM

The byte_offsets algorithm will typically result in close to a factor of two
reduction in data storage size relative to typical 2-byte diffraction
images. It should give similar gains in disk I/O and network transfer. It
also has the advantage that integer values up to 32 bits (31 bits unsigned)
may be stored efficiently without the need for special over-load tables. It
is a fixed algorithm which does not need to calculate any image statistics,
so is fast.

The algorithm works because of the following property of almost all
diffraction data and much other image data: The value of one element tends
to be close to the value of the adjacent elements, and the vast majority of
the differences use little of the full dynamic range. However, noise in
experimental data means that run-length encoding is not useful (unless the
image is separated into different bit-planes). If a variable length code is
used to store the differences, with the number of bits used being inversely
proportional to the probability of occurrence, then compression ratios of
2.5 to 3.0 may be achieved. However, the optimum encoding becomes dependent
of the exact properties of the image, and in particular on the noise. Here a
lower compression ratio is achieved, but the resulting algorithm is much
simpler and more robust.

The byte_offsets algorithm is the following:

  1. The first element of the array is stored as a 4-byte signed two's
     integer regardless of the raw array element type. The byte order for
     this and all subsequent multi-byte integers is little_endian regardless
     of the native computer architecture i.e. the first byte is the least
     significant, and the last byte the most. This value is the first
     reference value ("previous element") for calculating pixel to pixel
     differences.

  2. For all elements, including the first element, the value of the
     previous element is subtracted to produce the difference. For the first
     element on a line the value to subtract is the value of the first
     element of the previous line. For the first element of a subsequent
     image (or plane) the value to subtract is the value of the first
     element of the previous image (or plane).

  3. If the difference is less than +-127, then one byte is used to store
     the difference as a signed two's complement integer, otherwise the byte
     is set to -128 (80 in hex) and if the difference is less than +-32767,
     then the next two bytes are used to store the difference as a signed
     two byte two's complement integer, otherwise -32768 (8000 in hex, which
     will be output as 00 80 in little-endian format) is written into the
     two bytes and the following 4-bytes store the difference as a full
     signed 32-bit two's complement integer.

  4. The array element order follows the normal ordering as defined by the
     _array_structure_list entries index, precedence and direction.

It may be noted that one element value may require up to 7 bytes for
storage, however for almost all 16-bit experimental data the vast majority
of element values will be within +-127 units of the previous element and so
only require 1 byte for storage and a compression factor of close to 2 is
achieved.

The PREDICTOR_HUFFMAN ALGORITHM

Section to be added.

OTHER SECTIONS

Other sections will be added.

9.0 REFERENCES

1. S R Hall, F H Allen, and I D Brown, "The Crystallographic Information
File (CIF): a New Standard Archive File for Crystallography", Acta Cryst.,
A47, 655-685 (1991)

10.0 NOTES

(1) A pure CIF based format has been considered inappropriate given the
enormous size of many raw experimental data-sets and the desire for
efficient storage, and reading and writing. <<< However, an ASCII format is
helpful for debugging software and in understanding what has been written in
a CBF when problems arise, and there are other CIF application for which a
convenience binary format should be useful (e.g. illustrations in a
manuscript). <<<

(2) Some simple method of checking whether the file is a CBF or not is
needed. Ideally this would be right at the start of the file. Thus, a
program only needs to read in n bytes and should then know immediately if
the file is of the right type or not. Andy though this identifier should be
some straightforward and clear ASCII string. <<< With the use of binary
strings and MIME conventions identification of a CBF versus a CIF is less
critical than it was before, but the distinct header as a simple ASCII
string is still a good idea for the sake of the most efficient processing of
large files.<<<

The underscore character has been used to avoid any ambiguity in the spaces.

(Such an identifier should be long enough that it is highly unlikely to
occur randomly, and if it is ASCII text, should be very slightly obscure,
again to reduce the chances that it is found accidently. Hence I added the
three hashes, but some other form may be equally valid.)

(3) The format should maintain backward compatibility e.g. a version 1.0
file can be read in by a version 1.1, 3.0, etc. program, but to allow future
extensions the reverse cannot be guaranteed to be true. <<< However, prior
to actual adoption of version 1.0, we are not yet trying to ensure full
upwards compatibility, just that the effort to convert won't be
unreasonable. <<<

  ------------------------------------------------------------------------

EXAMPLE CBF

  ------------------------------------------------------------------------

This page was produced on 14 November 1998
by Herbert J. Bernstein (email: yaya@bernstein-plus-sons.com),
based on the the 8 July version and the page produced by Andy Hammersley
(E-mail: hammersley@esrf.fr).

Further modification is highly likely, especially after Andy reads this and
finds all the mistakes.

  ------------------------------------------------------------------------