<?xml version="1.0"?>
<!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN" 
               "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd">
<refentry id="gwymodule-tutorial-file" revision="$Id: module-tutorial-file.xml 20682 2017-12-18 18:39:00Z yeti-dn $">
  <refmeta>
    <refentrytitle>File Modules</refentrytitle>
    <manvolnum>3</manvolnum>
    <refmiscinfo>Gwyddion</refmiscinfo>
  </refmeta>

  <refnamediv>
    <refname>File Modules</refname>
    <refpurpose>
      More about file modules
    </refpurpose>
  </refnamediv>

  <refsect1>
    <title>Overview</title>
    <para>
      File modules implement loading and saving of SPM files.  The structure
      and operation of file modules will be explained on a simple but complete
      sample file module.  This module imports a sample data format that was
      construed for this tutorial and does not occur in practice.  However,
      the format resembles many (binary) data formats used in practice,
      except it contains no device and manufacturer information as they
      are not essential for the example.
    </para>
    <para>
      The sample file format our module will load looks as follows, all
      values are stored in little-endian:
      <informaltable frame='none'>
        <tgroup cols='4'>
          <thead>
            <row>
              <entry>Position</entry>
              <entry>Type</entry>
              <entry>Name</entry>
              <entry>Description</entry>
            </row>
          </thead>
          <tbody>
            <row>
              <entry>0x00</entry>
              <entry><type>char</type>[4]</entry>
              <entry><varname>magic</varname></entry>
              <entry>Magic header <literal>"SMPL"</literal></entry>
            </row>
            <row>
              <entry>0x04</entry>
              <entry><type>uint16</type></entry>
              <entry><varname>xres</varname></entry>
              <entry>Pixels per line</entry>
            </row>
            <row>
              <entry>0x06</entry>
              <entry><type>uint16</type></entry>
              <entry><varname>yres</varname></entry>
              <entry>Number of lines</entry>
            </row>
            <row>
              <entry>0x08</entry>
              <entry><type>float</type></entry>
              <entry><varname>measure</varname></entry>
              <entry>Size of one pixel [nm]</entry>
            </row>
            <row>
              <entry>0x0c</entry>
              <entry><type>float</type></entry>
              <entry><varname>z0</varname></entry>
              <entry>Value corresponding to 0 in raw data [nm]</entry>
            </row>
            <row>
              <entry>0x10</entry>
              <entry><type>float</type></entry>
              <entry><varname>gain</varname></entry>
              <entry>
                Conversion factor from raw data to physical values [nm]
              </entry>
            </row>
            <row>
              <entry>0x14</entry>
              <entry><type>uint16</type>[<varname>xres</varname>*<varname>yres</varname>]</entry>
              <entry><varname>data</varname></entry>
              <entry>Raw data values</entry>
            </row>
          </tbody>
        </tgroup>
      </informaltable>
      Furthermore we will assume files of this type get the extension
      <filename>.ssd</filename> (Simple SPM Data).  The complete module source
      code follows, it will be described in detail below.
    </para>
    <informalexample><programlisting>
&num;include &lt;string.h&gt;
&num;include &lt;stdio.h&gt;
&num;include &lt;glib/gstdio.h&gt;
&num;include &lt;libgwyddion/gwymacros.h&gt;
&num;include &lt;libgwyddion/gwymath.h&gt;
&num;include &lt;libprocess/stats.h&gt;
&num;include &lt;libgwymodule/gwymodule-file.h&gt;
&num;include &lt;app/gwymoduleutils-file.h&gt;

&num;include "err.h"

&num;define EXTENSION ".ssd"

&num;define MAGIC "SMPL"
&num;define MAGIC_SIZE (sizeof(MAGIC) - 1)

enum { HEADER_SIZE = MAGIC_SIZE + 2*2 + 3*4 };

typedef struct {
    guint xres;
    guint yres;
    gdouble measure;
    gdouble z0;
    gdouble gain;
} SimpleFile;

static gboolean      module_register(void);
static gint          simple_detect  (const GwyFileDetectInfo *fileinfo,
                                     gboolean only_name);
static GwyContainer* simple_load    (const gchar *filename,
                                     GwyRunType mode,
                                     GError **error);

static GwyModuleInfo module_info = {
    GWY_MODULE_ABI_VERSION,
    &amp;module_register,
    N_("Imports simple data files."),
    "J. Random Hacker &lt;hacker.jr&commat;example.org&gt;",
    "1.0",
    "Bit Rot Inc.",
    "2006",
};
GWY_MODULE_QUERY(module_info)

static gboolean
module_register(void)
{
    gwy_file_func_register("simple",
                           N_("Simple AFM files (.afm)"),
                           (GwyFileDetectFunc)&amp;simple_detect,
                           (GwyFileLoadFunc)&amp;simple_load,
                           NULL,
                           NULL);

    return TRUE;
}

static gint
simple_detect(const GwyFileDetectInfo *fileinfo,
              gboolean only_name)
{
    if (only_name)
        return g_str_has_suffix(fileinfo->name_lowercase, EXTENSION) ? 20 : 0;

    if (fileinfo->buffer_len > MAGIC_SIZE
        &amp;&amp; memcmp(fileinfo->head, MAGIC, MAGIC_SIZE) == 0)
        return 100;

    return 0;
}

static GwyContainer*
simple_load(const gchar *filename,
            G_GNUC_UNUSED GwyRunType mode,
            GError **error)
{
    SimpleFile simple;
    GwyContainer *container = NULL;
    GwySIUnit *unit;
    GwyDataField *dfield;
    guchar *buffer;
    const guchar *p;
    GError *err = NULL;
    gsize size, expected_size;
    const guint16 *rawdata;
    gdouble *data;
    gint i;

    if (!gwy_file_get_contents(filename, &amp;buffer, &amp;size, &amp;err)) {
        err_GET_FILE_CONTENTS(error, &amp;err);
        g_clear_error(&amp;err);
        return NULL;
    }

    if (size &lt;= HEADER_SIZE) {
        err_TOO_SHORT(error);
        gwy_file_abandon_contents(buffer, size, NULL);
        return NULL;
    }

    if (memcmp(buffer, MAGIC, MAGIC_SIZE) != 0) {
        err_FILE_TYPE(error, "Simple");
        gwy_file_abandon_contents(buffer, size, NULL);
        return NULL;
    }

    p = buffer + MAGIC_SIZE;
    simple.xres = gwy_get_guint16_le(&amp;p);
    simple.yres = gwy_get_guint16_le(&amp;p);
    simple.measure = gwy_get_gfloat_le(&amp;p);
    simple.z0 = gwy_get_gfloat_le(&amp;p);
    simple.gain = gwy_get_gfloat_le(&amp;p);

    expected_size = 2*simple.xres*simple.yres + HEADER_SIZE;
    if (size != expected_size) {
        err_SIZE_MISMATCH(error, expected_size, size);
        gwy_file_abandon_contents(buffer, size, NULL);
        return NULL;
    }

    simple.measure *= 1e-9;
    simple.z0 *= 1e-9;
    simple.gain *= 1e-9;
    dfield = gwy_data_field_new(simple.xres, simple.yres,
                                simple.xres*simple.measure,
                                simple.yres*simple.measure,
                                FALSE);
    gwy_convert_raw_data(p, simple.xres*simple.yres, 1,
                         GWY_RAW_DATA_UINT16, GWY_BYTE_ORDER_LITTLE_ENDIAN,
                         gwy_data_field_get_data(dfield),
                         simple.gain, simple.offset);

    unit = gwy_si_unit_new("m");
    gwy_data_field_set_si_unit_xy(dfield, unit);
    g_object_unref(unit);

    unit = gwy_si_unit_new("m");
    gwy_data_field_set_si_unit_z(dfield, unit);
    g_object_unref(unit);

    container = gwy_container_new<!--x-->();
    gwy_container_set_object_by_name(container, "/0/data", dfield);
    gwy_container_set_string_by_name(container, "/0/data/title",
                                     g_strdup("Topography"));
    g_object_unref(dfield);

    gwy_file_channel_import_log_add(container, 0, "simple", filename);

    gwy_file_abandon_contents(buffer, size, NULL);

    return container;
}
</programlisting></informalexample>
  </refsect1>

  <refsect1>
    <title>Administrative and Conventions</title>
    <para>
      Beside standard headers our module includes a special header that
      provides some common inline functions
    </para>
    <informalexample><programlisting>
&num;include "err.h"
    </programlisting></informalexample>
    <para>
      It is located in <filename>modules/file</filename> for the use by
      in-tree modules.  External modules can copy it (or only the
      actually used functions) to their source code.
    </para>
    <para>
      Header <filename>err.h</filename> contains common error reporting
      functions (called <function>err_ERROR_TYPE</function>).
      Its primary purpose it to ensure consistency of error
      messages between file modules and to ease translators' work.
    </para>
    <para>
      Next we define a few convenience symbols: the file extension, the magic
      header and its size, and header size
    </para>
    <informalexample><programlisting>
&num;define EXTENSION ".ssd"

&num;define MAGIC "SMPL"
&num;define MAGIC_SIZE (sizeof(MAGIC) - 1)

enum { HEADER_SIZE = MAGIC_SIZE + 2*2 + 3*4 };
    </programlisting></informalexample>
    <para>
      We also define a structure representing the imported file.  It is a bit
      superfluous in our module as we do not pass this information around, but
      it is useful in many modules
    </para>
    <informalexample><programlisting><![CDATA[
typedef struct {
    guint xres;
    guint yres;
    gdouble measure;
    gdouble z0;
    gdouble gain;
} SimpleFile;
]]></programlisting></informalexample>
  </refsect1>

  <refsect1>
    <title>Registration</title>
    <para>
      The feature registration is similar to data processing functions,
      there are just no menu paths and stock icons, on the other hand there are
      more functions the module can provide: one for each file operation.  Our
      module implements only file detection and import therefore we pass
      %NULL for the save and export functions.
    </para>
    <informalexample><programlisting><![CDATA[
static gboolean
module_register(void)
{
    gwy_file_func_register("simple",
                           N_("Simple AFM files (.afm)"),
                           (GwyFileDetectFunc)&simple_detect,
                           (GwyFileLoadFunc)&simple_load,
                           NULL,
                           NULL);

    return TRUE;
}
]]></programlisting></informalexample>
    <para>
      It is necessary to provide at least one of load, save and export
      operations, otherwise the file type would be rejected.  It is highly
      recommended to provide a detection function, although it is not
      required.  If no detection function exists for a file type the user
      has to always explicitly request the type to load and/or save files
      in this format.
    </para>
  </refsect1>

  <refsect1>
    <title>Detection</title>
    <para>
      Two types of detection exist: on load when we have the file
      available, and on save when the file does not exist and we know only
      the requested file name.  They are differentiated by
      <varname>only_name</varname> argument.  In both cases the detection
      function gets a #GwyFileDetectInfo structure, but if
      <varname>only_name</varname> is %FALSE, some fields are unset.
    </para>
    <informalexample><programlisting><![CDATA[
static gint
simple_detect(const GwyFileDetectInfo *fileinfo,
              gboolean only_name)
{
    if (only_name)
        return g_str_has_suffix(fileinfo->name_lowercase, EXTENSION) ? 20 : 0;

    if (fileinfo->buffer_len > MAGIC_SIZE
        && memcmp(fileinfo->head, MAGIC, MAGIC_SIZE) == 0)
        return 100;

    return 0;
}
]]></programlisting></informalexample>
    <para>
      The latter case is usually trivial, we just compare the extension with
      the normal extension of our file type.
    </para>
    <para>
      The former case can be more complex.  File name does not tell us much
      about file type: consider just the number of formats using extension
      <filename>.afm</filename> (not talking about <filename>.dat</filename>
      which can be essentially anything).  Therefore we use file names only
      as the last resort and base the detection on actual file contents.
    </para>
    <para>
      Many formats have well defined magic headers, precisely for the puropose
      of easy detection.  If out format has one, we are lucky and the detection
      code can look as simple as in the example.  To detect a particular
      version or type of a file format a more detailed analysis is necessary,
      but not substantially different from the simple case.
    </para>
    <para>
      The most problematic case arises when the format has no detectable magic
      header at all.  Often we can employ the following approach: read the
      header data and check whether it makes sense if interpreted as our
      file format.  For example we read the number of lines and pixels per
      line, calculate expected file size and compare it to the real file size.
      Or we read the physical dimensions and check they are positive (and
      approximately of expected magnitude).  If our detection routine requires
      significant processing or <structfield>head</structfield> and
      <structfield>tail</structfield> are not sufficient and we have to open
      the file ourselves, it is advisable to perform a quick check that can
      eliminate most files that are <emphasis>not</emphasis> of our type first,
      and then perform the sophisticated test only on the likely candidates.
    </para>
  </refsect1>

  <refsect1>
    <title>Loading</title>
    <para>
      The task of the load function is to read the file and create
      a #GwyContainer containing all importable data. It must not assume
      anything about the file properties and contents, not even that the file
      passed the detection because the user can request a file type
      explicitly, bypassing detection.  It has also deal with broken files,
      checking whether the file is long enough to contain the data it claims
      to and performing further parameter validation if necessary.
    </para>
    <para>
      The file contents can be often read with gwy_file_get_contents() which
      uses mmap() (if available).  The returned memory
      is read-only.  Sometimes destructive header parsing is useful, in such
      a case it is possible to use g_memdup() to duplicate the header,
      keeping the data part mapped read-only.
    </para>
    <informalexample><programlisting><![CDATA[
if (size <= HEADER_SIZE) {
    err_TOO_SHORT(error);
    gwy_file_abandon_contents(buffer, size, NULL);
    return NULL;
}
]]></programlisting></informalexample>
    <para>
      The header is parsed using the
      <link linkend="module-tutorial-file-portability">portable</link>
      binary buffer reading functions.
    </para>
    <informalexample><programlisting><![CDATA[
p = buffer + MAGIC_SIZE;
simple.xres = gwy_get_guint16_le(&p);
simple.yres = gwy_get_guint16_le(&p);
simple.measure = gwy_get_gfloat_le(&p);
simple.z0 = gwy_get_gfloat_le(&p);
simple.gain = gwy_get_gfloat_le(&p);
]]></programlisting></informalexample>
    <para>
      Then we can create the data field and read the data into it.  Since
      <application>Gwyddion</application> always works with values in base
      SI units, we have to convert the raw data values to physical values.
      If the file uses non-base units (which is common in SPM), we have to
      multiply the data by the corresponding factor: in this case with
      10<superscript>-9</superscript> as the physcial quantities are stored
      in nanometers in the file.
      Raw data conversion can be most easily performed with the convenience
      function gwy_convert_raw_data().  Sometimes it may be not sufficient
      so you can resort to the low-level GLib macros such as GUINT16_FROM_LE()
      or Gwyddion get-functions gwy_get_gdouble_be().
    </para>
    <informalexample><programlisting><![CDATA[
simple.measure *= 1e-9;
simple.z0 *= 1e-9;
simple.gain *= 1e-9;
dfield = gwy_data_field_new(simple.xres, simple.yres,
                            simple.xres*simple.measure,
                            simple.yres*simple.measure,
                            FALSE);
gwy_convert_raw_data(p, simple.xres*simple.yres, 1,
                     GWY_RAW_DATA_UINT16, GWY_BYTE_ORDER_LITTLE_ENDIAN,
                     gwy_data_field_get_data(dfield),
                     simple.gain, simple.offset);
]]></programlisting></informalexample>
    <para>
      We have to set also the lateral and value units of the created data
      field (by default, the dimensions and values are unitless).
    </para>
    <informalexample><programlisting><![CDATA[
unit = gwy_si_unit_new("m");
gwy_data_field_set_si_unit_xy(dfield, unit);
g_object_unref(unit);

unit = gwy_si_unit_new("m");
gwy_data_field_set_si_unit_z(dfield, unit);
g_object_unref(unit);
]]></programlisting></informalexample>
    <para>
      Our sample file format can only store topography channels, therefore
      the units are always meters.  For different types of channels the units
      have to be set accordingly.  Note that
    <informalexample><programlisting><![CDATA[
unit = gwy_si_unit_new("nA");
]]></programlisting></informalexample>
      is exactly the same as
    <informalexample><programlisting><![CDATA[
unit = gwy_si_unit_new("A");
]]></programlisting></informalexample>
      In other words, the prefixes are ignored because #GwySIUnit represents
      the unit type and contains no magnitude information.  However, if the
      format stores the units as a string including the prefix we can use
      gwy_si_unit_new_parse() to obtain the corresponding power of 10 (and
      subsequently multiply data field values with the correct multiplier).
    </para>
    <informalexample><programlisting><![CDATA[
unit = gwy_si_unit_new_parse("nA", &power10);
]]></programlisting></informalexample>
    <para>
      Finally we create a new #GwyContainer, put the data field at
      <literal>"/0/data"</literal> and set the channel title
      <literal>"/0/data/title"</literal> (if the file contains explicit
      channel names, they are good candidates for a channel title, otherwise
      we settle for channel type as Topography or Current), and return the
      container.  The application then takes care of adding it to the data
      browser and showing it in a data window.  Calling
      gwy_file_channel_import_log_add() for channel 0 adds a data operation
      log entry, noting that the data originated by import from the
      <literal>"simple"</literal> format.
    </para>
    <informalexample><programlisting>
container = gwy_container_new<!--x-->();
gwy_container_set_object_by_name(container, "/0/data", dfield);
gwy_container_set_string_by_name(container, "/0/data/title",
                                 g_strdup("Topography"));

g_object_unref(dfield);

gwy_file_channel_import_log_add(container, 0, "simple", filename);

gwy_file_abandon_contents(buffer, size, NULL);

return container;
  </programlisting></informalexample>
  </refsect1>

  <refsect1>
    <title>Loading More</title>
    <para>
      Real file formats often contain more information we would like to
      import.  First, they can contain several data fields (channels or even
      unrelated images).  We store the second data at
      <literal>"/1/data"</literal> and its title at
      <literal>"/1/data/title"</literal>, the third channel data at
      <literal>"/2/data"</literal>, etc.  The number sequence does not have
      to be contignuous but it is customary to make it so (in any case
      small integers should be used, do not start counting from 12758933).
    </para>
    <para>
      Auxiliary information about the data as date and time of scanning, scan
      speed, tunneling current, tip oscillation frequency or user-defined
      comments can be stored to metadata.  Metadata is a nested #GwyContainer
      at <literal>"/0/meta"</literal> (for the first channel, the number of
      course corresponds to channel id).  The keys are simply names of the
      values, the values are always strings.  Note both keys and values have
      to be converted to UTF-8.
    </para>
    <informalexample><programlisting>
meta = gwy_container_new<!--x-->();
gwy_container_set_string_by_name(meta, "Comment",
                                 g_strdup(myfile-&gt;user_comment));
gwy_container_set_string_by_name(meta, "Tip oscillation frequency",
                                 g_strdup_printf("&percnt;g Hz", myfile-&gt;freq));
gwy_container_set_object_by_name(container, "/0/meta", meta);
g_object_unref(meta);
    </programlisting></informalexample>
  </refsect1>

  <refsect1 id="module-tutorial-file-portability">
    <title>Portability</title>
    <para>
      Gwyddion runs on many platforms and can be compiled with various
      compilers.  File import is the area most affected by the differences
      between them, therefore certain care must be taken to make file modules
      work across all platforms.
    </para>
    <refsect2>
      <title>Text Data</title>
      <para>
        Text files can encode <emphasis>line ends</emphasis>
        differently from the platform Gwyddion is running on.  Therefore
        opening files as text may lead to unexpected results.  One usually
        opens all files as binary and uses gwy_str_next_line () to parse text
        files (or text parts of files) to lines.
      </para>
      <para>
        The text representation of <emphasis>real numbers</emphasis> is
        locale-dependent.  User's locale is unpredictable and a module must not
        set locale to arbitrary values because it would affect the rest of the
        application.  Therefore one has to use functions as g_ascii_strtod()
        to read real numbers portably.
      </para>
      <para>
        Many text strings, like comments and remarks, are stored in an
        arbitrary <emphasis>character encoding</emphasis>.
        Typically it is ISO 8895-1 (Latin1) or UTF-16, but sometimes more
        exotic encodings are used.  Since Gwyddion uses UTF-8 for all text data
        (like Gtk+ does) they must be converted, e.g., with g_convert().
      </para>
    </refsect2>
    <refsect2>
      <title>Binary Data</title>
      <para>
        The canonical bad example is
      </para>
      <informalexample><programlisting><![CDATA[
struct {
    char c;
    LONG i;
    wchar_t remark[20];
} header;

fread(&header, sizeof(header), 1, filehandle);
]]></programlisting></informalexample>
      <para>
        For start, <type>LONG</type> is unlikely to exist on non-Microsoft
        platforms.  So should we fix it to <type>long int</type>?  No, the
        <emphasis>type size</emphasis> of classic C types (<type>short</type>,
        <type>long</type>, etc.) is different on different architectures (more
        precisely ABIs) and the standard prescribes only the lower limit. If we
        mean `32bit integer' we also have to express it in our code:
        #gint32.
      </para>
      <para>
        Likewise, <type>wchar_t</type> size is not well-defined.   If the
        file comes from a Microsoft platform, where <type>wchar_t</type> is
        used a lot, it is probably 16bit, so we write #gunichar2 instead
        and use g_utf16_to_utf8() to get UTF-8.
      </para>
      <para>
        Then we have <emphasis>byte order</emphasis>.  If we are on a platform
        with different native byte order than the file uses, we have to reverse
        the bytes in each multi-byte value.  GLib provides macros such as
        GUINT16_FROM_LE() to simplify this task.
      </para>
      <para>
        And last but not least, different ABIs prescribe different
        <emphasis>structure padding</emphasis>.  Typically there can be
        from 0 to 7 unused bytes between <structfield>c</structfield> and
        <structfield>i</structfield> fields.  While most compilers provide
        some packing <literal>&num;pragma</literal>s or other means to
        specify desired structure padding, the portability of these
        constructs is questionable at best.
      </para>
      <para>
        The answer to (almost) all the problems above is: Don't do that then.
        Avoid direct reading of structures, read the data into a flat byte
        buffer instead and use the
        <link linkend="libgwyapp-file-module-utils">file module utils</link>
        functions to obtain individual values.
      </para>
    </refsect2>
  </refsect1>
</refentry>