============
Introduction
============

Use Cases
=========

The design of the library has been driven by the following requirements:

1. Deduce the DRS directory structure a DRS-compliant filename. 
2. Validate filenames and directory paths against the DRS.
3. Convert the CMIP3 directory structure into a DRS-compliant form.
4. Manage multiple versions of DRS publication-level datasets on the filesystem.
5. Detect the CMIP5 product DRS component during data publication.


Installation
============

drslib is written in Python_.  It requires one of the installation
tools setuptools_ or pip_.  Other requirements are downloaded
automatically during installation.  Download and install with:

.. code-block:: bash

  # Using pip
  $ pip install drslib

  # OR using setuptools
  $ easy_install drslib

If you are upgrading drslib you will need to add the "-U" option to
either pip or easy_install.

.. pull-quote::

  **Note:** To install onto an ESG datanode you can use use the
  ``easy_install`` in ``/usr/local/uv-cdat/bin``.  This will install
  ``drslib`` into ``/usr/local/uv-cdat/bin/python`` and install
  ``drs_tool`` as ``/usr/local/uv-cdat/bin/drs_tool``.




Installing for development
--------------------------

The source of drslib is available on github from the ESGF `github repo`_.

Once you have the code you must activate the distribution
in `develop` mode.  To do this execute the following using a python
interpreter with setuptools_ installed:

.. code-block:: bash

  $ python setup.py develop


__ esgf-drslib.gitweb_


Configuration
=============

``drslib`` will use CMIP5 MIP tables to deduce realm and other
metadata.  MIP tables are not required for other projects as selected
with the ``-s`` scheme switch.  You can point to your MIP table location
using the environment variable ``MIP_TABLE_PATH``.  All tables should
be named ``CMIP5_*``.

Alternatively you can use metaconfig_ to configure the location of
your MIP tables.  Create the following in your ``$HOME/.metaconfig.conf`` file:

.. code-block:: ini

  [metaconfig]
  configs = drslib

  [drslib:tables]
  path = /path/to/mip/tables
  model_table = /path/to/model_table

.. _metaconfig: http://pypi.python.org/pypi/metaconfig

The option ``model_table`` is optional and if specified should point
to the "CMIP5 Modelling Group" spreadsheet in CSV format.  This table
is used to map model names to institute names.  The drslib package is
distributed with a recent version of this file.

Default DRS attributes used by the ``drs_tool`` command can be set in
the ``drs`` section:

.. code-block:: ini

  [drslib:drs]
  root = /path/to/drs/activity
  institute = MOHC
  model = HadCM3

It is usually convenient to set at least ``root`` and ``activity`` in
the configuration file.

Overriding DRS vocabularies
----------------------

Drslib ships with reasonable defaults for the DRS vocabularies,
however sometimes you will want to override or extend the defaults
supplied.  At this time drslib supports extending the vocabularies of
institutes, models and experiments.

To define all CMIP5 experiments including individual decadal
experiments you can define the ``drslib:vocabularies`` section as follows:

.. code-block:: ini

  [drslib:vocabularies]
  experiments = 
    1pctto2x 2xco2 pdcntrl sresa1b 1pctto4x amip picntrl sresa2 20c3m commit 
    slabcntl sresb1
    decadal1960 decadal1961 decadal1962 decadal1963 decadal1964 decadal1965 
    decadal1966 decadal1967 decadal1968 decadal1969 decadal1970 decadal1971 
    decadal1972 decadal1973 decadal1974 decadal1975 decadal1976 decadal1977 
    decadal1978 decadal1979 decadal1980 decadal1981 decadal1982 decadal1983 
    decadal1984 decadal1985 decadal1986 decadal1987 decadal1988 decadal1989 
    decadal1990 decadal1991 decadal1992 decadal1993 decadal1994 decadal1995 
    decadal1996 decadal1997 decadal1998 decadal1999 decadal2000 decadal2001 
    decadal2002 decadal2003 decadal2004 decadal2005 decadal2006 decadal2007 
    decadal2008 decadal2009

Institutes and models can be defined with the ``institutes`` option.
This is interpreted as a newline separated list of lines, each line
being the institute name a colon then space separated list of models.

.. code-block:: ini

  institutes =
    NOAA-GFDL:GFDL-ESM2G
    MOHC:HadGEM2-ES HadCM3 HadGEM2-CC



Logging
=======

``drslib`` uses Python's standard logging infrastructure to give
details of it's operation.  Messages are sent to loggers under the
``drslib`` logger.  You can configure logging via metaconfig by
pointing to a separate logging configuration file:

.. code-block:: ini

  [metaconfig]
  configs = drslib
  logging = /path/to/logging.conf

The format of ``logging.conf`` should conform to the Python logging
`configuration file format`__.  An example logging configuration is
given below which will log product detection decisions separately from
general drslib warnings:

.. code-block:: ini

   #
   # Basic logging configuration for drs_tool
   #
   # This configuration prints product detection decisions to STDERR and logs
   # warnings to ./drs_tool.log
   #

   [loggers]
   keys=root,drslib,p_cmip5

   [handlers]
   keys=drslib_h,p_cmip5_h

   [formatters]
   keys=f1,f2

   #---------------------------------------------------------------------------
   # Loggers

   # No catch-all logging
   [logger_root]
   handlers=
   level=NOTSET

   [logger_drslib]
   qualname=drslib
   handlers=drslib_h

   [logger_p_cmip5]
   qualname=drslib.p_cmip5
   handlers=p_cmip5_h
   propagate=0

   #---------------------------------------------------------------------------
   # Handlers & Formatters

   [handler_drslib_h]
   class=FileHandler
   args=('./drs_tool.log', )
   formatter=f1
   level=INFO

   [handler_p_cmip5_h]
   class=StreamHandler
   args=(sys.stderr, )
   formatter=f2
   level=INFO

   [formatter_f1]
   format=%(asctime)s [%(levelname)s] %(name)s: %(message)s
   datefmt=

   [formatter_f2]
   format=[%(levelname)s] %(name)s: %(message)s
                                                                        

__ http://docs.python.org/library/logging.html#configuration-file-format



Testing
=======

drslib ships with a test suite compliant with nose_.  The suite can be
run in various ways.  The test suite uses the extension NoseXUnit_ to
produce XML reports of the test results.  NoseXUnit will be
automatically installed if you run the tests via ``setup.py``::

  $ python setup.py test

Or if the depencencies are satisfied you can run all tests with::

  $ nosetests


Reporting Bugs
==============

Please report bugs to the `github repo`_.

.. _CMIP5: http://cmip-pcmdi.llnl.gov/cmip5/
.. _DRS: http://cmip-pcmdi.llnl.gov/cmip5/docs/cmip5_data_reference_syntax.pdf
.. _nose: http://somethingaboutorange.com/mrl/projects/nose
.. _setuptools: http://pypi.python.org/pypi/setuptools
.. _pip: http://pypi.python.org/pypi/pip
.. _NoseXUnit: http://pypi.python.org/pypi/NoseXUnit
.. _esgf-drslib.git: git://esgf.org/esgf-drslib.git
.. _esgf-drslib.gitweb: http://esgf.org/gitweb/?p=esgf-drslib.git;a=summary
.. _CEDA: http://www.ceda.ac.uk
.. _`Stephen Pascoe`: mailto:Stephen.Pascoe@stfc.ac.uk
.. _Python: http://www.python.org
.. _`esgf.org bugzilla`: http://esgf.org/bugzilla/enter_bug.cgi?product=drslib
.. _`github repo`: http://github.com/ESGF/esgf-drslib