PnetCDF Installation Guide
=====================

1.  Getting Started
2.  Alternate Configure Options
3.  Testing the PnetCDF installation
4.  Reporting Installation or Usage Problems


-------------------------------------------------------------------------

1. Getting Started
==================

The following instructions take you through a sequence of steps to get the
default configuration of PnetCDF up and running.

(a) You will need the following prerequisites.

    - REQUIRED: This compressed tar file

    - REQUIRED: An MPI C compiler

    - REQUIRED: GNU m4 (https://www.gnu.org/software/m4/m4.html)

    - OPTIONAL: An MPI C++ compiler, if C++ applications are to be used.
      If you do not require support for C++ applications, you can disable this
      support using the configure option --disable-cxx (configuring PnetCDF is
      described in step 1(d) below).

    - OPTIONAL: An MPI Fortran 77 compiler, if Fortran 77 applications are to
      be used. If you do not require support for Fortran 77 applications, you
      can disable this support using --disable-fortran (configuring PnetCDF is
      described in step 1(d) below).

    - OPTIONAL: An MPI Fortran 90 compiler, if Fortran 90 applications are to
      be used. If you do not require support for Fortran 90 applications, you
      can disable this support using --disable-fortran. Note that Fortran 77
      support is a prerequisite for Fortran 90 support (configuring PnetCDF is
      described in step 1(d) below).

    Also, you need to know what shell you are using since different shell has
    different command syntax. Command "echo $SHELL" prints out the current
    shell used by your terminal program.

(b) Unpack the tar file and go to the top level directory, for example:

      gzip -dc pnetcdf-_PNETCDF_VERSION_.tar.gz | tar -xf -
      cd pnetcdf-_PNETCDF_VERSION_

(c) Choose an installation directory, for example $HOME/PnetCDF

(d) Configure PnetCDF specifying the installation directory:

      ./configure --prefix=$HOME/PnetCDF

    * To use an MPI C compiler installed in a non-default location, add and
      set the environment variable MPICC in the same command line, e.g.

      ./configure --prefix=$HOME/PnetCDF MPICC=/path/to/mpicc

    * Similarly, to use MPI C++ and Fortran compilers installed in a
      non-default location, add and set the environment variables MPICXX,
      MPIF77, or MPIF90 in the same command line, e.g.

      ./configure --prefix=$HOME/PnetCDF \
                  MPICC=/path/to/mpicc   MPICXX=/path/to/mpicxx \
                  MPIF77=/path/to/mpif77 MPIF90=/path/to/mpifort

(e) Build PnetCDF:

      make

   Or if "make" runs slow, try parallel make, e.g. (using 8 simultaneous jobs)

      make -j8

   For quiet mode, use the command below will produce minimum output on screen.

      make -s LIBTOOLFLAGS=--silent V=1 -j8

(f) Install PnetCDF

      make install

    If a non-default install directory is desired, use command:

      make install prefix=/OTHER/INSTALL/DIRECTORY

(g) Add the bin subdirectory of the installation directory to your path
    environment variable in your startup script (.bashrc for bash, .cshrc for
    csh, etc.):

    for csh and tcsh:

      setenv PATH $HOME/PnetCDF/bin:$PATH

    for bash and sh:

      PATH=$HOME/PnetCDF/bin:$PATH ; export PATH

    Check that everything is in order at this point by doing:

      which ncmpidump
      which ncmpidiff

    These commands should display the path to your bin subdirectory of your
    install directory.

If you have completed all of the above steps, you have successfully installed
PnetCDF.

-------------------------------------------------------------------------

2. Alternate Configure Options
=================

PnetCDF has a number of configure features.  A complete list of configuration
options can be found using:

   ./configure --help

Here lists a few important options:

  --prefix=PREFIX         install architecture-independent files in PREFIX
                          [/usr/local]
  --exec-prefix=EPREFIX   install architecture-dependent files in EPREFIX
                          [PREFIX]
  --disable-cxx           Turn off support for the C++ interface, if you only
                          need the C interface. [default: enabled]
  --disable-fortran       Turn off support for the Fortran interface, if you
                          only need the C interface. [default: enabled]
  --enable-shared[=PKGS]  build shared libraries [default=no]
  --enable-static[=PKGS]  build static libraries [default=yes]
  --enable-debug          Enable PnetCDF internal debug mode. This also enables
                          safe mode. [default: disabled]
  --(en/dis)able-in-place-swap
                          Enable/disable memory in-place byte swap on Little
                          Endian machines. [default: auto]
  --enable-subfiling      Enable subfiling support. [default: disabled]
  --enable-thread-safe    Enable thread-safe capability. [default: disabled]
  --disable-erange-fill   Disable use of fill value when out-of-range type
                          conversion causes NC_ERANGE error. [default: enabled]
  --disable-relax-coord-bound
                          Use stricter rule for error NC_EINVALCOORDS to
			  disallow coordinate start argument equal to dimension
			  size when argument count is zero. [default: enabled]
  --enable-doxygen        Enable generation of documentation. [default:
                          disabled]
  --enable-large-file-test
                          Enable testing for large (>4GB) file/variable I/O.
                          Note "make check" can run very slow. [default:
                          disabled]
  --enable-null-byte-header-padding
			  Enable check for null-byte header padding when
			  reading files in classic formats. [default: disabled]
  --enable-burst-buffering
			  Enable burst buffer driver support. [default:
			  disabled]
  --enable-profiling      Enable time and memory profiling. [default: disabled]
  --disable-versioning    Disable library versioning. [default: enabled]

Optional Packages:

  --with-mpi[=DIR]        Provide the MPI installation path in DIR.
  --with-pthread=DIR      Search Pthreads library within the supplied path
                          DIR, when --enable-thread-safe is enabled.
  --with-netcdf4[=INC,LIB | =DIR]
                          Enable NetCDF-4 feature and provide the NetCDF-4
                          installation path(s): --with-netcdf4=INC,LIB for
                          include and lib paths separated by a comma.
                          --with-netcdf4=DIR for the path containing include/
                          and lib/ subdirectories. [default: disabled]
  --with-adios[=DIR]      Enable ADIOS feature and provide the ADIOS
                          installation path in DIR. [default: disabled]

Some influential environment variables:
  RM          Command for deleting files or directories. [default: rm]
  MPICC       MPI C compiler, [default: CC]
  MPICXX      MPI C++ compiler, [default: CXX]
  MPIF77      MPI Fortran 77 compiler, [default: F77]
  MPIF90      MPI Fortran 90 compiler, [default: FC]
  CC          C compiler command (used only if MPICC is not set)
  CFLAGS      C compiler flags
  LDFLAGS     linker flags, e.g. -L<lib dir> if you have libraries in a
              nonstandard directory <lib dir>
  LIBS        libraries to pass to the linker, e.g. -l<library>
  CPPFLAGS    (Objective) C/C++ preprocessor flags, e.g. -I<include dir> if
              you have headers in a nonstandard directory <include dir>
  CXX         C++ compiler command (used only if MPICXX is not set)
  CXXFLAGS    C++ compiler flags
  LT_SYS_LIBRARY_PATH
              User-defined run-time library search path.
  CPP         C preprocessor
  CXXCPP      C++ preprocessor
  FC          Fortran compiler command (used only if MPIF90 is not set)
  FCFLAGS     Fortran compiler flags
  F77         Fortran 77 compiler command (used only if MPIF77 is not set)
  FFLAGS      Fortran 77 compiler flags
  GCOV        name/path for the gcov utility
  TESTSEQRUN  Run command (on one MPI process) for "make check" on a cross-
              compile environment. Example: "aprun -n 1". [default: none]
  TESTMPIRUN  MPI run command for "make ptest", [default: mpiexec -n NP]
  TESTOUTDIR  Output file directory for "make check" and "make ptest",
              [default: ./]
  SEQ_CC      C compiler for building sequential utility programs,
              [default: gcc]
  SEQ_CFLAGS  C compile flags for building sequential utility programs,
              [default: none]
  SEQ_LDFLAGS Linker flags for building sequential utility programs,
              [default: none]
  SEQ_LIBS    Libraries for building sequential utility programs,
              [default: none]

Use these variables to override the choices made by `configure' or to help
it to find libraries and programs with nonstandard names/locations.

PnetCDF can automatically detect the available MPI compilers and compile flags.
If alternate compilers or flags are desired, they can be specified by the
following environment variables and/or configure options.

Some influential environment variables:
    CFLAGS, CPPFLAGS, CXXFLAGS, FFLAGS, FCFLAGS, LDFLAGS and LIBS
    Setting these compile flags would result in the PnetCDF library being built
    with these flags.

    MPICC, MPICXX, MPIF77, MPIF90
    Setting these variables would result in the PnetCDF library being built
    with these compilers. CC, CXX, F77, FC, and F90 will be overwritten by
    their corresponding MPI compiler variables.

    Note the compile flags, such as -O2 or -g, should be given in CFLAGS and
    other flag environment variables. Please do not set them in compiler
    variable. For instance, setting MPICC="mpicc -O2" may result in the error
    of compiler not found.

 - For platform-specific build instructions, see one of the README.<ARCH>
   files in directory doc/.


----
Note on configure option "--(en/dis)able-in-place-swap"
----
   This option enables/disables the byte-swap operations running in-place on
   the user's write buffers. The purpose of providing this option is to deal
   with the problem when a Fortran program uses a immutable buffer for put
   APIs, e.g.  the buffer is declared as a PARAMETER, and in-place byte swap
   on this buffer causes segmentation fault. See discussion threads of
   http://lists.mcs.anl.gov/pipermail/parallel-netcdf/2013-July/001498.html

   Impacts:
      1. It takes effect only on Little Endian machines, as no byte swap is
         necessary on Big Endian machines.
      2. It only affects put/iput data APIs, but not attribute APIs.
      3. The INTENT of buffer arguments in all Fortran 90 put/iput APIs will be
         declared as "IN". Without this setting, the default is "INOUT".
      4. It has an impact on performance, as an extra internal temporary buffer
         will be allocated to copy data over from user's put buffer, so byte
         swap can be run on the temporary buffer.

   The default setting is auto. The three options are explained below.
      --enable-in-place-swap : perform byte swap on user I/O buffers whenever
      possible. This option results in the least amount of internal memory
      usage. However, if an immutable user buffer is used, segmentation fault
      may occur when byte swap is performed on user buffer in place.

      --disable-in-place-swap : when byte swap is required, an internal memory
      space will be allocated for byte swap without altering the contents of
      user buffer. This option will increase internal memory footprint.

      The default (neither the above option is used at the command line): an
      internal buffer is allocated for byte swap only when the I/O request
      size is less than 4 KiB. For larger requests, user buffers are used for
      in-place byte swap.  The mode can also be changed at the run time by
      setting the PnetCDF hint nc_in_place_swap.


   PnetCDF tries not to allocate additional memory space, due to performance
   concern. Users are discouraged to use Fortran PARAMETER buffers in put
   APIs.

----
Note on configure option "--disable-erange-fill"
----
   Prior to 1.8.0, data type conversion for out-of-bound elements proceeds
   even if the out-of-bound error is detected, and the contents (in both files
   and read buffers) are left up to the user to handle. This behavior conforms
   with NetCDF. See the URL below for details.
   http://www.unidata.ucar.edu/software/netcdf/docs/group__error.html

   Starting from 1.8.0, when the out-of-bound error is detected, instead of
   proceeding the type conversion, PnetCDF fills the out-of-bound data elements
   with their "fill values", as specified by the variable's attribute
   "_FillValue" or default CDF fill values if the attribute is not defined.
   Specifically, for PnetCDF put APIs, the value of attribute _FillValue, if
   defined, will be used when writing to files. For get APIs, the default CDF
   _FillValue corresponding to the internal data type will be used to fill the
   read buffer. This feature can be disabled by adding option
   "--disable-erange-fill" to the configure command line. In this case, the
   conventional NetCDF method described above is used.

----
Note on configure option "--enable-relax-coord-bound"
----
   Starting from version 1.8.0, a new configure option
   --enable-relax-coord-bound is added to provide a more relaxed boundary check
   for argument start. Traditionally, both NetCDF and PnetCDF detect the error
   code NC_EINVALCOORDS when the value of start is larger than or equal to the
   defined dimension size. This can be inconvenient for some applications that
   make a zero-length request with start being the dimension size, usually the
   last process. The new configure option relaxes the boundary check by
   allowing the above exception, i.e. start[i] can be the size of dimension i
   only when count[i] is zero. The default of this feature is disabled. For
   more detailed information, see discussion in:
   http://lists.mcs.anl.gov/pipermail/parallel-netcdf/2016-March/001839.html

   Starting from version 1.10.0, the default of this option is changed to
   enabled, in accord with NetCDF 4.7.0. See discussion in:
   https://github.com/Unidata/netcdf-c/issues/1010

-------------------------------------------------------------------------

3. Testing the PnetCDF installation
==================================

Two type of testings are implemented in PnetCDF: sequential and parallel.
* For testing sequential programs, the command is
     make check
* For testing parallel programs, the command is
     make ptest
  which uses 4 MPI processes to run all test programs
* For more elaborated parallel testing, command
     make ptests
  will use up to 10 MPI processes to run all test programs.

Note the above commands can be run before "make install".

Command "make tests" will build executables of all the test programs. This can
be handy if testing must run through a batch job system. Having the testing
executables built before submitting a batch job could save a lot of time.

There are three environment variables that can be used to run "make check" and
"make ptest" on a cross compile platform.
 - TESTMPIRUN : command to launch MPI jobs. default: mpiexec -n NP
 - TESTSEQRUN : command to run MPI executable sequentially. default: none
 - TESTOUTDIR : output directory. default: ./

Examples:
    make check TESTOUTDIR=/scratch
    make ptest TESTMPIRUN="aprun -n NP" TESTOUTDIR=/scratch

Note the keyword "NP" used in the environment variable string TESTMPIRUN.  It
will be replaced with the different numbers of MPI processes used in testing.
Currently, the testing uses up to 8 processes. Hence, please make sure the
process allocation at least contains 8 processes.

One can also run "make ptest" on batch queue systems. One example PBS script is
provide "pbs.script" in directory doc/. It is recommended to build all the
testing executables before submitting the batch job. This can be done by
running the below command.
    make tests

Note on setting TESTOUTDIR. In order to run parallel test correctly, the
output directory must be on a file system accessible to all MPI processes.  We
recommend to use parallel file systems or POSIX compliant file systems (Using
NFS will most likely fail the parallel test.)

Issue with older MPI-IO implementation.
    During "make check", one may encounter the following error message:
    "MPI error (MPI_File_delete) : MPI_ERR_IO: input/output error"
    This is due to the underneath MPI-IO libraries fail to return the correct
    error class MPI_ERR_NO_SUCH_FILE when trying to delete a non-existing file.
    This message can be ignored.

-------------------------------------------------------------------------

4. Reporting Installation or Usage Problems
===========================================

Please send an email to parallel-netcdf@mcs.anl.gov