%-------------------------------------------------------------------------------

% This file is part of Code_Saturne, a general-purpose CFD tool.
%
% Copyright (C) 1998-2018 EDF S.A.
%
% This program is free software; you can redistribute it and/or modify it under
% the terms of the GNU General Public License as published by the Free Software
% Foundation; either version 2 of the License, or (at your option) any later
% version.
%
% This program is distributed in the hope that it will be useful, but WITHOUT
% ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
% FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
% details.
%
% You should have received a copy of the GNU General Public License along with
% this program; if not, write to the Free Software Foundation, Inc., 51 Franklin
% Street, Fifth Floor, Boston, MA 02110-1301, USA.

%-------------------------------------------------------------------------------

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Short doc CS class corresponding to article
\documentclass[a4paper,10pt,twoside]{csshortdoc}
% Additional macros
\usepackage{csmacros}
\usepackage[usenames, dvipsnames]{color}
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Packages and commands for PDF documents and hyperlinks
\hypersetup{%
  pdftitle = {CodeSaturne installation guide},
  pdfauthor = {MFEE},
  pdfpagemode = UseOutlines
}
\pdfinfo{/CreationDate (D:20100802000000-01 00 )}
%
% To have thumbnails upon opening the document under ACROREAD
% pdfpagemode = UseThumbs
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Info for title pages
\titreCS{\CS version~\verscs installation guide}

\docassociesCS{}
\resumeCS{This document presents all the necessary elements to install
with \CS version \verscs.

\begin{center}
\large{WORK IN PROGRESS}
\end{center}
}
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Document start
\begin{document}

\def\contentsname{\textbf{\normalsize TABLE OF CONTENTS}\pdfbookmark[1]{Table of
contents}{contents}}

\renewcommand{\logocs}{cs_logo_wire_black}

\pdfbookmark[1]{Flyleaf}{pdg}
\large
\makepdgCS
\normalsize

\passepage

\begin{center}\begin{singlespace}
\tableofcontents
\end{singlespace}\end{center}

\section{\CS Automated or manual installation\label{sec:inst_types}}

\CS may be installed either directly through its GNU Autotools based
scripts (the traditional \texttt{configure}, \texttt{make}, \texttt{make install})
sequence), or using an automated installer (\texttt{install\_saturne.py}),
which generates an initial \texttt{setup} file when run a first time,
and builds and installs \CS and some optional libraries based on the
edited \texttt{setup} when run a second time. The use of this automated script
is briefly explained in the top-level \texttt{README} file of the \CS
sources, as well as in the comments of \texttt{setup} file. It is not detailed
further in this documentation, which details the manual installation, allowing
a finer control over installation options.

Note that when the automatic installer is run, it generates a build directory,
in which the build may be modified (re-running \texttt{configure}, possibly
adapting the command logged at the beginning of the \texttt{config.status}
file) and resumed.

\section{Installation basics\label{sec:install_basics}}

The installation scripts of \CS are based on the GNU Autotools,
(Autoconf, Automake, and Libtool), so it should be familiar for many
administrators. A few remarks are given here:

\begin{itemize}
\item As with most software with modern build systems, it is recommended
      to build the code in a separate directory from the sources. This
      allows multiple builds (for example production and debug), and is
      considered good practice. Building directly in the source tree is
      not regularly tested, and is not guaranteed to work, in addition
      to ``polluting'' the source directory with build files.
\item By default, optional libraries which may be used by \CS are
      enabled automatically if detected in default search paths
      (i.e. \texttt{/usr/} and \texttt{/usr/local}. To find libraries
      associated with a package installed in an alternate path,
      a \texttt{--with-<package>=...} option to the \texttt{configure} script
      must given. To disable the use of a library which would be
      detected automatically, a matching \texttt{--without-<package>} option
      must be passed to \texttt{configure} instead.
\item Most third-party libraries usable by \CS are considered optional,
      and are simply not used if not detected, but the libraries needed by
      the GUI are considered mandatory, unless the \texttt{--disable-gui}
      or \texttt{--disable-frontend} option is explicitly used.
\end{itemize}

When the prerequisites are available, and a build directory
created, building and installing \CS may be as simple as running:

\fbox{\begin{minipage}{\textwidth}\texttt{\\
\$ ../../code\_saturne-\verscs/configure\\
\$ make\\
\$ make install
}\end{minipage}}

The following chapters give more details on \CS's recommended
third-party libraries, configuration recommendations, troubleshooting,
and post-installation options.

\section{Compilers and interpreters\label{sec:compiler}}

For a minimal build of \CS on a Linux or Posix system, the requirements are:
\begin{itemize}
\item A C compiler, conforming at least to the C99 standard.
\item A Fortran compiler, conforming at least to the Fortran 95 standard
      and supporting the ISO\_C\_BINDING Fortran 2003 module.
\item A Python interpreter, with Python version 2.6 or above.
\end{itemize}

For parallel runs, an MPI library is also necessary (MPI-2 or MPI-3 conforming).
To build and use the GUI, PyQt 4 or 5 (which in turn requires
Qt 4 or 5 and SIP) are required.
Other libraries may be used for additional mesh format options,
as well as to improve performance. A list of those libraries
and their role is given in \S\ref{sec:list_ext_lib}.

For some external libraries, such as Catalyst (see \ref{sec:list_ext_lib}),
a C++ compiler is also required.

The SALOME platform is currently only compatible with Python 2,
so Python 3 should only be used when not building with SALOME support.

In practice, the code is known to build and function properly at least with the
GNU compilers 4.4 and above (up to 6.3 at this date), Intel compilers 11 and
above (up to 17 at this date), IBM XL Fortran 14 and C 12 compilers, and Clang
(tested with 3.7 or above).

Note also that while \CS makes heavy use of Python, this is for scripts and
for the GUI only; The solver only uses compiled code, so we may for example use
a 32-bit version of Python with 64-bit \CS libraries and executables.
Also, the version of Python used by ParaView/Catalyst may be independent
from the one used for building \CS.

\section{Third-Party libraries\label{sec:ext_lib}}

For a minimal build of \CS, a Linux or Posix system with C and Fortran compilers
(C99 and Fortran 95 with Fortran 2003 ISO C bindings conforming respectively),
a Python (2.6 or later) interpreter and a {\tt make} tool should be sufficient.
For parallel runs, an MPI library is also necessary (MPI-2 or MPI-3 conforming).
To build and use the GUI, Qt 4 or 5 with PyQt 4 or 5 Python bindings
(which in turn requires SIP) are required.
Other libraries may be used for additional mesh format options,
as well as to improve performance. A list of those libraries
and their role is given in \S\ref{sec:list_ext_lib}.

\subsection{Installing third-party libraries for \CS\label%
{sec:obtain_ext_lib}}

Third-Party libraries usable with \CS may be installed in several
ways:

\begin{itemize}
\item On many Linux systems, most of libraries listed in
      \S\ref{sec:list_ext_lib} are available through the distribution's
      package manager.\footnote{On Mac OS X systems, package managers such as
      Fink or MacPorts also provide package management, even though the
      base system does not.} This requires administrator privileges,
      but is by far the easiest way to install third-party libraries
      for \CS.

      Note that distributions usually split libraries or tools into runtime
      and development packages, and that although some packages are
      installed by default on many systems, this is generally not the
      case for the associated development headers. Development
      packages usually have the same name as the matching runtime package,
      with a \texttt{-dev} postfix added. Names might also differ
      slightly. For example, on a Debian system, the main package
      for Open MPI is \texttt{openmpi-bin},
      but \texttt{libopenmpi-dev} must also be installed for the \CS
      build to be able to use the former.

\item On many large compute clusters, Environment Modules allow
      the administrators to provide multiple versions of many
      scientific libraries, as well us compilers or MPI libraries,
      using the \texttt{module} command. More details on
      Environment Modules may be found at \url{http://modules.sourceforge.net}.
      When being configured and installed \CS checks for modules loaded
      with the \texttt{module} command, and records the list of loaded
      modules. Whenever running that build of \CS, the modules detected
      at installation time will be used, rather than those defined by
      default in the user's environment. This allows using versions of
      \CS built with different modules safely and easily, even if the
      user may be experimenting with other modules for various purposes.

\item If not otherwise available, third-party software may be compiled
      an installed by an administrator or a user. An administrator
      will choose where software may be installed, but for a user
      without administrator privileges or write access to
      \texttt{usr/local}, installation to a user account is often
      the only option. None of the third-party libraries usable
      by \CS require administrator privileges, so they may all be
      installed normally in a user account, provided the user
      has sufficient expertise to install them. This is usually not
      complicated (provided one reads the installation instructions,
      and is prepared to read error messages if something goes wrong),
      but even for an experienced user or administrator, compiling
      and installing 5 or 6 libraries as a prerequisite significantly
      increases the effort required to install \CS.

      Even though it is more time-consuming, compiling and installing
      third-party software may be necessary when no matching packages
      or Environment Modules are available, or when a more recent version
      or a build with different options is desired.

\item When \CS is configured to use the SALOME platform, some
      libraries inclued in that platform may be used directly;
      this is described in \S\ref{sec:config:salome}.
\end{itemize}

\subsection{List of third-party libraries usable by \CS\label%
{sec:list_ext_lib}}

The list of third-party software usable with \CS is provided here:

\begin{itemize}

\item PyQt version 4 or 5 is required by the \CS GUI. PyQt in turn requires
      Qt (4 or 5), Python, and SIP. Without this library, the GUI may not be
      built, although XML files generated with another install of \CS
      may be used.

      If desired, Using PySide instead of PyQt4/SIP should require a relatively
      small porting effort, as most of the preparatory work has been done.
      The development team should be contacted in this case.

\item HDF5 is necessary for MED, and may also be used by CGNS.

\item CGNSlib is necessary to read or write mesh and visualization files
      using the CGNS format, available as an export format with many
      third-party meshing tools. CGNS version 3.1 or above is required.

\item MED is necessary to read or write mesh and visualization files
      using the MED format, mainly used by the SALOME platform
      (\url{www.salome-platform.org}).

\item libCCMIO is necessary to read or write mesh and visualization files
      generated or readable by \starccmp using its native format.

\item \scotch or \ptscotch may be used to optimize mesh partitioning.
      Depending on the mesh, parallel computations with meshes partitioned
      with these libraries may be from 10\% to 50\% faster than using the
      built-in space-filling curve based partitioning.

      As \scotch and \ptscotch use symbols with the same names, only
      one of the 2 may be used. If both are detected, \ptscotch is used.
      Versions 6.0 and above are supported.

\item \metis or \parmetis are alternative mesh partitioning libraries.
      These libraries have a separate source tree, but some of their
      functions have identical names, so only one of the 2 may be used.
      If both are available, \parmetis will be used. Partitioning
      quality is similar to that obtained with \scotch or
      \ptscotch.

      Though broadly available, the \parmetis license is quite restrictive,
      so \ptscotch may be preferred (\CS may be built with both
      \metis and \scotch libraries). Also, the \metis license was changed
      in March 2013 to the Apache 2 license, so it would not be surprising
      for future \parmetis versions to follow. \metis 5.0 or above and
      \parmetis 4.0 or above are supported.

\item Catalyst (\url{http://www.paraview.org/in-situ/}) or full ParaView
      may be used for co-visualization or in-situ visualization.
      This requires ParaView 4.2 or above.

\item \texttt{eos-1.2} may be used for thermodynamic properties of fluids.
      it is not currently free, so usually available only to users at EDF,
      CEA, or organisms participating in projects with those entities.

\item \textbf{freesteam} (\url{http://freesteam.sourceforge.net}) is a
      free software thermodynamic properties library, implementing the
      IAPWS-IF97 steam tables, from the \href{http://www.iapws.org}{International
      Association for the Properties of Water and Steam} (IAPWS).
      Version 2.0 or above may be used.

\item CoolProp (\url{http://www.coolprop.org}) is a quite recent library
      open source library, which provides pure and pseudo-pure fluid
      equations of state and transport properties for 114 components
      (as of version 5.1), mixture properties using high-accuracy Helmholtz
      energy formulations (or cubic EOS), and correlations of properties
      of incompressible fluids and brines. Its validation is based at least
      in part on comparisons with REFPROP.

\item BLAS (Basic Linear Algebra Subroutines) may be used by the
      \texttt{cs\_blas\_test} unit test to compare the cost of operations
      such as vector sums and dot products with those provided
      by the code and compiler.
      If no third-party BLAS is provided, \CS reverts to its own
      implementation of BLAS routines, so no functionality is lost here.
      Optimized BLAS libraries such as Atlas, MKL, ESSL, or ACML may be very
      fast for BLAS3 (dense matrix/matrix operations), but the advantage is
      usually much less significant for BLAS 1 (vector/vector) operations, which
      are almost the only ones \CS has the opportunity of using.
      \CS uses its own dot product implementation (using a superblock algorithm,
      for better precision), and $y \leftarrow ax+y$ operations, so external
      BLAS1 are not used for computation, but only for unit testing (so as
      to be able to compare performance of built-in BLAS with external BLAS).
      The Intel MKL BLAS may also be used for matrix-vector products, so it
      is linked with the solver when available, but this is also currently only
      used in unit benchmark mode.
      Note that in some cases, threaded BLAS routines might oversubscribe
      processor cores in some MPI calculations, depending on the way both
      \CS and the BLAS were configured and interact, and this can actually
      lead to lower performance.
      Use of BLAS libraries is thus useful as a unit benchmarking feature,
      but has no influence on full calculations.

\item PETSc (Portable, Extensible Toolkit for Scientific Computation,
      \url{http://www.mcs.anl.gov/petsc/}) consists  of a variety of libraries,
      which may be used by \CS for the resolution of linear equation systems.
      In addition to providing many solver options, it may be used as a bridge
      to other major solver libraries. Version 3.6 has been tested with \CS.

\end{itemize}

For developers, the GNU Autotools (Autoconf, Automake, Libtool) as
well as gettext will be necessary. To build the documentation,
pdf\LaTeX{} and \texttt{fig2dev} (part of TransFig) will be necessary.

\subsection{Notes on some third-party tools and libraries}

\subsubsection{Python and PyQt\label{sec:ext:python}}

The GUI is written in PyQt (Python bindings for Qt), so Qt (version 4 or 5)
and the matching Python bindings must be available. On most modern
Linux distributions, this is available through the package manager,
which is by far the preferred solution.

On systems on which both PyQt4 and Pyqt5 are available, PyQt4 will be selected
by default, but the selection may be forced by defining
\texttt{QT\_SELECT=4} or \texttt{QT\_SELECT=5}.

When running on a system which does
not provide these libraries, there are several alternatives:

\begin{itemize}

\item build \CS without the GUI. XML files
      produced with the GUI are still usable, so if an install of \CS
      with the GUI is available on an other machine, the XML files
      may be copied on the current machine. This is certainly not an optimal
      solution, but in the case where users have a mix of desktop or virtual
      machines with modern Linux distributions and PyQt installed, and
      a compute cluster with an older system, this may avoid requiring
      a build of Qt and PyQt on the cluster if users find this too daunting.

\item Install a local Python interpreter, and add Qt5 bindings to this
      interpreter.

      Python (\url{http://www.python.org}) and Qt
      (\url{https://www.qt.io}) must be downloaded  and
      installed first, in any order. The installation instructions of
      both of these tools are quite clear, and though the installation of these
      large packages (especially Qt) may be a lengthy process in terms of
      compilation time, but is well automated and usually devoid of nasty
      surprises.

      Once Python is installed, the SIP bindings generator
      (\url{http://riverbankcomputing.co.uk/software/sip/intro})
      must also be installed. This is a small package, and configuring it
      simply requires running \texttt{python configure.py} in its source
      tree, using the Python interpreter just installed.

      Finally, the PyQt bindings
      (\url{http://riverbankcomputing.co.uk/software/pyqt/intro}) may be
      installed, in a manner similar to SIP.

      When this is finished, the local Python interpreter contains
      the PyQt bindings, and may be used by \CS's \texttt{configure}
      script by passing \texttt{PYTHON=<path\_to\_python\_executable}.

\item add Python Qt bindings as a Python extension module for an existing
      Python installation. This is a more elegant solution than the previous
      one, and avoids requiring rebuilding Python, but if the user does not
      have administrator privileges, the extensions will be placed in a
      directory that is not on the default Python extension search path, and
      that must be added to the \texttt{PYTHONPATH} environment variable.
      This works fine, but for all users using this build of \CS, the
      \texttt{PYTHONPATH} environment variable will need to be
      set.\footnote{In the future, the \CS installation scripts could check
      the \texttt{PYTHONPATH} variable and save its state in the build so as
      to ensure all the requisite directories are searched for.}

      The process is similar to the previous one, but SIP and PyQt
      installation requires a few additional configuration options
      in this case. See the SIP and PyQt reference guides for
      detailed instructions, especially the \emph{Building a Private
      Copy of the SIP Module} section of the SIP guide.

\end{itemize}

\subsubsection{\scotch and \ptscotch\label{sec:ext:scotch}}

Note that both \scotch and \ptscotch may be built from the same source
tree, and installed together with no name conflicts.

For better performance, \ptscotch may be built to use threads with concurrent
MPI calls. This requires initializing MPI with \texttt{MPI\_Init\_thread}
with \texttt{MPI\_THREAD\_MULTIPLE} (instead of the more restrictive
\texttt{MPI\_THREAD\_SERIALIZED}, \texttt{MPI\_THREAD\_FUNNELED}, or
\texttt{MPI\_THREAD\_SINGLE}, or simply using \texttt{MPI\_Init}).
As \CS does not support thread models in which different threads may call
MPI functions simultaneously, and the use of \texttt{MPI\_THREAD\_MULTIPLE}
may carry a performance penalty, we prefer to sacrifice some of
\ptscotch's performance by requiring that it be compiled without the
\texttt{-DSCOTCH\_PTHREAD} flag. This is not detected at compilation time,
but with recent MPI libraries, \ptscotch will complain at run time
if it notices that the MPI thread safety level in insufficient.

Detailed build instructions, including troubleshooting instructions,
are given in the source tree's \texttt{INSTALL.txt} file.
In case of trouble, note especially the explanation relative to the
\texttt{dummysizes} executable, which is run to determine the
sizes of structures. On machines with different front-end and
compute node architectures, it may be necessary
to start the build process, let it fail, run this executable
manually using \texttt{mpirun}, then pursue the build process.

\subsubsection{\med\label{sec:ext:med}}

The Autotools installation of MED is simple on most machines,
but a few remarks may be useful for specific cases.

Note that as of MED 3.3.0, HDF5 1.8 is required, but HDF5 1.10 is
not supported yet.

MED has a C API, is written in a mix of C and C++ code,
and provides both a C (\texttt{libmedC}) and an Fortran API
(\texttt{libmed}). Both libraries are always built, so a Fortran
compiler is required, but \CS only links using the C API, so using
a different Fortran compiler to build MED and \CS is possible.

MED does require a C++ runtime library, which is usually transparent
when shared libraries are used. When built with static libraries
only, this is not sufficient, so when testing for a MED library,
the \CS \texttt{configure} script also tries linking with a C++
compiler if linking with a C compiler fails. This must be the
same compiler that was used for MED, to ensure the runtime matches.
The choice of this C++ compiler may be defined passing the
standard \texttt{CXX} variable to \texttt{configure}.

Also, when building MED in a cross-compiling situation,
\texttt{--med-int=int} or \texttt{--med-int=int64\_t} (depending
on whether 32 or 64 bit ids should be used) should be
passed to its \texttt{configure} script to avoid a run-time
test.

\subsubsection{libCCMIO\label{sec:ext:libccmio}}

Different versions of this library may use different build
systems, and use different names for library directories,
so using both the \texttt{--with-ccm=} or \texttt{--with-ccm-include=}
and \texttt{--with-ccm-lib=} options to \texttt{configure} is
usually necessary.
Also, the include directory should be the toplevel library,
as header files are searched under a \texttt{libccmio}
subdirectory\footnote{this is made necessary by libCCMIO version
2.6.1, in which this is hard-coded in headers including other
headers. In more recent versions such as 2.06.023, this is not the
case anymore, and an \texttt{include} subdirectory is present, but
it does not contain the \texttt{libccmioversion.h} file, which is
found only under the \texttt{libccmio} subdirectory, and is required
by \CS to handle differences between versions, so that source
directory is preferred to the installation \texttt{include}.}

A libCCMIO distribution usually contains precompiled
binaries, but recompiling the library is recommended.
Note that at least for version 2.06.023, the build will fail
building dump utilities, due to the \texttt{-l adf} option
being placed too early in the link command. To avoid this,
add \texttt{LDLIBS=-ladf} to the makefile command, for example:

\texttt{make -f Makefile.linux SHARED=1 LDLIBS=-ladf}

(\texttt{SHARED=1} and \texttt{DEBUG=1} may be used to force
shared library or debug builds respectively).

Finally, if using libCCMIO 2.6.1, remove the \texttt{libcgns*}
files from the libCCMIO libraries directory if also building
\CS with CGNS support, as those libraries are not required
for CCMIO, and are are an obsolete version of CGNS, which
may interfere with the version used by \CS.

Note that libCCMIO uses a modified version of CGNS's ADF library,
which may not be compatible with that of CGNS. When building
with shared libraries, the reader for libCCMIO uses a plugin
architecture to load the library dynamically. For a static
build with both libCCMIO and CGNS support, reading ADF-based
CGNS files may fail. To work around this issue, CGNS files
may be converted to HDF5 using the \texttt{adf2hdf} utility
(from the CGNS tools). By default, CGNS post-processing output
files use HDF5, so this issue is rarer on output.

\subsubsection{freesteam\label{sec:ext:freesteam}}

This library's build instructions mention bindings with ascend,
but those are not necessary in the context of \CS, so building
without them is simplest. Its build system is based on scons,
and builds on relatively recent systems with Python 2.7 should
be straightforward.

With Python versions lower than 2.6, the command-line arguments
allowing to choose the installation prefix (so as to place it
in a user directory) are ignored, and its \texttt{SConstruct}
file is not complete enough to allow setting flags for linking
with an alternative, user-installed Python library outside
the default linker search path. In this case, editing the
\texttt{SConstruct} file to change the default paths is
an ugly, but simple solution.

\subsubsection{CoolProp\label{sec:ext:coolprop}}

This library's build system is based on CMake, and building it
is straightforward, though some versions seem to have build
issues (the 5.1.0 release is missing a file, while 5.1.1 release
builds fine). CoolProp uses submodules which are downloaded
using \texttt{git clone https://github.com/CoolProp/CoolProp.git --recursive},
but may be missing when downloading a zip file.

Its user documentation is good, but its installation documentation is poor,
so recommendations are provided here

To download and prepare CoolProp for build, using an out-of-tree build
(so as to avoid polluting the source tree with cache files), the
following commands are recommended:

\fbox{\begin{minipage}{\textwidth}\texttt{\\
\$ git clone https://github.com/CoolProp/CoolProp.git --recursive\\
\$ cd CoolProp\\
\$ git checkout release\\
\$ cd ..\\
\$ mkdir CoolProp\_build\\
\$ cd CoolProp\_build
}\end{minipage}}

Then configure, build, and install, run:

\fbox{\begin{minipage}{\textwidth}\texttt{\\
\$ cmake
\textbackslash \\
\textcolor{Violet}{-DCOOLPROP\_INSTALL\_PREFIX}=\textcolor{OliveGreen}{\$INSTALL\_PATH}/arch/\textcolor{OliveGreen}{\$machine\_name}
\textbackslash \\
\textcolor{Violet}{-DCOOLPROP\_SHARED\_LIBRARY}=ON
\textbackslash \\
\textcolor{OliveGreen}{\$COOLPROP\_SRC\_PATH}
}\end{minipage}}

Followed by:

\fbox{\begin{minipage}{\textwidth}\texttt{\\
\$ make \\
\$ make install\\
\$ make clean
}\end{minipage}}

CoolProp's installer only installs one C wrapper header, not the
C++ headers required for lower-level access, so the following commands
must also be run:

\fbox{\begin{minipage}{\textwidth}\texttt{\\
\$ cp -rp \textcolor{OliveGreen}{\$COOLPROP\_SRC\_PATH}/include
{\$INSTALL\_PATH}/arch/\textcolor{OliveGreen}{\$machine\_name}\\
\$ rm -f {\$INSTALL\_PATH}/arch/\textcolor{OliveGreen}{\$machine\_name}/CoolPropLib.h
}\end{minipage}}

Alternatively, to copy less files and avoid changing the structure provided
by CoolProp:

\fbox{\begin{minipage}{\textwidth}\texttt{\\
\$ cp -r \textcolor{OliveGreen}{\$COOLPROP\_SRC\_PATH}/include/CoolProp.h
\textbackslash \\
{INSTALL\_PATH}/arch/\textcolor{OliveGreen}{\$machine\_name} \\
\$ cp -r \textcolor{OliveGreen}{\$COOLPROP\_SRC\_PATH}/include/PlatFormDetermination.h
\textbackslash \\
{INSTALL\_PATH}/arch/\textcolor{OliveGreen}{\$machine\_name}
}\end{minipage}}

\sloppy

To install CoolProp's Python bindings (used by the GUI when available),
the straigthforward method is to go into the CoolProp source directory,
into the wrappers/Python subdirectory, then run:

\fbox{\begin{minipage}{\textwidth}\texttt{\\
\$ export PYTHONPATH=$COOLPROP\_INSTALL\_PREFIX/lib/{\$python\_version}/site-packages:$PYTHONPATH
\$ python setup.py install --prefix=\$COOLPROP\_INSTALL\_PREFIX
}\end{minipage}}

Although this is not really an out-of-tree build, the Python
setup also cleans the directory.

\subsubsection{Paraview or Catalyst\label{sec:ext:catalyst}}

By default, this library is built with a GUI, but it may also be be
built using OSMesa for offscreen rendering. The build documentation
on the ParaView website and Wiki details this. For use with \CS,
the recommended solution is to build or use a standard ParaView build
for interactive visualization, and to use its CatalystScriptGeneratorPlugin
to generate Python co-processing scripts. A second build, using OSMesa,
may be used for in-situ visualization. This is the Version \CS will be linked
to. A recommended cmake command for this build contains:

\fbox{\begin{minipage}{\textwidth}\texttt{\\
\$ cmake
\textbackslash \\
\textcolor{Violet}{-DCMAKE\_INSTALL\_PREFIX}=\textcolor{OliveGreen}{\$INSTALL\_PATH}/arch/\textcolor{OliveGreen}{\$machine\_name}\_osmesa
\textbackslash \\
\textcolor{Violet}{-DPARAVIEW\_BUILD\_QT\_GUI}=OFF
\textbackslash \\
\textcolor{Violet}{-DPARAVIEW\_USE\_MPI}=ON
\textbackslash \\
\textcolor{Violet}{-DPARAVIEW\_ENABLE\_PYTHON}=ON
\textbackslash \\
\textcolor{Violet}{-DPARAVIEW\_INSTALL\_DEVELOPMENT\_FILES}=ON
\textbackslash \\
\textcolor{Violet}{-DVTK\_USE\_X}=OFF
\textbackslash \\
\textcolor{Violet}{-DOPENGL\_INCLUDE\_DIR}=IGNORE
\textbackslash \\
\textcolor{Violet}{-DOPENGL\_xmesa\_INCLUDE\_DIR}=IGNORE
\textbackslash \\
\textcolor{Violet}{-DOPENGL\_gl\_LIBRARY}=IGNORE
\textbackslash \\
\textcolor{Violet}{-DOSMESA\_INCLUDE\_DIR}=\textcolor{OliveGreen}{\$MESA\_INSTALL\_PREFIX}/include
\textbackslash \\
\textcolor{Violet}{-DOSMESA\_LIBRARY}=\textcolor{OliveGreen}{\$MESA\_INSTALL\_PREFIX}/lib/libOSMesa.so
\textbackslash \\
\textcolor{Violet}{-DVTK\_OPENGL\_HAS\_OSMESA}=ON
\textbackslash \\
\textcolor{Violet}{-DVTK\_USE\_OFFSCREEN}=OFF
\textbackslash \\
\textcolor{OliveGreen}{\$PARAVIEW\_SRC\_PATH}
}\end{minipage}}

More info may also be found on the ParaView Wiki:
(\url{http://www.paraview.org/Wiki/ParaView/ParaView_And_Mesa_3D}).

Catalyst editions
(\url{http://www.paraview.org/Wiki/ParaView/Catalyst/BuildCatalyst})
may be used instead of a full ParaView build, but some
coprocessing scripts may not work depending on what is included in the
editions, so this is recommended only for advanced users.

On some systems, loading the Catalyst module as a plug-in (which is the
default) seems to interfere with the detection of required OpenGL2 features
or extensions required by ParaView 5.2 an above. In this case, Catalyst
support may be linked in the standard manner by using the
\texttt{--disable-catalyst-as-plugin} configuration option.
A less extreme option is to use the \texttt{--enable-dlopen-rtld-global}
option, which changes the system options with which libraries are loaded
(possibly impacting all plugins). This seems to be sufficient with
OSMesa 17.x versions. Using the \texttt{DL\_PRELOAD} environment variable
at runtime to preload the OSMesa library also avoids the issue.

\section{Preparing for build\label{sec:prepare}}

If the code was obtained as an archive, it must be unpacked:

\texttt{tar xvzf saturne.tar.gz}

If for example you unpacked the directory in a directory
named \texttt{/home/user/Code\_Saturne}, you will now
have a directory named \texttt{/home/user/Code\_Saturne/saturne}.

It is recommended to build the code in a separate directory from the source.
This also allows multiple builds, for example, building both an
optimized and a debugging version. In this case, choose a consistent
naming scheme, using an additional level of sub-directories,
for example:

\fbox{\begin{minipage}{\textwidth}\texttt{\\
\$ mkdir saturne\_build\\
\$ cd saturne\_build\\
\$ mkdir prod\\
\$ cd prod%
}\end{minipage}}

Some older system's {\tt make} command may not support compilation
in a directory different from the source directory ({\tt VPATH}
support). In this case, installing and using the GNU {\tt gmake}
tool instead of the native {\tt make} is recommended.

\subsection{Source trees obtained through a source code repository\label{sec:preparerepo}}

For developers obtaining the code was obtained through a version control
system such as Subversion, an additional step is required:

\fbox{\begin{minipage}{\textwidth}\texttt{\\
\$ cd saturne\\
\$ ./sbin/bootstrap\\
\$ cd ..
}\end{minipage}}

In this case, additional tools need to be available:

\begin{itemize}
\item GNU Autotools: Autoconf, Automake, Libtool (2.2 or 2.4), and Gettext.
\item Bison (or Yacc) and Flex (or Lex)
\item PdfLaTeX and TransFig
\item Doxygen (1.8.7 or more recent). The path to Doxygen can be specified during
the configure phase with \code{configure DOXYGEN=PATH\_TO\_DOXYGEN}.
\end{itemize}

These tools are not necessary for builds from tarballs; they
are called when building the tarball (using {\tt make dist}), so
as to reduce the number of prerequisites for regular users, while
developers building the code from a repository can be expected to
need a more complete development environment.

Also, to build and install the documentation when building the code
from a repository instead of a tarball, the following stages are required:

\fbox{\begin{minipage}{\textwidth}\texttt{\\
\$ make doc\\
\$ make install-doc
}\end{minipage}}

\section{Configuration\label{sec:config}}

\CS uses a build system based on the GNU Autotools, which includes
its own documentation.

To obtain the full list of available configuration options,
run: {\tt configure~--help}.

Note that for all options starting with {\tt --enable-},
there is a matching option with {\tt --disable-}. Similarly,
for every {\tt --with-}, {\tt --without-} is also possible.

Select configuration options, then run {\tt configure}, for example:

\fbox{\begin{minipage}{\textwidth}\texttt{\\
\$ /home/user/Code\_Saturne/\verscs/src/code\_saturne-\verscs/configure \textbackslash \\
\textcolor{Violet}{--prefix}=/home/user/Code\_Saturne/\verscs/arch/prod
\textbackslash \\
\textcolor{Violet}{--with-med}=/home/user/opt/med-3.3 \textbackslash \\
\textcolor{red}{CC}=/home/user/opt/mpich-3.2/bin/mpicc \textcolor{red}{FC}=gfortran
}\end{minipage}}

In the rest of this section, we will assume that we are in
a build directory separate from sources, as described in
\S\ref{sec:prepare}. In different examples, we assume
that third-party libraries used by \CS are either available
as part of the base system (i.e. as packages in a Linux distribution),
as Environment Modules, or are installed under a separate path.

\subsection{Debug builds\label{sec:config:shared}}

It may be useful to install debug builds alongside production
builds of \CS, especially when user subroutines are used
and the risk of crashes due to user programming error is high.
Running the code using a debug build is significantly
slower, but more information may be available in the case
of a crash, helping understand and fix the problem faster.

Here, having a consistent and practical naming scheme is useful.
For a side-by-side debug build for the example above, we simply replace \texttt{prod} by
\texttt{dbg} in the \texttt{--prefix} option, and add
\texttt{--enable-debug} to the configure command:

\fbox{\begin{minipage}{\textwidth}\texttt{\\
\$ cd ..\\
\$ mkdir dbg\\
\$ cd dbg\\
\$ ../../code\_saturne-\verscs/configure \textbackslash \\
\textcolor{Violet}{--prefix}=/home/user/Code\_Saturne/\verscs/arch/\textcolor{Magenta}{dbg}
\textbackslash \\
\textcolor{Violet}{--with-med}=/home/user/opt/med-3.3 \textbackslash \\
\textcolor{Magenta}{--enable-debug} \textbackslash \\
\textcolor{red}{CC}=/home/user/opt/mpich-3.2/bin/mpicc \textcolor{red}{FC}=gfortran
}\end{minipage}}

\subsection{Shared or static builds\label{sec:config:shared}}

By default, on most architectures, \CS will be built with shared libraries.
Shared libraries may be disabled (in which case static libraries
are automatically enabled) by adding  {\tt --disable-shared} to the options
passed to {\tt configure}.
On some systems, such as BlueGene/Q, the build may default to static libraries
instead.

It is possible to build both shared and static libraries by adding
{\tt --disable-static} to the {\tt configure} options, but the
executables will be linked with the shared version of the libraries,
so this is rarely useful (the build process is also slower in this case, as
each file is compiled twice).

In some cases, a shared build may fail due to some dependencies
on static-only libraries. In this case, {\tt --disable-shared}
will be necessary. Disabling shared libraries has also been seen
to avoid issues with linking on Mac OSX systems.

In any case, be careful if you switch from one option to the other: as
linking will be done with shared libraries by default, a build
with static libraries only will not completely overwrite a build using
shared libraries, so uninstalling the previous build first
is recommended.

\subsection{Relocatable builds\label{sec:config:relocatable}}

By default, a build of \CS is not movable, as not only
are library paths hard-coded using \emph{rpath} type info,
but the code's scripts also contain absolute paths.

To ensure a build is movable, pass the \texttt{--enable-relocatable} option
to {\tt configure}.

Movable builds assume a standard directory hierarchy, so when running
{\tt configure}, the \texttt{--prefix} option may be used, but fine tuning
of installation directories using options such as \texttt{--bindir},
\texttt{--libdir}, or \texttt{--docdir} must not be used
(these options are useful to install to strict directory hierarchies,
such as when packaging the code for a Linux distribution,
in which case making the build relocatable would be nonsense anyways,
so this is not an issue.
\footnote{In the special case of packaging the code, which
may require both fine-grained control of the installation directories
and the possibility to support options such as \texttt{dpgg}'s
\texttt{--instdir}, it is assumed the packager has sufficient knowledge to
update both \emph{rpath} information and paths in scripts in the executables
and python package directories of a non-relocatable build, and that the
packaging mechanism includes the necessary tools and scripts to enable this.}

\subsection{Compiler flags and environment variables\label{sec:config:flags}}

As usual when using an Autoconf-based \texttt{configure} script,
some environment variables may be used. \texttt{configure --help}
will provide the list of recognized variables.
\texttt{CC} and \texttt{FC} allow selecting the C and Fortran compiler
respectively (possibly using an MPI compiler wrapper).

Compiler options are usually defined automatically, based on
detection of the compiler (and depending on whether \texttt{--enable-debug}
was used). This is handled in a \texttt{config/cs\_auto\_flags.sh}
and \texttt{libple/config/ple\_auto\_flags.sh} scripts.
These files are sourced when running \texttt{configure}, so
any modification to it will be effective as soon as \texttt{configure} is run.
When installing on an exotic machine, or with a new compiler, adapting this
file is useful (and providing feedback to the \CS development team
will enable support of a broader range of compilers and systems in the
future.

The usual \texttt{CPPFLAGS}, \texttt{CFLAGS},
\texttt{FCCFLAGS}, \texttt{LDFLAGS}, and \texttt{LIBS} environment variables
may also be used, an flags provided by the user are appended to the automatic
flags. To completely disable automatic setting of flags,
the \texttt{--disable-auto-flags} option may be used.

\subsection{MPI compiler wrappers\label{sec:config:mpicc}}

MPI environments generally provide compiler wrappers, usually
with names similar to \texttt{mpicc} for C, \texttt{mpicxx} for C++,
and \texttt{mpif90} for Fortran 90. Wrappers conforming to the
MPI standard recommendations should provide a \texttt{-show}
option, to show which flags are added to the compiler so as to
enable MPI. Using wrappers is fine as long as several third-party tools
do not provide their own wrappers, in which case either
a priority must be established. For example, using HDF5's
\texttt{h5pcc} compiler wrapper includes the options used by
\texttt{mpicc} when building HDF5 with parallel IO, in addition to
HDF5's own flags, so it could be used instead of \texttt{mpicc}.
On the contrary, when using a serial build of HDF5 for a parallel
build of \CS, the \texttt{h5cc} and \texttt{mpicc} wrappers
contain different flags, so they are in conflict.

Also, some MPI compiler wrappers may include optimization options
used to build MPI, which may be different from those we wish to use
that were passed.

To avoid issues with MPI wrappers, it is possible to select an
MPI library using the \texttt{--with-mpi} option to \texttt{configure}.
For finer control, \texttt{--with-mpi-include} and \texttt{--with-mpi-lib}
may be defined separately.

Still, this may not work in all cases, as a fixed list of libraries
is tested for, so using MPI compiler wrappers is the simplest and safest
solution. Simply use a \texttt{CC=mpicc} or similar option instead
of \texttt{--with-mpi}.

There is no need to use an \texttt{FC=mpif90} or equivalent option:
in \CS, MPI is never called directly from Fortran code,
so Fortran MPI bindings are not necessary.

\subsection{Environment Modules\label{sec:config:envmode}}

As noted in \S\ref{sec:obtain_ext_lib}, on systems providing
Environment Modules with the {\tt module} command, \CS's {\tt configure}
script detects which modules are loaded and saves
this list so that future runs of the code use that same environment,
rather than the user's environment, so as to allows using versions of
\CS built with different modules safely and easily.

Given this, it is recommended that when configuring and installing
\CS, only the modules necessary for that build of for
profiling or debugging be loaded. Note that as \CS uses the module
environment detected and runtime instead of the user's current
module settings, debuggers requiring a specific module may
not work under a standard run script if they were not loaded when
installing the code.

The detection of environment modules may be disabled using the
\texttt{--without-modules} option,
or the use of a specified (colon-separated) list of modules
may be forced using the \texttt{--with-modules=} option.

\subsection{Remarks for very large meshes\label{sec:config:largemesh}}

If \CS is to be run on large meshes, several precautions regarding
its configuration and that of third-party software must be taken.

in addition to local connectivity arrays, \CS uses global element ids
for some operations, such as reading and writing meshes and restart files,
parallel interface element matching, and post-processing output.
For a hexahedral mesh with $N$ cells,
the number of faces is about $3N$ (6 faces per cell, shared by 2 cells each).
With 4 cells per face, the $face \rightarrow vertices$ array is of size
of the order of $4\times3N$, so global ids used in that array's index
will reach $2^{31}$ for a mesh in the range of $2^{31} / 12 \approxeq 178.10^6$.
In practice, we have encountered a limit with slightly smaller meshes,
around 150 million cells.

Above 150 million hexahedral cells or so, it is thus imperative to configure
the build to use 64-bit global element ids. This is the default.
Local indexes use the default {int} size. To slightly decrease memory
consumption if meshes of this size are never expected (for example on a workstation
or a small cluster), the {\tt --disable-long-gnum} option may be used.

Recent versions of some third-party libraries may also optionally use 64-bit ids,
independently of each other or of \CS.
This is the case for the \scotch and \metis, MED and
CGNS libraries. In the case of graph-based partitioning, only
global cell ids are used, so 64-bit ids should not in theory be necessary
for meshes under 2 billion cells. In a similar vein, for post-processing output
using nodal connectivity, 64-bit global ids should only be an imperative
when the number of cells or vertices approaches 2 billion.
Practical limits may be lower, if some intermediate internal counts
reach these limits earlier.

Partitioning a 158 million hexahedral mesh using serial \metis 5 or \scotch
on a front-end node with 128 Gb memory is possible,
but partitioning the same mesh on cluster nodes with ``only'' 24 Gb each
may not, so using parallel partitioning \ptscotch or \parmetis
should be preferred.

\subsection{Installation with the SALOME platform\label{sec:config:salome}}

To enable SALOME platform (\url{http://www.salome-platform.org}) integration,
the \texttt{--with-salome} configuration option should be used, so as to
specify the directory of the SALOME installation (note that this should be
the main installation directory, not the default application directory,
also generated by SALOME's installers).

With SALOME support enabled, both the CFDSTUDY salome module
(available by running \texttt{code\_saturne salome)} after install)
and the \textit{code\_aster} coupling adapter should be available.

Note that the CFDSTUDY module will only be usable with a PyQt
version similar to that used in SALOME. PyQt5 is used by SALOME
versions 8 and above, while PyQt4 is used for older versions.
Other aspects of SALOME integration should not be impacted by an incompatible
PyQt version.

Also, SALOME expects a specific directory tree when loading modules,
so the CFDSTUDY and \textit{code\_aster} coupling adapter my be
ignored when installing with a specified (i.e. non-default)
\texttt{--datarootdir} path in the \CS \texttt{configure} options.

Note that specifying a SALOME directory does not automatically
force the \CS \texttt{configure} script to find some libraries
which may be available in the SALOME distribution, such as HDF5,
MED, or CGNS. To indicate that the versions from SALOME should be used,
without needing to provide the full paths, the following configuration
options may be used for HDF5, CGNS, and MED respectively:

 \texttt{--with-hdf5=salome}\\
 \texttt{--with-cgns=salome}\\
 \texttt{--with-med=salome}

As CGNS and MED file formats are portable, MED or CGNS files produced
by either \CS or SALOME remain interoperable.\footnote{At the least,
files produced with a given version of CGNS or MED should be readable
with the same or a newer version of that library.}

Unless a specific\texttt{--with-medcoupling} option is given, a compatible
MEDCoupling library is also searched for in the SALOME distribution.

Also note that for SALOME builds containing their own Python interpreter and library,
using that same interpreter for \CS may avoid some issues, but may then require
sourcing the SALOME environment or at least its Python-related
\texttt{LD\_LIBRARY\_PATH} for the main \CS script to be usable.

\subsection{Example configuration commands\label{sec:config:examples}}

Most available prerequisites are auto-detected, so to install the
code to the default \texttt{/usr/local} sub-directory,
a command such as:

\texttt{\$ ../../code\_saturne-\verscs/configure}

should be sufficient.

For the following examples, Let us define environment variables respectively
reflecting the \CS source path, installation path, and a path where optional
libraries are installed:

\fbox{\begin{minipage}{\textwidth}\texttt{\\
\$ \textcolor{OliveGreen}{SRC\_PATH}=/home/projects/Code\_Saturne/\verscs \\
\$ \textcolor{OliveGreen}{INSTALL\_PATH}=/home/projects/Code\_Saturne/\verscs \\
\$ \textcolor{OliveGreen}{CS\_OPT}=/home/projects/opt
}\end{minipage}}

For an install on which multiple
versions and architectures of the code should be available,
configure commands with all bells and whistles (except SALOME support) for a
build on a cluster named \texttt{athos}, using the Intel compilers
(made available through environment modules) may look like this:

\fbox{\begin{minipage}{\textwidth}\texttt{\\
\$ \textcolor{Magenta}{module purge} \\
\$ \textcolor{Magenta}{module load} intel\_compilers/2016.0.047 \\
\$ \textcolor{Magenta}{module load} open\_mpi/gcc/2.0.1 \\
\$ \textcolor{OliveGreen}{\$SRC\_PATH}/code\_saturne-\verscs/configure \textbackslash \\
\textcolor{Violet}{--prefix}=\textcolor{OliveGreen}{\$INSTALL\_PATH}/arch/athos\_ompi
\textbackslash \\
\textcolor{Violet}{--with-blas}=/opt/mkl-2016.0.047/mkl \textbackslash \\
\textcolor{Violet}{--with-hdf5}=\textcolor{OliveGreen}{\$CS\_OPT}/hdf5-1.8.17/arch/athos
\textbackslash \\
\textcolor{Violet}{--with-med}=\textcolor{OliveGreen}{\$CS\_OPT}/med-3.3/arch/athos
\textbackslash \\
\textcolor{Violet}{--with-cgns}=\textcolor{OliveGreen}{\$CS\_OPT}/cgns-3.3/arch/athos \textbackslash \\
\textcolor{Violet}{--with-ccm}=\textcolor{OliveGreen}{\$CS\_OPT}/libccmio-2.06.23/arch/athos \textbackslash \\
\textcolor{Violet}{--with-scotch}=\textcolor{OliveGreen}{\$CS\_OPT}/scotch-6.0/arch/athos\_ompi \textbackslash \\
\textcolor{Violet}{--with-metis}=\textcolor{OliveGreen}{\$CS\_OPT}/parmetis-4.0/arch/athos\_ompi \textbackslash \\
\textcolor{Violet}{--with-eos}/\textcolor{OliveGreen}{\$CS\_OPT}/eos-1.2.0/arch/athos\_ompi \textbackslash \\
\textcolor{red}{CC}=mpicc \textcolor{red}{FC}=ifort \textcolor{red}{CXX}=icpc
}\end{minipage}}

In the example above, we have appended the \texttt{\_ompi} postfix
to the architecture name for libraries using MPI, in case we intend
to install 2 builds, with different MPI libraries (such as Open MPI and MPICH-based Intel MPI).
Note that optional libraries using MPI must also use the same MPI
library. This is the case for \ptscotch or \parmetis, but also HDF5,
CGNS, and MED if they are built with MPI-IO support.
Similarly, C++ and Fortran libraries, and even C libraries built
with recent optimizing C compilers, may require runtime libraries
associated to that compiler, so if versions using different compilers
are to be installed, it is recommended to use a naming scheme
which reflects this.
In this example, HDF5, CGNS and MED were built without MPI-IO support,
as \CS does not yet exploit MPI-IO for these libraries.

To avoid copying platform-independent data (such as the documentation)
from different builds multiple times, we may use the same
\texttt{--datarootdir} option for each build so as to install that
data to the same location for each build.

\subsection{Cross-compiling}

On machines with different front-end and compute node architectures,
such as IBM Blue Gene/Q, cross-compiling is necessary.
To install and run \CS, 2 builds are required:

\begin{itemize}
\item a ``front-end'' build, based on front-end node's architecture. This is
      the build whose \texttt{code\_saturne} command, GUI, and documentation
      will be used, and with which meshes may be imported (i.e. whose
      Preprocessor will be used). This build is not intended for calculations,
      though it could be used for mesh quality criteria checks.
      This build will thus usually not need  MPI.
\item a ``compute'' build, cross-compiled to run on the compute nodes.
      This build does not need to include the GUI, documentation, or
      the \pcs.
\end{itemize}

A debug variant of the compute build is also recommended, as always.
Providing a debug variant of the front-end build is not generally useful.

A post-install step (see \S\ref{sec:post_install}) will allow
the scripts of the front-end build to access the compute build in a transparent
manner, so it will appear to the users that they are simply working with that
build.

Depending on their role, optional third-party libraries should be installed
either for the front-end, for the compute nodes, or both:

\begin{itemize}
\item BLAS will be useful only for the compute nodes, and are generally
      always available on large compute facilities.
\item Python and PyQt will run on the front-end node only.
\item HDF5, MED, CGNSlib, and libCCMIO may be used by the Preprocessor on
      the front-end node to import meshes, and by the main solver on the
      compute nodes to output visualization meshes and fields.
\item \scotch or \metis may be used by a front-end node build of the
      solver, as serial partitioning of large meshes requires a lot of memory.
\item \ptscotch or \parmetis may be used by the main solver on the
      compute nodes.
\end{itemize}

\subsubsection{Cross-compiling configuration for Blue Gene/Q}

In our example, the front-end node is based on an IBM Power architecture,
runnning under Red Hat Enterprise Linux 6, on which the Python/Qt4
environment should be available as an RPM package, and installed by the
administrators.

On the compute nodes, the IBM XL compilers produce static object files
by default, so specifying the \texttt{--disable-shared} option is not necessary
for libraries using Autotools-based installs when using those compilers,
though using \texttt{--build=ppc64 --host=bluegeneq} in this case ensures
the cross-compiling environment is detected. This environment
is the suggested default, and prevents shared library builds.
To allow building with shared libraries while still ensuring the
cross-compiling environment is detected,
use \texttt{--host=powerpc64-bgq-linux} instead.

For the compute nodes, the following remarks may be mode for prerequisites:

\begin{itemize}
\item HDF5: building this library with its \texttt{configure} script
is a pain\footnote{It requires running a \texttt{yodconfigure}
script and adapting other scripts (see documentation), then running
this as a submitted job (or under a SLURM allocation if you are lucky
enough to use this resource manager).}, but installing HDF5 1.8.9 or above
using CMake is as simple as on a workstation, and simply requires
choosing the correct compilers and possibly a few other options (in
the EDF \CS build, the GCC compilers were chosen to reduce risks, and
the Fortran wrappers were not needed, so not built).
\item CGNS: building CGNS 3.1 or 3.2 is based on CMake, and no specific
problems have been observed.
\item MED: building MED 3.0.5 or above for \CS is easier than previous versions,
as a new \\ \texttt{--disable-fortan} option is available for the
\texttt{configure} script. Both the C and C++ compiler wrappers
must be specified, and the link may fail with the GNU compilers, due
to some shared library issue (trying to force \texttt{--disable-shared}).
With the IBM XL compilers, the same build works fine, as long as the
\texttt{CXXFLAGS=-qlanglvl=redefmac}) option is passed. Adding
the HDF5 tools path to the \texttt{\$PATH} environment variable for
the  configuration stage may also be required.
\item CoolProp: the GNU (not XL) C++ compiler must be used, and the additional
\texttt{-DCMAKE\_CXX\_FLAGS=-U\_\_powerpc\_\_} option must be passed to
the \texttt{cmake} command.
\end{itemize}

For an example, let us start with the front-end build:

\fbox{\begin{minipage}{\textwidth}\texttt{\\
\$ \textcolor{OliveGreen}{\$SRC\_PATH}/code\_saturne-\verscs/configure \textbackslash \\
\textcolor{Violet}{--prefix}=\textcolor{OliveGreen}{\$INSTALL\_PATH}/arch/frontend
\textbackslash \\
\textcolor{Violet}{--with-hdf5}=\textcolor{OliveGreen}{\$CS\_OPT}/hdf5-1.8.17/arch/frontend
\textbackslash \\
\textcolor{Violet}{--with-med}=\textcolor{OliveGreen}{\$CS\_OPT}/med-3.3/arch/frontend
\textbackslash \\
\textcolor{Violet}{--with-cgns}=\textcolor{OliveGreen}{\$CS\_OPT}/cgns-3.3/arch/frontend \textbackslash \\
\textcolor{Violet}{--with-scotch}=\textcolor{OliveGreen}{\$CS\_OPT}/scotch-6.0/arch/frontend \\
}\end{minipage}}

For the compute node, we use the same version of Python (which
is used only for the GUI and scripts, which only run on the front-end
or service nodes), but the compilers are cross-compilers for the
compute nodes:

\fbox{\begin{minipage}{\textwidth}\texttt{\\
\$ \textcolor{OliveGreen}{\$SRC\_PATH}/code\_saturne-\verscs/configure \textbackslash \\
\textcolor{Violet}{--prefix}=\textcolor{OliveGreen}{\$INSTALL\_PATH}/arch/bgq
\textbackslash \\
\textcolor{Violet}{--without-blas} \\
\textcolor{Violet}{--with-hdf5}=\textcolor{OliveGreen}{\$CS\_OPT}/hdf5-1.8.13/arch/bgq \textbackslash \\
\textcolor{Violet}{--with-med}=\textcolor{OliveGreen}{\$CS\_OPT}/med-3.3/arch/bgq \textbackslash \\
\textcolor{Violet}{--with-cgns}=\textcolor{OliveGreen}{\$CS\_OPT}/cgns-3.3/arch/bgq \textbackslash \\
\textcolor{Violet}{--with-scotch}=\textcolor{OliveGreen}{\$CS\_OPT}/scotch-6.0/arch/bgq \textbackslash \\
\textcolor{Violet}{
--disable-sockets --disable-dlloader -disable-nls \textbackslash \\
--disable-frontend --enable-long-gnum \textbackslash
} \\
\textcolor{Magenta}{--build}=ppc64
\textcolor{Magenta}{--host}=bluegeneq \textbackslash \\
\textcolor{red}{CC}=/bgsys/drivers/ppcfloor/comm/xl/bin/mpixlc\_r \textbackslash \\
\textcolor{red}{CXX}=/bgsys/drivers/ppcfloor/comm/xl/bin/mpixlcxx\_r \textbackslash \\
\textcolor{red}{FC}=bgxlf95\_r \\
}\end{minipage}}

The C++ compiler is specified, as it will be needed for
the link stage due to C++ dependencies in the MED library,
which is a static library in this example (see \S\ref{sec:ext:med}).

Note that in the above examples, we specified an install of the \scotch
partitioning library both for the front-end and for the compute nodes.
The implies a serial build of \scotch on the front-end node, and a parallel
build (\ptscotch) on the compute nodes. Both are optional, and the
serial partitioning on the front-end nodes should only be used as a
backup or as a reference for parallel partitioning. Unless robustness
or quality issues are encountered with parallel partitioning, it
should supercede serial partitioning, as it allows for a simpler
toolchain even for large meshes.
Similarly, \metis could be used on the front-end node, and \parmetis
on the compute nodes.

\subsubsection{Compiling for Cray X series}

For Cray X series, when using the GNU compilers, installation should
be similar to that on standard clusters. Using The Cray compilers,
options such as in the following example are recommended:

\fbox{\begin{minipage}{\textwidth}\texttt{\\
\$ \textcolor{OliveGreen}{\$SRC\_PATH}/code\_saturne-\verscs/configure \textbackslash \\
\textcolor{Violet}{--prefix}=\textcolor{OliveGreen}{\$INSTALL\_PATH}/arch/xc30
\textbackslash \\
\textcolor{Violet}{--with-hdf5}=\textcolor{OliveGreen}{\$CS\_OPT}/hdf5-1.8.17/arch/xc30 \textbackslash \\
\textcolor{Violet}{--with-med}=\textcolor{OliveGreen}{\$CS\_OPT}/med-3.3/arch/xc30 \textbackslash \\
\textcolor{Violet}{--with-cgns}=\textcolor{OliveGreen}{\$CS\_OPT}/cgns-3.3/arch/xc30 \textbackslash \\
\textcolor{Violet}{--with-scotch}=\textcolor{OliveGreen}{\$CS\_OPT}/scotch-6.0/arch/xc30 \textbackslash \\
\textcolor{Violet}{
--disable-sockets --disable-nls \textbackslash \\
--disable-shared \textbackslash
} \\
\textcolor{Magenta}{--host}=x86\_64-unknown-linux-gnu \textbackslash \\
\textcolor{red}{CC}=cc \textbackslash \\
\textcolor{red}{CXX}=CC \textbackslash \\
\textcolor{red}{FC}=ftn
}\end{minipage}}

In case the automated environment modules handling causes issues,
adding the \texttt{--without-modules} option may be necessary.
In that case, caution must be exercised so that the user will load
the same modules as those used for installation. This is not an issue
if modules for \CS is also built, and the right dependencies
handled at that level.

Note that to build without OpenMP with the Cray compilers,
\texttt{CFLAGS=\"-h noomp\"} and \texttt{FCFLAGS=\"-h noomp\"}
need to be added.

\subsection{Troubleshooting\label{sec:config:troubleshoot}}

If \texttt{configure} fails and reports an error, the message should
be sufficiently clear in most case to understand the cause of the
error and fix it. Do not forget that for libraries installed using
packages, the development versions of those packages are also
necessary, so if configure fails to detect a package which you
believe is installed, check the matching development package.

Also, whether it succeeds or fails, \texttt{configure} generates
a file named \texttt{config.log}, which contains details on tests
run by the script, and is very useful to troubleshoot
configuration problems. When \texttt{configure} fails due to a given
third-party library, details on tests relative to that library
are found in the \texttt{config.log} file. The interesting information
is usually in the middle of the file, so you will need to search
for strings related to the library to find the test that failed
and detailed reasons for its failure.

\section{Compile and install\label{sec:compile}}

Once the code is configured, it may be compiled and installed;
for example, to compile the code (using 4 parallel threads),
then install it:

\texttt{\$ make -j 4 \&\& make install}

To compile the documentation, add:

\texttt{\$ make pdf \&\& make install-pdf}

To clean the build directory, keeping the configuration,
use \texttt{make clean};
To uninstall an installed build, use \texttt{make uninstall}.
To clear all configuration info, use \texttt{make distclean}
(\texttt{make uninstall} will not work after this).

\subsection{Installing to a system directory\label{sec:sys_install}}

When installing to a system directory, such as \texttt{/usr}
or \texttt{/usr/local}, some Linux systems may require running
\texttt{ldconfig} as root or sudoer for the code to work correctly.

\section{Post-install\label{sec:post_install}}

Once the code is installed, a post-install step may
be necessary for computing environments using a batch system,
for separate front-end and compute systems (such as Blue Gene
systems), or for coupling with \syrthes 4 or code\_aster.
The global default MPI execution commands and options may also be overridden.

Copy or rename the \texttt{<install-prefix>/etc/code\_saturne.cfg.template} to \\
\texttt{<install-prefix>/etc/code\_saturne.cfg},
and uncomment and define the applicable sections.

If used, the name of the batch system should match one of the templates \\
in \texttt{<install-prefix>/share/code\_saturne/batch},
and those may also be edited if necessary to match the local
batch configuration\footnote{Some batch systems allow a wide
range of alternate and sometimes incompatible options or keywords,
and it is for all practical purposes impossible to determine
which options are allowed for a given setup, so editing the
batch template to match the local setup may be necessary.}

Also, the \texttt{compute\_versions} section allows the administrator
to define one or several alternate builds which will be used for
compute stages. This is especially useful for installation
on BlueGene type machines, where 2 separate builds are required
(one for the front-end nodes and one for the compute nodes).
The compute-node build may be configured using the
\texttt{--disable-frontend} option so as only to build and install
the components required to run on compute-nodes,
while the front-end build may be configured without MPI support.
The front-end build's post-install step allows definition of
the associated compute build.

All default MPI execution commands and options may be overriden using the
\texttt{mpi} section. Note that only the options defined in this section
are overridden; defaults continue to apply for all others.

For relocatable builds using ParaView/Catalyst, a \texttt{CATALYST\_ROOT\_DIR}
environment variable may be used to specify the Catalyst location
in case that was moved also.

\section{Installing for \syrthes coupling\label{sec:syrthes}}

Coupling with \syrthes 4 requires defining the path to \syrthes 4
at the post-install stage.

When coupling with \syrthes 4, both \CS and \syrthes must
use the same MPI library, and must use the same version of the
PLE (Parallel Location and Exchange) library from \CS. By default, PLE
is built as a sub-library of \CS, but a standalone version may be
configured and built, using the \texttt{libple/configure} script
from the \CS source tree, instead of the top-level \texttt{configure}
script. \CS may then be configured to use the existing install of PLE
using the \texttt{--with-ple} option. Similarly, \syrthes must also
be configured to use PLE.

Alternatively, \syrthes 4 may simply be configured to use the PLE
library from an existing \CS install.

\section{Shell configuration}

If \CS is installed in a non-default system directory (i.e. outside
\texttt{/usr} or \texttt{/usr/local}, it is recommended to define
an alias (in the user's \texttt{.alias} or \texttt{.profile} file, so as to
avoid needing to type the full path when using the code:

\texttt{alias code\_saturne="\$prefix/code\_saturne-\$version/bin/code\_saturne"}

Note that when multiple versions of the code are installed side by side, using
a different alias for each will allow using them simultaneously, with no risk
of confusion.

If using the bash shell, you may also source a bash completion file,
so as to benefit from shell completion for \CS commands
and options, either using

\texttt{. <install-prefix>/etc/bash\_completion.d/code\_saturne}

or

\texttt{source <install-prefix>/etc/bash\_completion.d/code\_saturne}

On some systems, only the latter syntax is effective. For greater
comfort, you should save this setting in your \texttt{.bashrc}
or \texttt{.bash\_profile} file.

\section{Caveats}

\subsubsection{Moving an existing installation}

\textcolor{Red}{\emph{Never move a non-relocatable installation}} of \CS.
Using \texttt{LD\_LIBRARY\_PATH} or \texttt{LD\_PRELOAD}
may allow the executable to run despite \emph{rpath} info
not being up-to-date, but in environments where different library,
versions are available, there is a strong risk of not using
the correct library. In addition, the scripts will not work
unless paths in the installed scripts are updated.

To build a relocatable installation, see section
\ref{sec:config:relocatable}.

If you are packaging the code and need both fine-grained control of
the installation directories, and the possibility to support
options such as \texttt{dpgg}'s \texttt{--instdir}, it is assumed
you have sufficient knowledge to update both \emph{rpath} information
and paths in scripts in the executables and python package directories,
and that the packaging mechanism includes the necessary tools and
scripts to enable this.

In any other case, you should not even think about moving a
non-relocatable build.

If you need to test an installation in a test directory before
installing it in a production directory, use the
\texttt{make install DESTDIR=<test\_prefix>} provided
by the Autotools mechanism rather than configuring an install for a
test directory and then moving it to a production directory.
Another less elegant but safe solution is to configure the build for
installation to a test directory, and once it is tested,
re-configure the build for installation to the final production
directory, and rebuild and install.

\subsubsection{Dynamic linking and path issues on some systems}

On Linux systems and Unix-like, there are several ways for a library or executable
to find dynamic libraries, listed here in decreasing priority:

\begin{itemize}
\item the \texttt{LD\_PRELOAD} environment variable explicitly lists
      libraries to be loaded with maximum priority, before the libraries
      otherwise specified (useful mainly for instrumentation
      and debugging, and should be avoided otherwise);
\item the \texttt{RPATH} binary header of the dependent library or executable;
      (if both are present, the library has priority);
\item the \texttt{LD\_LIBRARY\_PATH} environment variable;
\item the \texttt{RUNPATH} binary header of executable;
\item \texttt{/etc/ld.so.cache};
\item base library directories (\texttt{/lib} and /\texttt{/usr/lib});
\end{itemize}

Note that changing the last two items usually require administrative privileges,
and we have encountered cases where installing to /\texttt{/usr/lib} was not
sufficient without updating \texttt{/etc/ld.so.cache}. We do not consider
\texttt{LD\_PRELOAD} here, as it has other specific uses.

So basically, when using libraries in non-default paths, the remaining
options are between \texttt{RPATH} or \texttt{RUNPATH} binary headers, or the
\texttt{LD\_LIBRARY\_PATH} environment variable.

The major advantage of using binary headers is that the executable
can be run without needing to source a specific environment, which
is very useful, especially when running under MPI (where the propagation
of environment variables may depend on the MPI library and batch system's
configuration), or running under debuggers (where library paths would have
to be sourced first).

In addition, the \texttt{RPATH} binary header has priority over
\texttt{LD\_LIBRARY\_PATH}, allowing the installation to be ``protected''
from settings in the user environment required by other tools.
this is why the \CS installation chooses this mode by default,
unless the \texttt{--enable-relocatable} option is passed to
{\tt configure}.

Unfortunately, the ELF library spec indicates that the use of the
\texttt{DT\_RPATH} entry (for \texttt{RPATH}) has been superseded by the
\texttt{DT\_RUNPATH} (for \texttt{RUNPATH}). Most systems still use
\texttt{RPATH}, but some (such as SUSE and Gentoo) have defaulted to
\texttt{RUNPATH}, which provides no way of ``protecting'' an executable
or library from external settings.

Also, the \texttt{--enable-new-dtags} linker option allows replacing
\texttt{RPATH} with \texttt{RUNPATH}, so adding \texttt{-Wl,--enable-new-dtags}
to the {\tt configure} options will do this.

The addition of \texttt{RUNPATH} to the ELF specifications may have
corrected the oversight of not being able to supersede an executable's
settings when needed (though \texttt{LD\_PRELOAD} is usually sufficient
for debugging, but the bright minds who decided that it should
replace \texttt{RPATH} and not simply supplement it did not provide
a solution for the following scenario:

\begin{enumerate}
\item \CS in installed, along with the MED and HDF libraries,
       on a system where \texttt{--enable-new-dtags} is enabled by default.
\item Another code is installed, with its own (older) versions of MED and HDF
      libraries; this second code requires sourcing environment variables
      including \texttt{LD\_LIBRARY\_PATH} to work at all, so the user
      adds those libraries to his environment (or the admins add it to
      environment modules).
\item \CS now uses the older libraries, and is not capable of reading
      files generated with more recent versions
\end{enumerate}

The aforementioned scenario occurs with \CS and Syrthes, on some
machines, and could occur with \CS and some SALOME libraries, and there is
no way around it short of changing the installation logic of these other tools,
or using a cumbersome wrapper to launch \CS, which could still fail when
\CS needs to load a Syrthes or SALOME environment for coupled cases.
A wrapper would lead to its own problems, as for example Qt is needed
by the GUI but not the executable, so to avoid \emph{causing} issues with
a debugger using its own version of Qt, separate sections would need
to be defined. None of those issues exist with \texttt{RPATH}.

Some administrators consider text-based wrappers a cleaner solution
then hard-coded paths, as they can be updated, but that argument is
in bad faith, as it transfers the problems to the code developers,
while utilities on various systems allow editing those headers.
In addition, in our experience, those administrators with the
strongest stance against use of \texttt{RPATH} are the ones who
most often exceed their own abilities by modifying installations
in ways defying all attempts at quality control (rather than re-installing,
which provides a clean trace), so none has succeeded so far in
presenting us a consistently better solution, though we certainly
would be interested.

So hopefully, you will not be affected by such issues on your
system, but if you are, you may need to define an
\texttt{LD\_LIBRARY\_PATH} environment variable, ugly a solution
as it may be. {\emph{If this causes further issues, please
do not complain to the \CS support team, but please ask those
(administrators or Linux distribution packagers) who decided upon this
policy to solve those issues they created}.

\end{document}