\documentclass[12pt,a4paper]{article}
\def\version{5.1}
\def\qe{{\sc Quantum ESPRESSO}}

\usepackage{html}

% BEWARE: don't revert from graphicx for epsfig, because latex2html
% doesn't handle epsfig commands !!!
\usepackage{graphicx}

\textwidth = 17cm
\textheight = 24cm
\topmargin =-1 cm
\oddsidemargin = 0 cm

\def\pwx{\texttt{pw.x}}
\def\cpx{\texttt{cp.x}}
\def\phx{\texttt{ph.x}}
\def\nebx{\texttt{neb.x}}
\def\configure{\texttt{configure}}
\def\PWscf{\texttt{PWscf}}
\def\PHonon{\texttt{PHonon}}
\def\CP{\texttt{CP}}
\def\PostProc{\texttt{PostProc}}
\def\NEB{\texttt{PWneb}} % to be decided
\def\make{\texttt{make}}

\begin{document} 
\author{}
\date{}

\def\qeImage{quantum_espresso.pdf}
\def\democritosImage{democritos.pdf}

\begin{htmlonly}
\def\qeImage{quantum_espresso.png}
\def\democritosImage{democritos.png}
\end{htmlonly}

\title{
  \includegraphics[width=5cm]{\qeImage} \hskip 2cm
  \includegraphics[width=6cm]{\democritosImage}\\
  \vskip 1cm
  % title
  \Huge User's Guide for \qe\ \smallskip
  \Large (version \version)
}
%\endhtmlonly

%\latexonly
%\title{
% \epsfig{figure=quantum_espresso.png,width=5cm}\hskip 2cm
% \epsfig{figure=democritos.png,width=6cm}\vskip 1cm
%  % title
%  \Huge User's Guide for \qe \smallskip
%  \Large (version \version)
%}
%\endlatexonly

\maketitle

\tableofcontents

\section{Introduction}

This guide gives a general overview of the contents and of the installation 
of \qe\ (opEn-Source Package for Research in Electronic Structure, Simulation,
and Optimization), version \version.

The \qe\ distribution contains the core packages \PWscf\ (Plane-Wave 
Self-Consistent Field) and \CP\ (Car-Parrinello) for the calculation
of electronic-structure properties within
Density-Functional Theory (DFT), using a Plane-Wave (PW) basis set 
and pseudopotentials. It also includes other packages for
more specialized calculations:
\begin{itemize}
  \item \NEB:
        energy barriers and reaction pathways through the Nudged Elastic Band 
        (NEB) method.
  \item \PHonon:
        vibrational properties  with Density-Functional Perturbation Theory.
  \item \PostProc: 
        codes and utilities for data postprocessing.
  \item \texttt{PWcond}: 
        ballistic conductance.
  \item \texttt{XSPECTRA}:
        K-edge X-ray adsorption spectra.
  \item \texttt{TD-DFPT}:
        spectra from Time-Dependent 
        Density-Functional Perturbation Theory.
\end{itemize}
The following auxiliary packages are included as well:
\begin{itemize}
\item \texttt{PWgui}:
      a Graphical User Interface, producing input data files for 
      \PWscf\ and some \PostProc\ codes.
\item \texttt{atomic}:
      atomic calculations and pseudopotential generation.
\item \texttt{QHA}:
      utilities for the calculation of projected density of states (PDOS)
      and of the free energy in the Quasi-Harmonic Approximation (to be
      used in conjunction with \PHonon).
\item \texttt{PlotPhon}:
      phonon dispersion plotting utility (to be
      used in conjunction with \PHonon).
\end{itemize}
A copy of required external libraries is also included.
Finally, several additional packages that exploit data produced by \qe\ 
or patch some \qe\ routines can be installed as {\em plug-ins}:
\begin{itemize}
\item \texttt{Wannier90}:
      maximally localized Wannier functions.
\item \texttt{WanT}:
      quantum transport properties with Wannier functions.
\item \texttt{YAMBO}:
      electronic excitations within Many-Body Perturbation Theory:
      GW and Bethe-Salpeter equation.
\item \texttt{PLUMED}:
      calculation of free-energy surface through metadynamics.
\item \texttt{GIPAW} (Gauge-Independent Projector Augmented Waves):
      NMR chemical shifts and EPR g-tensor.
\item \texttt{GWL}: electronic excitations within GW Approximation.
\end{itemize}
Documentation on single packages can be found in the \texttt{Doc/} or
\texttt{doc/} directory of each package. A detailed description of input
data is available for most packages in files \texttt{INPUT\_*.txt} and 
\texttt{INPUT\_*.html}.

The \qe\ codes work on many different types of Unix machines,
including parallel machines using both OpenMP and MPI 
(Message Passing Interface) and GPU-accelerated machines.
\qe\ also runs on Mac OS X and MS-Windows machines:
see section \ref{Sec:Installation}. A GPU-enabled version
of most important codes is available on GitHub:
\texttt{https://github.com/fspiga/QE-GPU}.

Further documentation, beyond what is provided in this guide, can be found in:
\begin{itemize}
  \item the \texttt{Doc/} directory of the \qe\ distribution;
  \item the \qe\ web site \texttt{www.quantum-espresso.org};
  \item the archives of the  mailing list:
   See section \ref{SubSec:Contacts}, ``Contacts'', for more info.
\end{itemize}
People who want to contribute to \qe\ should read the 
Developer Manual: \texttt{Doc/developer\_man.pdf}.

This guide does not explain the basic Unix concepts (shell, execution 
path, directories etc.) and utilities needed to run \qe; it does not 
explain either solid state physics and its computational methods.
If you want to learn the latter, you should first read a good textbook, 
such as e.g. the book by Richard Martin:
{\em Electronic Structure: Basic Theory and Practical Methods},
Cambridge University Press (2004); or:
{\em Density functional theory: a practical introduction}, 
D. S. Sholl, J. A. Steckel (Wiley, 2009); or
{\em Electronic Structure Calculations for Solids and Molecules:
Theory and Computational Methods}, 
J. Kohanoff (Cambridge University Press, 2006). Then you should consult
the documentation of the package you want to use for more specific references.

All trademarks mentioned in this guide belong to their respective owners.

\subsection{People}

The maintenance and further development of the \qe\ distribution
is promoted by the DEMOCRITOS National Simulation Center 
of IOM-CNR under the coordination of
Paolo Giannozzi (Univ.Udine, Italy) and Layla Martin-Samos 
(Univ.Nova Gorica) with the strong support
of the CINECA National Supercomputing Center in Bologna under 
the responsibility of Carlo Cavazzoni.

Main contributors to \qe, in addition to the authors of the paper 
mentioned in Sect.\ref{SubSec:Terms}, are acknowledged in the 
documentation of each package. An alphabetic list of further
contributors who answered questions on the mailing list, found 
bugs, helped in porting to new architectures, wrote some code,
contributed in some way or another at some stage, follows:
\begin{quote}
  {\AA}ke Sandgren, Audrius Alkauskas, Alain Allouche, Francesco Antoniella,
  Uli Aschauer,  Francesca Baletto, Gerardo Ballabio, Mauro Boero, Pietro
  Bonf\`a, Claudia Bungaro, Paolo Cazzato, Gabriele Cipriani, Ismaila Dabo,
  Jiayu Dai, Cesar Da Silva, Alberto Debernardi, Gernot Deinzer, Marco Govoni,
  Thomas Gruber, Martin Hilgeman, Yosuke Kanai, Konstantin Kudin,
  Nicolas Lacorne, Stephane Lefranc, Sergey Lisenkov, Kurt Maeder,
  Andrea Marini, Giuseppe Mattioli, Nicolas Mounet, William Parker,
  Pasquale Pavone, Mickael Profeta, Guido Roma, Kurt Stokbro, 
  David Strubbe, Sylvie Stucki, Paul Tangney,  Pascal Thibaudeau, 
  Antonio Tilocca, Jaro Tobik, Malgorzata Wierzbowska, Vittorio Zecca,
  Silviu Zilberman, Federico Zipoli,
\end{quote}
and let us apologize to everybody we have forgotten.
 
\subsection{Contacts}
\label{SubSec:Contacts}

The web site for \qe\ is \texttt{http://www.quantum-espresso.org/}.
Releases and patches can be downloaded from this
site or following the links contained in it. The main entry point for 
developers is the QE-forge web site:
\texttt{http://qe-forge.org/}, and in particular the page dedicated to
the \qe\ project: \texttt{qe-forge.org/gf/project/q-e/}.

The recommended place where to ask questions about installation 
and usage of \qe, and to report problems, is the \texttt{pw\_forum} 
mailing list: \texttt{pw\_forum@pwscf.org}.
Here you can obtain help from the developers and from 
knowledgeable users. You have to be subscribed (see ``Contacts'' 
section of the web site) in order to post to the  \texttt{pw\_forum} 
list. Please read the guidelines for posting, section \ref{SubSec:Guidelines}!
NOTA BENE: only messages that appear to come from the 
registered user's e-mail address, in its {\em exact form}, will be
accepted. Messages "waiting for moderator approval" are
automatically deleted with no further processing (sorry, too 
much spam). In case of trouble, carefully check that your return 
e-mail is the correct one (i.e. the one you used to subscribe).

Since \texttt{pw\_forum} has a sizable traffic, an alternative
low-traffic list, \texttt{pw\_users@pwscf.org}, is provided for
those interested only in \qe-related news, such as e.g. announcements 
of new versions, tutorials, etc.. You can subscribe (but not post) to 
this list from the web site, ``Contacts'' section.

If you need to contact the developers for {\em specific} questions 
about coding, proposals, offers of help, etc., please send a message
to the developers' mailing list: \texttt{q-e-developers@qe-forge.org}.
Do not post general questions: they will be ignored.

\subsection{Guidelines for posting to the mailing list}
\label{SubSec:Guidelines}
Life for subscribers of \texttt{pw\_forum} will be easier if everybody 
complies with the following guidelines:
\begin{itemize}
\item Before posting, {\em please}: browse or search the archives -- 
  links are available in the ``Contacts'' section  of the   web site.
  Most questions are asked over and over again. Also: make an attempt
  to search the
  available documentation, notably the FAQs and the User Guide(s).
  The answer to most questions is already there.
\item Reply to both the mailing list and the author or the post, using 
  ``Reply to all'' (not ``Reply'': the Reply-To: field no longer points
   to the mailing list).
\item Sign your post with your name and affiliation.
\item Choose a meaningful subject. Do not use "reply" to start a new
  thread:
  it will confuse the ordering of messages into threads that most mailers
  can do. In particular, do not use "reply" to a Digest!!!
\item Be short: no need to send 128 copies of the same error message just
  because you this is what came out of your 128-processor run. No need to
  send the entire compilation log for a single error appearing at the end.
\item Avoid excessive or irrelevant quoting of previous messages. Your
  message must be immediately visible and easily readable, not hidden
  into a sea of quoted text.
\item Remember that even experts cannot guess where a problem lies in
  the absence of sufficient information. One piece of information that
  must {\em always} be provided is the version number of \qe.
\item Remember that the mailing list is a voluntary endeavor: nobody is 
  entitled to an answer, even less to an immediate answer.
\item Finally, please note that the mailing list is not a replacement
  for your own work, nor is it a replacement for your thesis director's 
  work.
\end{itemize}
 
\subsection{Terms of use}
\label{SubSec:Terms}

\qe\ is free software, released under the 
GNU General Public License. See
\texttt{http://www.gnu.org/licenses/old-licenses/gpl-2.0.txt}, 
or the file License in the distribution).
    
We shall greatly appreciate if scientific work done using \qe\ distribution will 
contain an explicit acknowledgment and the following reference:
\begin{quote}
P. Giannozzi, S. Baroni, N. Bonini, M. Calandra, R. Car, C. Cavazzoni,
D. Ceresoli, G. L. Chiarotti, M. Cococcioni, I. Dabo, A. Dal Corso,
S. Fabris, G. Fratesi, S. de Gironcoli, R. Gebauer, U. Gerstmann,
C. Gougoussis, A. Kokalj, M. Lazzeri, L. Martin-Samos, N. Marzari,
F. Mauri, R. Mazzarello, S. Paolini, A. Pasquarello, L. Paulatto,
C. Sbraccia, S. Scandolo, G. Sclauzero, A. P. Seitsonen, A. Smogunov,
P. Umari, R. M. Wentzcovitch, J.Phys.:Condens.Matter 21, 395502 (2009),
http://arxiv.org/abs/0906.2569
\end{quote}
Note the form \qe\ for textual citations of the code.
Please also see package-specific documentation for
further recommended citations.
Pseudopotentials should be cited as (for instance)
\begin{quote}
[ ] We used the pseudopotentials C.pbe-rrjkus.UPF
and O.pbe-vbc.UPF from\\
\texttt{http://www.quantum-espresso.org}.
\end{quote}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Installation}

For machines with GPU acceleration, see the page
\texttt{qe-forge.org/gf/project/q-e-gpu/} and the
file \texttt{README.GPU} in the GPU-enabled distribution
for more specific information.

\subsection{Download}
\label{SubSec:Download}
 
Presently, \qe\ is distributed in source form; some precompiled 
executables (binary files) are provided for \texttt{PWgui}.
Packages for the Debian Linux distribution are however 
made available by \texttt{debichem} developers.
Stable releases of the \qe\ source package (current version 
is \version) can be downloaded from the Download section
of \texttt{www.quantum-espresso.org}. If you plan to run
on GPU machines, download the GPU-enabled version, also reachable 
from the same link. 

Uncompress and unpack the base distribution using the command:
\begin{verbatim}
     tar zxvf espresso-X.Y.Z.tar.gz
\end{verbatim}
(a hyphen before "zxvf" is optional) where \texttt{X.Y.Z} stands for the
version number. If your version of \texttt{tar} 
doesn't recognize the "z" flag:
\begin{verbatim}
     gunzip -c espresso-X.Y.Z.tar.gz | tar xvf -
\end{verbatim}
A directory \texttt{espresso-X.Y.Z/} will be created. 

Additional packages that are not included in the base distribution
will be downloaded on demand at compile time, using \texttt{make} 
(see Sec.\ref{SubSec:Compilation}).
Note however that this will work only if the computer you are 
installing on is directly connected to the internet and has
either \texttt{wget} or \texttt{curl} installed and working. 
If you run into trouble, manually download each required package 
into subdirectory \texttt{archive/}, {\em not unpacking or
uncompressing it}:
command \texttt{make} will take care of this during installation. 

Package \texttt{GWL} needs a manual download and installation:
please follow the instructions given at \texttt{gww.qe-forge.org}.

% Occasionally, patches for the current version, fixing some errors and bugs,
% may be distributed as a "diff" file. In order to install a patch (for 
% instance):
% \begin{verbatim}
%    cd espresso-X.Y.Z/
%    patch -p1 < /path/to/the/diff/file/patch-file.diff
% \end{verbatim}
%If more than one patch is present, they should be applied in the correct order.

% Daily snapshots of the development version can be downloaded from the
%developers' site \texttt{qe-forge.org}: follow the link ''Quantum ESPRESSO'', 
%then ''SCM''.

The bravest may access the development version via anonymous access to the
Subversion (SVN) repository: \texttt{qe-forge.org/gf/project/q-e/scmsvn}, 
link ''Access Info'' on the left. See also the Developer Manual
(\texttt{Doc/developer\_man.pdf}), section ''Using SVN''.
Beware: the development version 
is, well, under development: use at your own risk! 

The \qe\ distribution contains several directories. Some of them are
common to all packages:

\begin{tabular}{ll}
\texttt{Modules/} &  source files for modules that are common to all programs\\
\texttt{include/} &  files *.h included by fortran and C source files\\
\texttt{clib/}    &  external libraries written in C\\
\texttt{flib/}    &  external libraries written in Fortran\\
\texttt{install/} &  installation scripts and utilities\\
\texttt{pseudo}/  &  pseudopotential files used by examples\\
\texttt{upftools/}&  converters to unified pseudopotential format (UPF)\\
\texttt{Doc/}     &  general documentation\\
\texttt{archive/} &  contains plug-ins in .tar.gz form\\
\end{tabular}
\\
while others are specific to a single package:

\begin{tabular}{ll}
\texttt{PW/}      & \PWscf\ package\\
\texttt{NEB/}     & \NEB\ package\\
\texttt{PP/}      & \PostProc\ package\\
\texttt{PHonon/}  & \PHonon\ package\\
\texttt{PWCOND/}  & \texttt{PWcond}\ package\\
\texttt{CPV/}     & \CP\ package\\
\texttt{atomic/}  & \texttt{atomic} package\\
\texttt{GUI/}     & \texttt{PWGui} package\
\end{tabular}

Finally, directory \texttt{COUPLE/} contains code and documentation
that is useful to call \qe\ programs from external codes.
\subsection{Prerequisites}
\label{Sec:Installation}

To install \qe\ from source, you need first of all a minimal Unix 
environment: basically, a command shell (e.g.,
bash or tcsh) and the utilities \make, \texttt{awk}, \texttt{sed}.
 MS-Windows users need to have Cygwin (a UNIX environment which
 runs under Windows) installed:
see \texttt{http://www.cygwin.com/}. Note that the scripts contained 
in the distribution assume that the local  language is set to the 
standard, i.e. "C"; other settings 
may break them. Use \texttt{export LC\_ALL=C} (sh/bash) or
\texttt{setenv LC\_ALL C} (csh/tcsh) to prevent any problem 
when running scripts (including installation scripts).

Second, you need C and Fortran-95 compilers. For parallel 
execution, you will also need MPI libraries and a parallel
(i.e. MPI-aware) compiler. For massively parallel machines, or 
for simple multicore parallelization, an OpenMP-aware compiler
and libraries are also required.

Big machines with
specialized hardware (e.g. IBM SP, CRAY, etc) typically have a
Fortran-95 compiler with MPI and OpenMP libraries bundled with 
the software. Workstations or ``commodity'' machines, using PC 
hardware, may or may not have the needed software. If not, you need 
either to buy a commercial product (e.g Portland) or to install
an open-source compiler like gfortran or g95. 
Note that several commercial compilers are available free of charge
under some license for academic or personal usage (e.g. Intel, Sun). 

\subsection{\configure}

To install the \qe\ source package, run the \configure\
script. This is actually a wrapper to the true \configure,
located in the \texttt{install/} subdirectory. \configure\
will (try to) detect compilers and libraries available on
your machine, and set up things accordingly. Presently it is expected
to work on most Linux 32- and 64-bit PCs (all Intel and AMD CPUs) and 
PC clusters, SGI Altix, IBM SP and BlueGene machines, NEC SX, Cray XT
machines, Mac OS X, MS-Windows PCs, and (for experts!) on several 
GPU-accelerated hardware. Detailed installation instructions for some
specific HPC machines can be found in files \texttt{install/README.}{\em sys},
where {\em sys} is the machine name.

Instructions for the impatient:
\begin{verbatim}
    cd espresso-X.Y.Z/
    ./configure
     make all
\end{verbatim}
Symlinks to executable programs will be placed in the
\texttt{bin/}
subdirectory. Note that both C and Fortran compilers must be in your execution
path, as specified in the PATH environment variable.
Additional instructions for special machines:

\begin{tabular}{ll}
    \texttt{./configure ARCH=crayxt4}& for CRAY XT machines \\
    \texttt{./configure ARCH=necsx}   & for NEC SX machines \\
    \texttt{./configure ARCH=ppc64-mn}& PowerPC Linux + xlf (Marenostrum) \\
    \texttt{./configure ARCH=ppc64-bg}& IBM BG/P (BlueGene)
\end{tabular}

\noindent \configure\ generates the following files:

\begin{tabular}{ll}
\texttt{make.sys} &      compilation rules and flags (used by \texttt{Makefile})\\
\texttt{install/configure.msg} & a report of the configuration run (not needed for compilation)\\
\texttt{install/config.log} & detailed log of the configuration run (may be needed for debugging)\\
\texttt{include/fft\_defs.h} &    defines fortran variable for C pointer (used only by FFTW)\\
\texttt{include/c\_defs.h} &      defines C to fortran calling convention\\
                           & and a few more definitions used by C files\\
\end{tabular}\\
NOTA BENE: unlike previous versions, \configure\ no longer runs the 
\texttt{makedeps.sh} shell script that updates dependencies. If you modify the  
sources, run \texttt{./install/makedeps.sh} or type \texttt{make depend}
to update files \texttt{make.depend} in the various subdirectories.
    
You should always be able to compile the \qe\ suite
of programs without having to edit any of the generated files. However you
may have to tune \configure\ by specifying appropriate environment variables
and/or command-line options. Usually the tricky part is to get external
libraries recognized and used: see Sec.\ref{Sec:Libraries}
for details and hints.

Environment variables may be set in any of these ways:
\begin{verbatim}
     export VARIABLE=value; ./configure             # sh, bash, ksh
     setenv VARIABLE value; ./configure             # csh, tcsh
     ./configure VARIABLE=value                     # any shell
\end{verbatim}
Some environment variables that are relevant to \configure\ are:

\begin{tabular}{ll}
\texttt{ARCH}& label identifying the machine type (see below)\\
\texttt{F90, F77, CC} &names of Fortran 95, Fortran 77, and C compilers\\
\texttt{MPIF90} &       name of parallel Fortran 95 compiler (using MPI)\\
\texttt{CPP} &          source file preprocessor (defaults to \$CC -E)\\
\texttt{LD} &           linker (defaults to \$MPIF90)\\
\texttt{(C,F,F90,CPP,LD)FLAGS}& compilation/preprocessor/loader flags\\
\texttt{LIBDIRS}&     extra directories where to search for libraries\\
\end{tabular}\\
For example, the following command line:
\begin{verbatim}
     ./configure MPIF90=mpf90 FFLAGS="-O2 -assume byterecl" \
                  CC=gcc CFLAGS=-O3 LDFLAGS=-static
\end{verbatim}
instructs \configure\ to use \texttt{mpf90} as Fortran 95 compiler 
with flags \texttt{-O2 -assume byterecl}, \texttt{gcc} as C compiler with 
flags \texttt{-O3}, and to link with flag \texttt{-static}. 
Note that the value of \texttt{FFLAGS} must be quoted, because it contains
spaces. NOTA BENE: do not pass compiler names with the leading path
included. \texttt{F90=f90xyz} is ok, \texttt{F90=/path/to/f90xyz} is not. 
Do not use
environmental variables with \configure\ unless they are needed! try
\configure\ with no options as a first step.

If your machine type is unknown to \configure, you may use the 
\texttt{ARCH}
variable to suggest an architecture among supported ones. Some large
parallel machines using a front-end (e.g. Cray XT) will actually
need it, or else \configure\ will correctly recognize the front-end
but not the specialized compilation environment of those
machines. In some cases, cross-compilation requires to specify the target machine with the
\texttt{--host} option. This feature has not been extensively
tested, but we had at least one successful report (compilation 
for NEC SX6 on a PC). Currently supported architectures are:

\begin{tabular}{ll}
\texttt{ia32}&    Intel 32-bit machines (x86) running Linux\\
\texttt{ia64}&    Intel 64-bit (Itanium) running Linux\\
\texttt{x86\_64}&  Intel and AMD 64-bit running Linux - see note below\\
\texttt{aix}&     IBM AIX machines\\
\texttt{solaris}& PC's running SUN-Solaris\\
\texttt{sparc}&   Sun SPARC machines\\
\texttt{crayxt4}& Cray XT4/XT5/XE machines\\
\texttt{mac686}&  Apple Intel machines running Mac OS X\\
\texttt{cygwin}&  MS-Windows PCs with Cygwin\\
\texttt{mingw32}& Cross-compilation for MS-Windows, using mingw, 32 bits\\
\texttt{mingw64}& As above, 64 bits\\
\texttt{necsx}&   NEC SX-6 and SX-8 machines\\
\texttt{ppc64}&   Linux PowerPC machines, 64 bits\\
\texttt{ppc64-mn}&as above, with IBM xlf compiler\\
\texttt{ppc64-bg}&IBM BlueGene\\
\texttt{arm}     &ARM machines (with gfortran)
\end{tabular}\\
{\em Note}: \texttt{x86\_64} replaces \texttt{amd64} since v.4.1. 
Cray Unicos machines, SGI 
machines with MIPS architecture, HP-Compaq Alphas are no longer supported
since v.4.2; PowerPC Macs are no longer
supported since v.5.0.
Finally, \configure\ recognizes the following command-line options:\\
\begin{tabular}{ll}
\texttt{--enable-parallel}&     compile for parallel (MPI) execution if possible (default: yes)\\
\texttt{--enable-openmp}&       compile for OpenMP execution if possible (default: no)\\
\texttt{--enable-shared}&       use shared libraries if available (default: yes;\\
                        &       "no" is implemented, untested, in only a few cases)\\
\texttt{--enable-debug}&        compile with debug flags (only for selected cases; default: no)\\
\texttt{--disable-wrappers}&    disable C to fortran wrapper check (default: enabled)\\
\texttt{--enable-signals}&      enable signal trapping (default: disabled)\\
\end{tabular}\\
and the following optional packages:\\
\begin{tabular}{ll}
\texttt{--with-internal-blas}&    compile with internal BLAS (default: no)\\
\texttt{--with-internal-lapack}&  compile with internal LAPACK (default: no)\\
\texttt{--with-scalapack=no}&     do not use ScaLAPACK (default: yes)\\
\texttt{--with-scalapack=intel}&  use ScaLAPACK for Intel MPI (default:OpenMPI)\\
\end{tabular}\\
If you want to modify the \configure\ script (advanced users only!), 
see the Developer Manual.

\subsubsection{Manual configuration}
\label{SubSec:manconf}
If \configure\ stops before the end, and you don't find a way to fix
it, you have to write working \texttt{make.sys}, \texttt{include/fft\_defs.h}
and \texttt{include/c\_defs.h} files. 
For the latter two files, follow the explanations in 
\texttt{include/defs.h.README}.

If \configure\ has run till the end, you should need only to
edit \texttt{make.sys}. A few sample \texttt{make.sys} files
are provided in \texttt{install/Make.}{\em system}. The template used 
by \configure\ is also found there as \texttt{install/make.sys.in} 
and contains explanations of the meaning
of the various variables. Note that you may need 
to select appropriate preprocessing flags
in conjunction with the desired or available
libraries (e.g. you need to add \texttt{-D\_\_FFTW} to \texttt{DFLAGS}
if you want to link internal FFTW). For a correct choice of preprocessing 
flags, refer to the documentation in \texttt{include/defs.h.README}.

NOTA BENE: If you change any settings (e.g. preprocessing,
compilation flags) 
after a previous (successful or failed) compilation, you must run 
\texttt{make clean} before recompiling, unless you know exactly which 
routines are affected by the changed settings and how to force their recompilation.

\subsection{Libraries}
\label{Sec:Libraries}

\qe\ makes use of the following external libraries:
\begin{itemize}
\item BLAS (\texttt{http://www.netlib.org/blas/}) and 
\item LAPACK (\texttt{http://www.netlib.org/lapack/}) for linear algebra 
\item FFTW (\texttt{http://www.fftw.org/}) for Fast Fourier Transforms
\end{itemize}
A copy of the needed routines is provided with the distribution. However,
when available, optimized vendor-specific libraries should be used: this
often yields huge performance gains.

\paragraph{BLAS and LAPACK} 
\qe\ can use the following architecture-specific replacements for BLAS and LAPACK:\\
\begin{quote}
MKL for Intel Linux PCs\\
ACML for AMD Linux PCs\\
ESSL for IBM machines\\
SCSL for SGI Altix\\
SUNperf for Sun
\end{quote}
If none of these is available, we suggest that you use the optimized ATLAS library: see \\
\texttt{http://math-atlas.sourceforge.net/}. Note that ATLAS is not
a complete replacement for LAPACK: it contains all of the BLAS, plus the
LU code, plus the full storage Cholesky code. Follow the instructions in the
ATLAS distributions to produce a full LAPACK replacement.
    
Sergei Lisenkov reported success and good performances with optimized
BLAS by Kazushige Goto. The library is now available under an 
open-source license: see the GotoBLAS2 page at \\
\texttt{http://www.tacc.utexas.edu/tacc-software/gotoblas2/}.

\paragraph{FFT}
\qe\ has an internal copy of an old FFTW version, and it 
can use the following vendor-specific FFT libraries:
\begin{quote}
      IBM ESSL\\
      SGI SCSL\\
      SUN sunperf\\
      NEC ASL
\end{quote}
\configure\ will first search for vendor-specific FFT libraries;
if none is found, it will search for an external FFTW v.3 library;
if none is found, it will fall back to the internal  copy of FFTW.

If you have recent versions (v.10 or later) of MKL installed, you 
may use the FFTW3 interface provided with MKL. This can be directly
linked in MKL distributed with v.12 of the Intel compiler. In earlier 
versions, only sources are distributed: you have to compile them and 
to modify file \texttt{make.sys} accordingly 
(MKL must be linked {\em after} the FFTW-MKL interface).

\paragraph{MPI libraries} 
MPI libraries are usually needed for parallel execution 
(unless you are happy with OpenMP multicore parallelization).
In well-configured machines, \configure\ should find the appropriate
parallel compiler for you, and this should find the appropriate
libraries. Since often this doesn't 
happen, especially on PC clusters, see Sec.\ref{SubSec:LinuxPCMPI}.

\paragraph{Other libraries}
\qe\ can use the MASS vector math
library from IBM, if available (only on AIX).

\paragraph{If optimized libraries are not found}
The \configure\ script attempts to find optimized libraries, but may fail
if they have been installed in non-standard places. You should examine
the final value of \texttt{BLAS\_LIBS, LAPACK\_LIBS, FFT\_LIBS, MPI\_LIBS} (if needed),
\texttt{MASS\_LIBS} (IBM only), either in the output of \configure\ or in the generated
\texttt{make.sys}, to check whether it found all the libraries that you intend to use.
    
If some library was not found, you can specify a list of directories to search
in the environment variable \texttt{LIBDIRS}, 
and rerun \configure; directories in the
list must be separated by spaces. For example:
\begin{verbatim}
   ./configure LIBDIRS="/opt/intel/mkl70/lib/32 /usr/lib/math"
\end{verbatim}
If this still fails, you may set some or all of the \texttt{*\_LIBS} variables manually
and retry. For example:
\begin{verbatim}
   ./configure BLAS_LIBS="-L/usr/lib/math -lf77blas -latlas_sse"
\end{verbatim}
Beware that in this case, \configure\ will blindly accept the specified value,
and won't do any extra search. 
    
\subsection{Compilation}
\label{SubSec:Compilation}

There are a few adjustable parameters in \texttt{Modules/parameters.f90}. 
The
present values will work for most cases. All other variables are dynamically
allocated: you do not need to recompile your code for a different system.
    
At your choice, you may compile the complete \qe\ suite of programs 
(with \texttt{make all}), or only some specific programs. \texttt{make} with no arguments yields a list of valid compilation targets:
\begin{itemize}
\item \texttt{make pw}  compiles the self-consistent-field package \PWscf
\item \texttt{make cp}  compiles the Car-Parrinello package \CP
\item \texttt{make neb} downloads \NEB\ package from \texttt{qe-forge}
			unpacks it and compiles it. All executables are linked
			in main \texttt{bin} directory
\item \texttt{make ph}  downloads \PHonon\ package from \texttt{qe-forge}
			unpacks it and compiles it. All executables are linked
			in main \texttt{bin} directory
\item \texttt{make pp}  compiles the postprocessing package \PostProc
\item \texttt{make pwcond} downloads the balistic conductance package \texttt{PWcond}
			from \texttt{qe-forge}
			unpacks it and compiles it. All executables are linked
			in main \texttt{bin} directory
\item \texttt{make pwall} produces all of the above.
\item \texttt{make ld1}  downloads the pseudopotential generator package \texttt{atomic} 
			from \texttt{qe-forge}
			unpacks it and compiles it. All executables are linked
			in main \texttt{bin} directory
\item \texttt{make xspectra} downloads the package \texttt{XSpectra} 
			from \texttt{qe-forge}
			unpacks it and compiles it. All executables are linked
			in main \texttt{bin} directory
\item \texttt{make upf} produces utilities for pseudopotential conversion in
                        directory \texttt{upftools/}
\item \texttt{make all} produces all of the above
\item \texttt{make plumed} unpacks \texttt{PLUMED}, patches several routines
                           in \texttt{PW/}, \texttt{CPV/} and \texttt{clib/},
                           recompiles \PWscf\ and \CP\ with \texttt{PLUMED}
                           support
\item \texttt{make w90} downloads \texttt{wannier90}, unpacks it, copies an appropriate
                       \texttt{make.sys} file,  produces all executables
                       in \texttt{W90/wannier90.x} and in \texttt{bin/}
\item \texttt{make want} downloads \texttt{WanT} from \texttt{qe-forge}, 
			 unpacks it, runs its \configure,
                         produces all executables for \texttt{WanT} in 
                         \texttt{WANT/bin}.
\item \texttt{make yambo} downloads \texttt{yambo} from \texttt{qe-forge}, 
			  unpacks it, runs its \configure,
                          produces all \texttt{yambo} executables in 
                          \texttt{YAMBO/bin}
\item \texttt{make gipaw} downloads \texttt{GIPAW} from \texttt{qe-forge},
                          unpacks it, runs its \configure,
                          produces all \texttt{GIPAW} executables in 
                          \texttt{GIPAW/bin} and in main \texttt{bin} directory.
\end{itemize}
For the setup of the GUI, refer to the \texttt{PWgui-X.Y.Z /INSTALL} file, where
X.Y.Z stands for the version number of the GUI (should be the same as the
general version number). If you are using the SVN sources, see
the \texttt{GUI/README} file instead.

If \texttt{make} refuses for some reason to download additional 
packages, manually download them into subdirectory 
\texttt{archive/}, {\em not unpacking or or uncompressing them},
and try \texttt{make} again. Also see Sec.(\ref{SubSec:Download}).
   
\subsection{Running tests and examples}
\label{SubSec:Examples}

% You should first of all ensure that you have downloaded 
% and correctly unpacked the package containing examples (since v.4.1 in a
% separate package):
% \begin{verbatim}
%      tar -zxvf /path/to/package/espresso-X.Y.Z-examples.tar.gz
% \end{verbatim}
% will unpack several subdirectories into \texttt{espresso-X.Y.Z/}. 

As a final check that compilation was successful, you may want to run some or
all of the examples. There are two different types of examples: 
\begin{itemize}
\item automated tests. Quick and exhaustive, but not
meant to be realistic, implemented only for \PWscf\ and \CP.
\item examples.
Cover many more programs and features of the \qe\ distribution,
but they require manual inspection of the results. 
\end{itemize}
Instructions for the impatient:
\begin{verbatim}
   cd PW/tests/
   ./check_pw.x.j
\end{verbatim}
for \PWscf;
\texttt{PW/tests/README} contains a list of what is tested.
For \CP:
\begin{verbatim}
   cd CPV/tests/
   ./check_cp.x.j
\end{verbatim}
Instructions for all others: edit file \texttt{environment\_variables},
setting the following variables as needed.
\begin{quote}
   BIN\_DIR: directory where executables reside\\
   PSEUDO\_DIR: directory where pseudopotential files reside\\
   TMP\_DIR: directory to be used as temporary storage area
\end{quote}
The default values of BIN\_DIR and PSEUDO\_DIR should be fine, 
unless you have installed things in nonstandard places. TMP\_DIR 
must be a directory where you have read and write access to, with 
enough available space to host the temporary files produced by the 
example runs, and possibly offering high I/O performance (i.e., don't 
use an NFS-mounted directory). NOTA BENE: do not use a
directory containing other data: the examples will clean it!

If you have compiled the parallel version of \qe\ (this
is the default if parallel libraries are detected), you will usually
have to specify a launcher program (such as \texttt{mpirun} or 
\texttt{mpiexec}) and the number of processors: see Sec.\ref{Sec:para} for
details. In order to do that, edit again the \texttt{environment\_variables} 
file
and set the PARA\_PREFIX and PARA\_POSTFIX variables as needed. 
Parallel executables will be run by a command like this: 
\begin{verbatim}
      $PARA_PREFIX pw.x $PARA_POSTFIX -i file.in > file.out
\end{verbatim}
For example, if the command line is like this (as for an IBM SP):
\begin{verbatim}
      poe pw.x -procs 4 -i file.in > file.out
\end{verbatim}
you should set PARA\_PREFIX="poe", PARA\_POSTFIX="-procs
4". Furthermore, if your machine does not support interactive use, you
must run the commands specified above through the batch queuing
system installed on that machine. Ask your system administrator for
instructions. For execution using OpenMP on N threads, 
you should set PARA\_PREFIX to \texttt{"env OMP\_NUM\_THREADS=N ... "}.

Notice that most tests and examples are devised to be run serially 
or on a small number of processors; do not use tests and examples
to benchmark parallelism, do not try to run on too many processors.

To run an example, go to the corresponding directory (e.g.
 \texttt{PW/examples/example01}) and execute: 
\begin{verbatim}
      ./run_example
\end{verbatim}
This will create a subdirectory \texttt{results/}, containing the input and
output files generated by the calculation. Some examples take only a
few seconds to run, while others may require several minutes depending
on your system.

In each example's directory, the \texttt{reference/} subdirectory contains
verified output files, that you can check your results against. They
were generated on a Linux PC using the Intel compiler. On different
architectures the precise numbers could be slightly different, in
particular if different FFT dimensions are automatically selected. For
this reason, a plain diff of your results against the reference data
doesn't work, or at least, it requires human inspection of the results. 

The example scripts stop if an error is detected. You should look {\em inside}
the last written output file to understand why.

\subsection{Installation tricks and problems}

\subsubsection{All architectures}
\begin{itemize}
\item
Working Fortran-95 and C compilers are needed in order
to compile \qe. Most ``Fortran-90'' compilers actually
implement the Fortran-95 standard, but older versions 
may not be Fortran-95 compliant. Moreover, 
C and Fortran compilers must be in your PATH.
If \configure\ says that you have no working compiler, well,
you have no working compiler, at least not in your PATH, and
not among those recognized by \configure.
\item
If you get {\em Compiler Internal Error} or similar messages: your
compiler version is buggy. Try to lower the optimization level, or to
remove optimization just for the routine that has problems. If it
doesn't work, or if you experience weird problems at run time, try to 
install patches for your version of the compiler (most vendors release
at least a few patches for free), or to upgrade to a more recent
compiler version.
\item
If you get error messages at the loading phase that look like 
{\em file XYZ.o: unknown / not recognized/ invalid / wrong
file type / file format / module version},
one of the following things have happened:
\begin{enumerate}
\item you have leftover object files from a compilation with another
  compiler: run \texttt{make clean} and recompile. 
\item \make\ did not stop at the first compilation error (it may 
happen in some software configurations). Remove the file *.o
that triggers the error message, recompile, look for a 
compilation error. 
\end{enumerate}
If many symbols are missing in the loading phase: you did not specify the
location of all needed libraries (LAPACK, BLAS, FFTW, machine-specific
optimized libraries), in the needed order. 
If only symbols from \texttt{clib/} are missing, verify that
you have the correct C-to-Fortran bindings, defined in 
\texttt{include/c\_defs.h}.
Note that \qe\ is self-contained (with the exception of MPI libraries for 
parallel compilation): if system libraries are missing, the problem is in
your compiler/library combination or in their usage, not in \qe.
\item
If you get an error like {\em Can't open module file global\_version.mod}:
your machine doesn't like the script that produces file \texttt{version.f90}
with the correct version and revision. Quick solution: copy 
\texttt{Modules/version.f90.in} to \texttt{Modules/version.f90}.
\item
If you get mysterious errors ("Segmentation faults" and the like) 
in the provided tests and examples:
your compiler, or your mathematical libraries, or MPI libraries,
or a combination thereof, is very likely buggy, or there is some
form of incompatibility (see below). Although the 
presence of subtle bugs in \qe\ that are not revealed during 
the testing phase can never be ruled out, it is very unlikely
that this happens on the provided tests and examples. 
\end{itemize}

\subsubsection{Cray XE and XT machines}

For Cray XE machines:
\begin{verbatim}
$ module swap PrgEnv-cray PrgEnv-pgi
$ ./configure --enable-openmp --enable-parallel --with-scalapack
$ vim make.sys
\end{verbatim}
then manually add \texttt{-D\_\_IOTK\_WORKAROUND1} at the end of \texttt{DFLAGS} line.

''Now, despite what people can imagine, every CRAY machine deployed can
have different environment. For example on the machine I usually use 
for tests [...] I do have to unload some modules to make QE running 
properly. On another CRAY [...] there is also Intel compiler as option
and the system is slightly different compared to the other. 
So my recipe should work, 99\% of the cases.
I strongly suggest you to use PGI, also for a performance point of view.''
(Info by Filippo Spiga, Sept. 2012)

For Cray XT machines, use \texttt{./configure ARCH=crayxt4} or else 
\configure\ will not recognize the Cray-specific software environment. 

Older Cray machines: T3D, T3E, X1, are no longer supported.

\subsubsection{IBM AIX}

v.4.3.1 of the CP code, Wannier-function dynamics, crashes with
``segmentation violation'' on some AIX v.6 machines.
Workaround: compile it with \texttt{mpxlf95} instead of 
\texttt{mpxlf90}. (Info by Roberto Scipioni, June 2011)

On IBM machines with ESSL libraries installed, there is a 
potential conflict between a few LAPACK routines that are also part of ESSL, 
but with a different calling sequence. The appearance of run-time errors like {\em
    ON ENTRY TO ZHPEV  PARAMETER NUMBER  1 HAD AN ILLEGAL VALUE}
is a signal that you are calling the bad routine. If you have defined 
\texttt{-D\_\_ESSL} you should load ESSL before LAPACK: see
variable LAPACK\_LIBS in make.sys.

\subsubsection{IBM BlueGene}

The current \configure\ is tested and works on the machines at CINECA 
and at J\"ulich. For other sites, you may need something like
\begin{verbatim}
  ./configure ARCH=ppc64-bg BLAS_LIBS=...  LAPACK_LIBS=... \
              SCALAPACK_DIR=... BLACS_DIR=..."
\end{verbatim}
where the various *\_LIBS and *\_DIR "suggest" where the various libraries 
are located.

\subsubsection{Linux PC}

Both AMD and Intel CPUs, 32-bit and 64-bit, are supported and work,
either in 32-bit emulation and in 64-bit mode. 64-bit executables 
can address a much larger memory space than 32-bit executable, but
there is no gain in speed.
Beware: the default integer type for 64-bit machine is typically
32-bit long. You should be able to use 64-bit integers as well, 
but it is not guaranteed to work and will not give 
any advantage anyway.

Currently the following compilers are supported by \configure:
Intel (ifort), Portland (pgf90), gfortran, g95, Pathscale (pathf95), 
Sun Studio (sunf95),  AMD Open64 (openf95). The ordering approximately
reflects the quality of support. Both Intel MKL and AMD acml mathematical
libraries are supported. Some combinations of compilers and of libraries
may however require manual editing of \texttt{make.sys}.

It is usually convenient to create semi-statically linked executables (with only
libc, libm, libpthread dynamically linked). If you want to produce a binary
that runs on different machines, compile it on the oldest machine you have
(i.e. the one with the oldest version of the operating system).

If you get errors like {\em IPO Error: unresolved : \_\_svml\_cos2}
at the linking stage, your compiler is optimized to use the SSE
version of sine, cosine etc. contained in the SVML library. Append
\texttt{-lsvml} to the list of libraries in your \texttt{make.sys} 
file (info by Axel Kohlmeyer, oct.2007). 

\paragraph{Linux PCs with Portland compiler (pgf90)}

\qe\ does not work reliably, or not at all, with many old
versions ($< 6.1$) of the Portland Group compiler (pgf90).
 Use the latest version of each 
release of the compiler, with patches if available (see
the Portland Group web site, \texttt{http://www.pgroup.com/}).

\paragraph{Linux PCs with Pathscale compiler}

Version 3.1 and version 4 (open source!) of the Pathscale EKO compiler
work (info by Cezary Sliwa, April 2011, and Carlo Nervi, June 2011).
In case of mysterious errors while compiling \texttt{iotk},
remove all lines like:
\begin{verbatim}
# 1 "iotk_base.spp"
\end{verbatim}
from all \texttt{iotk} source files.

\paragraph{Linux PCs with gfortran}

Only recent versions (at least v.4.4) of gfortran properly compile \qe.
Older versions often produce nonfunctional phonon executables 
(segmentation faults and the like); other versions miscompile iotk 
(the executables work but crash with a mysterious iotk
error when reading from data files). 

"There is a known incompatibility problem between the calling 
convention for Fortran functions that return complex values: there is the
convention used by 
g77/f2c, where in practice the compiler converts such functions to subroutines 
with a further parameter for the return value; gfortran instead produces a 
normal function returning a complex value.
If your system libraries were compiled using g77 (which may happen for
system-provided libraries in not-too-recent Linux distributions),
and you instead use gfortran to compile \qe, your code
may crash or produce random results. This typically happens 
during calls to \texttt{zdotc}, which is one the most commonly used
complex-returning functions of BLAS+LAPACK.

For further details see for instance this link:\\
\texttt{http://www.macresearch.org/lapackblas-fortran-106\#comment-17071}\\
or read the man page of gfortran under the flag \texttt{-ff2c}.

If your code crashes during a call to \texttt{zdotc},
try to recompile \qe\ using the internal BLAS and LAPACK
routines (using the \texttt{--with-internal-blas} and 
\texttt{--with-internal-lapack} parameters of the configure script)
to see if the problem disappears; or, add the \texttt{-ff2c} flag"
(info by Giovanni Pizzi, Jan. 2013).

Note that a similar problem with complex functions exists with MKL libraries
as well: if you compile with gfortran, link \texttt{-lmkl\_gf\_lp64},
not \texttt{-lmkl\_intel\_lp64}, and the like for other architectures.
Since v.5.1, you may use the following workaround:
add preprocessing option \texttt{-Dzdotc=zdotc\_wrapper} to \texttt{DFLAGS}.

If you experience problems in reading files produced by previous versions
of \qe: ``gfortran used 64-bit record markers to allow writing of records 
larger than 2 GB. Before with 32-bit record markers only records $<$2GB 
could be written. However, this caused problems with older files and 
inter-compiler operability. This was solved in GCC 4.2 by using 32-bit 
record markers but such that one can still store $>$2GB records (following 
the implementation of Intel). Thus this issue should be gone. See 4.2 
release notes (item ``Fortran") at 
\texttt{http://gcc.gnu.org/gcc-4.2/changes.html}."
(Info by Tobias Burnus, March 2010).

``Using gfortran v.4.4 (after May 27, 2009) and 4.5 (after May 5, 2009) can 
produce wrong results, unless the environment variable
GFORTRAN\_UNBUFFERED\_ALL=1 is set. Newer 4.4/4.5 versions
(later than April 2010) should be OK. See\\
\texttt{http://gcc.gnu.org/bugzilla/show\_bug.cgi?id=43551}."
(Info by Tobias Burnus, March 2010).

\paragraph{Linux PCs with g95}

g95 v.0.91 and later versions (\texttt{http://www.g95.org}) work. 
The executables that produce are however slower (let us say 20\% or so) 
that those produced by gfortran, which in turn are slower 
(by another 20\% or so) than those produced by ifort.

\paragraph{Linux PCs with Sun Studio compiler}

``The Sun Studio compiler, sunf95, is free (web site:
\texttt{http://developers.sun.com/sunstudio/} and comes  
with a set of algebra libraries that can be used in place of the slow 
built-in libraries. It also supports OpenMP, which g95 does not. On the 
other hand, it is a pain to compile MPI with it. Furthermore the most
recent version has a terrible bug that totally miscompiles the iotk 
input/output library (you'll have to compile it with reduced optimization).''
(info by Lorenzo Paulatto, March 2010).

\paragraph{Linux PCs with AMD Open64 suite}

The AMD Open64 compiler suite, openf95 (web site:
\texttt{http://developer.amd.com/cpu/open64/pages/default.aspx})
can be freely downloaded from the AMD site.
It is recognized by \configure\ but little tested. It sort of works 
but it fails to pass several tests (info by Paolo Giannozzi, March 2010).
"I have configured for Pathscale, then switched to the Open64 compiler by 
editing make.sys. "make pw" succeeded and pw.x did process my file, but with 
"make all" I get an internal compiler error [in CPV/wf.f90]" (info by Cezary 
Sliwa, April 2011).

\paragraph{Linux PCs with Intel compiler (ifort)}

The Intel compiler, ifort, is available for free for personal 
usage (\texttt{http://software.intel.com/}). It produces fast executables, 
at least on Intel CPUs, but not all versions work as expected.
ifort versions $<9.1$ are not recommended, due to the presence of subtle 
and insidious bugs. In case of trouble, update your version with 
the most recent patches,
available via Intel Premier support (registration free of charge for Linux):
\texttt{http://software.intel.com/en-us/articles/intel-software-developer-support}.
Since each major release of ifort
differs a lot from the previous one, compiled objects from different 
releases may be incompatible and should not be mixed.    

If \configure\ doesn't find the compiler, or if you get 
{\em Error loading shared libraries} at run time, you may have 
forgotten to execute the script that
sets up the correct PATH and library path. Unless your system manager has
done this for you, you should execute the appropriate script -- located in
the directory containing the compiler executable -- in your
initialization files. Consult the documentation provided by Intel. 
    
The warning: {\em feupdateenv is not implemented and will always fail}, 
showing up in recent versions, can be safely ignored. Warnings on
"bad preprocessing option" when compiling iotk
and complains about ``recommanded formats''
should also be ignored.

{\bf ifort v.12}: release 12.0.0 miscompiles iotk, leading to 
mysterious errors when reading data files. Workaround: increase 
the parameter BLOCKSIZE to e.g. 131072*1024 when opening files in 
\texttt{iotk/src/iotk\_files.f90} (info by Lorenzo Paulatto,
Nov. 2010). Release 12.0.2 seems to work and to produce faster executables
than previous versions on 64-bit CPUs (info by P. Giannozzi, March 2011).

{\bf ifort v.11}: Segmentation faults were reported for the combination 
ifort 11.0.081, MKL 10.1.1.019, OpenMP 1.3.3. The problem disappeared
with ifort 11.1.056 and MKL 10.2.2.025 (Carlo Nervi, Oct. 2009).

\paragraph{Linux PCs with MKL libraries}
On Intel CPUs it is very convenient to use Intel MKL libraries.
They can be also used for AMD CPU, selecting the appropriate machine-optimized
libraries, and also together with non-Intel compilers. Note however
that recent versions of MKL (10.2 and following) do not perform
well on AMD machines.

\configure\ should recognize properly installed MKL libraries.
By default the non-threaded version of MKL is linked, unless option
\texttt{configure --with-openmp} is specified. In case of trouble,
refer to the following web page to find the correct way to link MKL:\\
\texttt{http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/}.

Recent MKL versions also contain optimized FFT routines and a FFTW interface.

For parallel (MPI) execution on multiprocessor (SMP) machines, set the
environmental variable OMP\_NUM\_THREADS to 1 unless you know what you 
are doing. See Sec.\ref{Sec:para} for more info on this
and on the difference between MPI and OpenMP parallelization. 

\paragraph{Linux PCs with ACML libraries}
For AMD CPUs, especially recent ones, you may find convenient to 
link AMD acml libraries (can be freely downloaded from AMD web site). 
\configure\ should recognize properly installed acml libraries,
together with the compilers most frequently used on AMD systems:
pgf90, pathscale, openf95, sunf95.

\subsubsection{Linux PC clusters with MPI}
\label{SubSec:LinuxPCMPI}
PC clusters running some version of MPI are a very popular
computational platform nowadays. \qe\ is known to work
with at least two of the major MPI implementations (MPICH, LAM-MPI),
plus with the newer MPICH2 and OpenMPI implementation. 
\configure\ should automatically recognize a properly installed
parallel environment and prepare for parallel compilation. 
Unfortunately this not always happens. In fact:
\begin{itemize}
\item \configure\ tries to locate a parallel compiler in a logical
  place with a logical name,  but if it has a strange names or it is
  located  in a strange location, you will have to instruct \configure\ 
  to find it. Note that in many PC clusters (Beowulf), there is no
  parallel Fortran-95 compiler in default installations:  you have to
  configure an appropriate script, such as mpif90. 
\item \configure\ tries to locate libraries (both mathematical and
  parallel libraries) in the usual places with usual names, but if
  they have strange names or strange locations, you will have to
  rename/move them, or to instruct \configure\ to find them. If MPI
  libraries are not found,
  parallel compilation is disabled. 
\item \configure\ tests that the compiler and the libraries are
  compatible (i.e. the compiler may link the libraries without
  conflicts and without missing symbols). If they aren't and the
  compilation fails, \configure\ will revert to serial compilation. 
\end{itemize}

Apart from such problems, \qe\ compiles and works on all non-buggy, properly
configured hardware and software combinations. In some cases you may have to
recompile MPI libraries: not all MPI installations contain support for
the fortran-90 compiler of your choice (or for any fortran-90 compiler
at all!). 

If \qe\ does not work for some reason on a PC cluster,
try first if it works in serial execution. A frequent problem with parallel
execution is that \qe\ does not read from standard input,
due to the configuration of MPI libraries: see Sec.\ref{SubSec:badpara}.
If you are dissatisfied with the performances in parallel execution,
see Sec.\ref{Sec:para} and in particular Sec.\ref{SubSec:badpara}.

\subsubsection{Mac OS}

Newer Mac OS-X machines (10.4 and later) with Intel CPUs are supported 
by \configure,
with gcc4+g95, gfortran, and the Intel compiler ifort with MKL libraries.
Parallel compilation with OpenMPI also works. 

Mysterious crashes, occurring when \texttt{zdotc} is called, are due
to the same incompatibility of complex functions with some optimized 
BLAS as reported in the "Linux PCs with gfortran" paragraph. Workaround:
add preprocessing option \texttt{-Dzdotc=zdotc\_wrapper} to \texttt{DFLAGS}.

\paragraph{Detailed installation instructions for Mac OS X 10.6}

(Instructions for 10.6.3 by Osman Baris Malcioglu, tested as of May 2010)
Summary for the hasty: 
\begin{itemize}
\item GNU fortran:
Install macports compilers, 
Install MPI environment,
Configure \qe\  using
\begin{verbatim}
  ./configure CC=gcc-mp-4.3 CPP=cpp-mp-4.3 CXX=g++-mp-4.3 F77=g95 FC=g95
\end{verbatim}
\item Intel compiler:
Use Version $>11.1.088$,
Use 32 bit compilers,
Install MPI environment,
install macports provided cpp (optional),
Configure \qe\ using
\begin{verbatim}
 ./configure CC=icc CXX=icpc F77=ifort F90=ifort FC=ifort CPP=cpp-mp-4.3
\end{verbatim}
\end{itemize}

\paragraph{Compilation with GNU compilers}.
The following instructions use macports version of gnu compilers due to some
issues in mixing gnu supplied fortran compilers with apple modified gnu compiler
collection. For more information regarding macports please refer to:
\texttt{http://www.macports.org/}  

First install necessary compilers from macports
\begin{verbatim}
   port install gcc43
   port install g95
\end{verbatim}
The apple supplied MPI environment has to be overridden since there is
a new set of compilers now (and Apple provided mpif90 is just an empty 
placeholder since Apple does not provide fortran compilers). I have used
OpenMPI for this case. Recommended minimum configuration line is:
\begin{verbatim}
  ./configure CC=gcc-mp-4.3 CPP=cpp-mp-4.3 CXX=g++-mp-4.3 F77=g95 FC=g95
\end{verbatim}
of course, installation directory should be set accordingly if a multiple
compiler environment is desired. The default installation directory of 
OpenMPI overwrites apple supplied MPI permanently!\\
Next step is \qe\ itself. Sadly, the Apple supplied optimized BLAS/LAPACK
libraries tend to misbehave under different tests, and it is much safer to
use internal libraries. The minimum recommended configuration
line is (presuming the environment is set correctly):
\begin{verbatim}
  ./configure CC=gcc-mp-4.3 CXX=g++-mp-4.3 F77=g95 F90=g95 FC=g95 \
              CPP=cpp-mp-4.3 --with-internal-blas --with-internal-lapack
\end{verbatim}
\paragraph{Compilation with Intel compilers}.
Newer versions of Intel compiler (>11.1.067) support Mac OS X 10.6, and furthermore they are
bundled with intel MKL. 32 bit binaries obtained using 11.1.088 are tested and no problems
have been encountered so far. Sadly, as of 11.1.088 the 64 bit binary misbehave 
under some tests. Any attempt to compile 64 bit binary using v.$<11.1.088$ will result in
very strange compilation errors.   

Like the previous section, I would recommend installing macports compiler suite. 
First, make sure that you are using the 32 bit version of the compilers,
i.e.  
\begin{verbatim}
. /opt/intel/Compiler/11.1/088/bin/ifortvars.sh ia32
\end{verbatim}
\begin{verbatim}
. /opt/intel/Compiler/11.1/088/bin/iccvars.sh ia32
\end{verbatim}
will set the environment for 32 bit compilation in my case. 

Then, the MPI environment has to be set up for Intel compilers similar to previous 
section. 

The recommended configuration line for \qe\ is: 
\begin{verbatim}
 ./configure CC=icc CXX=icpc F77=ifort F90=ifort FC=ifort CPP=cpp-mp-4.3
\end{verbatim}
MKL libraries will be detected automatically if they are in their default locations. 
Otherwise, mklvars32 has to be sourced before the configuration script. 

Security issues: 
MacOs 10.6 comes with a disabled firewall. Preparing a ipfw based firewall is recommended. 
Open source and free GUIs such as "WaterRoof" and "NoobProof" are available that may help 
you in the process.

\newpage

\section{Parallelism}
\label{Sec:para}

\subsection{Understanding Parallelism}

Two different parallelization paradigms are currently implemented 
in \qe:
\begin{enumerate}
\item {\em Message-Passing (MPI)}. A copy of the executable runs 
on each CPU; each copy lives in a different world, with its own
private set of data, and communicates with other executables only
via calls to MPI libraries. MPI parallelization requires compilation 
for parallel execution, linking with MPI libraries, execution using 
a launcher program (depending upon the specific machine). The number of CPUs used
is specified at run-time either as an option to the launcher or
by the batch queue system. 
\item {\em OpenMP}.  A single executable spawn subprocesses
(threads) that perform in parallel specific tasks. 
OpenMP can be implemented via compiler directives ({\em explicit} 
OpenMP) or via {\em multithreading} libraries  ({\em library} OpenMP).
Explicit OpenMP require compilation for OpenMP execution;
library OpenMP requires only linking to a multithreading
version of mathematical libraries, e.g.:
ESSLSMP, ACML\_MP, MKL (the latter is natively multithreading).
The number of threads is specified at run-time in the environment 
variable OMP\_NUM\_THREADS. 
\end{enumerate}

MPI is the well-established, general-purpose parallelization.
In \qe\ several parallelization levels, specified at run-time
via command-line options to the executable, are implemented
with MPI. This is your first choice for execution on a parallel 
machine.

Library OpenMP is a low-effort parallelization suitable for
multicore CPUs. Its effectiveness relies upon the quality of 
the multithreading libraries and the availability of 
multithreading FFTs. If you are using MKL,\footnote{Beware: 
MKL v.10.2.2 has a buggy \texttt{dsyev} yielding wrong results 
with more than one thread; fixed in v.10.2.4}
you may want to select FFTW3 (set \texttt{CPPFLAGS=-D\_\_FFTW3...}
in \texttt{make.sys}) and to link with the MKL interface to FFTW3. 
You will get a decent speedup ($\sim 25$\%) on two cores.

Explicit OpenMP is a recent addition, still under
development, devised to increase scalability on
large multicore parallel machines. Explicit OpenMP can be used
together with MPI and also together with library OpenMP. Beware
conflicts between the various kinds of parallelization! 
If you don't know how to run MPI processes
and OpenMP threads in a controlled manner, forget about mixed 
OpenMP-MPI parallelization.

\subsection{Running on parallel machines}

Parallel execution is strongly system- and installation-dependent. 
Typically one has to specify:
\begin{enumerate}
\item a launcher program (not always needed), 
such as \texttt{poe}, \texttt{mpirun}, \texttt{mpiexec},
  with the  appropriate options (if any);
\item the number of processors, typically as an option to the launcher
  program, but in some cases to be specified after the name of the
  program to be
  executed; 
\item the program to be executed, with the proper path if needed; 
\item other \qe-specific parallelization options, to be
  read and interpreted by the running code.
\end{enumerate}  
Items 1) and 2) are machine- and installation-dependent, and may be 
different for interactive and batch execution. Note that large
parallel machines are  often configured so as to disallow interactive
execution: if in doubt, ask your system administrator.
Item 3) also depend on your specific configuration (shell, execution path, etc). 
Item 4) is optional but it is very important
for good performances. We refer to the next
section for a description of the various 
possibilities.

\subsection{Parallelization levels}

In \qe\ several MPI parallelization levels are 
implemented, in which both calculations
and data structures are distributed across processors.
Processors are organized in a hierarchy of groups, 
which are identified by different MPI communicators level.
The groups hierarchy is as follow:
\begin{itemize}
\item {\bf world}: is the group of all processors (MPI\_COMM\_WORLD).
\item 
{\bf images}: Processors can then be divided into different "images", each corresponding to a
different self-consistent or linear-response
calculation, loosely coupled to others.
\item
{\bf pools}: each image can be subpartitioned into
"pools", each taking care of a group of k-points.
\item
{\bf bands}: each pool is subpartitioned into
"band groups", each taking care of a group
of Kohn-Sham orbitals (also called bands, or
wavefunctions) (still experimental)
\item
{\bf PW}: orbitals in the PW basis set,
as well as charges and density in either 
reciprocal or real space, are distributed
across processors.
This is usually referred to as "PW parallelization".
All linear-algebra operations on array of  PW / 
real-space grids are automatically and effectively parallelized.
3D FFT is used to transform electronic wave functions from
reciprocal to real space and vice versa. The 3D FFT is
parallelized by distributing planes of the 3D grid in real
space to processors (in reciprocal space, it is columns of
G-vectors that are distributed to processors). 
\item
{\bf tasks}: 
In order to allow good parallelization of the 3D FFT when 
the number of processors exceeds the number of FFT planes,
FFTs on Kohn-Sham states are redistributed to 
"task" groups so that each group 
can process several wavefunctions at the same time.
\item
{\bf linear-algebra group}:
A further level of parallelization, independent on
PW or k-point parallelization, is the parallelization of
subspace diagonalization / iterative orthonormalization.
 Both operations required the diagonalization of 
arrays whose dimension is the number of Kohn-Sham states
(or a small multiple of it). All such arrays are distributed block-like
across the ``linear-algebra group'', a subgroup of the pool of processors,
organized in a square 2D grid. As a consequence the number of processors
in the linear-algebra group is given by $n^2$, where $n$ is an integer;
$n^2$ must be smaller than the number of processors in the PW group.
The diagonalization is then performed
in parallel using standard linear algebra operations.
(This diagonalization is used by, but should not be confused with,
the iterative Davidson algorithm). The preferred option is to use
ScaLAPACK; alternative built-in algorithms are anyway available.
\end{itemize}
Note however that not all parallelization levels 
are implemented in all codes! 

\paragraph{About communications}
Images and pools are loosely coupled and processors communicate
between different images and pools only once in a while, whereas
processors within each pool are tightly coupled and communications
are significant. This means that Gigabit ethernet (typical for
cheap PC clusters) is ok up to 4-8 processors per pool, but {\em fast}
communication hardware (e.g. Mirynet or comparable) is absolutely 
needed beyond 8 processors per pool.

\paragraph{Choosing parameters}:
To control the number of processors in each group,
command line switches: 
\texttt{-nimage}, \texttt{-npools}, \texttt{-nband},
\texttt{-ntg}, \texttt{-ndiag} or \texttt{-northo}
(shorthands, respectively: \texttt{-ni}, \texttt{-nk}, \texttt{-nb},
\texttt{-nt}, \texttt{-nd})
are used.
As an example consider the following command line:
\begin{verbatim}
mpirun -np 4096 ./neb.x -ni 8 -nk 2 -nt 4 -nd 144 -i my.input
\end{verbatim}
This executes a NEB calculation on 4096 processors, 8 images (points in the configuration
space in this case) at the same time, each of 
which is distributed across 512 processors.
k-points are distributed across 2 pools of 256 processors each, 
3D FFT is performed using 4 task groups (64 processors each, so
the 3D real-space grid is cut into 64 slices), and the diagonalization
of the subspace Hamiltonian is distributed to a square grid of 144
processors (12x12).

Default values are: \texttt{-ni 1 -nk 1 -nt 1} ; 
\texttt{nd} is set to 1 if ScaLAPACK is not compiled,
it is set to the square integer smaller than or equal to  half the number 
of processors of each pool.

\paragraph{Massively parallel calculations}
For very large jobs (i.e. O(1000) atoms or more) or for very long jobs,
to be run on massively parallel  machines (e.g. IBM BlueGene) it is
crucial to use in an effective way all available parallelization levels.
Without a judicious choice of parameters, large jobs will find a 
stumbling block in either memory or CPU requirements. Note that I/O
may also become a limiting factor.

Since v.4.1, ScaLAPACK can be used to diagonalize block distributed
matrices, yielding better speed-up than the internal algorithms for
large ($ > 1000\times 1000$) matrices, when using a large number of processors 
($> 512$). You need to have \texttt{-D\_\_SCALAPACK} added to DFLAGS 
in \texttt{make.sys}, LAPACK\_LIBS set to something like:
\begin{verbatim}
    LAPACK_LIBS = -lscalapack -lblacs -lblacsF77init -lblacs -llapack
\end{verbatim}
The repeated \texttt{-lblacs} is not an error, it is needed! 
\configure\ tries to find a ScaLAPACK  library, unless 
\texttt{configure --with-scalapack=no} is specified.
If it doesn't, inquire with your system manager
on the correct way to link it.

A further possibility to expand scalability, especially on machines
like IBM BlueGene, is to use mixed MPI-OpenMP. The idea is to have
one (or more) MPI process(es) per multicore node, with OpenMP
parallelization inside a same node. This option is activated by  \texttt{configure --with-openmp},
which adds preprocessing flag \texttt{-D\_\_OPENMP}
and one  of the following compiler options:

\begin{tabular}{ll}
 ifort& \texttt{-openmp}\\
 xlf&   \texttt{-qsmp=omp}\\
 PGI&   \texttt{-mp}\\
 ftn&   \texttt{-mp=nonuma}\\
\end{tabular}

OpenMP parallelization is currently implemented and tested for the following combinations of FFTs
and libraries:

\begin{tabular}{ll}
 internal FFTW copy &requires \texttt{-D\_\_FFTW}\\
 ESSL& requires \texttt{-D\_\_ESSL} or \texttt{-D\_\_LINUX\_ESSL}, link 
 with \texttt{-lesslsmp}\\
\end{tabular}

Currently, ESSL (when available) are faster than internal FFTW.

\subsubsection{Understanding parallel I/O}
In parallel execution, each processor has its own slice of data
(Kohn-Sham orbitals, charge density, etc), that have to be written
to temporary files during the calculation,
or to data files at the end of the calculation.
This can be done in two different ways:
\begin{itemize}
\item ``distributed'': each processor
writes its own slice to disk in its internal 
format to a different file. 
\item ``collected'': all slices are
collected by the code to a single processor
that writes them to disk, in a single file,
using a format that doesn't depend upon 
the number of processors or their distribution.
\end{itemize}

The ``distributed'' format is fast and simple,
but the data so produced is readable only by 
a job running on the same number of processors,
with the same type of parallelization, as the
job who wrote the data, and if all 
files are on a file system that is visible to all
processors (i.e., you cannot use local scratch
directories: there is presently no way to ensure 
that the distribution of processes across
processors will follow the same pattern 
for different jobs). 

Currently, \CP\ uses the ``collected'' format;
\PWscf\ uses the ``distributed'' format, but 
has the option to write the final data file in 
``collected'' format (input variable \texttt{wf\_collect})
so that it can be easily read by \CP\ and by other
codes running on a different  number of processors.

In addition to the above, other restrictions to file
interoperability apply: e.g., \CP\ can read only files
produced by \PWscf\ for the $k=0$ case. 

The directory for data is specified in input variables
\texttt{outdir} and \texttt{prefix} (the former can be specified
as well in environment variable ESPRESSO\_TMPDIR):
\texttt{outdir/prefix.save}. A copy of pseudopotential files
is also written there. If some processor cannot access the
data directory, the pseudopotential files are read instead
from the pseudopotential directory specified in input data.
Unpredictable results may follow if those files
are not the same as those in the data directory!

{\em IMPORTANT:}
Avoid I/O to network-mounted disks (via NFS) as much as you can! 
Ideally the scratch directory \texttt{outdir} should be a modern 
Parallel File System. If you do not have any, you can use local
scratch disks (i.e. each node is physically connected to a disk
and writes to it) but you may run into trouble anyway if you 
need to access your files that are scattered in an unpredictable
way across disks residing on different nodes.

You can use input variable \texttt{disk\_io} to reduce the the
amount of I/O done by \pwx. Since v.5.1, the dafault value is 
\texttt{disk\_io='low'}, so the code will store wavefunctions 
into RAM and not on disk during the calculation. Specify
\texttt{disk\_io='medium'} only if you have too many k-points 
and you run into trouble with memory; choose \texttt{disk\_io='none'} 
if you do not need to keep final data files. 

For very large \cpx\ runs, you may consider using 
\texttt{wf\_collect=.false.}, \texttt{memory='small'} and
\texttt{saverho=.false.} to reduce I/O to the strict minimum.

\subsection{Tricks and problems}
\label{SubSec:badpara}

\paragraph{Trouble with input files}
Some implementations of the MPI library have problems with input 
redirection in parallel. This typically shows up under the form of
mysterious errors when reading data. If this happens, use the option 
\texttt{-i} (or \texttt{-in}, \texttt{-inp}, \texttt{-input}), 
followed by the input file name. 
Example:
\begin{verbatim}
   pw.x -i inputfile -nk 4 > outputfile
\end{verbatim} 
Of course the 
input file must be accessible by the processor that must read it
(only one processor reads the input file and subsequently broadcasts
its contents to all other processors).

Apparently the LSF implementation of MPI libraries manages to ignore or to
confuse even the \texttt{-i/in/inp/input} mechanism that is present in all
\qe\ codes. In this case, use the \texttt{-i} option of \texttt{mpirun.lsf}
to provide an input file.

\paragraph{Trouble with MKL and MPI parallelization}
If you notice very bad parallel performances with MPI and MKL libraries, 
it is very likely that the OpenMP parallelization performed by the latter 
is colliding with MPI. Recent versions of MKL enable autoparallelization
by default on multicore machines.  You must set the environmental variable
OMP\_NUM\_THREADS to 1 to disable it. 
Note that if for some reason the correct setting  of variable
OMP\_NUM\_THREADS  
does not propagate to all processors, you may equally run into trouble. 
Lorenzo Paulatto (Nov. 2008) suggests to use the \texttt{-x} option to \texttt{mpirun} to 
propagate OMP\_NUM\_THREADS to all processors.
Axel Kohlmeyer suggests the following (April 2008): 
"(I've) found that Intel is now turning on multithreading without any
warning and that is for example why their FFT seems faster than
FFTW. For serial and OpenMP based runs this makes no difference (in
fact the multi-threaded FFT helps), but if you run MPI locally, you
actually lose performance. Also if you use the 'numactl' tool on linux
to bind a job to a specific cpu core, MKL will still try to use all
available cores (and slow down badly). The cleanest way of avoiding
this mess is to either link with
\begin{quote}
\texttt{-lmkl\_intel\_lp64 -lmkl\_sequential -lmkl\_core} (on 64-bit: 
x86\_64, ia64)\\
\texttt{-lmkl\_intel -lmkl\_sequential -lmkl\_core} (on 32-bit, i.e. ia32 )
\end{quote}
or edit the \texttt{libmkl\_'platform'.a} file. I'm using now a file 
\texttt{libmkl10.a} with:
\begin{verbatim}
  GROUP (libmkl_intel_lp64.a libmkl_sequential.a libmkl_core.a)
\end{verbatim}
It works like a charm". UPDATE: Since v.4.2, \configure\ links by
default MKL without multithreaded support.

\paragraph{Trouble with compilers and MPI libraries}
Many users of \qe, in particular those working on PC clusters,
have to rely on themselves (or on less-than-adequate system managers) for 
the correct configuration of software for parallel execution. Mysterious and
irreproducible crashes in parallel execution are sometimes due to bugs
in \qe, but more often than not are a consequence of buggy
compilers or of buggy or miscompiled MPI libraries.

\end{document}