\documentclass[12pt,a4paper]{article}
\def\version{5.0}
\def\qe{{\sc Quantum ESPRESSO}}
\def\qeforge{\texttt{qe-forge.org}}
\textwidth = 17cm
\textheight = 24cm
\topmargin =-1 cm
\oddsidemargin = 0 cm

\usepackage{html}

% BEWARE: don't revert from graphicx for epsfig, because latex2html
% doesn't handle epsfig commands !!!
\usepackage{graphicx}


% \def\htmladdnormallink#1#2{#1}

\def\configure{\texttt{configure}}
\def\configurac{\texttt{configure.ac}}
\def\autoconf{\texttt{autoconf}}

\def\qeImage{quantum_espresso.pdf}
\def\democritosImage{democritos.pdf}

\begin{htmlonly}
\def\qeImage{quantum_espresso.png}
\def\democritosImage{democritos.png}
\end{htmlonly}

\begin{document} 
\author{}
\date{}
\title{
  \includegraphics[width=5cm]{\qeImage} \hskip 2cm
  \includegraphics[width=6cm]{\democritosImage}\\
  \vskip 1cm
  % title
  \Huge Developer's Manual for \qe (v. \version) \smallskip
}
\maketitle

\tableofcontents

\section{Introduction}
\subsection{Who should read (and who should {\em write}) this guide}

The intended audience of this guide is everybody who wants to:
\begin{itemize}
\item know how \qe\ works, including its internals;
\item modify/customize/add/extend/improve/clean up \qe;
\item know how to read data produced by \qe.
\end{itemize}
The same category of people should also {\em write} this guide, of course.

\subsection{Who may read this guide but will not necessarily profit from it}

People who want to know about the capabilities of \qe,
or who want just to use it, should read the User Guide.

People who want to know about the methods or the physics
behind \qe\ should read first the relevant  
literature (some pointers in the User Guide).

\subsection{How to contribute to \qe}

You can contribute to a better \qe, even if you are
not a full-fledged developer, by

\begin{itemize}
\item answering other people's questions on the mailing list (correct
  answers are strongly preferred to wrong ones). 
\item porting to new/unsupported architectures or configurations: see
  section \ref{SubSec:Inst}, "Installation mechanism". You should
  not need to add new preprocessing flags, but if you do, 
  see section \ref{SubSec:CPP}, "Preprocessing".
\item pointing out bugs in the software and in the documentation
  (reports of real bugs are strongly preferred to reports of
  nonexistent bugs). See section \ref{SubSec:Bugs}, "Guidelines 
  for reporting bugs".
\item improving the documentation (generic complaints or suggestions
  that "there should be this and that" do not qualify as improvements). 
\item  suggesting changes: note however that suggestions requiring a
  significant amount of work are welcome only if accompanied by
  implementation or by a promise of future implementation (fulfilled
  promises are strongly preferred to forgotten ones). 
\item adding new features to the code. If you like to have something
  added to \qe, contact the developers via the
 \texttt{q-e-developers} mailing list, hosted on \qeforge.
 Unless there are technical reasons not to include your changes, we 
 will try to make you happy (no warranty that we will actually succeed).
\end{itemize}

\subsection{How to make major contributions to \qe}
If you want to get involved as a developer and contribute serious
or nontrivial stuff, you should register for the \qe\ project on 
\qeforge. You may also consider the idea of opening
your own project on \qeforge. There is a developers' 
mailing list, \texttt{q-e-developers}, to which you should subscribe,
and a mailing list receiving commit message, \texttt{q-e-commits}.
The ideal procedure to add extensive changes is as follows:
\begin{itemize}
\item download the current SVN version (see 
section \ref{Sec:SVN}, "Using SVN", and work on that version
\item when you are happy with your modified version, make a copy of
  it, then update your copy with \texttt{svn update} 
\item if you get no conflicts and everything is still working, you
  have won. Send the modified files to the developers, or save
  your changes if you are one.
\item if you get conflicts, or if the updated code doesn't work any
  longer, you haven't yet won. See section
  \ref{SubSec:Conflicts} for hints on how to 
  remove conflicts and on how to figure out
   what went wrong.
\end{itemize}

\subsection{About \qeforge}
\qeforge\ is the portal for \qe\ developers, contributors,
and for anybody else wanting to develop a project in the
field of atomistic simulations.  \qeforge\ provides a CVS 
or SVN repository, mailing lists, a wiki, upload space, a
bug tracking facility, various other tools that are useful 
for developers. You can use either CVS or SVN but not both
together. Note that the usage of the wiki provided by \qeforge\ 
is currently disabled for security reasons.

You can open your own project, retaining all rights on it (including 
the right not to release anything); or else, you can register as a 
developer in an existing project (or both).

Currently \qe\ uses the following development tools:
\begin{itemize}
\item SVN server
\item Bug Tracking facility
\item Upload space (with download counter)
\item Mailing lists \texttt{q-e-commits} and \texttt{q-e-developers}.
\end{itemize}
Everybody is encouraged to explore other capabilities of \qeforge.

Once you are registered, you need to register your SSH keys in order
to have read-write access the CVS or SVN repository (if you have been
allowed by the project leader). The procedure is as follows:
\begin{itemize}
\item login to your \qeforge\ account
\item click on My account (top right corner)
\item click on Edit Keys, follow the instructions to add your keys
\item go back to your home page, click on SCM, follow instruction
\end{itemize}
Specific to the \qe\ project:
\begin{itemize}
\item If you want to receive an e-mail at each commit, subscribe to
\texttt{q-e-commits@qe-forge.org}. This will occasionally 
flood you with e-mails, because each changed directory generates 
an e-mail. 
\item If you want to be kept informed on what is going on, please
subscribe to \texttt{q-e-developers@qe-forge.org} (low traffic)
\end{itemize}

\subsection{Hints, Caveats, Do's and Dont's for developers}

\begin{itemize}
\item Before doing anything, inquire whether it is already there, or under development.
\item Before starting writing code, inquire whether you can reuse code that is already available in the distribution. Avoid redundancy: the only bug-free software line is the one that doesn't exist.
\item When you make some change:
\begin{itemize}
\item Check that are not spoiling other people's work. In particular, search the distribution for codes using the routine or module you are modifying and change its usage or its calling sequence everywhere.
\item Do not forget that your change must work on many different combinations of hardware and software, in both serial and parallel execution.
\end{itemize}
\item Document your changes:
\begin{itemize}
\item If you modify what a code can do, or introduce
incompatibilities with previous versions (e.g. old data file 
no longer readable, old input no longer valid), report it in
file \texttt{Doc/release-notes}.
\item If you add/modify/remove input variables, document
it in the appropriate file \\
\texttt{INPUT\_*.def}; if
you remove an input variable, update tests and examples
accordingly.
\item All newly introduced features or variables must be 
accompanied by an example or a test or both (either a 
new one or a modified existing test or example).
\end{itemize}
\item Please do not include files (any kind, including
pseudopotential files) with DOS \^{}M characters or 
tabulators \^{}I. 
\end{itemize}
When you modify the program sources, run the
\texttt{install/makedeps.sh}  script  or type \texttt{make depend} 
to update files \texttt{make.depend} in the various 
subdirectories.

\subsection{Guidelines for reporting bugs}
\label{SubSec:Bugs}
\begin{itemize}
\item Before deciding that a problem is due to a bug in the codes, 
verify if it is reproducible on different machines/architectures/phases 
of the moon: erratic or irreproducible problems, especially in parallel
execution, are often an indication of buggy compilers or libraries
\item Bug reports should preferably be filed using the bug tracking 
facility at \qeforge:\\
\texttt{http://qe-forge.org/tracker/?group\_id=10}
\item Bug reports should include enough information to be reproduced: 
typically, version number, hardware/software combination(s) for which 
the problem arises, whether it happens in serial or parallel
execution or both, and, most important, an input and output
exhibiting such behavior (fast to execute if possible). The error
message alone is usually not a sufficient piece of information. 
\item If a bug is found in a stable (released) version of \qe, it must
be reported in the \texttt{Doc/release-notes} file.
\end{itemize}
\section{ Structure of the distribution}
% \subsection{Contents of the various directories}
% \subsubsection{ Modules}
% \subsubsection{ Sources}
% \subsubsection{ Utilities}
\subsubsection{ Libraries}

Subdirectory \texttt{flib/} contains libraries written in fortran77 
(\texttt{*.f}) and in fortran-90 (\texttt{*.f90}).
The latter should not depend on any module, except for modules
\texttt{kinds} and \texttt{constants}.

Subdirectory \texttt{clib/} contains libraries written in C 
(\texttt{*.c}). Functions that are called by fortran
should be preprocessed using the macros:
\begin{enumerate}
\item \texttt{F77\_FUNC (func,FUNC)} for function \texttt{func}, not containing underscore(s) in name 
\item \texttt{F77\_FUNC\_(f\_nc,F\_NC)} for function \texttt{f\_nc}, containing underscore(s) in name
\end{enumerate}
These macros are defined in file \texttt{include/c\_defs.h}. This file must be included
by all \texttt{*.c} files. The macros are automagically generated by 
\configure\ and 
choose the correct case 
(lowercase or uppercase) and the correct number of final underscores. 
See file \texttt{include/defs.h.README} for more info.

\subsection{Installation mechanism}

\label{SubSec:Inst}
The code contains C-style preprocessing directives. There are two ways to do preprocessing of fortran files:
\begin{itemize}
\item directly with the fortran compiler, if supported;
\item by first pre-compiling with the C preprocessor \texttt{cpp}.
\end{itemize}

In the first case, one needs to specify in the \texttt{make.sys} file the fortran compiler option that tells the compiler to pre-process first. In the second case, one needs to
specify the C precompiler and options (if needed) in \texttt{make.sys}.
Normally, \configure\ should take care of this.

\subsubsection{ How to edit the \configure\ script}

The \configure\ script is generated from its source file
\configurac\ by the GNU \autoconf\ utility
(\texttt{http://www.gnu.org/software/autoconf/}).  Don't edit \configure\
directly: whenever it gets regenerated, your changes will be lost.
Instead, go to the \texttt{install/} directory, edit \configurac, 
then run \autoconf\ to regenerate \configure. If you want 
to keep the old \configure, make a copy
first.

GNU \autoconf\ is installed by default on most Unix/Linux systems.  If
you don't have it on your system, you'll have to install it. You will
need a recent version (e.g. v.2.65) of \autoconf, because our \configurac
file uses recent syntax.

\configurac\ is a regular Bourne shell script (i.e., "sh" -- not csh!), 
except that:
\begin{itemize}
\item[--] capitalized names starting with "AC\_" are \autoconf\ macros.  Normally you shouldn't have to touch them.
\item[--] square brackets are normally removed by the macro processor.  If you need a square bracket (that should be very rare), you'll have to write two.
\end{itemize}

You may refer to the GNU \autoconf\ Manual for more info.

\texttt{make.sys.in} is the source file for \texttt{make.sys}, that
\configure\ generates: you might want to edit that file as well. 
The generation procedure is as follows: if \configurac\ contains the macro
"AC\_SUBST(name)", then every occurrence of "@name@" in the source
file will be substituted with the value of the shell variable "name"
at the point where AC\_SUBST was called.

Similarly, \configure\texttt{.msg} is generated from \configure\texttt{.msg.in}: this
file is only used by \configure\ to print its final report, and isn't
needed for the compilation.  We did it this way so that our
\configure\ may also be used by other projects, just by replacing the
\qe-specific \configure\texttt{.msg.in} by your own.

\configure\ writes a detailed log of its operation to \texttt{config.log}.
When any configuration step fails, you may look there for the relevant
error messages.  Note that it is normal for some checks to fail.

\subsubsection{How to add support for a new architecture}

In order to support a previously unsupported architecture, first you
have to figure out which compilers, compilation flags, libraries
etc. should be used on that architecture.
In other words, you have to write a \texttt{make.sys} that works: you may use
the manual configuration procedure for that (see the 
User Guide).  Then, you have to modify \configure\ so that it can
generate that \texttt{make.sys} automatically.

To do that, you have to add the case for your architecture in several
places throughout \configurac:
\begin{enumerate}
\item Detect architecture

Look for these lines:
\begin{verbatim}
  if test "$arch" = ""
  then
          case $host in
                  ia64-*-linux-gnu )      arch=ia64   ;;
                  x86_64-*-linux-gnu )    arch=x86_64 ;;
                  *-pc-linux-gnu )        arch=ia32   ;;
                  etc.
\end{verbatim}
Here you must add an entry corresponding to your architecture and
operating system.  Run \texttt{config.guess} to obtain the string identifying
your system.
For instance on a PC it may be "i686-pc-linux-gnu", while on IBM SP4
"powerpc-ibm-aix5.1.0.0".  It is convenient to put some asterisks to
account for small variations of the string for different machines of
the same family.  For instance, it could be "aix4.3" instead of
"aix5.1", or "athlon" instead of "i686"...

\item  Select compilers

Look for these lines:

\begin{verbatim}
  # candidate compilers and flags based on architecture
  case $arch in
  ia64 | x86_64 )
        ...
  ia32 )
        ...
  aix )
        ...
  etc.
\end{verbatim}

Add an entry for your value of \$arch, and set there the appropriate
values for several variables, if needed (all variables are assigned
some reasonable default value, defined before the "case" block):

- "try\_f90" should contain the list of candidate Fortran 90 compilers,
in order of decreasing preference (i.e. configure will use the first
it finds).  If your system has parallel compilers, you should list
them in "try\_mpif90".

- "try\_ar", "try\_arflags": for these, the values "ar" and "ruv" should
be always fine, unless some special flag is required (e.g., -X64
With sp4).  

- you should define "try\_dflags" if there is
any
"\#ifdef" specific to your machine: for instance, on IBM machines,
"try\_dflags=-D\_\_AIX" . A list of such flags can be found in file 
\texttt{include/defs.h.README}.

You shouldn't need to define the following:
- "try\_iflags" should be set to the appropriate "-I" option(s)
needed by the preprocessor or by the compiler to locate *.h files
to be included; try\_iflags="-I../include" should be good for most cases

For example, here's the entry for IBM machines running AIX:
\begin{verbatim}
   aix )
        try_mpif90="mpxlf90_r mpxlf90"
        try_f90="xlf90_r xlf90 $try_f90"
        try_arflags="-X64 ruv"
        try_arflags_dynamic="-X64 ruv"
        try_dflags="-D__AIX -D__XLF"
        ;;
\end{verbatim}
The following step is to look for both serial and parallel fortran
compilers:
\begin{verbatim}
  # check serial Fortran 90 compiler...
  ...
  AC_PROG_F77($f90)
  ...
        # check parallel Fortran 90 compiler
  ...
        AC_PROG_F77($mpif90)
  ...
  echo setting F90... $f90
  echo setting MPIF90... $mpif90
\end{verbatim}
A few compilers require some extra work here: for instance, if the
Intel Fortran compiler was selected, you need to know which version
because different versions need different flags.

At the end of the test,
 
- \$mpif90 is the parallel compiler, if any; if no parallel compiler is found or if \texttt{--disable-parallel} was specified, \$mpif90 is the serial compiler

- \$f90 is the serial compiler

Next step: the choice of (serial) C and Fortran 77 compilers.
Look for these lines:
\begin{verbatim}
  # candidate C and f77 compilers good for all cases
  try_cc="cc gcc"
  try_f77="$f90"

  case "$arch:$f90" in
  *:f90 )
        ....
  etc.
\end{verbatim}
Here you have to add an entry for your architecture, and since the 
correct choice of C and f77 compilers may depend on the fortran-90 
compiler, you may need to specify the f90 compiler as well.
Again, specify the compilers in try\_cc and try\_f77 in order of 
decreasing preference.  At the end of the test, 

- \$cc is the C compiler

- \$f77 is the Fortran 77 compiler, used to compile *.f files
(may coincide with \$f90)

\item Specify compilation flags.

Look for these lines:
\begin{verbatim}
  # check Fortran compiler flags
  ...
  case "$arch:$f90" in
  ia64:ifort* | x86_64:ifort* )
        ...
  ia64:ifc* )
        ...
  etc.
\end{verbatim}
Add an entry for your case and define:

- "try\_fflags": flags for Fortran 77 compiler.

- "try\_f90flags": flags for Fortran 90 compiler.
In most cases they will be the same as in Fortran 77 plus some
others.  In that case, define them as "\$(FFLAGS) -something\_else".

- "try\_fflags\_noopt": flags for Fortran 77 with all optimizations
turned off: this is usually "-O0".
These flags must be used for compiling flib/dlamch.f (part of our
version of Lapack): it won't work properly with optimization.

- "try\_ldflags": flags for the linking phase (not including the list
of libraries: this is decided later). 

- "try\_ldflags\_static": additional flags to select static compilation
(i.e., don't use shared libraries).

- "try\_dflags": must be defined if there is in the code any \#ifdef
specific to your compiler (for instance, -D\_\_INTEL for Intel
compilers).  Define it as "\$try\_dflags -D..." so that pre-existing
flags, if any, are preserved.

- if the Fortran 90 compiler is not able to invoke the C preprocessor
automatically before compiling, set "have\_cpp=0" (the opposite case
is the default). The appropriate compilation rules will be generated 
accordingly. If the compiler requires that any flags be specified in
order to invoke the preprocessor (for example, "-fpp " -- note the 
space), specify them in "pre\_fdflags".

For example, here's the entry for ifort on Linux PC:
\begin{verbatim}
  ia32:ifort* )
          try_fflags="-O2 -tpp6 -assume byterecl"
          try_f90flags="\$(FFLAGS) -nomodule"
          try_fflags_noopt="-O0 -assume byterecl"
          try_ldflags=""
          try_ldflags_static="-static"
          try_dflags="$try_dflags -D__INTEL"
          pre_fdflags="-fpp "
          ;;
\end{verbatim}
Next step: flags for the C compiler. Look for these lines:
\begin{verbatim}
  case "$arch:$cc" in
  *:icc )
        ...
  *:pgcc )
        ...
  etc.
\end{verbatim}
Add an entry for your case and define:

- "try\_cflags": flags for C compiler.

- "c\_ldflags": flags for linking, when using the C compiler as linker.
This is needed to check for libraries written in C, such as FFTW.

- if you need a different preprocessor from the standard one (\$CC -E),
define it in "try\_cpp".

For example for XLC on AIX:
\begin{verbatim}
  aix:mpcc* | aix:xlc* | aix:cc )
          try_cflags="-q64 -O2"
          c_ldflags="-q64"
          ;;
\end{verbatim}
Finally, if you have to use a nonstandard preprocessor, look for these
lines:
\begin{verbatim}
  echo $ECHO_N "setting CPPFLAGS... $ECHO_C"
  case $cpp in
        cpp) try_cppflags="-P -traditional" ;;
        fpp) try_cppflags="-P"              ;;
        ...
\end{verbatim}
and set "try\_cppflags" as appropriate.

\item Search for libraries

To instruct \configure\ to search for libraries, you must tell it two
things: the names of libraries it should search for, and where it
should search.

The following libraries are searched for:

- BLAS or equivalent. 
Some vendor replacements for BLAS that are supported by \qe\ are:
\begin{quote}
    MKL on Linux, 32- and 64-bit Intel CPUs\\
    ACML on Linux, 64-bit AMD CPUs\\
    essl on AIX\\
    SCSL on sgi altix\\
    SUNperf on sparc
\end{quote}
Moreover, ATLAS is used over BLAS if available.

- LAPACK or equivalent. Some vendor replacements for LAPACK that are supported by \qe\ are:
\begin{quote}
    mkl on linux
    SUNperf on sparc
\end{quote}

- FFTW (version 3) or another supported FFT library. The latter include:
\begin{quote}
    essl on aix
    ACML on Linux, 64-bit AMD CPUs
    SUNperf on sparc
\end{quote}

- the MASS vector math library on aix

- an MPI library. This is often automatically linked by the compiler

If you have another replacement for the above libraries, you'll have
to insert a new entry in the appropriate place.

This is unfortunately a little bit too complex to explain.
Basic info: \\
"AC\_SEARCH\_LIBS(function, name, ...)" looks for symbol
"function" in library "libname.a".  If that is found, "-lname" is
appended to the LIBS environment variable (initially empty).
The real thing is more complicated than just that because the
"-Ldirectory" option must be added to search in a nonstandard
directory, and because a given library may require other libraries as
prerequisites (for example, Lapack requires BLAS).
\end{enumerate}

% \subsection{Adding new directories or routines}

\section{ Algorithms}
% \subsection{Diagonalization}
% \subsection{Self-consistency}
% \subsection{Structural optimization}
% \subsection{Symmetrization}
\subsection{Gamma tricks}

In calculations using only the $\Gamma$ point (k=0),
the Kohn-Sham orbitals can be chosen to be real functions in 
real space, so that 
$
  \psi(G) = \psi^*(-G).
$
This allows us to store only half of the Fourier components.
Moreover, two real FFTs can be performed as a single complex FFT.
The auxiliary complex function $\Phi$ is introduced:
$
    \Phi(r) = \psi_j(r)+ i \psi_{j+1}(r)
$
whose Fourier transform $\Phi(G)$ yields

$
   \psi_j    (G) =  {\Phi(G) + \Phi^*(G)\over 2},
   \psi_{j+1}(G) =  {\Phi(G) - \Phi^*(G)\over 2i}.
$

A side effect on parallelization is that $G$ and $-G$ must
reside on the same processor. As a consequence, pairs of columns
with $G_{n'_1,n'_2,n'_3}$ and $G_{-n'_1,-n'_2,n'_3}$
(with the exception of the case $n'_1=n'_2=0$),
must be assigned to the same processor.

\section{ Structure of the code}
% \subsection{Modules and global variables}
% \subsection{Meaning of the most important variables}
% \subsection{Conventions for indices}
\subsection{Preprocessing}
\label{SubSec:CPP}
The code contains C-style preprocessing directives. Most fortran compilers directly support them; some don't, and preprocessing is ''hand-made'' by the makefile using the C preprocessor \texttt{cpp}. The C preprocessor may:
\begin{itemize}
\item assign a value to a given expression. For instance, command \texttt{\#define THIS that}, or the option in the command line: \texttt{-DTHIS=that}, will replace all occurrence of \texttt{THIS} with \texttt{that}.
\item include file (command \texttt{\#include})
\item expand macros (command \texttt{\#define})
\item execute conditional expressions such as
\begin{verbatim}
  #ifdef __expression
    ...code A...
  #else
    ...code B...
  #endif
\end{verbatim}
If ''expression'' is defined (with a \texttt{\#define} command 
or from the command line with option \texttt{\-D\_\_expression}), 
then  \texttt{...code A...} is sent to output; otherwise 
\texttt{...code B...} is sent to output.

\end{itemize}
The file \texttt{include/defs.h.README} contains a list of definitions that are used in the code. In order to make  preprocessing options easy to see, preprocessing variables should start with 
two underscores, as \texttt{\_\_expression} in the above example. Traditionally ''preprocessed'' variables are also written in uppercase.

% \subsection{Performance issues}
% \subsection{Portability issues}

\section{Format of arrays containing charge density, potential, etc.}
The index of arrays used to store functions defined on 3D meshes is
actually a shorthand for three indices, following the FORTRAN convention
("leftmost index runs faster"). An example will explain this better.
Suppose you have a 3D array \texttt{psi(nr1x,nr2x,nr3x)}. FORTRAN
compilers store this array sequentially  in the computer RAM in the following way:
\begin{verbatim}
        psi(   1,   1,   1)
        psi(   2,   1,   1)
        ...
        psi(nr1x,   1,   1)
        psi(   1,   2,   1)
        psi(   2,   2,   1)
        ...
        psi(nr1x,   2,   1)
        ...
        ...
        psi(nr1x,nr2x,   1)
        ...
        psi(nr1x,nr2x,nr3x)
etc
\end{verbatim}
Let \texttt{ind} be the position of the \texttt{(i,j,k)} element in the above list:
the following relation
\begin{verbatim}
        ind = i + (j - 1) * nr1x + (k - 1) *  nr2x * nr1x
\end{verbatim}
holds. This should clarify the relation between 1D and 3D indexing. In real
space, the \texttt{(i,j,k)} point of the FFT grid with dimensions
\texttt{nr1} ($\le$\texttt{nr1x}),
\texttt{nr2}  ($\le$\texttt{nr2x}), , \texttt{nr3} ($\le$\texttt{nr3x}), is
$$
r_{ijk}=\frac{i-1}{nr1} \tau_1  +  \frac{j-1}{nr2} \tau_2 +
\frac{k-1}{nr3} \tau_3
$$
where the $\tau_i$ are the basis vectors of the Bravais lattice.
The latter are stored row-wise in the \texttt{at} array:
$\tau_1 = $ \texttt{at(:, 1)},
$\tau_2 = $ \texttt{at(:, 2)},
$\tau_3 = $ \texttt{at(:, 3)}.

The distinction between the dimensions of the FFT grid,
\texttt{(nr1,nr2,nr3)} and the physical dimensions of the array,
\texttt{(nr1x,nr2x,nr3x)} is done only because it is computationally
convenient in some cases that the two sets are not the same.
In particular, it is often convenient to have \texttt{nrx1}=\texttt{nr1}+1
to reduce memory conflicts.

\section{Parallelization}

In parallel execution (MPI only), PW starts N independent processes 
(do not start more than one per processor!) that communicate via calls
to MPI libraries. Each process has its own set of variables and knows
nothing about other processes' variables. Variables that take little memory 
are replicated, those that take a lot of memory (wavefunctions, G-vectors, 
R-space grid) are distributed.
    
\subsection{Tricks and pitfalls}

\begin{itemize}
\item
Replicated calculations may either be performed independently on 
each processor, or performed on one processor and broadcast to all
others. The first approach requires less programming, but it is unsafe:
in principle all processors should yield exactly the same results, if 
they work on the same data, but sometimes they don't (depending on the
machine, compiler, and libraries). Even a tiny difference in the last 
significant digit can eventually cause serious trouble if allowed to
 build up, especially when a replicated check is performed (in which
case the code may ''hang'' if the check yields different results on 
different processors). Never assume that the value of a variable produced 
by replicated calculations is exactly the same on all processors: when in 
doubt, broadcast the value calculated on a specific processor (the ''root'' 
processor) to all others.
\item
Routine \texttt{errore} should be called in parallel by all processors,
or else it will hang
\item
I/O operations: file opening, closing, and so on, are as a rule performed 
only on processor \texttt{ionode}. The correct way to check for errors is 
the following:
\begin{verbatim}
IF ( ionode ) THEN
   OPEN ( ..., IOSTAT=ierr )
   ...
END IF
CALL mp_bcast( ierr, ... )
CALL errore( 'routine','error', ierr )
\end{verbatim}
The same applies to all operations performed on a single processor,
or a subgroup of processors: any error code must be broadcast before
the check.
\end{itemize}

\subsection{ Data distribution}

Quantum ESPRESSO employ arrays whose memory requirements fall 
into three categories.
\begin{itemize}
\item {\em Fully Scalable}: 
Arrays that are distributed across processors of a pool.
Fully scalable arrays are typically large to very large and contain one 
of the following dimensions:
\begin{itemize}
\item number of plane waves, npw (or max number, npwx)
\item number of Gvectors, ngm
\item number of grid points in the R space, nrxx
\end{itemize}
Their size decreases linearly with the number of processors in a pool. 

\item {\em Partially Scalable}: 
Arrays that are distributed across processors of the
ortho or diag group. Typically they are much smaller than fully scalable
array, and small in absolute terms for moderate-size system. Their size
however increases quadratically with the number of atoms in the system,
so they have to be distributed for large systems (hundreds to thousands
atoms). Partially scalable arrays contain none of the dimensions listed 
above, two of the following dimensions:
\begin{itemize}
\item number of states, nbnd
\item number of atomic states, natomwfc
\item number of projectors, nkb
\end{itemize}
Their size decreases linearly with the number of processors in a ortho
or diag group. 

\item
{\em Nonscalable}: All the remaining arrays, that are not distributed across
processors. These are typically small arrays, having dimensions like for
instance:
\begin{itemize}
\item number of atoms, nat
\item number of species of atoms, nsp
\end{itemize}
The size of these arrays is independent on the number of processors.
\end{itemize}

% \subsubsection{ Parallel fft}

\section{ File Formats}
\subsection{Data file(s)}

\qe\ restart file specifications:
Paolo Giannozzi scripsit AD 2005-11-11,
Last modified by Andrea Ferretti 2006-10-29

\subsubsection{Rationale}

Requirements: the data file should be
\begin{itemize}
\item efficient (quick to read and write)
\item easy to read, parse and write without special libraries
\item easy to understand (self-documented)
\item portable across different software packages
\item portable across different computer architectures 
\end{itemize}
Solutions:
\begin{itemize}
\item use binary I/O for large records
\item exploit the file system for organizing data
\item use XML
\item use a small specialized library (iotk) to read, parse, write 
\item ensure the possibility to convert to a portable formatted file
\end{itemize}
Integration with other packages:
\begin{itemize}
\item provide a self-standing (code-independent) library to read/write this format
\item the use of this library is intended to be at high level, hiding low-level details
\end{itemize}

\subsubsection{General structure}

Format name: QEXML \\
Format version: 1.4.0 \\

The "restart file" is actually a "restart directory", containing several files and sub-directories. 
For CP/FPMD, the restart directory is created as "\$prefix\_\$ndw/", where \$prefix is the value of the 
variable "prefix". \$ndw the value of variable ndw, both read in input; it is read from "\$prefix\_\$ndr/", 
where \$ndr the value of variable ndr, read from input.
For PWscf, both input and output directories are called 
"\$prefix.save/".

The content of the restart directory is as follows:
\begin{verbatim}
data-file.xml          which contains:
                       - general information that doesn't require large data set: 
                         atomic structure, lattice, symmetries, parameters of the run, ...
                       - pointers to other files or directories containing bulkier data:
                         such as grids, wavefunctions, charge density, potentials, ...
  
charge_density.dat     contains the charge density
spin_polarization.dat  contains the spin polarization (rhoup-rhodw) (LSDA calculations)
magnetization.x.dat    
magnetization.y.dat    contain the spin polarization along x,y,z (noncollinear calculations)  
magnetization.z.dat 
lambda.dat             contains occupations (Car-Parrinello dynamics only
mat_z.1                contains occupations (ensemble-dynamics only)

<pseudopotentials>     A copy of all pseudopotential files given in input
    
<k-point dirs>         Subdirectories K00001/, K00002/, etc, one per k-point.
\end{verbatim}
Each k-point directory contains:
\begin{verbatim}
    evc.dat                wavefunctions for spin-unpolarized calculations, OR
    evc1.dat
    evc2.dat               spin-up and spin-down wavefunctions, respectively, 
                           for spin polarized (LSDA) calculations;
    gkvectors.dat          the details of specific k+G grid;
    eigenval.xml           eigenvalues for the corresponding k-point
                           for spin-unpolarized calculations, OR
    eigenval1.xml          spin-up and spin-down eigenvalues,
    eigenval2.xml          for spin-polarized calculations;
\end{verbatim}
in a molecular dynamics run, also wavefunctions at the preceding time step:
\begin{verbatim}
    evcm.dat               for spin-unpolarized calculations OR
    evcm1.dat
    evcm2.dat              for spin polarized calculations;
\end{verbatim}

\begin{itemize}
\item All files "*.xml" are XML-compliant, formatted file;
\item Files "mat\_z.1", "lambda.dat" are unformatted files, containing a single record;
\item All other files "*.dat", are XML-compliant files, but they contain an unformatted record.
\end{itemize}

\subsubsection{ Structure of file "data-file.xml"}
\begin{verbatim}
XML Header: whatever is needed to have a well-formed XML file

Body: introduced by <Root>, terminated by </Root>. Contains first-level tags
      only. These contain only other tags, not values. XML syntax applies.

First-level tags: contain either
     second-level tags, OR
     data tags:   tags containing data (values for a given variable), OR
     file tags:   tags pointing to a file
\end{verbatim}
data tags syntax ( [...] = optional ) :
\begin{verbatim}
      <TAG type="vartype" size="n" [UNIT="units"] [LEN="k"]>
      values (in appropriate units) for variable corresponding to TAG:
      n elements of type vartype (if character, of lenght k)
      </TAG>
\end{verbatim}
where TAG describes the variable into which data must be read;\\
"vartype" may be "integer", "real", "character", "logical";\\
if type="logical", LEN=k" must be used to specify the length
of the variable character; size="n" is the dimension.\\
Acceptable values for "units" depend on the specific tag.

Short syntax, used only in a few cases:
\begin{verbatim}
      <TAG attribute="something"/> . 
\end{verbatim}
For instance:
\begin{verbatim}
      <FFT_GRID nr1="NR1" nr2="NR2" nr3="NR3"/>
\end{verbatim}
defines the value of the FFT grid parameters nr1, nr2, nr3
for the charge density

\subsubsection{Sample}
Header:
\begin{verbatim}
 <?xml version="1.0"?>
 <?iotk version="1.0.0test"?>
 <?iotk file_version="1.0"?>
 <?iotk binary="F"?> 
\end{verbatim}
These are meant to be used only by iotk (actually they aren't)

First-level tags:
\begin{verbatim}
  - <HEADER>         (global information about fmt version)
  - <CONTROL>        (miscellanea of internal information)
  - <STATUS>         (information about the status of the CP simulation)
  - <CELL>           (lattice vector, unit cell, etc)
  - <IONS>           (type and positions of atoms in the unit cell etc)
  - <SYMMETRIES>     (symmetry operations)
  - <ELECTRIC_FIELD> (details for an eventual applied electric field)
  - <PLANE_WAVES>    (basis set, cutoffs etc)
  - <SPIN>           (info on spin polarizaztion)
  - <MAGNETIZATION_INIT>     (info about starting or constrained magnetization)
  - <EXCHANGE_CORRELATION>
  - <OCCUPATIONS>    (occupancy of the states)
  - <BRILLOUIN_ZONE> (k-points etc)
  - <PHONON>         (info for phonon calculations)  
  - <PARALLELISM>    (specialized info for parallel runs)
  - <CHARGE-DENSITY>
  - <TIMESTEPS>      (positions, velocities, nose' thermostats)
  - <BAND_STRUCTURE_INFO>    (dimensions and basic data about band structure)
  - <EIGENVALUES>    (eigenvalues and related data)
  - <EIGENVECTORS>   (eigenvectors and related data)

  
* Tag description

  <HEADER> 
     <FORMAT>    (name and version of the format)
     <CREATOR>   (name and version of the code generating the file)
  </HEADER>

  <CONTROL>
     <PP_CHECK_FLAG>    (whether the file can be used for post-processing)
     <LKPOINT_DIR>      (whether kpt-data are written in sub-directories)
     <Q_REAL_SPACE>     (whether augmentation terms are used in real space)
  </CONTROL>

  <STATUS>  (optional)
     <STEP>   (number $n of steps performed, i.e. we are at step $n)
     <TIME>   (total simulation time)
     <TITLE>  (a job descriptor)
     <ekin>   (kinetic energy)
     <eht>    (hartree energy)
     <esr>    (Ewald term, real-space contribution)
     <eself>  (self-interaction of the Gaussians)
     <epseu>  (pseudopotential energy, local)
     <enl>    (pseudopotential energy, nonlocal)
     <exc>    (exchange-correlation energy)
     <vave>   (average of the potential)
     <enthal> (enthalpy: E+PV)
  </STATUS>

  <CELL>
     <BRAVAIS_LATTICE>
     <LATTICE_PARAMETER>
     <CELL_DIMENSIONS>  (cell parameters)
     <DIRECT_LATTICE_VECTORS>
        <UNITS_FOR_DIRECT_LATTICE_VECTORS>
        <a1>
        <a2>
        <a3>
     <RECIPROCAL_LATTICE_VECTORS>
        <UNITS_FOR_RECIPROCAL_LATTICE_VECTORS>
        <b1>
        <b2>
        <b3>
  </CELL>

  <IONS>
     <NUMBER_OF_ATOMS>
     <NUMBER_OF_SPECIES>
     <UNITS_FOR_ATOMIC_MASSES>
     For each $n-th species $X:
        <SPECIE.$n>
           <ATOM_TYPE>
           <MASS>
           <PSEUDO>
        </SPECIE.$n>
     <PSEUDO_DIR>
     <UNITS_FOR_ATOMIC_POSITIONS>
     For each atom $n of species $X:
        <ATOM.$n SPECIES="$X">
  </IONS>

  <SYMMETRIES>
     <NUMBER_OF_SYMMETRIES>
     <INVERSION_SYMMETRY>
     <NUMBER_OF_ATOMS>
     <UNITS_FOR_SYMMETRIES>
     For each symmetry $n:
        <SYMM.$n>
           <INFO>
           <ROTATION>
           <FRACTIONAL_TRANSLATION>
           <EQUIVALENT_IONS>
        </SYMM.$n>
  </SYMMETRIES>

  <ELECTRIC_FIELD>  (optional)
     <HAS_ELECTRIC_FIELD> 
     <HAS_DIPOLE_CORRECTION>
     <FIELD_DIRECTION>
     <MAXIMUM_POSITION>
     <INVERSE_REGION>
     <FIELD_AMPLITUDE>
  </ELECTRIC_FIELD>  

  <PLANE_WAVES>
     <UNITS_FOR_CUTOFF>
     <WFC_CUTOFF>
     <RHO_CUTOFF>
     <MAX_NUMBER_OF_GK-VECTORS>
     <GAMMA_ONLY>
     <FFT_GRID>
     <GVECT_NUMBER>
     <SMOOTH_FFT_GRID>
     <SMOOTH_GVECT_NUMBER>
     <G-VECTORS_FILE>       link to file "gvectors.dat"
     <SMALLBOX_FFT_GRID>
  </PLANE_WAVES>

  <SPIN>
     <LSDA>
     <NON-COLINEAR_CALCULATION>
     <SPIN-ORBIT_CALCULATION>
     <SPIN-ORBIT_DOMAG>
  </SPIN>

  <EXCHANGE_CORRELATION>
     <DFT>
     <LDA_PLUS_U_CALCULATION>
     if LDA_PLUS_U_CALCULATION
        <NUMBER_OF_SPECIES>
        <HUBBARD_LMAX>
        <HUBBARD_L>
        <HUBBARD_U>
        <HUBBARD_ALPHA>
     endif
  </EXCHANGE_CORRELATION>

  if hybrid functional
      <EXACT_EXCHANGE>
        <x_gamma_extrapolation>
        <nqx1>
        <nqx2>
        <nqx3>
        <exxdiv_treatment>
        <yukawa>
        <ecutvcut>
        <exx_fraction>
        <screening_parameter>
      </EXACT_EXCHANGE>
  endif 

  <OCCUPATIONS>
     <SMEARING_METHOD>
     if gaussian smearing
        <SMEARING_TYPE>
        <SMEARING_PARAMETER>
     endif
     <TETRAHEDRON_METHOD>
     if use tetrahedra
        <NUMBER_OF_TETRAHEDRA>
        for each tetrahedron $t
           <TETRAHEDRON.$t>
     endif
     <FIXED_OCCUPATIONS>
     if using fixed occupations
        <INFO>
        <INPUT_OCC_UP>
        if lsda
           <INPUT_OCC_DOWN>
        endif
     endif
  </OCCUPATIONS>

  <BRILLOUIN_ZONE>
     <NUMBER_OF_K-POINTS>
     <UNITS_FOR_K-POINTS>
     <MONKHORST_PACK_GRID>
     <MONKHORST_PACK_OFFSET>
     For each k-point $n:
        <K-POINT.$n>
  </BRILLOUIN_ZONE>

  <PHONON> 
     <NUMBER_OF_MODES>
     <UNITS_FOR_Q-POINT>
     <Q-POINT>
  </PHONON>

  <PARALLELISM>
     <GRANULARITY_OF_K-POINTS_DISTRIBUTION>
  </PARALLELISM>

  <CHARGE-DENSITY>
      link to file "charge_density.rho"
  </CHARGE-DENSITY>

  <TIMESTEPS>  (optional)
     For each time step $n=0,M
       <STEP$n>
          <ACCUMULATORS>
          <IONS_POSITIONS>
             <stau>
             <svel>
             <taui>
             <cdmi>
             <force>
          <IONS_NOSE>
             <nhpcl>
             <nhpdim>
             <xnhp>
             <vnhp>
          <ekincm>
          <ELECTRONS_NOSE>
             <xnhe>
             <vnhe>
          <CELL_PARAMETERS>
             <ht>
             <htve>
             <gvel>
          <CELL_NOSE>
             <xnhh>
             <vnhh>
          </CELL_NOSE>
  </TIMESTEPS>

  <BAND_STRUCTURE_INFO>
     <NUMBER_OF_BANDS>
     <NUMBER_OF_K-POINTS>
     <NUMBER_OF_SPIN_COMPONENTS>
     <NON-COLINEAR_CALCULATION>
     <NUMBER_OF_ATOMIC_WFC>
     <NUMBER_OF_ELECTRONS>
     <UNITS_FOR_K-POINTS>
     <UNITS_FOR_ENERGIES>
     <FERMI_ENERGY>
  </BAND_STRUCTURE_INFO>

  <EIGENVALUES>
     For all kpoint $n:
         <K-POINT.$n>
             <K-POINT_COORDS>
             <WEIGHT>
             <DATAFILE>                  link to file "./K$n/eigenval.xml"
         </K-POINT.$n>
  </EIGENVALUES>

  <EIGENVECTORS>
     <MAX_NUMBER_OF_GK-VECTORS>
     For all kpoint $n:
         <K-POINT.$n>
             <NUMBER_OF_GK-VECTORS>
             <GK-VECTORS>                link to file "./K$n/gkvectors.dat"
             for all spin $s
                <WFC.$s>                 link to file "./K$n/evc.dat"
                <WFCM.$s>                link to file "./K$n/evcm.dat" (optional)
                                         containing wavefunctions at preceding step
         </K-POINT.$n>
  </EIGENVECTORS>
\end{verbatim}

\subsection{Restart files}

\section{Modifying/adding/extending \qe}

\subsection{Programming style (or lack of it)}

Guidelines for developers:
\begin{itemize}
\item preprocessing options should be capitalized and start with two underscores. Examples: \_\_AIX, \_\_LINUX, ...
\item fortran commands should be capitalized: 
CALL something( )
\item variable names should be lowercase: \texttt{foo = bar/2}
\item indent DO's and IF's with three white spaces (editors like emacs will do this automatically for you)
\item do not write crammed code: leave spaces, insert empty separation lines
\item comments (introduced by a !) should be used to explain what is not obvious from the code, not to repeat what is already evident. Obscure comments serve no purpose.
\item do not use machine-dependent extensions or sloppy syntax. Standard f90 
requires that a \& is needed both at end of line AND at the beginning of 
continuation line if there is a character variable (inside ' ' or " ")
spanning two lines. Some compilers do not complain if the latter \& is 
missing, others do.
\item use DP (defined in module ''kinds'') to define the type of real and complex variables
\item all constants should be defined to be of kind DP.  Preferred syntax: 0.0\_dp.
\item use "generic" intrinsic functions: SIN, COS, etc.
\item conversions should be explicitely indicated. For conversions to real, 
use DBLE, or else REAL(...,KIND=dp). For conversions to complex, use 
CMPLX(...,...,KIND=dp). For complex conjugate, use CONJG.  For imaginary part, 
use AIMAG.  IMPORTANT: Do not use REAL or CMPLX without KIND=dp, or else you 
will lose precision (except when you take the real part of a 
double precision complex number).
\item Do not use automatic arrays (e.g. \texttt{REAL(DP) :: A(N)} 
in a subroutine) except if you are sure that the array is small in all
cases: you may easily exceed the stack size if the arrays are large.
\item Do not use pointers unless you have a good reason to: 
pointers may hinder optimization, allocatable arrays should be used instead.
\item If you use pointers, nullify them before performing tests on their
status.
\item Do not pass unallocated arrays as arguments, even in those cases where
they are not actually used inside the subroutine.
\item Do not use any construct that is susceptible to be flagged as 
out-of-bounds error, even if no actual out-of-bound error takes place.
\end{itemize}

\subsection{Adding or modifying input variables}

New input variables should be added to 
''Modules/input\_parameters.f90'',
then copied to the code internal variables in the ''input.f90''
subroutine. The namelists and cards parsers are in
''Modules/read\_namelists.f90'' and ''Modules/read\_cards.f90''.
Files ''input\_parameters.f90'', ''read\_namelists.f90'',
''read\_cards.f90'' are shared by all codes, while each code
has its own version of ''input.f90''  used to copy input values
into internal variables

EXAMPLE:
suppose you need to add a new input variable called ''pippo''
to the namelist control, then:

\begin{enumerate}
\item add pippo to the input\_parameters.f90 file containing the
namelist control 
\begin{verbatim}
              INTEGER :: pippo = 0
              NAMELIST / control / ....., pippo
\end{verbatim}
Remember: always set an initial value!

\item add pippo to the control\_default subroutine (contained in
module read\_namelists.f90 ) 
\begin{verbatim}
               subroutine control_default( prog )
              ...
              IF( prog == 'PW' ) pippo = 10
              ...
              end subroutine
\end{verbatim}
This routine sets the default value for pippo (can be different in
different codes) 

\item add pippo to the control\_bcast subroutine (contained in module
read\_namelists.f90 ) 
 \begin{verbatim}
                subroutine control_bcast( )
                ...
                call mp_bcast( pippo )
                ...
                end subroutine
\end{verbatim}
\end{enumerate}
 
\section{ Using SVN}
\label{Sec:SVN}
The package is available read-only using anonymous access to the Subversion 
(SVN) repository. Developers can have read-write access when needed. Note 
that the latest (development) version may not work properly, and sometimes 
not even compile properly. Use at your own risk. 

Subversion, also known as SVN, is a software that allows many
developers to work and maintain a single copy of a software in a
central location (repository).
It is installed by default on many Unix machines, or otherwise 
it can be very easily installed.
For the end user, SVN is rather similar to CVS:
if no advanced features are used, the basic commands are the same.
More information on SVN can be found here: 
\texttt{http://subversion.apache.org/}.

Follow the instructions in the SCM page in \qeforge\ for the \qe\
project: \texttt{http://qe-forge.org/scm/?group\_id=10},
to check out (i.e.  download) the SVN repository in either 
read-write or anonymous mode.
The distribution will appear in directory \texttt{trunk/espresso/}.
Branches (i.e. sub-versions) will appear as separate directories.

\subsection{SVN operations}

To update the code to the current version: 
\begin{verbatim}
  svn update
\end{verbatim}
in the directory containing the distribution.
To see the difference between the current version and your modified 
copy:
\begin{verbatim}
  svn diff
\end{verbatim}
To save your modified version into the repository:
(read-write access only):
\begin{verbatim}
  svn commit
\end{verbatim}
If you also want to add a new file, before commiting give command
\begin{verbatim}
  svn add
\end{verbatim}

\subsection{Removing conflicts}
\label{SubSec:Conflicts}
  Look into the conflicting section: in
  most cases conflicts are trivial (format changes, white spaces) or
  easily solved (the part of the code you were modifying has been
  moved to another place, for instance). Sometimes, somebody else has
  done changes that are incompatible with yours during the same
  period. Use the web-svn interface:
\begin{verbatim}
   http://qe-forge.org/scm/browser.php?group\_id=10
\end{verbatim}
  You can select the repository at a given date or with a given ``tag'': 
  this may be useful if if you suspect that the incompatibile 
  change happened after that date. 

\section{bibliography}

Fortran books:
\begin{itemize}
\item 
M. Metcalf, J. Reid, Fortran 95/2003 Explained, Oxford University Press (2004) 
\item
S. J. Chapman, Fortran 95/2003 for Scientists and Engineers, McGraw Hill (2007) 
\item
J. C. Adams, W. S. Brainerd, R. A. Hendrickson, R. E. Maine, J. T. Martin,
B. T. Smith, The Fortran 2003 Handbook, Springer (2009) 
\item
W. S. Brainerd, Guide to Fortran 2003 Programming, Springer (2009)
\end{itemize}
On-line tutorials:
\begin{itemize}
\item Fortran:
http://www.cs.mtu.edu/\~{}shene/COURSES/cs201/NOTES/fortran.html
\item Make:  
http://en.wikipedia.org/wiki/Make\_(software)
\item Configure script:
http://en.wikipedia.org/wiki/Configure\_script
\end{itemize}
(info courtesy of Goranka Bilalbegovic)
\end{document}
