%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Copyright (c) 2003-2018 by The University of Queensland
% http://www.uq.edu.au
%
% Primary Business: Queensland, Australia
% Licensed under the Apache License, version 2.0
% http://www.apache.org/licenses/LICENSE-2.0
%
% Development until 2012 by Earth Systems Science Computational Center (ESSCC)
% Development 2012-2013 by School of Earth Sciences
% Development from 2014 by Centre for Geoscience Computing (GeoComp)
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{Execution of an {\it escript} Script}
\label{EXECUTION}

\section{Overview}
A typical way of starting your {\it escript} script \file{myscript.py} is with
the \program{run-escript} command\index{run-escript}\footnote{The
\program{run-escript} launcher is not supported under \WINDOWS.}.
This command was renamed from \program{escript} (used in previous releases) to
avoid clashing with an unrelated program installed by default on some systems.
To run your script, issue\footnote{For this discussion, it is assumed that
\program{run-escript} is included in your \env{PATH} environment. See the
installation guide for details.}
\begin{verbatim}
run-escript myscript.py
\end{verbatim}
as already shown in \Sec{FirstSteps}.
In some cases it can be useful to work interactively, e.g. when debugging a
script, with the command 
\begin{verbatim}
run-escript -i myscript.py
\end{verbatim}
This will execute \var{myscript.py} and when it completes (or an error occurs),
a \PYTHON prompt will be provided.
To leave the prompt press \kbd{Control-d} (\kbd{Control-z} on \WINDOWS).

To run the script using four threads (e.g. if you have a multi-core processor)
you can use
\begin{verbatim}
run-escript -t 4 myscript.py
\end{verbatim}
This requires {\it escript} to be compiled with \OPENMP\cite{OPENMP} support.
To run the script using \MPI\cite{MPI} with 8 processes use
\begin{verbatim}
run-escript -p 8 myscript.py
\end{verbatim}
If the processors which are used are multi-core processors or you are working
on a multi-processor shared memory architecture you can use threading in
addition to \MPI.
For instance to run 8 \MPI processes with 4 threads each, use the command
\begin{verbatim}
run-escript -p 8 -t 4 myscript.py
\end{verbatim}
In the case of a supercomputer or a cluster, you may wish to distribute the
workload over a number of nodes\footnote{For simplicity, we will use the term
\emph{node} to refer to either a node in a supercomputer or an individual
machine in a cluster}.
For example, to use 8 nodes with 4 \MPI processes per node, write
\begin{verbatim}
run-escript -n 8 -p 4 myscript.py
\end{verbatim}
Since threading has some performance advantages over processes, you may
specify a number of threads as well:
\begin{verbatim}
run-escript -n 8 -p 2 -t 4 myscript.py
\end{verbatim}
This runs the script on 8 nodes, with 2 processes per node and 4 threads per
process.

\section{Options}
The general form of the \program{run-escript} launcher is as follows:

%%%%
% If you are thinking about changing this please remember to update the man page as well
%%%%

\program{run-escript} 
\optional{\programopt{-n \var{nn}}} 
\optional{\programopt{-p \var{np}}}
\optional{\programopt{-t \var{nt}}}
\optional{\programopt{-f \var{hostfile}}}
\optional{\programopt{-x}}
\optional{\programopt{-V}}
\optional{\programopt{-e}}
\optional{\programopt{-h}}
\optional{\programopt{-v}}
\optional{\programopt{-o}}
\optional{\programopt{-c}}
\optional{\programopt{-i}}
\optional{\programopt{-b}}
\optional{\programopt{-m \var{tool}}}
\optional{\var{file}}
\optional{\var{ARGS}}

where \var{file} is the name of a script and \var{ARGS} are the arguments to
be passed to the script.
The \program{run-escript} program will import your current environment variables. 
If no \var{file} is given, then you will be presented with a regular \PYTHON
prompt (see \programopt{-i} for restrictions).

The options have the following meaning:
\begin{itemize}
\item[\programopt{-n} \var{nn}] the number of compute nodes \var{nn} to be used.
    The total number of processes being used is $\var{nn} \cdot \var{np}$.
    This option overrides the value of the \env{ESCRIPT_NUM_NODES}
    environment variable. 
    If a \var{hostfile} is given (see below), the number of nodes needs to
    match the number of hosts given in that file.
    If $\var{nn}>1$ but {\it escript} is not compiled for \MPI, a warning is
    printed but execution is continued with $\var{nn}=1$.
    If \programopt{-n} is not set the number of hosts in the host file is
    used. The default value is 1.
 
\item[\programopt{-p} \var{np}] the number of \MPI processes (per node).
    The total number of processes to be used is $\var{nn} \cdot \var{np}$.
    This option overwrites the value of the \env{ESCRIPT_NUM_PROCS}
    environment variable.
    If $\var{np}>1$ but {\it escript} is not compiled for \MPI, a warning is
    printed but execution is continued with $\var{np}=1$.
    The default value is 1.

\item[\programopt{-t} \var{nt}] the number of threads used per process.
    The option overwrites the value of the \OPENMP environment variable \env{ESCRIPT_NUM_THREADS}.
    If $\var{nt}>1$ but {\it escript} is not compiled for \OPENMP, a warning
    is printed but execution is continued with $\var{nt}=1$.
    The default value is 1.

\item[\programopt{-f} \var{hostfile}] the name of a file with a list of host names.
    Some systems require to specify the addresses or names of the compute
    nodes where \MPI processes should be spawned.
    These addresses or names of the compute nodes are listed in the file with
    the name \var{hostfile}.
    If \programopt{-n} is set, the number of different hosts defined in \var{hostfile}
    must be equal to the number of requested compute nodes \var{nn}.
    The option overwrites the value of the \env{ESCRIPT_HOSTFILE} environment
    variable. By default no host file is used.

\item[\programopt{-c}] prints information about the settings used to compile {\it escript} and stops execution.

\item[\programopt{-V}] prints the version of {\it escript} and stops execution.

\item[\programopt{-h}] prints a help message and stops execution.

\item[\programopt{-i}] executes the script \var{file} and switches to
    interactive mode after the execution is finished or an exception has occurred.
    This option is useful for debugging a script.
    The option cannot be used if more than one process ($\var{nn} \cdot \var{np}>1$) is used.

\item[\programopt{-b}] do not invoke python. This is used to run non-python
    programs within an environment set for {\it escript}.

\item[\programopt{-e}] shows additional environment variables and commands
    used to set up the {\it escript} environment.
    This option is useful if users wish to execute scripts without using
    the \program{run-escript} command.

\item[\programopt{-o}] enables the redirection of messages printed by
    processors with \MPI rank greater than zero to the files
    \file{stdout_\var{r}.out} and \file{stderr_\var{r}.out} where \var{r} is
    the rank of the processor.
    The option overwrites the value of the \env{ESCRIPT_STDFILES} environment
    variable.

\item[\programopt{-x}] runs everything within a new \emph{xterm} instance.

\item[\programopt{-v}] prints some diagnostic information.

\item[\programopt{-m} \var{tool}] runs under \emph{valgrind}. The argument
    \var{tool} must be one of \var{m} (for memcheck), \var{c} (for callgrind),
    or \var{h} (for cachegrind). Valgrind output is written to a file under
    \file{valgrind_logs} as reported when {\it escript} terminates.
\end{itemize}

\subsection{Notes}
The \program{run-escript} script is generated at build time taking into
account the \var{prelaunch}, \var{launcher}, and \var{postlaunch} settings
passed to \program{scons}. This makes it possible to easily customize the
script for different environments, such as batch systems (PBS, SLURM) and
different implementations of MPI (Intel, SGI, OpenMPI, etc.).

\section{Input and Output}
When \MPI is used on more than one process ($\var{nn} \cdot \var{np} >1$) no
input from the standard input is accepted.
Standard output on any process other than the master process (\var{rank}= 0)
will be silently discarded by default.
Error output from any processor will be redirected to the node where \program{run-escript} has been invoked.
If the \programopt{-o} option or \env{ESCRIPT_STDFILES} is set\footnote{That is, it has a non-empty value.},
then the standard and error output from any process other than the master
process will be written to files of the names \file{stdout_R.out}
and \file{stderr_R.out} (where \var{R} is the rank of the process).

If files are created or read by individual \MPI processes with information
local to the process (e.g. in the \function{dump} function)  and more than one
process is used ($\var{nn} \cdot \var{np} >1$), the \MPI process rank is
appended to the file names.
This is to avoid problems if processes are using a shared file system.
Files which collect data that are global for all \MPI processors are created
by the process with \MPI rank 0 only.
Users should keep in mind that if the file system is not shared among the
processes, then a file containing global information which is read by all
processors needs to be copied to the local file system(s) before \program{run-escript} is invoked.
 
\section{Hints for MPI Programming}
In general a script based on the \escript module does not require
modifications to run under \MPI.
However, one needs to be careful if other modules are used. 

When \MPI is used on more than one process ($\var{nn} \cdot \var{np} >1$) the
user needs to keep in mind that several copies of his script are executed at
the same time\footnote{In the case of \OPENMP only one copy is running
but {\it escript} temporarily spawns threads.} while data exchange is
performed through the \escript module.

This has three main implications:
\begin{enumerate}
 \item most arguments (\var{Data} excluded) should have the same values on all
     processors, e.g. \var{int}, \var{float}, \var{str} and \numpy parameters.
 \item the same operations will be called on all processors.
 \item different processors may store different amounts of information.
\end{enumerate}

With a few exceptions\footnote{\var{getTupleForDataPoint}}, values of
types \var{int}, \var{float}, \var{str} and \numpy returned by \escript will
have the same value on all processors.
If values produced by other modules are used as arguments, the user has to
make sure that the argument values are identical on all processors.
For instance, the usage of a random number generator to create argument values
bears the risk that the value may depend on the processor. 

Some operations in \escript require communication with all processors
executing the job. It is not always obvious which operations these are.
For example, \var{Lsup} returns the largest value on all processors.
\var{getValue} on \var{Locator} may refer to a value stored on another processor.
For this reason it is better if scripts do not have conditional operations
(which manipulate data) based on which processor the script is on.
Crashing or hanging scripts can be an indication that this has happened.

It is not always possible to divide data evenly amongst processors.
In fact some processors might not have any data at all.
Try to avoid writing scripts which iterate over data points, instead try to
describe the operation you wish to perform as a whole.

Special attention is required when using files on more than one processor as
several processors access the file at the same time. Opening a file for
reading is safe, however the user has to make sure that the variables which
are set from reading data from files are identical on all processors.

When writing data to a file it is important that only one processor is writing
to the file at any time. As all values in \escript are global it is sufficient
to write values on the processor with \MPI rank $0$ only.
The \class{FileWriter} class provides a convenient way to write global data
to a simple file.  The following script writes to the file \file{test.txt} on
the processor with rank 0 only:
\begin{python}
  from esys.escript import FileWriter
  f = FileWriter('test.txt')
  f.write('test message')
  f.close()
\end{python}
We strongly recommend using this class rather than \PYTHON's built-in \function{open}
function as it will guarantee a script which will run in single processor mode
as well as under \MPI.

If the situation occurs that one of the processors throws an exception, for
instance when opening a file for writing fails, the other processors are not
automatically made aware of this since \MPI does not handle exceptions.
However, \MPI will still terminate the other processes but may not inform the
user of the reason in an obvious way.
The user needs to inspect the error output files to identify the exception.