.TH MPI_Comm_spawn 3 "6/24/2006" "LAM/MPI 7.1.4" "LAM/MPI"
.SH NAME
MPI_Comm_spawn \-  Spawn a dynamic MPI process 
.SH SYNOPSIS
.nf
#include <mpi.h>
int
MPI_Comm_spawn(char* command, char** argv, int maxprocs, MPI_Info info, 
               int root, MPI_Comm comm, MPI_Comm *intercomm, 
               int *errcodes)
.fi
.SH INPUT PARAMETERS
.PD 0
.TP
.B command 
- Name of program to spawn (only significant at root)
.PD 1
.PD 0
.TP
.B argv 
- arguments to command (only significant at root)
.PD 1
.PD 0
.TP
.B maxprocs 
- max number of processes to start (only significant at root)
.PD 1
.PD 0
.TP
.B info 
- startup hints
.PD 1
.PD 0
.TP
.B root 
- rank of process to perform the spawn
.PD 1
.PD 0
.TP
.B comm 
- parent intracommunicator
.PD 1

.SH OUTPUT PARAMETERS
.PD 0
.TP
.B intercomm 
- child intercommunicator containing spawned processes
.PD 1
.PD 0
.TP
.B errcodes 
- one code per process
.PD 1

.SH DESCRIPTION

A group of processes can create another group of processes with
.I MPI_Comm_spawn
\&.
This function is a collective operation over the
parent communicator.  The child group starts up like any MPI
application.  The processes must begin by calling 
.I MPI_Init
, after
which the pre-defined communicator, 
.I MPI_COMM_WORLD
, may be used.
This world communicator contains only the child processes.  It is
distinct from the 
.I MPI_COMM_WORLD
of the parent processes.

.I MPI_Comm_spawn_multiple
is used to manually specify a group of
different executables and arguments to spawn.  
.I MPI_Comm_spawn
is
used to specify one executable and set of arguments (although a
LAM/MPI appschema(5) can be provided to 
.I MPI_Comm_spawn
via the
"lam_file" info key).

Communication With Spawned Processes

The natural communication mechanism between two groups is the
intercommunicator.  The second communicator argument to
.I MPI_Comm_spawn
returns an intercommunicator whose local group
contains the parent processes (same as the first communicator
argument) and whose remote group contains child processes. The child
processes can access the same intercommunicator by using the
.I MPI_Comm_get_parent
call.  The remote group size of the parent
communicator is zero if the process was created by 
.I mpirun
(1) instead
of one of the spawn functions.  Both groups can decide to merge the
intercommunicator into an intracommunicator (with the
.I MPI_Intercomm_merge
function) and take advantage of other MPI
collective operations.  They can then use the merged intracommunicator
to create new communicators and reach other processes in the MPI
application.

Resource Allocation

LAM/MPI offers some MPI_Info keys for the placement of spawned
applications.  Keys are looked for in the order listed below.  The
first key that is found is used; any remaining keys are ignored.

.I lam_spawn_file

The value of this key can be the filename of an appschema(1).  This
allows the programmer to specify an arbitrary set of LAM CPUs or nodes
to spawn MPI processes on.  In this case, only the appschema is used
to spawn the application; 
.I command
, 
.I argv
, and 
.I maxprocs
are all
ignored (even at the root).  Note that even though 
.I maxprocs
is
ignored, 
.I errcodes
must still be an array long enough to hold an
integer error code for every process that tried to launch, or be the
MPI constant 
.I MPI_ERRCODES_IGNORE
\&.
Also note that
.I MPI_Comm_spawn_multiple
does 
.I not
accept the "lam_spawn_file" info
key.  As such, the "lam_spawn_file" info key to 
.I MPI_Comm_spawn
is
mainly intended to spawn MPMD applications and/or specify an arbitrary
number of nodes to run on.

Also note that this "lam_spawn_file" key is 
.I not
portable to other
MPI implementations; it is a LAM/MPI-specific info key.  If specifying
exact LAM nodes or CPUs is not necessary, users should probably use
.I MPI_Comm_spawn_multiple
to make their program more portable.

.I file

This key is a synonym for "lam_spawn_file".  Since "file" is not a
LAM-specific name, yet this key carries a LAM-specific meaning, its
use is deprecated in favor of "lam_spawn_file".

.I lam_spawn_sched_round_robin

The value of this key is a string representing a LAM CPU or node
(using standard LAM nomenclature -- see mpirun(1)) to begin spawning
on.  The use of this key allows the programmer to indicate which
node/CPU for LAM to start spawning on without having to write out a
temporary app schema file.

The CPU number is relative to the boot schema given to
lamboot(1).  Only a single LAM node/CPU may be specified, such as "n3"
or "c1".  If a node is specified, LAM will spawn one MPI process per
node.  If a CPU is specified, LAM will scedule one MPI process per
CPU.  An error is returned if "N" or "C" is used.

Note that LAM is not involved with run-time scheduling of the MPI
process -- LAM only spawns processes on indicated nodes.  The
operating system schedules these processes for executation just like
any other process.  No attempt is made by LAM to bind processes to
CPUs.  Hence, the "cX" nomenclature is just a convenicence mechanism
to inidicate how many MPI processes should be spawned on a given node;
it is not indicative of operating system scheduling.

For "nX" values, the first MPI process will be spawned on the
indicated node.  The remaining (maxprocs - 1) MPI processes will be
spawned on successive nodes.  Specifically, if X is the starting node
number, process i will be launched on "nK", where K = ((X + i) %
total_nodes).  LAM will modulus the node number with the total number
of nodes in the current LAM universe to prevent errors, thereby
creating a "wraparound" effect.  Hence, this mechanism can be used for
round-robin scheduling, regardless of how many nodes are in the LAM
universe.

For "cX" values, the algorithm is essentially the same, except that
LAM will resolve "cX" to a specific node before spawning, and
successive processes are spawned on the node where "cK" resides, where
K = ((X + i) % total_cpus).

For example, if there are 8 nodes and 16 CPUs in the current LAM
universe (2 CPUs per node), a "lam_spawn_sched_round_robin" key is
given with the value of "c14", and maxprocs is 4, LAM will spawn MPI
.SH PROCESSES ON

.nf
CPU  Node  MPI_COMM_WORLD rank
---  ----  -------------------
c14  n7    0
c15  n7    1
c0   n0    2
c1   n0    3
.fi


.I lam_no_root_node_schedule

This key is used to designate that the spawned processes must not be
spawned or scheduled on the "root node" (the node doing the spawn). There
is no specific value associated with this key, but it should be given some
non-null/non-empty dummy value.

It is a node-specific key and not a CPU-specific one. Hence if the root node
has multiple CPUs, none of the CPUs on this root node will take part in the
scheduling of the spawned processes.

No keys given

If none of the info keys listed above are used, the value of
.I MPI_INFO_NULL
should be given for 
.I info
(all other keys are
ignored, anyway - there is no harm in providing other keys).  In this
case, LAM schedules the given number of processes onto LAM nodes by
starting with CPU 0 (or the lowest numbered CPU), and continuing
through higher CPU numbers, placing one process on each CPU.  If the
process count is greater than the CPU count, the procedure repeats.

Predefined Attributes

The pre-defined attribute on 
.I MPI_COMM_WORLD
, 
.I MPI_UNIVERSE_SIZE
,
can be useful in determining how many CPUs are currently unused.  For
example, the value in 
.I MPI_UNIVERSE_SIZE
is the number of CPUs that
LAM was booted with (see MPI_Init(1)).  Subtracting the size of
.I MPI_COMM_WORLD
from this value returns the number of CPUs in the
current LAM universe that the current application is not using (and
are therefore likely not being used).

Process Terminiation

Note that the process[es] spawned by 
.I MPI_COMM_SPAWN
(and
.I MPI_COMM_SPAWN_MULTIPLE
) effectively become orphans.  That is, the
spawnning MPI application does not wait for the spawned application to
finish.  Hence, there is no guarantee the spawned application has
finished when the spawning completes.  Similarly, killing the spawning
application will also have no effect on the spawned application.

User applications can effect this kind of behavior with 
.I MPI_BARRIER
between the spawning and spawned processed before 
.I MPI_FINALIZE
\&.


Note that 
.I lamclean
will kill *all* MPI processes.

Process Count

The 
.I maxprocs
parameter to 
.I MPI_Comm_spawn
specifies the exact
number of processes to be started.  If it is not possible to start the
desired number of processes, 
.I MPI_Comm_spawn
will return an error
code.  Note that even though 
.I maxprocs
is only relevant on the root,
all ranks must have an 
.I errcodes
array long enough to handle an
integer error code for every process that tries to launch, or give MPI
constant 
.I MPI_ERRCODES_IGNORE
for the 
.I errcodes
argument.  While
this appears to be a contradiction, it is per the MPI-2 standard.  :-\\

Frequently, an application wishes to chooses a process count so as to
fill all processors available to a job.  MPI indicates the maximum
number of processes recommended for a job in the pre-defined
attribute, 
.I MPI_UNIVERSE_SIZE
, which is cached on 
.I MPI_COMM_WORLD
\&.

The typical usage is to subtract the value of 
.I MPI_UNIVERSE_SIZE
from
the number of processes currently in the job and spawn the difference.
LAM sets 
.I MPI_UNIVERSE_SIZE
to the number of CPUs in the user's LAM
session (as defined in the boot schema [bhost(5)] via 
.I lamboot
(1)).

See MPI_Init(3) for other pre-defined attributes that are helpful when
spawning.

Locating an Executable Program

The executable program file must be located on the node(s) where the
process(es) will run.  On any node, the directories specified by the
user's PATH environment variable are searched to find the program.

All MPI runtime options selected by 
.I mpirun
(1) in the initial
application launch remain in effect for all child processes created by
the spawn functions.

Command-line Arguments

The 
.I argv
parameter to 
.I MPI_Comm_spawn
should not contain the
program name since it is given in the first parameter.  The command
line that is passed to the newly launched program will be the program
name followed by the strings in 
.I argv
\&.


.SH USAGE WITH IMPI EXTENSIONS

The IMPI standard only supports MPI-1 functions.  Hence, this function
is currently not designed to operate within an IMPI job.

.SH ERRORS

If an error occurs in an MPI function, the current MPI error handler
is called to handle it.  By default, this error handler aborts the
MPI job.  The error handler may be changed with 
.I MPI_Errhandler_set
;
the predefined error handler 
.I MPI_ERRORS_RETURN
may be used to cause
error values to be returned (in C and Fortran; this error handler is
less useful in with the C++ MPI bindings.  The predefined error
handler 
.I MPI::ERRORS_THROW_EXCEPTIONS
should be used in C++ if the
error value needs to be recovered).  Note that MPI does 
.I not
guarantee that an MPI program can continue past an error.

All MPI routines (except 
.I MPI_Wtime
and 
.I MPI_Wtick
) return an error
value; C routines as the value of the function and Fortran routines
in the last argument.  The C++ bindings for MPI do not return error
values; instead, error values are communicated by throwing exceptions
of type 
.I MPI::Exception
(but not by default).  Exceptions are only
thrown if the error value is not 
.I MPI::SUCCESS
\&.


Note that if the 
.I MPI::ERRORS_RETURN
handler is set in C++, while
MPI functions will return upon an error, there will be no way to
recover what the actual error value was.
.PD 0
.TP
.B MPI_SUCCESS 
- No error; MPI routine completed successfully.
.PD 1
.PD 0
.TP
.B MPI_ERR_COMM 
- Invalid communicator.  A common error is to use a
null communicator in a call (not even allowed in 
.I MPI_Comm_rank
).
.PD 1
.PD 0
.TP
.B MPI_ERR_SPAWN 
- Spawn error; one or more of the applications
attempting to be launched failed.  Check the returned error code
array.  
.PD 1
.PD 0
.TP
.B MPI_ERR_ARG 
- Invalid argument.  Some argument is invalid and is not
identified by a specific error class.  This is typically a NULL
pointer or other such error.
.PD 1
.PD 0
.TP
.B MPI_ERR_ROOT 
- Invalid root.  The root must be specified as a rank
in the communicator.  Ranks must be between zero and the size of the
communicator minus one.
.PD 1
.PD 0
.TP
.B MPI_ERR_OTHER 
- Other error; use 
.I MPI_Error_string
to get more
information about this error code.
.PD 1
.PD 0
.TP
.B MPI_ERR_INTERN 
- An internal error has been detected.  This is
fatal.  Please send a bug report to the LAM mailing list (see
.I http://www.lam-mpi.org/contact.php
). 
.PD 1
.PD 0
.TP
.B MPI_ERR_NO_MEM 
- This error class is associated with an error code
that indicates that free space is exhausted.
.PD 1

.SH SEE ALSO
appschema(5), bhost(5), lamboot(1), MPI_Comm_get_parent(3), MPI_Intercomm_merge(3), MPI_Comm_spawn_multiple(3), MPI_Info_create(3), MPI_Info_set(3), MPI_Info_delete(3), MPI_Info_free(3), MPI_Init(3), mpirun(1)
.br

.SH MORE INFORMATION

For more information, please see the official MPI Forum web site,
which contains the text of both the MPI-1 and MPI-2 standards.  These
documents contain detailed information about each MPI function (most
of which is not duplicated in these man pages).

.I http://www.mpi-forum.org/


.SH ACKNOWLEDGEMENTS

The LAM Team would like the thank the MPICH Team for the handy program
to generate man pages ("doctext" from
.I ftp://ftp.mcs.anl.gov/pub/sowing/sowing.tar.gz
), the initial
formatting, and some initial text for most of the MPI-1 man pages.
.SH LOCATION
spawn.c