File: INSTALL

package info (click to toggle)
lam 6.5.6-6
  • links: PTS
  • area: main
  • in suites: woody
  • size: 15,464 kB
  • ctags: 11,670
  • sloc: ansic: 101,210; cpp: 12,146; sh: 10,821; makefile: 4,050; fortran: 218
file content (1323 lines) | stat: -rw-r--r-- 51,036 bytes parent folder | download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
Copyright 1998-2001, University of Notre Dame.
Authors: Jeffrey M. Squyres, Arun Rodrigues, and Brian Barrett with
         Kinis L. Meyer, M. D. McNally, and Andrew Lumsdaine

This file is part of the Notre Dame LAM implementation of MPI.

You should have received a copy of the License Agreement for the Notre
Dame LAM implementation of MPI along with the software; see the file
LICENSE.  If not, contact Office of Research, University of Notre
Dame, Notre Dame, IN 46556.

Redistribution and use in source and binary forms, with or without
modification, are permitted subject to the conditions specified in the
LICENSE file.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT,
INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.

Additional copyrights may follow.


Installation instructions for LAM 6.5.6
=========================================

This file contains the installation instructions for LAM/MPI version
6.5.6.  There are also some tips for writing/developing/running
parallel programs, especially in parallel environments that are
clusters of workstations.  If you have problems configuring/compiling
LAM, see the "Troubleshooting" section, below.

Here's a brief table of contents (### indicates updates for this
version):

 ### * For the impatient
 ### * Unpacking the distribution
 ### * Architecture-specific notes
     * Configuration
 ###   - ROMIO issues
 ###   - MPI 2 C++ issues
 ###   - VPATH builds
 ###   - ./configure switches
 ###   - 64 bit LAM
 ### * Building LAM
 ### * Building LAM, ROMIO, and MPI 2 C++ examples
 ### * Boot schema
     * Using LAM
       - Typical usage
 ###   - The LAMHOME and TROLLIUSHOME environment variables
       - Starting LAM
       - Common filesystems
       - Using LAM with AFS
       - Using LAM with ssh
 ### * Troubleshooting
 ###   - Problems with building LAM
       - Problems with running LAM and/or user programs
 ###   - Insufficient shared resources
     * Clearing disk space
     * Tuning LAM
       - Short/long protocol
 ###   - Shortcircuit send/receive 
       - TCP transport
       - Usysv and sysv transports
       - Use of the global pool
       - Synchronization
       - Usysv transport spin-locks
       - Sysv transport semaphores


For the impatient
-----------------

If you don't want to read the rest of the instructions, the following
should do the trick for most situations:

     % gunzip -c lam-6.5.6.tar.gz | tar xf -
     % cd lam-6.5.6
     % ./configure --prefix=/path/to/install/in
     [...lots of output...]
     % make
     [...lots of output...]
     % make install
     [...lots of output...]

     # Ensure that $prefix/bin is in in your $path so that LAM's
     # newly-created "mpicc" can be found

     % make examples   # This step is optional
     [...lots of output...]

If you do not specify a prefix, LAM will first look for "lamclean" in
your path.  If lamclean is found, it will use the parent of the
directory where lamclean is located as the prefix.  Otherwise,
/usr/local is used (like most GNU software).

Now go read the RELEASE_NOTES file; it contains all the information
about the new features of this release of LAM/MPI.

Common causes of failure:

     - No C++ compiler installed; use --without-mpi2cpp configure option
     - C++ compiler does not support required C++ features; use
       --without-mpi2cpp configure option
     - No Fortran compiler installed; user --without-fc configure option
     - "mpicc" cannot be found, or other kinds of failures when
       compiling the examples; ensure that the
       newly-compiled-and-installed "mpicc" is found *first* in your
       path (i.e., if mpicc appears in your path multiple times, the
       new one must be first)

Unpacking the distribution
--------------------------

The LAM distribution is packaged as a compressed tape archive,
lam-6.5.6.tar.Z, lam-6.5.6.tar.gz, or lam-6.5.6.tar.bz2.  It is
available from the main LAM web site: http://www.lam-mpi.org/

Uncompress the archive and extract the sources. 

     % gunzip -c lam-6.5.6.tar.gz | tar xf -

or 

     % uncompress -c lam-6.5.6.tar.Z | tar xf -

or 

     % bunzip2 -c lam-6.5.6.tar.bz2 | tar xf -


Architecture-specific notes
---------------------------

LAM/MPI will build on just about any POSIX system.  There are,
however, a few restrictions:

--- Microsoft Windows

Microsoft Windows is not a POSIX platform.  LAM/MPI currently will not
build in a Windows environment.

--- AIX

It appears that GNU libtool does not presently support building shared
libraries on AIX.  This has been tested on AIX 4.3.3; it is not known
if GNU libtool builds shared libraries on other versions of AIX.

Additionally, in some cases, GNU libtool apparently does not function
completely properly when using the "xlc" compiler.  Use "cc", instead
(they are both the same compiler anyway).

Finally, there have been repeatable problems with AIX's "make" when
building ROMIO.  This does not appear to be ROMIO's fault -- it
appears to be a bug in AIX's "make".  The LAM Team suggests that you
use GNU "make" (ftp://ftp.gnu.org/gnu/make/) when building on AIX
platforms to avoid these problems.

--- Various BSD systems

The version of "make" that is distributed on some BSD systems (e.g.,
FreeBSD) requires the use of the "-i" parameter to some of LAM's make
targets.  For example:

     make -i clean

--- HP-UX

It appears that the default C++ compiler on HP-UX (CC) is a pre-ANSI
standard C++ compiler.  As such, it will not build the C++ bindings
package.  The C++ compiler "aCC" should be used to build
the C++ bindings package.  The C++ compiler can be specified by
specifying "--with-cxx=aCC" as an option to configure.


Configuration
-------------

LAM uses a GNU configure script to perform site and architecture
specific configuration.

Change directory to the top level LAM directory (lam-6.5.6) and run the
configure script.

     % ./configure {options}

or 

     % sh ./configure {options}

By default the configure script sets the LAM install directory to the
parent of where lamclean is found (if it is in your path), or
/usr/local if lamclean is not in your path.  This can be overridden
with the --prefix option (see below).


ROMIO issues
------------

Note that the ROMIO package does not currently support many GNU-like
configure switches.  In particular, attempting to use any of the
directory-specifying options (other than --prefix) will not work as
expected with ROMIO.  ROMIO installs everything under
$(DESTDIR)/$prefix.  Hence, if you attempt to use switches such as
--libdir, --bindir, etc. to LAM's configure, all of the LAM (and the
C++ bindings) will install as expected, but ROMIO will still install
itself under $prefix.

The ROMIO authors have been notified of this issue.


MPI 2 C++ issues
----------------

--- C++ Exceptions

The default is to build LAM/MPI with the C++ bindings, but without C++
exception support.

Enabling C++ exceptions typically entails a slight degradation of
run-time performance because of extra bootstrapping required for every
function call (particularly with gcc/g++).  As such, they are disabled
by default, and the MPI::ERRORS_THROW_EXCEPTIONS error handler will
only print out error messages.  If full exception handling
capabilities are desired, LAM must be configured with the
"--with-exceptions" flag.  It should be noted that some C++ (and C and
Fortran) compilers need additional command line flags to properly
enable exception handling.

For example, with gcc/g++ 2.95.2 and later, gcc, g77, and g++ all
require the command line flag "-fexceptions".  gcc and gf77 require
"-fexceptions" so that they can pass C++ exceptions through C and
Fortran functions properly.  As such, *all* of LAM/MPI must be
compiled with the appropriate compiler options, not just the C++
bindings.  Using MPI::ERRORS_THROW_EXCEPTIONS without having compiled
LAM with proper exception support will cause undefined behavior (read:
core dumps and other Bad Things).

If building with IMPI or the C++ bindings, LAM's configure script will
automatically guess the necessary compiler exception support command
line flags for the gcc/g++ and KCC compilers.  That is, if a user
selects to build the MPI 2 C++ bindings and/or the IMPI extensions,
and also selects to build exception support, and g++ or KCC is
selected as the C++ compiler, the appropriate exceptions flags will
automatically be used.

Users with other compilers that require command line flags for
exception support should use the "--with-exflags=FLAGS" command line
switch to configure.

Note that this also applies even if you do not build the C++ bindings
-- if LAM is to call C++ functions that may throw exceptions
(e.g,. from an MPI error handler or other callback function), you need
to build LAM with the appropriate exceptions compiler flags.


--- Mixing Vendor Compilers

A single vendor product line should be used to compile all of the C,
Fortran, and C++ code.  That is, if gcc is used to compile LAM, g++
should be used to compile the C++ bindings, and gcc/g++/g77 should be
used to compile any user programs.  Mixing multiple vendors' compilers
between different components of LAM/MPI and/or to compile user MPI
programs, particularly when using the C++ MPI bindings, is almost
guaranteed not to work.

C++ compilers are not link-compatible -- compiling the C++ bindings
with one C++ compiler and compiling a user program that uses the MPI
C++ bindings will almost certainly produce linker errors.  

Indeed, if exception support is enabled in the C++ bindings, it will
only work if the C and/or Fortran code knows how to pass C++
exceptions through their code.  This will only happen properly when
the same compiler (or a single vendor's compiler product line, such as
gcc, g77, and g++) is used to compile *all* components -- LAM/MPI, the
C++ bindings, and the user program.  Using multiple vendor compilers
with C++ exceptions will almost certainly not work (read: core dumps
and other Bad Things).

The one possible exception to this rule (pardon the pun) is the KCC
compiler.  Since KCC turns C++ code to C code and then gives it to the
back end "native" C compiler, KCC may work properly with the native C
and Fortran compilers.


VPATH builds
------------

Alternatively, LAM supports the "VPATH" building mechanism.  If
LAM/MPI is to be installed in multiple environments that require
different options to configure, or require different compilers (such
as compiling for multiple architectures/operating systems), the
following form can be used to configure LAM:

     % cd /some/temp/directory
     % LAMTOP/configure {options}

where LAMTOP is the directory where the LAM/MPI distribution tarball
was expanded.  This form will build the LAM executables and libraries
under /some/temp/directory and will not produce any files in the
LAMTOP tree.  It allows multiple, concurrent builds of LAM/MPI from
the same source tree.  

Note that you must have a VPATH-enabled "make" in order to use this
form.  The GNU "make" (ftp://ftp.gnu.org/gnu/make/) supports VPATH
builds, for example, but the Solaris Workshop 5.0 "make" does not.
Parts of LAM/MPI may compile correctly in a VPATH build without a
VPATH-enabled compiler, but ROMIO will not.


./configure switches
--------------------

The configure script will create several configuration files,
including share/include/lam_config.h.  You may wish to inspect this
file for a sanity check, but ./configure usually guesses correctly.

There are many options available from the configure script.  You can
use the command "./configure --help" to list them all.  An explanation
of each follows (shown here in alphabetical order):

--disable-static

     Do not build static libraries.  This flag is only meaningful when
--enable-shared is specified; if this flag is specified without
--enable-shared, it is ignored, and static libraries are created.

--enable-echo

     Will echo all of the commands that configure executes.  This is
usually for debugging purposes only, and is not recommended for end
users.

--enable-shared

     Build shared libraries.  Note that this option is incompatible
with --with-romio (which is the default) and --with-mpi2cpp (which is
also the default) because (among other reasons) ROMIO expects to find
libmpi.a, not libmpi.so.

     Also note that enabling building shared libraries does *not*
disable building the static libraries.  Specifying --enable-shared
without --disable-static will result in a build taking twice as long,
and installing both the static and shared libraries.

     Finally, note that neither ROMIO nor the MPI 2 C++ bindings do
not currently support shared libraries.  They will always be built as
static libraries.

--prefix=PREFIX

     Sets the installation location for the LAM binaries, libraries,
etc., to the directory PREFIX.  PREFIX must be specified as an
absolute directory name.

--with-cc=CC

     Use the C compiler CC.  The C compiler can also be selected by
setting the "CC" environment variable before running configure.  This
compiler will be used both to compile LAM, and as the default compiler
for the hcc(1) and mpicc(1) wrapper compilers.

--with-cflags=CFLAGS

     Use the C compiler flags CFLAGS.  The flags passed to the C
compiler can also be selected by setting the "CFLAGS" environment
variable before running configure.  These flags are used to compile
LAM, ROMIO, and some example programs that come with LAM.  If CFLAGS
are not specified, ./configure will pick optimization flags to use.

     These flags are *not* used as default flags in any of the wrapper
compilers.

--with-cxx=CXX

     Use the C++ compiler CXX.  The C++ compiler can also be selected
by setting the "CXX" environment variable before running configure.
This compiler will be used to compile the MPI 2 C++ bindings, IMPI
support, and will be used as the default compiler for the hcp(1) and
mpiCC(1) wrapper compilers.

--with-cxxflags=CXXFLAGS

     Use the C++ compiler flags CXXFLAGS.  The flags passed to the C++
compiler can also be selected by setting the "CXXFLAGS" environment
variable before running configure.  These flags will be used when
compiling the MPI 2 C++ bindings, IMPI support, as well as some
example programs that come with LAM.  If CXXFLAGS are not specified,
./configure will pick optimization flags to use.

     These flags are *not* used as default flags in any of the wrapper
compilers.

--with-cxxldflags=CXXLDFLAGS

     Use the C++ linker flags CXXLDFLAGS.  These flags will be used
when compiling the MPI 2 C++ bindings, IMPI support, as well as some
example programs that come with LAM.  If CXXFLAGS are not
specified. ./configure will pick optimization flags to use.

     These flags are *not* used as default flags in any of the wrapper
compilers.

--with-exceptions

      Used to enable exception handling support in the C++ bindings
for MPI.  Exception handling support (i.e., the
MPI::ERRORS_THROW_EXCEPTIONS error handler) is disabled by default.
See the section "MPI 2 C++ Issues", above.

--with-exflags=FLAGS

      Used to specify any command line arguments that are necessary
for the C, C++, and Fortran compilers to enable C++ exception support.
This switch is ignored unless --with-exceptions is also specified.

      This switch is unnecessary for gcc/g77/g++ version 2.95 and
above -- "-fexceptions" will automatically be used (when building
--with-exceptions).  Additionally, this switch is unnecessary if the
KCC compiler is used -- "-x" is automatically used.

      See the section entitled "MPI 2 C++ Issues", above.

--with-fc=FC

     Use the Fortran compiler FC.  Specify FC=no (or --without-fc) to
disable Fortran support if you do not have a Fortran compiler or do
not require such support.  This compiler will be used both to compile
LAM, and as the default compiler for the hf77(1) and mpif77(1) wrapper
compilers.

--with-fflags=FFLAGS

     Use the Fortran compiler flags FFLAGS when compiling LAM.  The
flags passed to the Fortran compiler can also be selected by setting
the "FFLAGS" environment variable before running configure.  These
flags will be used only when compiling some example programs that come
with LAM.  If FFLAGS are not specified, ./configure will pick
optimization flags to use.

     These flags are *not* used as default flags in any of the wrapper
compilers.

--with-impi

     Use this switch to enable the IMPI extensions.  The IMPI
extensions are still consdered experimental, and are disabled by
default.

--with-lamd-ack=SEC

     Number of seconds until an ACK is resent between LAM daemons.
You probably shouldn't need to change this; the default is one a
second.

--with-lamd-hb=SEC

    Number of seconds between heartbeat messages in the LAM daemon
(only applicable when running in fault tolerant mode).  You probably
shouldn't need to change this; the default is 120 seconds.

--with-lamd-boot=SEC

     Set the default number of seconds to wait before a process started
on a remote node is considered to have failed (e.g., during lamboot).
You probably shouldn't need to change this; the default is 60 seconds.

--with-ldflags=LDFLAGS

     Use the LD linker flags LDFLAGS.  If this flag is not set on the
./configure command line, the value for CFLAGS is used.  These flags
are used to link LAM executables and all example programs that come
with LAM.  If LDFLAGS (and CFLAGS) are not specified, ./configure will
pick optimization flags to use.

     These flags are *not* used as default flags in any of the wrapper
compilers.

--without-mpi2cpp

     Build LAM without the MPI-2 C++ bindings (see chapter 10 of the
MPI-2 standard); the default is to build them.  The C++ bindings
require some advanced features of the C++ compiler.  While most modern
C++ compilers now support all the required features, you may encounter
problems on some platforms.  Consult the mpi2c++/README file for more
information.

--without-profiling

     Build LAM/MPI without the MPI profiling layer.  The default is to
build this layer, since ROMIO uses it.  See the --without-romio option
for more details.

--with-pthread-lock

     Use a process shared pthread mutex to lock access to the shared
memory pool rather than the default SYSV semaphore.  This option is
only valid with the "usysv" RPI, and on systems which support process
shared pthread mutexes.

--with-purify

     Causes LAM to zero out all data structures before using them.
This option is not necessary to make LAM function correctly (LAM
already zeros out relevant structure members when necessary), but it
is very helpful when running MPI programs through memory checking
debuggers, such as purify and the Solaris Workshop bcheck program.
See the "Zeroing out LAM buffers before use" section of the
RELEASE_NOTES file for more information.  The default is to not enable
this option.

--without-romio

     Build LAM without ROMIO support (ROMIO provides the MPI-2 I/O
support, see chapter 9 of the MPI-2 standard); the default is to build
*with* ROMIO support.  ROMIO is known to only work on certain systems.
Consult the romio/README file for more information.  Note that this
option is incompatible with --with-shared, because (among other
reasons) ROMIO expects to find libmpi.a, not libmpi.so.

     Note also that building ROMIO implies building the profiling
layer.  ROMIO makes extensive use of the MPI profiling layer; that is
you cannot select --without-profiling without also specifying
--without-romio. 

--with-romio-flags=FLAGS

     Pass FLAGS to ROMIO's configure script when it is invoked during
the build process.  This switch is to effect specific behavior in
ROMIO, such as building for a non-default file system (e.g., PVNFS).
Note that LAM already sends the following switches to ROMIO's
configure script -- the --with-romio-flags switch should not be used
to override them:

     --prefix
     -mpi
     -mpiincdir
     -cc
     -fc
     -debug (if -g is specified in CFLAGS)
     -cflags
     -fflags
     -nof77 (if --without-fc is selected in LAM)
     -make
     -mpilib

--with-rpi=RPI

     Build with request progression interface (RPI) transport layer
RPI [RPI=tcp]. RPI must one of: tcp, sysv, or usysv.  If this option
is not specified, the RPI transport layer defaults to tcp.  Please
refer to the RELEASE_NOTES file for descriptions of the RPI transport
layers.

--with-rsh=RSH

     Use RSH as the remote shell command. For example if you want to
use the secure shell ssh then specify --with-rsh="ssh -x" (note that
the "-x" is necessary to prevent the ssh 1.x series of clients from
sending its standard banner information to standard error, which will
cause recon/lamboot/etc. to fail).  This shell command will be used to
launch commands on remote nodes from binaries such as lamboot, wipe,
etc.  The command can be one or more shell words, such as a command
and multiple command line switches.

This value can be overridden at recon/lamboot/etc. run time with the
LAMRSH environment variable.  See the RELEASE_NOTES file for more
details.

--with-select-yield

     Force the use of select() to yield the processor. 

--with-shm-maxalloc=BYTES

     Use BYTES as the size of the maximum allocation from the shared
memory pool.  If no value is specified, configure will set the size
according to the value of shm-poolsize (below).  See "Usysv and Sysv
transports", below.

--with-shm-poolsize=BYTES

     Use BYTES as the size of the shared memory pool.  If no size is
specified, configure will determine a suitably large size to use.  See
"Ususv and Sysv transports", below.

--with-shm-short=BYTES

     Use BYTES as the maximum size of a short message when
communicating via shared memory.  Default is 8 KB.

--without-shortcircuit

     Disable the send/receive short circuiting optimization. The short
circuit optimization has proven to be fairly stable, and this option
is not usually necessary.  It remains for hysterical raisins.

--with-signal=SIGNAL

     Use SIGNAL as the signal used internally by LAM. The default
value is "SIGUSR2". To set the signal to "SIGUSR1" for example,
specify --with-signal=SIGUSR1.

--with-tcp-short=BYTES

     Use BYTES as the maximum size of a short message when
communicating over TCP.  Default is 64 KB.  This is relevant to all
RPIs, since the shared memory RPIs are multi-protocol -- they will use
TCP when communicating with MPI ranks that are not in the same node.

--with-thread

     This option is not yet supported.  Do not use it.

--with-trillium

     Build and install the Trillium support executables, header files,
and man pages.  These extra Trillium executables, header files, and
man pages are not necessary for normal MPI operation; they are
intended for Trillium developers and certain third party products that
interact with the lower layer of LAM/MPI.  Building XMPI
(http://www.lam-mpi.org/software/xmpi/), for example, requires that
all the Trillium header files were previously installed.  Hence, if
you intend to compile XMPI after installing LAM/MPI, you should use
this option.

     Building the extra Trillium executables and installing the
Trillium header files and man pages used to be the default in prior
versions of LAM/MPI.  However, since few users actually used them, it
has been relegated to an option.


Example: 

     % ./configure --with-rpi=usysv --with-cc=/bin/cc \
         --with-cflags=-O4 -without-fc

Compile for the usysv RPI using the C compiler /bin/cc with options
-O4 and disable Fortran support.



64 bit LAM
----------

LAM has been verified as being 64 bit clean under Solaris 7, AIX
4.3.3, IRIX 6.5, and Alpha/Linux 2.2.x.  To compile LAM with the 64
bit architecture, you will likely need to add compiler and linker
flags with configure.  For example, if you are using the Solaris
Workshop 5.0 compilers on Solaris 7, you can use the following:

     % ./configure --with-cflags='-xarch=v9' --with-ldflags='-xarch=v9'

Other compilers/architectures will have their own flags to enable 64
bit compilation; consult the documentation for your compiler.  Of
course, you can also add in any debugging/optimization flags in the
cflags and ldflags strings as well.


Building LAM
------------

Once the configuration step has completed, build LAM by doing:

     % make

in the top level LAM directory. This will build the LAM binaries and
libraries within the distribution source tree.  Once they have
compiled properly, you can install them with:

     % make install

*** NOTE ***: Previous version of LAM included "make install" in the
default "make".  THIS IS NO LONGER TRUE.  You *must* execute "make
install" to install the LAM executables, libraries, and header files
to the location specified by the --prefix option to configure.


-- Building LAM, ROMIO, and MPI 2 C++ examples

LAM and the ROMIO and MPI-2 C++ packages all include example code that
can be built with a single top-level "make examples".  Note that the
examples can *only* be built after a successful "make install", and
$prefix/bin has been placed in your $path.

     % make examples

This will do the following (where TOPDIR is the top-level directory of
the LAM source tree):

  1. Build the LAM examples.  They are located in:

     TOPDIR/examples

  2. If LAM was configured to build the C++ examples (i.e., if you did
     not configure with --without-mpi2cpp), the MPI 2 C++ examples
     will be built.  They are located in:

     TOPDIR/mpi2c++/contrib

  3. If you configured LAM with ROMIO support (i.e., if you did not
     configure with --without-romio), the ROMIO examples will be
     built.  See the notes about ROMIO in the RELEASE_NOTES file.
     They are located in:

     TOPDIR/romio/test

Additionally, the following three commands can be used to build each
of the packages' examples separately (provided that support for each
was compiled in to LAM) from TOPDIR:

     % make lam-examples
     % make romio-examples
     % make mpi2c++-examples


Boot schema
-----------

A boot schema is a description of a multicomputer on which LAM will be
run.  You can create boot schema files (see bhost(5) for syntax) for
typical configurations of the local multicomputer(s).  Place these
files under etc/ in the installation directory.  They will be found by
LAM tools such as lamboot(1), recon(1) and wipe(1) if you do not
specify a filename on the command line to use instead of the default.

The default etc/lam-bhost.def file comes with a single line:

     localhost

So that if you simply do "lamboot", you will get a LAM with one node
(the localhost) booted.

You can re-write the etc/lam-bhost.def file if you are frequently
going to boot LAM to the same configuration.  For example, if you
frequently use 4 workstations: inky, blinky, pinky, and clyde, you can
have a etc/lam-bhost.def files as follows:

     inky
     blinky
     blinky
     blinky
     blinky
     pinky cpu=2
     clyde user=lamrocks

Note that "blinky" is listed 4 times.  This tells LAM/MPI that blinky
has 4 CPUs (relevant for the "C" notation to the mpirun command; see
mpirun(1)).  An alternate (and equivalent) notation is used for pinky
-- "cpu=2" specifies that pinky has 2 CPUs.

You can also specify different remote usernames on the remote nodes;
the username "lamrocks" is used on the machine "clyde" in the above
example.


Using LAM
---------

If the LAM installation directory is moved after it is built, users
must set the LAMHOME environment variable to the new location.  This
is the *only* case where the LAMHOME environment variable should be
set -- otherwise, it should be left unset.  See "The LAMHOME and
TROLLIUSHOME environment variables", below.

On each UNIX machine, users must add the LAM executable directory to
their shell's search path.  LAM executables are found under
$prefix/bin.  These steps must be taken on each and every machine that
might be part of a multicomputer running LAM.  Set the variables in
the shell's start-up file, **not the .login file***.


--- Typical usage

LAM is a daemon-based implementation of MPI.  This means that a daemon
process is launched on each machine that will be in the parallel
environment.  Once the daemons have been launched, LAM is ready to be
used.  A typical usage scenario is as follows:

     - Boot LAM on all the nodes
     - Run MPI programs
     - Shut down LAM

LAM does not need to be booted in order to compile MPI programs.

LAM is a user-based MPI environment; each user who wishes to use LAM
must boot their own LAM environment.  LAM is not a client-server
environment where a single LAM daemon can service all LAM users on a
given machine.  There are no future plans to make LAM client-server
oriented (unless someone volunteers to write it :-).

As a side-effect of this design, each user must have an account on
each machine that they wish to use LAM on. 


--- The LAMHOME and TROLLIUSHOME environment variables

Note that it is typically *not necessary* to set the LAMHOME and/or
TROLLIUSHOME environment variables.  These variables are *only*
necessary if the $prefix of the LAM installation is moved after "make
install" was run.

As such, there are very few cases when one would need to set LAMHOME
or TROLLIUSHOME.  The LAM Team recomends that you leave these
variables unset.


--- Starting LAM

The recon(1) tool checks if LAM can be started on the given boot
schema.  There are several prerequisites that enable LAM to be started
on a remote machine:

     * The machine must be reachable and operational. 
     * The user must have an account on the machine. 
     * The user must be able to rsh(1) (or use an rsh substitute --
       see above for details on how to specify a different remote
       shell) to the machine (typically, permissions must be set in
       the user's .rhosts file on the machine).
     * The user must be able to write to /tmp.
     * The LAM executables must be locatable on that machine, using
       the shell's search path and possibly the LAMHOME environment
       variable, as described above.
     * The shell's start-up script must not print anything on standard
       error. The user can take advantage of the fact that rsh(1) will
       start the shell non-interactively. The start-up script can exit
       early in this case, before executing many commands relevant
       only to interactive sessions and likely to generate output.

*All* of these prerequisites must be met before LAM will function
properly.  If recon does not complete successfully, the "-d" option
will give verbose descriptions of what it tried to do, and suggestions
to fix the problem.

Also keep in mind that just because recon works, lamboot itself may
still fail.  This usually happens when the "hboot" program (that
lamboot invokes on remote nodes) fails for some reason.  Again, the
"-d" option to lamboot will enable extremely verbose output, and
suggest solutions to common problems.

Users should read the lam(7) manual page to get started using LAM
tools and libraries.

Additionally, the University of Notre Dame offers a "Getting Started
with LAM" tutorial, that, although somewhat biased towards the Notre
Dame computing environment, is a good starting point to getting
familiar with LAM.  

		http://www.lam-mpi.org/tutorials/lam/


--- Common filesystems

A common environment to run LAM in is a Beowulf-class or other
workstation cluster.  Simply stated, LAM can run on a group of
workstations connected by a network.  As mentioned above, there are
several prerequisites, however (the user must have an account on all
the machines, the user can rsh [or ssh, or whatever other remote shell
transport capability is desired -- see above for how to change the
underlying remote shell transport] to all the machines, etc.).

This raises the question for LAM system administrators: where to
install the LAM binaries, header files, etc.?  There are two main
choices:

1. Have a common filesystem, such as NFS, between all the machines to
be used.  Install the LAM files such that the LAM executables can be
found in the *same directory* on each node.  This will *greatly*
simplify user's .cshrc/.profile scripts -- the value of the $PATH can
be set without checking which machine the user is on.  It also
simplifies the system administrator's job; when the time comes to
patch or otherwise upgrade LAM, only one copy needs to be modified.

For example, consider a cluster of four machines: inky, blinky, pinky,
and clyde.  If the LAM binaries et al. are installed on inky's local
hard drive in the directory /home/lam, the system administrator has
two main choices:

  - mount inky:/home/lam on the remaining three machines, such that
/home/lam on all machines is effectively "the same".  That is, the
following directories all contain the LAM binaries:

     inky:/home/lam
     blinky:/home/lam
     pinky:/home/lam
     clyde:/home/lam

  - mount inky:/usr/local/src/lam-6.5.6 on *all four* machines in some
other common location, such as /home/lam (a symbolic link can be
installed on inky instead of a mount point for efficiency).  This
strategy is typically used for environments where one tree is NFS
exported, but another tree is typically used for the location of
binaries.  For example, the following directories all contain the LAM
binaries:

     inky:/home/lam
     blinky:/home/lam
     pinky:/home/lam
     clyde:/home/lam

Notice that there are the same four directories as the previous
example, but on inky, the directory is *actually* located in
/usr/local/src/lam-6.5.6.  There is a bit of a disadvantage in this
approach; each of the remote nodes have to incur NFS (or whatever
filesystem is used) delays to access the LAM directory tree.  However,
both the administration ease and low cost (relatively speaking) of
using a networked file system usually greatly outweighs the cost.


2. If you are concerned with networked filesystem costs of accessing
the LAM binaries, you can install LAM on the local hard drive of each
node in your system.  Again, it is *highly* advisable to install LAM
in the *same* directory on each node so that user's $PATH can be set
to the same value, regardless of the node that a user has logged on
to.

This approach will save some network latency of accessing the LAM
binaries, but is only used where users are very concerned about
squeezing every spare cycle out of their machines.


--- Using LAM with AFS

AFS has some peculiarities, especially with file permissions when
using rsh.  However, most sites tend to install the Transarc rsh
replacement (i.e., the one that passes tokens to the remote machine)
as the default rsh, so when you "rsh" to a remote machine (with recon
or lamboot), your AFS token will be passed to the remote LAM daemon
automatically.  If your site does not install the Transarc replacement
rsh as the default, consult the documentation on "--with-rsh" (above)
to see how to set the path to the rsh that LAM will use.

Once you use the replacement rsh, you should get a token on the other
side.  This means that your LAM daemons are running with your AFS
token, and you should be able to run any program that you wish,
including those that are not system:anyuser accessible.  You will even
be able to write into your filespace (as you would expect).

Keep in mind, however, that AFS tokens have limited lives, and will
eventually expire.  This means that your LAM daemons (and user MPI
programs) will lose their AFS permissions after some specified time
unless you renew your token (with the "klog" command, for example) on
the originating machine before the token runs out.  This can play
havoc with long-running MPI programs that periodically write out file
results; if you lose your AFS token in the middle of a run, and your
program tries to write out to a file, it won't have permission to,
which may cause Bad Things to happen.

If you need to run long MPI jobs with LAM on AFS, it is usually
advisable to ask your AFS administrator to increase your default token
life time to a large value, such as 2 weeks.


--- Using LAM with ssh

Note that you can change the remote transport agent that LAM uses to
spawn the LAM daemons.  While rsh is the default, it can be changed to
other agents, such as ssh.  

ssh is a popular choice because of the added security that it provides
over the .rhosts security provided by rsh.  And since ssh can pass AFS
tokens, it presents an attractive, highly secure, yet
fully-AFS-authenticated method, for invoking LAM.

If you choose to use ssh, the 1.x series of ssh will require the use
of the "-x" command line flag to prevent ssh from printing its
standard banner information to stderr.  lamboot/recon/etc. interprets
information on stderr to mean that a remote invocation has failed;
ssh's "-x" will prevent this.  (We do not have access to SSH 2.x
clients -- they may require a similar command line flag).

Note that using ssh (or any other agent) only changes the way that LAM
is *invoked*.  Once LAM is invoked, it sets up its own sockets for
communication that are outside of ssh (and are therefore not
encrypted).  ssh provides stronger security only during lamboot and
wipe.  Once the LAM daemons are launched, all MPI meta information is
passed through separate channels (such as startup of user programs)
which are independent of ssh.


Troubleshooting
---------------

--- Problems with building LAM

It is highly recommended that you execute the following steps *in
order*.  Many people have similar problems with configuration and
initial setup of LAM, and most common problems have already been
answered in one way or another.

1. Check the LAM FAQ:

     http://www.lam-mpi.org/faq/

2. Check the mailing list archives.  Use the "search" features to
check old posts and see if others have asked the same question and
had it answered:

     http://www.lam-mpi.org/MailArchives/lam/

3. If you do not find a solution to your problem in the above
resources, and your problem specifically has to do with *building*
LAM, send the following information to the LAM mailing list (see the
next section below about sending mail to the LAM mailing list):

- The result of "uname -a" on your system
- The result of "./config/config.guess" from the top-level LAM source
  directory. 
- Output from when you ran "./configure" to configure LAM
- The config.log file from the top-level LAM directory
- The share/include/lam_config.h file
- Output from when you ran "make" to build LAM

To capture the output of the configure and make steps you can use the
script command or the following technique if using a csh style shell:

     % ./configure {options} |& tee config.LOG
     % make install          |& tee make.LOG

or if using a Bourne style shell:

     % ./configure {options} 2>&1 | tee config.LOG
     % make install 2>&1          | tee make.LOG


--- The LAM/MPI Mailing Lists

There are two mailing lists: one for LAM/MPI announcements, and
another for questions and user discussion of LAM/MPI.

1. Announcement list.

This is a low-volume list that is used to announce new version of
LAM/MPI, important patches, etc.  To subscribe to the LAM announcement
list, visit its list information page (you can also use that page to
unsubscribe or change your subscription options):

     http://www.lam-mpi.org/mailman/listinfo.cgi/lam-announce

2. General discussion/user list.

This list is used for general questions and discussion of LAM/MPI.
User can post questions, comments, etc. to this list.  Due to problems
with spam, only subscribers are allowed to post to the list.  To
subscribe or unsubscribe from the list, visit the list information
page:

     http://www.lam-mpi.org/mailman/listinfo.cgi/lam

After you have subscribed (and received a confirmation e-mail), you
can send mail to the list at the following address:

     lam@lam-mpi.org

NOTE: People tend to only reply to the list; if you subscribe, post,
and then unsubscribe from the list, you will likely miss replies.

Also please be aware that lam@lam-mpi.org is a list that goes to
several hundred people around the world -- it is not uncommon to move
a high-volume exchange off the list, and only post the final
resolution of the problem/bug fix to the list.  This prevents
exchanges like "Did you try X?", "Yes, I tried X, and it did not
work.", "Did you try Y?", etc. from cluttering up peoples' inboxes.


--- Problems with running LAM and/or user programs

Check the LAM FAQ and mailing list archive resources mentioned in the
previous section (Problems with building LAM).  If you do not find the
solution to your problem there, send mail to the LAM mailing list:
lam@lam-mpi.org.

Some typical problems with rsh include the following:

     * Incorrect permissions on a user's home directory
     * Incorrect permissions on $HOME/.rhosts
     * No entry (or incorrect entry) in $HOME/.rhosts

Some typical problems with a user's environment include the following:

     * User's .cshrc/.profile does not put $prefix/bin in the path
     * Inaccessible permissions on the program that you are trying to
       run
     * Inaccessible permissions on the /tmp directory


--- Insufficient shared resources

When using the sysv or usysv RPIs, the operating system may run out of
shared memory and/or semaphores.  This is typically indicated by
failing to run an MPI program, or failing to run more than X copies of
an MPI program on a single node.

To fix this problem, your operating system settings need to be
modified to increate the allowable shared semaphores/memory.  

For Linux, teconfiguration can only be done by building a new kernel.
First modify the appropriate constants in
include/asm-<arch>/shmparam.h.  Increasing SHMMAX will allow larger
shared segments and increasing _SHM_ID_BITS allows for more shared
memory identifiers (this information is likely from 2.0 linux kernels;
it may or may not have changed in more recent versions).

For Solaris, reconfiguration can be done by modifying /etc/system and
then rebooting. See the Solaris man page system(4).
 
For example to set the maximum shared memory segment size to 32 MB
put the following in /etc/system:

     set shmsys:shminfo_shmmax=0x2000000

If you are using the sysv transport and are running out of semaphores then
the following tunables can be set.

     set semsys:seminfo_semmap=32
     set semsys:seminfo_semmni=128
     set semsys:seminfo_semmns=1024

Please consult your system documentation for help in determining the
correct values for your systems.


Clearing disk space
-------------------

After LAM has been built, all of the objects can be removed by running
the make(1) utility with the "clean" target in the source directory.

     % make clean 

NOTE: If you are using a really picky version of make (such as
OpenBSD's make), you may need to use "make -i clean".

If you're *really* desperate for more space, a bit more space can be
reclaimed by running:

     % make distclean

NOTE: Again, if you are using a really picky version of make (such as
OpenBSD's make), you may need to use "make -i distclean".

If further space is required, the entire source directory can be taken
off-line (indeed, "make distclean" returns the LAM source tree to the
same state as it was when it was unpacked from the original
distribution tarball).  Only the installation directory need be
maintained on-line.


Tuning LAM
----------

There are various constants defined in the LAM header files which
relate to message transfer protocols, shared memory allocation, and so
on.  Some of these are configurable via the configure script; it is
hoped that in time, more and more options will be configurable.

This section is intended to describe some of these constants so that
LAM users can experiment with tuning the MPI library.  It also
provides some description of the transport layer internals which may
help LAM users better understand the behavior and performance they see
from the LAM MPI library.


--- Short/long protocol

LAM MPI uses a short/long message protocol. If a message is "short",
it is sent together with a header in one transfer to the destination
process.  If the message is "long", then a header (possibly with some
data) is sent to the destination.  The sending process then waits for
an acknowledgment from the receiver before sending the rest of the
message data.  The receiving process sends the acknowledgment when a
matching receive is posted.

The crossover point from "short" to "long" message is configurable in
each transport.  See the transport specific section tcp, sysv, or
usysv for further information.


-- Shortcircuit send/receive

Typically, when a message is sent or received, LAM creates a request
structure, fills it with information about the message, links the
request into a list of messages, and calls a progression "engine" to
effect the data transfer.

When there are no active requests and a blocking (standard mode) send
or receive is done, the overhead of creating the request and linking it
into the list can be bypassed (shortcircuited) and the progression
"engine" called directly to effect the transfer.

In prior versions of LAM/MPI, this option was not the default.  It is
now used by default, unless specifically disabled via the configure
script.

--- TCP transport

The crossover point from "short" to "long" message is configurable via
the constant TCPSHORTMSGLEN in share/include/lam_config.h (relative to
the top of the LAM build tree).  It can also be set from the configure
script via the --with-tcp-short option.  The default is 64KB.

This number is relevant to all the RPIs.  The shared memory RPIs are
multi-protocol; they will use LAM/MPI use TCP to communicate with
ranks that are not on the same node.


--- Usysv and sysv transports

Descriptions of the usysv and sysv transports can be found in the "RPI
transport layers" section of the RELEASE_NOTES file.

Configuration constants for the usysv and sysv transports are found in
share/include/rpi.shm.h (from the top of the LAM build directory).

In these transports, processes on different nodes communicate via TCP
sockets.  The crossover point from "short" to "long" messages for
these communications is configurable via the constant TCPSHORTMSGLEN.
It can also be set from the configure script via the --with-tcp-short
option.  The default is 64KB.

Processes located on the same node communicate via shared memory.  The
transport allocates one SYSV shared segment shared by all processes in
the tasks which are on the node.  This segment is logically divided
into two areas.

The "postbox" area contains postboxes for "short" message
communication.  A postbox is used for communication one-way between
two processes.  The space allocated per postbox is SHMSHORTMSGLEN +
CACHELINESIZE.  SHMSHORTMSGLEN is configurable (via the configure
option --with-shm-short).  It is the the crossover point from "short"
to "long" messages in shared memory communication; the default value
is 8 KB.

CACHELINESIZE must be the size of a cache line or a multiple thereof.
The default setting is 64 bytes.  You shouldn't need to change it.
CACHELINESIZE bytes in the postbox are used for a cache-line sized
synchronization location.

The size of the postbox area is np (np-1) (SHMSHORTMSGLEN +
CACHELINESIZE) bytes.

The rest of the shared memory area is used as a global pool from which
space for long message transfers is allocated.  Allocation from this
pool is locked.  The default lock mechanism is a SYSV semaphore but
the configure option --with-pthread-lock can be used to change this to
a process shared pthread mutex lock.  The size of this pool is
configurable via the constant LAM_MPI_SHMPOOLSIZE, and by the configure
option --with-shm-poolsize.

The configure script will try to determine a size for the pool if none
is explicitly specified.  You should always check this to see if it is
reasonable.  Larger values should improve performance especially when
an application passes large messages, but will also increase the
system resources used by each task.

The total size of the shared segment allocated is 2 CACHELINESIZE +
LAM_MPI_SHMPOOLSIZE + np (np-1) (SHMSHORTMSGLEN + CACHELINESIZE).  The
2 CACHELINESIZE bytes are for the global pool lock.


--- Use of the global pool

When a message larger than 2 SHMSHORTMSGLEN is sent, the transport
sends SHMSHORTMSGLEN bytes with the first packet.  When the
acknowledgment is received, it allocates (message length -
SHMSHORTMSGLEN) bytes from the global pool to transfer the rest of the
message.

To prevent a single large message transfer from monopolizing the
global pool, allocations from the pool are actually restricted to a
maximum of LAM_MPI_SHMMAXALLOC bytes.  Even with this restriction, it
is possible for the global pool to temporarily become exhausted.  In
this case, the transport will fall back to using the postbox area to
transfer the message.  Performance will be degraded, but the
application will progress.

LAM_MPI_SHMMAXALLOC is configurable via the configure option
--with-shm-maxalloc or editing rpi.shm.h.


--- Synchronization

The usysv and sysv transports differ only in the mechanism used to
synchronize the transfer of messages via shared memory.  The usysv
transport uses spin locks with back-off, while the sysv transport uses
SYSV semaphores.

Both transports use a few SYSV semaphores for synchronizing the
deallocation of shared structures or for synchronizing access to the
shared pool.

The usysv transport should be superior to the sysv transport on
multiprocessors.  On uniprocessors, which is better depends on the OS
and the means used for processor yielding.  On a Linux uniprocessor,
for example, using semaphores (sysv transport) appears to be vastly
superior to spin-locking.


--- Usysv transport spin-locks

The usysv transport uses spin locks with back-off.  When a process
backs off, it attempts to yield the processor.  If the configure
script found a system provided yield function such as yield() or
sched_yield(), this is used. If no such function is found, then
select() on NULL file descriptor sets with a timeout of 10us is used.

The use of select() to yield can be forced by the --with-select-yield
option to the configure script.


--- Sysv transport semaphores

The sysv transport allocates a semaphore set (of size 6) for each
process pair communicating via shared memory.  On some systems, you
may need to reconfigure the system to allow for more semaphore sets if
running tasks with many processes communicating via shared memory.