File: RELEASE_NOTES

package info (click to toggle)
lam 6.5.6-6
  • links: PTS
  • area: main
  • in suites: woody
  • size: 15,464 kB
  • ctags: 11,670
  • sloc: ansic: 101,210; cpp: 12,146; sh: 10,821; makefile: 4,050; fortran: 218
file content (1195 lines) | stat: -rw-r--r-- 45,469 bytes parent folder | download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
Copyright 1998-2001, University of Notre Dame.
Authors: Jeffrey M. Squyres, Arun Rodrigues, and Brian Barrett with
         Kinis L. Meyer, M. D. McNally, and Andrew Lumsdaine

This file is part of the Notre Dame LAM implementation of MPI.

You should have received a copy of the License Agreement for the Notre
Dame LAM implementation of MPI along with the software; see the file
LICENSE.  If not, contact Office of Research, University of Notre
Dame, Notre Dame, IN 46556.

Redistribution and use in source and binary forms, with or without
modification, are permitted subject to the conditions specified in the
LICENSE file.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT,
INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.

Additional copyrights may follow.


Release notes for LAM 6.5.6
============================

This file contains the release notes for LAM/MPI version 6.5.6.
Here's a brief table of contents (### indicates updates for this
version):

     * Installation instructions
 ### * Supported systems
     * Caveats
     * Backward compatibility
     * LAM and Linux
 ### * New feature overview
 ### * PBS support
 ### * LAM help file
     * Zeroing out LAM buffers before use
 ### * Trillium executables / header files / man pages
 ### * Shared library support
 ### * mpirun behavior
 ###   - SMP-aware command line syntax
       - Ability to pass environment variables
 ###   - Pseudo-tty support
       - Current working directory behavior
       - Ability to run shell scripts/debuggers/etc.
 ###   - The default behavior of mpirun has changed  
 ### * lamexec
     * hcc / hcp / hf77 / mpicc / mpiCC / mpif77
 ### * lamboot / recon / wipe / lamhalt
 ### * lamnodes
     * Root execution disallowed
     * RPI transport layers
       - tcp
       - usysv
       - sysv
 ### * IMPI extensions
     * Signal catching
     * Internal signal
     * New basic datatypes
     * MPI-2 Support
       - MPI-IO / ROMIO 
       - C++ bindings
       - Inter-language interoperability
       - One-sided communication
 ###   - Dynamic processes
       - MPI_Info
       - Communicator and window error handling
       - Handle conversions
       - Attributes on communicators, datatypes, and windows
       - New derived type constructors and type inquiry functions
 ###   - Miscellaneous


Installation instructions
-------------------------

Please see the installation guide for instructions on installing LAM
6.5.6 in the INSTALL file.


Supported systems
-----------------

LAM 6.5.6 has been officiall tested on the following systems:

        AIX 4.3.3
        IRIX 6.5
        Linux 2.2.16
        OpenBSD 2.8
        Solaris 2.7

Even if your operating system is not listed here, LAM will likely
compile/run properly.  The source code is fairly portable -- it should
work properly under most POSIX-like systems.

We request that you also download the "lamtest" package from the LAM
web site and run it on your system.  If any errors are encountered,
please send us all the output (see the README file in that package).

We would greatly appreciate your time and effort in helping to verify
LAM/MPI on a wide variety of systems.


Caveats
-------

LAM is fully MPI-1 complaint with the exception of MPI_CANCEL.
MPI_CANCEL works properly for receives, but will almost never work on
sends.  MPI_CANCEL is most frequently used with unmatched MPI_IRECV's
that were made "in case" a matching message arrived.  This simply
entails removing the receive request from the local queue, and is
fairly straightforward to implement.

Actually canceling a send operation is much more difficult because
some meta information about a message is usually sent immediately.  As
such, the message is usually at least partially sent before an
MPI_CANCEL is issued.  Trying to chase down all the particular cases
is a nightmare, to say the least.

As such, the LAM Team decided not to implement MPI_CANCEL on sends,
and instead concentrate on other features.  

But in true MPI Forum tradition, we would be happy to discuss any code
that someone would like to submit that fully implements MPI_CANCEL.
:-)


Backward compatibility
----------------------

LAM provides full source code backward compatibility with previous
versions of LAM.  Old applications that compiled properly with older
versions of LAM can simply be recompiled with this version of LAM.

Binary compatibility, however, is not provided -- applications that
have been compiled with previous versions of LAM will need to be
recompiled in order to run properly with this version of LAM.  If
applications are not re-compiled with this LAM, their behavior will be
unpredictable (read: they just won't work).

Specifically -- users need to ensure that they use the new versions of
all the LAM commands in order to run properly.  For example, mixing
LAM 6.3.2 and 6.5.6 commands (e.g., running the "wrong" version of
lamd) will result in unpredictable behavior.


LAM and Linux
-------------

LAM is frequently used on Linux-based machines (iX86 and otherwise).
It works correctly under 2.0.36 (we didn't test under 2.0.37, but we
have no reason to believe that it wouldn't work under that version as
well, since it is really only minor changes from 2.0.36) and 2.2.x.
We haven't explicitly tested under 2.4, but we don't expect any
problems with it.

However, versions 2.2.0 through 2.2.9 had some TCP/IP performance
problems.  It seems that version 2.2.10 fixed these problems; if you
are using a Linux version between 2.2.0 and 2.2.9, LAM may exhibit
poor C2C performance due to the Linux TCP/IP kernel bugs.  We recommend
that you upgrade to 2.2.10 (or the latest version).  

See http://www.lam-mpi.org/linux/ for a full discussion of the
problem.


New feature overview
--------------------

     LAM 6.5

     * Added man pages for all MPI-1 and MPI-2 functions.
     * Added some primitive syslog messages to the lamd (when launched
       via "lamboot -d ...") to help the troubleshooting process when
       the lamd seems to be involved.
     * lamboot, recon, and [deprecated] wipe are noticably faster when
       the "-b" switch is used.  "-b" can be used in situations where
       the user's shell is the on the remote nodes as on the local
       node.
     * Allow a single user to have multiple LAM instances on a single
       node; this is useful for batch kinds of systems.  LAM does this
       automatically when runnning LAM in a PBS job, but users can
       also do this manually.
     * New SMP-aware "mpirun" command line syntax and boot schema
       syntax for "lamboot".
     * "-pty", pseudo tty support, is now enabled by default with
       mpirun.  Use the new option "-npty" to disable pseudo tty
       support.
     * MPI_UNIVERSE_SIZE now reports total number of CPUs, not nodes.
     * Slightly changed the internal and install directory structures;
       now fully using automake and libtool for building LAM.
     * Full support for VPATH builds.
     * Shared library support (only if not using ROMIO or the C++
       bindings).
     * Added MPI_Info key to MPI_COMM_SPAWN to specify a schema file
       for MPMD spawns.
     * Added "lamhalt" command to quickly shut down LAM
     * Added "lamnodes" command to retrieve hostnames from nX and cX
       nomenclature.
     * Added MPI_ALLOC_MEM and MPI_FREE_MEM, and 2 optional RPI calls
       to support them, mainly in anticipation of Myrinet and VIA
       RPIs.
     * Added "-s" option to lamboot to allow "rsh somenode lamboot -s
       hostfile" to complete normally.
     * Expanded many error messages to be more descriptive and
       generally user-friendly.
     * Updated MPI 2 C++ bindings distribution.
     * Updated and patched ROMIO distribution.  Building ROMIO is now 
       the default.
     * Building the profiling layer is now the default.
     * Supplemental Trillium executables and .h files that are not
       necessary for MPI operation are no longer built/installed by
       default.
     * Made the lamtests package easier to port to other
       implementations of MPI.
     *  Added MPI_INIT_THREAD, MPI_IS_THREAD_MAIN, MPI_QUERY_THREAD.
     * Oodles of bug fixes.

     LAM 6.3 series

     * Fixed critical bug in sysv and usysv RPIs
     * Added ROMIO MPI I/O package (chapter 9 from the MPI-2 standard)
       from the Argonne National Laboratory
     * Added the MPI-2 C++ bindings package (chapter 10 from the MPI-2
       standard) from the Laboratory for Scientific Computing at the
       University of Notre Dame
     * Pseudo-tty support for remote IO (e.g., line buffered output).
     * Ability to pass environment variables through mpirun.
     * Ability to mpirun shell scripts/debuggers/etc. (that eventually
       run LAM/MPI programs).
     * New default present working directory behavior for mpirun
     * Ability to execute non-MPI programs across the multicomputer.
     * Added configurable ability to zero-fill internal LAM buffers before
       they are used (for development tools such as Purify).
     * Greatly expanded error messages; provided for customizable
       local help files.
     * Expanded and updated documentation.
     * Wrapper compilers now use same compilers as in ./configure. 
     * Made -lmpi implicit in wrapper compilers.
     * Fixed compatibility problems with some compilers.
     * Fixed problem with some compilers not getting prototypes
       properly from <mpi.h>.
     * Various bug fixes (see the HISTORY file).

     LAM 6.2 series

     * GNU ./configure script for installing LAM.
     * Full standard conforming support for MPI-2 dynamic processes.
     * Support for MPI-2 one-sided communication.
     * Many MPI-2 support functions implemented.
     * More basic datatypes supported (including MPI-2 datatypes such
       as MPI_TYPE_DARRAY).
     * Direct TCP and shared-memory message passing transports.
     * Optimized the common send and receive cases.
     * All command line arguments are transparently passed to the user
       program when main() is invoked (no additional command line
       arguments are introduced by LAM).
     * New user level locking shared memory RPI transport. 
     * Signal catching to aid the termination of batch jobs.
     * Support for shared libraries on Linux and Solaris.
     * Various small bug fixes. 


PBS support
-----------

LAM is now "somewhat aware" of the Portable Batch System (PBS).  That
is, the following problems have been solved:

- The "multiple LAM" problem has been fixed.  When running under PBS,
LAM will take precautions to isolate itself from other instances of
LAM in concurrent PBS jobs.  That is, the multiple LAM instances from
the same user can exist on the same machine when executing under PBS.
This allows a user to submit as many LAM jobs as necessary, and not
have any fear that jobs will be killed.  

This "multiple LAM" behavior is *only* exhibited under a batch
environment (i.e., PBS).  Other batch systems can easily be supported
-- let the LAM Team know if you'd like to see support for others
included.  Additionally, manually setting the environment variable
LAM_MPI_SOCKET_SUFFIX on the node where lamboot is run achieves the
same ends.

- PBS will generate a $PBS_NODEFILE with the same hostname multiple
times when multiple vnodes from the same node are allocated by the
scheduler.  LAM will now correctly parse this file, and take it to
mean that the node has multiple CPUs available.  See the section on
lamboot and mpirun, below.

One major feature that is still lacking with respect to PBS is to use
PBS's implementation of the Task Management (TM) extensions.  Once LAM
uses the TM extensions:

- "epilog" scripts will no longer be necessary to guarantee the
  cleanup of LAM jobs

- accounting information will accurately reflect all usage (including
  on nodes other than where the mother superior resides)

- the world will be a better place


LAM help file
-------------

The following LAM binaries have had their help messages greatly
expanded:

        mpicc / mpiCC / mpif77
        lamboot
        lamhalt
        recon

The messages should be much more helpful in trying to diagnose
problems, especially for first-time users.  The help messages
generally try to identify the problem and suggest solutions.  It is
possible for multiple cascading error messages to be printed; one
failure may cause other failures.  As such, the first error message is
generally (but not always) the most relevant message -- solving that
error may solve the rest.

Additionally, much more information is now output when the "-d" switch
is used on all of these program (which enables debugging output).

The help messages are all contained in a single ASCII file which is
initially installed into the following file (where $prefix is the
option supplied to --prefix in the ./configure script):

        $prefix/etc/lam-6.5.6-helpfile

The format of the file is simple; simple delimiter lines separate help
topic messages.  It should be quite obvious which message corresponds
to which program/topic name combination.

This file allows system administrators to customize help messages for
their users according to the local environment.  When LAM tries to
find the helpfile to print out a help message, it actually searches
for the file in the following order:

        $LAMHELPFILE
        $HOME/lam-helpfile
        $HOME/lam-6.5.6-helpfile
        $HOME/etc/lam-helpfile
        $HOME/etc/lam-6.5.6-helpfile
        $LAMHELPDIR/lam-helpfile
        $LAMHELPDIR/lam-6.5.6-helpfile
        $LAMHOME/etc/lam-helpfile
        $LAMHOME/etc/lam-6.5.6-helpfile
        $TROLLIUSHOME/etc/lam-helpfile
        $TROLLIUSHOME/etc/lam-6.5.6-helpfile
        $prefix/etc/lam-helpfile
        $prefix/etc/lam-6.5.6-helpfile

This seemingly-over complicated scheme will allow for maximum
flexibility for system administrators and/or users to define the
location of customized help files.


Zeroing out LAM buffers before use
----------------------------------

LAM has several structures that are used in many different situations.
One example is LAM's internal "struct nmsg" -- one of the underlying
message constructs used to pass data between LAM entities.  But since
the "struct nmsg" is used in so many places, it is a generalized
structure and contains fields that are not used in every situation.

By default, LAM only initializes relevant struct members before using
a structure.  Using a structure may involve sending the entire
structure (including uninitialized members) to a remote host.  This is
not a problem, because the remote host will also ignore the irrelevant
struct members (depending on the specific function being invoked).
More to the point -- LAM was designed this way to avoid setting
variables that will not be used; this is a slight optimization in
run-time performance.

Memory-checking debuggers are quite popular (such as purify and the
Solaris Workshop/Forte bcheck program), and quite useful to find
memory leaks, indexing past the end of arrays, and other types of
Heisenbugs.  Since LAM "uses" uninitialized memory, it tends to
generate many warnings with these types of debuggers.

The --with-purify option has been added to the ./configure script that
will force LAM to zero out *all* memory before it is used.  This will
eliminate the "read from uninitialized" types of warnings that
memory-checking debuggers will identify deep inside LAM.  However,
this option invokes a slight overhead penalty in the run-time
performance of LAM, so it is not the default.


Trillium executables / header files / man pages
-----------------------------------------------

In prior versions of LAM/MPI, several additional Trillium executables
were built during the default build (Trillium is the built-in lower
layer of the LAM/MPI run-time system).  Additionally, several
Trillium-specific header files and man pages were installed by
default.

These executables, header files, and man pages are actually intended
for LAM/Trillium developers, and are not necessary for normal MPI
use.  That is -- these executables, header files, and man pages are
not necessary to develop, compile, and run normal MPI programs under
LAM/MPI.  Third party packages may require the Trillium header files
and/or executables, however.  For example, compiling XMPI will require
that the Trillium header files were previously installed.

As such, these executables have been removed from the default build.
Correspondingly, the header files and man pages have been removed from
the default install.  There is an option to configure to enable both
the Trillium executables to be built as well as the header files and
man pages to be installed.  See the INSTALL file for details.


Shared library support
----------------------

Since LAM/MPI is now build with automake/libtool, the two libraries
necessary for MPI programs (liblam and libmpi) can be built as shared
or static libraries.  See the INSTALL file for details.

However, the two third party packages that LAM/MPI includes -- the MPI
2 C++ bindings and ROMIO -- do *not* support shared library builds.
If you select to build either of these packages (both are enabled by
default; see the INSTALL file for how to disable them), building
LAM/MPI with shared library support is disallowed.

Additionally, GNU libtool also does not appear to build shared
libraries correctly on all systems.  For example, libtool fails to
build the LAM shared libraries properly under AIX 4.3.3.  Hence, LAM
can only be built statically on such platforms.  This is a limitation
of libtool, not LAM.  Since LAM uses libtool to build itself, libtool
limitations can constrain the build options for LAM/MPI.  libtool
cannot also reliably build C++ libraries (it *usually* works for
static linking of C++ libraries that have no templates, but shared
libraries typically don't work with non-GNU compilers).

It is hoped that future versions of the GNU libtool package will allow
buildings C++ libraries and correct building of shared libraries on
all platforms.  Write to your congressman/local government
representative!  ;-)


mpirun
------

Several new features have been added to mpirun.


--- SMP-aware command line syntax

New "C" syntax has been added to mpirun (note that this section also
applies to the "lamexec" command).  When running on SMP machines, it
is frequently desirable to group as many adjoining ranks as possible
on a single node in order to maximize shared memory message passing.
When used in conjunction with the extended bootschema syntax (that
allows the specification of number of CPUs available on each host),
the mpirun "C" syntax will run one executable on each available CPU,
and will group adjoining MPI_COMM_WORLD ranks on the same nodes.  For
example, when running on two SMPs, the first having four CPUs, and the
second having two CPUs, the following command:

      % mpirun C foo

will run four copies of "foo" on the four-way SMP (MPI_COMM_WORLD
ranks 0 through 3), and will run two copies of "foo" on the two-way
SMP (MPI_COMM_WORLD ranks 4 and 5).

Just like the "N" syntax in mpirun, the "C" syntax can also be used to
indicate specific CPUs.  For example:

      % mpirun c4,5 foo

runs "foo" on the fourth and fifth CPUs (i.e., the two-way SMP from
the previous examples).  "C" and "cX" syntax can also be combined:

      % mpirun c0 C master-slave

could be used to launch a "master" process (i.e., rank 0 in
MPI_COMM_WORLD) on CPU zero, a slave on each CPU (including rank 0.
This may be desirable, for example, in situations where the master
rank does very little computation).

The behavior of "-np" has been altered to match the "C" semantics.
"-np" now schedules across CPUs, not nodes.  Using "-np 6" in the
previous example would be the same as "C"; using "-np 4" would run one
four copies of "foo" on the four-way SMP.

Also note that "N", "nX", C", and "cX" syntax can all be used
simultaneously, although it is not clear that this is really useful.


--- Ability to pass environment variables

All environment variables named LAM_MPI_* are now automatically
passed to remote notes (unless disabled via the "-nx" option to
mpirun).  The "-x" option enabled exporting of specific environment
variables to the remote nodes:

     % LAM_MPI_FOO="green eggs and ham"
     % export LAM_MPI_FOO
     % mpirun C -x DISPLAY,SEUSS=author samIam

This will launch the "samIam" application on all available CPUs.  The
LAM_MPI_FOO, DISPLAY, and SEUSS variables will be created on all nodes
before the user's program is invoked.

Note that the parser for the "-x" option is currently not very
sophisticated -- it cannot even handle quoted values when defining new
environment variables.  Users are advised to set variables in the
environment prior to invoking mpirun, and only use "-x" to export the
variables to the remote nodes (not to define new variables), if
possible.


--- Pseudo-tty support

The "-pty" option to mpirun enabled pseudo tty support.  Among other
things, this gives line-buffered output from the remote nodes (which
is probably what you want).  "-pty" is now the default mode for
mpirun.  Use the "-npty" option to disable pseudo tty support.


--- Current working directory behavior

The default behavior of mpirun has changed with respect to the
directory that processes will be started in.

The "-wd" option to mpirun allows the user to change to an arbitrary
directory before their program is invoked.  It can also be used in
application schema files to specify working directories on specific
nodes and/or for specific applications.  

If the "-wd" option appears both in a schema file and on the command
line, the schema file directory will override the command line value.
"-wd" is mutually exclusive with "-D".

If neither "-wd" nor "-D" are specified, the local node will send the
directory name where mpirun was invoked from to each of the remote
nodes.  The remote nodes will then try to change to that directory.
If they fail (e.g., if the directory does not exist on that node),
they will start from the user's home directory.

All directory changing occurs before the user's program is invoked; it
does not wait until MPI_INIT is called.  


--- Ability to run shell scripts/debuggers/etc.

mpirun can now also run non-LAM/MPI programs.  That is, one can mpirun
a shell script, debugger, or any other program that will eventually
either exec a LAM/MPI program or spawn a LAM/MPI program as a child.

This is extremely helpful for batch systems and debugging
environments.  For example:

     % mpirun C gdb

However, one must be careful when running programs on remote nodes
that expect the use of stdin.  Recall that stdin on remote nodes is
redirected to /dev/null.  It is probably more advantageous to export
the DISPLAY environment variable, and run a shell script that invokes
an xterm with "gdb" running in it on each node.  For example:

     % mpirun C -x DISPLAY xterm-gdb.csh


--- The default behavior of mpirun has changed

The default options now correspond to "-w -c2c -nger".  That is:

        - wait for the application to terminate
        - use the fast client-to-client communication mode
        - disable GER. 

To get the old behavior, use the options "-lamd -ger -nw".

mpirun now recognizes command lines of the form

     % mpirun -np <nprocs> {LAM specific mpirun args} <program> {program args}

For example, 

     % mpirun -np 4 -lamd n0 n1 /bin/foobar 12 a b c

runs 4 copies of the /bin/foobar program on nodes n0 and n1 (2 copies
on each node), passing the arguments, 12 a b c, to the program.  The
new syntax is equivalent to the following in the "-c" syntax which is
still supported.

     % mpirun -c <nprocs> {LAM specific mpirun args} <program> -- {program args}


lamexec
-------

The lamexec command has been added to LAM/MPI's reporitore.  It is an
"mpirun clone", but is specifically for running non-MPI programs.
That is, one can do the following:

     % lamexec N hostname

which will run "hostname" on all nodes in the multicomputer.  It can
take most of the same command line arguments as mpirun; it does not
support the flags that do not make sense for non-MPI programs (e.g.,
-c2c, -lamd, etc.).  See lamexec(1) for more details.

Note that the "C" syntax for mpirun also applies to lamexec.


hcc / hcp / hf77 / mpicc / mpiCC / mpif77
-----------------------------------------

The hcc, hcp, and hf77 wrapper compilers have previously not
automatically passed the "-lmpi" option to the underlying compiler.
The rationale behind this decision was that the "mpicc" and "mpif77"
wrapper compilers added this functionality; the "h" wrappers were
intended as Trollius compilers, not LAM/MPI compilers.

But hcc, hcp, and hf77 have become the de facto wrapper compilers
(vs. mpicc and mpif77).  Indeed, some users have been confused about
why -lmpi is not implicit to the "h" wrapper compilers.  

Hence, "-lmpi" is now automatically passed to the underlying compiler
in the hcc, hcp, and hf77 wrapper compilers.  The mpicc and mpif77
compilers are now symbolic links to hcc and hf77, respectively. 

For symmetry, mpiCC has been created as a symbolic link to hcp.


lamboot / recon / wipe / lamhalt
--------------------------------

The help messages in the lamboot, lamhalt, and wipe commands have been
greatly enhanced.  It is hoped that this will be very useful to new
users of LAM.

- lamboot, recon, and wipe are now noticably faster when used with the
"-b" switch.  Without "-b", these commands each use two remote shell
invocations (e.g., rsh or ssh) to each node.  The first remote shell
invocation determines if the user's default shell was the Bourne/Korn
shell or not.  The second remote shell invocation actually runs the
specificed LAM binary.

Using the "-b" switch, lamboot/recon/wipe will assume that the user's
shell on all remote nodes is the same as the user's shell on the local
node.  This means that only *one* remote shell invocation will be
executed for each remote node, resulting in a nice speedup.

In either case, if the user's shell is not csh, tcsh, or bash, .profile
will be executed before other LAM binaries are executed.

- Setting the LAM_MPI_SOCKET_SUFFIX environment variable before
running lamboot will a user to have multiple instances of the LAM/MPI
run-time environment on a single machine.  This is useful for batch
and/or automated execution types of environments where the same node
may be used multiple times.

- The wipe command is now deprecated in favor of the lamhalt command.
The lamhalt command is identical in functionality to wipe except that
it does not require a hostfile argument to shut down an entire LAM.
That is, to shut down LAM, you can simply execute:

      % lamhalt

lamhalt is considerably faster and does not require a hostfile.  wipe
should only be used if something happened to "damage" the currently
running LAM such that lamhalt fails to shut it down properly.

- lamboot has been extended to understand multiple CPUs on a single
host, and is intended to be used in conjunction with the new "C"
mpirun syntax for running on SMP machines (see the section on mpirun).
Multiple CPUs can be indicated in two ways: list a hostname multiple
times, or add a "cpu=N" phrase to the host line (where "N" is the
number of CPUs available on that host).  For example, the following
hostfile:

        blinky
        blinky
        blinky
        blinky
        pinky cpu=2

indicates that there are four CPUs available on the "blinky" host, and
that there are two CPUs available on the "pinky" host.  Note that this
works nicely in a PBS environment, because PBS will list a host
multiple times when multiple vnodes on a single node have been
allocated by the scheduler.

Note that different usernames can be specified for specific hosts as
well.  For example:

       blinky cpu=2 user=lamguest

specifies that the username "lamguest" should be used to login to the
machine "blinky".  This is different than previous syntax for
specifying usernames for remote nodes; although its use is deprecated,
the "-o" option to recon, lamboot, and wipe can be used to obtain
previous parsing behavior.

- The "-s" option to lamboot will now close the stdout/stderr of the
locally-launched LAM daemon.  Normally, the stdout/stderr of the local
lamd is left open so that LAM's tstdio(3) emulation package works
properly.  Using "-s" does not affect the behavior of mpirun; for
example:

        # csh-based example
        mpirun -np 4 foo >& foo.out

still works as expected, by redirecting the output of the parallel run
of "foo" to the file foo.out.  The "-s" option does allow the lamboot
command to be executed remotely by a remote agent such as rsh:

        rsh somenode lamboot -s hostfile

rsh will wait for stdout/stderr to close before returning.  So even if
lamboot completes, rsh will hang while waiting for stdout/stderr of
all launched processes to close.  The "-s" option will force lamboot
to close stdout/stderr and allow rsh to complete normally.  While most
programs will not be affected, be sure to note that "-s" will break
tstdio functionality.

- The wipe command is now symmetric to lamboot in that if you do not
specify a hostfile on the command line, wipe will use the default boot
schema ($prefix/etc/lam-bhost.def, where $prefix is the top-level
directory where LAM was installed).

Previous versions of wipe would ask for a filename if none were
specified on the command line; this option has been removed.

- Additionally, the debugging capability of lamboot has been enhanced.
If the "-d" switch is given to lamboot, lamboot (and its helper
programs) become quite verbose, showing exactly what is being executed
(or not) -- which is a great help when attempting to determine the
exact source of user problems.  When started up in debugging mode, the
lamd will now output some diagnostic messages to the syslog when
starting and when launching user programs via mpirun or lamexec.  The
messages are output at the LOG_INFO severity, and will be prefixed
with "lamd" and the lamd's PID.

- Finally, the LAMRSH environment variable can be set at invocation
time to override the default remote shell command that
recon/lamboot/wipe/etc. use to invoke commands on remote nodes (e.g.,
rsh, ssh, etc.).  The default value is wherever rsh is found in the
path at ./configure time (or specified with the --with-rsh option);
setting the LAMRSH environment variable will override this setting.

The LAMRSH environment variable can contain a binary name and any
number of arguments.  For example (for the C shell and its
derivatives): 

     % setenv LAMRSH "ssh -x"


lamnodes
--------

LAM was specifically designed to abstract away hostnames once lamboot
has completed successfully.  However, for various reasons (usually
related to system-administration concerns, and/or for creating
human-readable reports), it can be desirable to retrieve the hostnames
of LAM nodes long after lamboot.

A new command, "lamnodes", can be used for this purpose.  It accepts
both the "N" and "C" syntax from mpirun, and will return the
corresponding names of the specified nodes.  For example:

      % lamnodes C

will return the node that each CPU is located on, the hostname of that
node, and the total number of CPUs on each (lamnodes' command line
arguments can be used to customize its output).

      % lamnodes n0,3

will return the node, hostname, and number of CPUs on n0 and n3.


Root execution disallowed
-------------------------

It is a Very Bad Idea to run the LAM executables as root.

LAM was designed to be run by individual users; it was *not* designed
to be run as a root-level service where multiple users use the same
LAM daemons in a client-server fashion (see "Typical Usage" in the
INSTALL file).  LAM should be booted by each individual user who
wishes to run MPI programs.  There are a wide array of security issues
when root runs a service-level daemon; LAM does not even attempt to
address any of these issues.

Especially with today's propensity for hackers to scan for root-owned
network daemons, it could be tragic to run this program as root.
While LAM is known to be quite stable, and LAM does not leave network
sockets open for random connections after the initial setup, several
factors should strike fear into system administrator's hearts if LAM
were to be constantly running for all users to utilize:

        1. LAM leaves a unix domain socket open on each machine in the
           /tmp directory.  So if someone breaks into root on one
           machine, they effectively have root on all machines that
           are connected via LAM.  

        2. Indeed, there must have been a .rhosts (or some other trust
           mechanism) for root which must have allowed you to run LAM
           on remote nodes.  Depending on your local setup, this may
           not be safe.

        3. LAM has never been checked for buffer overflows and other
           malicious input types of errors.  We don't *think* that
           there are any buffer-overflow types of situations in LAM,
           we've never checked explicitly (hence, per Mr. Murphy,
           there are certainly some hiding somewhere).

        4. LAM programs are not audited or tracked in any way.  This
           could present a sneaky way to execute binaries without log
           trails (especially as root).

Hence, it's a Very Bad Idea to run LAM as root.  LAM binaries will
quit immediately if root runs them.  Login as a different user to run
LAM.


RPI transport layers
--------------------

LAM currently provides three client-to-client transport layers which
implement the request progression interface (RPI).  As in LAM 6.1, the
LAM daemon RPI transport (lamd) is always available.  It is no longer
the default transport and must be explicitly invoked via the -lamd
option to mpirun.

The three client-to-client transports are:

--- tcp

     The tcp transport uses TCP sockets for all interprocess
communication.

--- usysv

     The usysv transport is multi-protocol. Processes on the same node
communicate via SYSV shared memory and processes on different nodes
communicate via TCP sockets.  It uses spin-locks for shared memory
synchronization as well as a SYSV semaphore or pthread mutex for
synchronizing access to a per node global shared memory pool.

The spin-locks require that the architecture has strongly ordered
writes and this transport is only supported on such platforms. It
should be relatively easy to modify this transport to work on systems
with weakly ordered writes by adding memory barriers in appropriate
places.

The usysv transport should give the best performance on SMPs. Please
refer to the tuning notes for more on the performance of the various
transports and on tuning them.

--- sysv

     The sysv transport is the same as the usysv transport except that
SYSV semaphores are used for message synchronization rather than
spin-locks. On some uniprocessor systems (e.g. Linux), the blocking
nature of semaphores can lead to better performance than when using
spin-locks.


IMPI extensions
---------------

The IMPI extensions are still considered experimental, and are
disabled by default in LAM.  They must be enabled when LAM is
configured and built -- see the INSTALL file for details.

Since the IMPI extensions are still experimental, relevant
instructions and release notes are in the top-level README.impi file
in the LAM distribution.



Signal catching
---------------

LAM MPI now catches the signals SEGV, BUS, FPE, and ILL.  The signal
handler terminates the application. This is useful in batch jobs to
help ensure that mpirun returns if an application process dies. To
disable the catching of signals use the -nsigs option to mpirun.


Internal signal
---------------

The signal used internally by LAM has been changed from SIGUSR1 to
SIGUSR2 to reduce the chance of conflicts with the Linux pthreads
library. The signal used is configurable.  See the installation guide
for the specific ./configure flag that can be used to change the
internal signal.


New basic datatypes
-------------------

Support has been added for the MPI_LONG_LONG_INT,
MPI_UNSIGNED_LONG_LONG and MPI_WCHAR basic datatypes.


MPI-2 Support
-------------

LAM 6.5.6 has support for many MPI-2 features.


--- MPI-IO / ROMIO

MPI-IO support has been added by including the ROMIO package from
Argonne National Labs (http://www.mcs.anl.gov/romio/), version 1.0.3.
The MPI-IO functions are described in chapter 9 of the MPI-2 standard,
which can be found at http://www.mpi-forum.org/.

The ROMIO package can be compiled with LAM, and the LAM wrapper
compilers (mpicc/mpiCC/mpif77) will automatically do "the right
things" to compile/link user programs that use ROMIO function calls.

The ROMIO group at Argonne has now included support for LAM in their
package.  However, there are still "one or two" things that the LAM
Team had to do to make ROMIO behave nicely with LAM/MPI.  As such, if
you try to install the ROMIO package manually with LAM/MPI, you will
experience some difficulties.

ROMIO support can be disabled via the LAM ./configure script; see the
INSTALL file for specific instructions.

There are some important limitations to ROMIO that are discussed in
the romio/README file.

One limitation that is not currently listed in the ROMIO README file
is that atomic file access will not work with AFS.  This is because of
file locking problems with AFS (i.e., AFS does not support file
locking).  The ROMIO test program "atomicity" will fail if you specify
an output file on AFS.

Since building ROMIO is now the default for LAM/MPI, building LAM's
profiling layer is also the default.  You cannot build ROMIO without
profiling support (i.e., if you attempt to configure LAM to not build
the profiling layer and still build ROMIO, the configure script will
complain).  

Additionally, ROMIO does not [yet] support shared libraries.  Even
though LAM can now support shared libraries, if you select to build
ROMIO (the default), shared library support is disabled.


Additionally, ROMIO does not support the following LAM functionality:

- LAM MPI-2 datatypes cannot be used with ROMIO; ROMIO makes the
  fundamental assumption that MPI-2 datatypes are built upon MPI-1
  datatypes.  LAM builds MPI-2 datatypes natively -- ROMIO cannot
  presently handle this case.

  This will hopefully be fixed in some future release of ROMIO.  The
  ROMIO test programs "coll_test", "fcoll_test", "large_array", and
  "coll_perf" will fail because they use the MPI-2 datatype
  MPI_DARRAY.

Please see the sections "ROMIO Users Mailing List" and "Reporting Bugs"
in romio/README for how to submit questions and bug reports about
ROMIO (that do not specifically pertain to LAM).


--- C++ bindings

C++ bindings for MPI-1 are provided from the MPI-2 C++ bindings
package from the University of Notre Dame
(http://www.mpi.nd.edu/research/mpi2c++/), version 1.0.6.  The MPI-1
C++ bindings are described in Chapter 10 and Appendix B of the MPI-2
standard, which can be found at http://www.mpi-forum.org/.

The C++ bindings package is compiled, by default, with LAM, and the
LAM wrapper compilers (mpicc/mpiCC/mpif77) will automatically do "the
right things" to compile/link user programs that use MPI C++ bindings
function calls.

Note that the C++ bindings have requirements on the degree of
conformance that your C++ compiler supports; see the file
mpi2c++/README for more details.  If your C++ compiler cannot support
the requirements of the C++ bindings package, it is safest just to
disable MPI C++ bindings support in LAM (however, most modern C++
compilers now support all the required functionality).

MPI C++ bindings support can be disabled via the LAM ./configure
script; see the INSTALL file for specific instructions.

Please see the "Contact Information" section of the mpi2c++/README
file for how to submit questions and bug reports about the MPI 2 C++
bindings package (that do not specifically pertain to LAM).


--- Inter-language interoperability

Inter-language interoperability is supported. It is now possible to
initialize LAM MPI from either C or Fortran and mix MPI calls from
both languages.


--- One-sided communication

Support is provided for get/put/accumulate data transfer operations
and for the post/wait/start/complete and fence synchronization
operations. No support is provided for window locking.

The datatypes used in the get/put/accumulate operations are restricted
to being basic datatypes or single level contiguous/vectors of basic
datatypes.

The implementation of the one-sided operations is layered on top of
the point-to-point functions, and will thus perform no better than
them.  Nevertheless it is hoped that providing this support will aid
developers in developing and debugging codes using one-sided
communication.

The following functions related to one-sided communication have been
implemented.

     MPI_Win_create
     MPI_Win_free 
     MPI_Win_get_group 

     MPI_Get
     MPI_Put
     MPI_Accumulate

     MPI_Win_fence 
     MPI_Win_post 
     MPI_Win_wait 
     MPI_Win_start 
     MPI_Win_complete 


--- Dynamic processes

The dynamic process support provided in LAM 6.2b has been extended and
the function names changed to conform to the final MPI-2 standard. The
following functions related to dynamic process support are provided.

     MPI_Comm_spawn
     MPI_Comm_spawn_multiple
     MPI_Comm_get_parent 
     MPI_Comm_accept
     MPI_Comm_connect
     MPI_Comm_disconnect
     MPI_Comm_join

     MPI_Lookup_name 
     MPI_Publish_name 
     MPI_Unpublish_name 

     MPI_Open_port 
     MPI_Close_port 

As requested by LAM users, MPI_Comm_spawn and MPI_Comm_spawn_multiple
can now take an MPI_Info argument specifying an appschema file for
MPMD spawns.  The key name is "file", and the value should be the
filename of the appschema (relative to the present working directory
on the root, or an absolute pathname accessible on the root).

These functions supersede the MPIL_Comm_spawn function that LAM/MPI
introduced in version 6.2b.  MPIL_Comm_spawn is considered
depreciated and should not be used (although it is still available for
backwards compatibility).

--- MPI_Info

Full support for info objects is provided. 

     MPI_Info_create 
     MPI_Info_free 
     MPI_Info_delete 
     MPI_Info_dup 
     MPI_Info_get 
     MPI_Info_get_nkeys 
     MPI_Info_get_nthkey 
     MPI_Info_get_valuelen 
     MPI_Info_set 


--- Communicator and window error handling

The new communicator error handler functions are supported, and window
error handlers are also supported.

     MPI_Comm_create_errhandler
     MPI_Comm_get_errhandler
     MPI_Comm_set_errhandler

     MPI_Win_create_errhandler
     MPI_Win_get_errhandler 
     MPI_Win_set_errhandler 


--- Handle conversions

Handle conversions for inter-language interoperability are supported. 

     MPI_Comm_f2c 
     MPI_Comm_c2f 

     MPI_Group_f2c 
     MPI_Group_c2f 

     MPI_Type_f2c 
     MPI_Type_c2f 

     MPI_Request_f2c 
     MPI_Request_c2f 

     MPI_Info_f2c 
     MPI_Info_c2f 

     MPI_Win_f2c 
     MPI_Win_c2f 

     MPI_Status_f2c
     MPI_Status_c2f


--- Attributes on communicators, datatypes, and windows

Attributes may now be set on and retrieved from datatypes and
windows. The new communicator attribute handling functions are also
supported.

     MPI_Comm_create_keyval
     MPI_Comm_free_keyval 
     MPI_Comm_delete_attr 
     MPI_Comm_get_attr 
     MPI_Comm_set_attr 

     MPI_Type_create_keyval
     MPI_Type_free_keyval 
     MPI_Type_delete_attr 
     MPI_Type_get_attr 
     MPI_Type_set_attr 

     MPI_Win_create_keyval
     MPI_Win_free_keyval 
     MPI_Win_delete_attr 
     MPI_Win_get_attr 
     MPI_Win_set_attr 


--- New derived type constructors and type inquiry functions

Support has been added for the following new derived type constructors.

     MPI_Type_create_struct
     MPI_Type_create_hindexed 
     MPI_Type_create_hvector
     MPI_Type_dup 
     MPI_Type_create_resized
     MPI_Type_create_subarray
     MPI_Type_create_darray

Support has also been added for the new type inquiry functions.

     MPI_Type_get_contents
     MPI_Type_get_envelope
     MPI_Type_get_extent 
     MPI_Type_get_true_extent


--- Miscellaneous

Implementations of the following functions are provided. LAM 6.5.6
reports its MPI version as 1.2.

     MPI_Alloc_mem
     MPI_Finalized
     MPI_Free_mem
     MPI_Get_version
     MPI_Get_address 

The MPI_Alloc_mem and MPI_Free_mem functions (and their two
corresponding RPI functions) have been added in anticipation of RPIs
that can take advantage of "special" memory for fast message passing.
They are simply wrappers to malloc() and free() (respectively) in the
currently provided RPIs, but can be used for portability and future
performance gains.