1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
|
.. _Software-Hierarchy-label:
How to use a Software Module hierarchy
======================================
Libraries built with one compiler need to be linked with applications
built with the same compiler version. For High Performance Computing there
are libraries called Message Passing Interface (MPI) that allow for
efficient communication between tasks on a distributed memory computer
with many processors. Parallel libraries and applications must be
built with a matching MPI library and compiler. To make this
discussion clearer, suppose we have the intel compiler version 15.0.1
and the gnu compiler collection version 4.9.2. Also we have two MPI
libraries: mpich version 3.1.2 and openmpi version 1.8.2. Finally we
have a parallel library HDF5 version 1.8.13 (phdf5).
Of the many possible ways of specifying a module layout, this flat
layout of modules is a reasonable way to do this::
$ module avail
--------------- /opt/apps/modulefiles ----------------------
gcc/4.9.2 phdf5/gcc-4.9-mpich-3.1-1.8.13
intel/15.0.2 phdf5/gcc-4.9-openmpi-15.0-1.8.13
mpich/gcc-4.9-3.1.2 phdf5/intel-15.0-mpich-3.1-1.8.13
mpich/intel-15.0-3.1.2 phdf5/intel-15.0-openmpi-15.0-1.8.13
openmpi/gcc-4.9-1.8.2
openmpi/intel-15.0-1.8.2
In order for users to load the matching set of compilers and MPI libraries,
they will have to load the matching set of modules. For example this
would be correct::
$ module load gcc/4.9.2 openmpi/gcc-4.9-1.8.2 phdf5/gcc-4.9-openmpi-15.0-1.8.13
But it is quite easy to load an incompatible set of modules. Now, it is
possible that the system administrators at your site might have set up
``conflict`` s to avoid loading mismatched modules. However, using
conflicts can be fragile. What happens if a site adds a new compiler
such as clang or pgi or a new mpi stack? All those module file
conflict statements will have to be updated.
A different strategy is to use a software hierarchy. In this approach
a user loads a compiler which extends the **MODULEPATH** to make
available the modules that are built with the currently loaded
compiler (similarly for the mpi stack).
Our modulefile hierarchy is stored under
``/opt/apps/modulefiles/{Core,Compiler,MPI}``. The Core directory is for
modules that are not dependent on Compiler or MPI implementations. The
Compiler directory is for packages which are only Compiler
dependent. Lastly, the MPI directory is packages which dependent on
MPI-Compiler pairing. The modulefiles for the compilers are placed in the
Core directory. For example the gcc version 4.9.2 file is in Core/gcc/4.9.2.lua
and contains::
-- Setup Modulepath for packages built by this compiler
local mroot = os.getenv("MODULEPATH_ROOT")
local mdir = pathJoin(mroot,"Compiler/gcc", "4.9")
prepend_path("MODULEPATH", mdir)
This code asks the environment for **MODULEPATH_ROOT** which is
``/opt/apps/modulefiles`` and the last two lines prepend
``/opt/apps/modulefiles/Compiler/gcc/4.9`` to the **MODULEPATH** .
The modulefiles for the MPI implementations are placed under the
Compiler directory because they only depend on a compiler. The
openmpi module file for the gcc-4.9.2 compiler is then stored at
``/opt/apps/modulefiles/Compilers/gcc/4.9/openmpi/1.8.2.lua`` and it
contains::
-- Setup Modulepath for packages built by this MPI stack
local mroot = os.getenv("MODULEPATH_ROOT")
local mdir = pathJoin(mroot,"MPI/gcc", "4.9","openmpi","1.8")
prepend_path("MODULEPATH", mdir)
The above code will prepend
``/opt/apps/modulefiles/MPI/gcc/4.9/openmpi/1.8`` to the
**MODULEPATH**.
The above description is a suggested way to handle the modulefile
software hierarchy. The software packages themselves can be stored in
many ways. For software packages, but not the modulefiles, we store
them in another software hierarchy as follows:
#. Core packages: **/opt/apps/pkgName/version**
#. Compiler dependent packages: **/opt/apps/compilerName-version/pkgName/version**
#. MPI-Compiler dependent packages: **/opt/apps/compilerName-version/mpiName-version/pkgName/version**
The modulefiles also need to be stored in a software hierarchy as
well.
#. Compiler dependent modulefiles: **/opt/apps/modulefiles/Compilers/compilerName/compiler-version/pkgName/version**
#. MPI dependent modulefiles: **/opt/apps/modulefiles/MPI/compilerName/compiler-version/mpiName/mpi-version/pkgName/version**
The regression testing suite that comes with the Lmod source has many
examples of a software hierarchy. See the directory
**rt/hierarchy/mf/** from the Lmod source tree.
When **MODULEPATH** changes, Lmod unloads any modules which are not
currently in the **MODULEPATH** and then tries to reload all the
previously loaded modules. Any modules which are not available are
marked as inactive. Those inactive modules become active if found with
new **MODULEPATH** changes.
.. Note::
In all of the examples above, we used only the first two
version numbers. In other words, we used 4.9 instead of 4.9.2 and
similarly 1.8 instead of 1.8.2. It is our view that for at least
compilers and MPI stacks, the third digit is typically a bug
fix and doesn't require rebuilding all the dependent
packages. Y.M.M.V.
|
