1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
|
.. _label-frameworks:
Internal frameworks
===================
The Modular Component Architecture (MCA) is the backbone of Open MPI
-- most services and functionality are implemented through MCA
components.
MPI layer frameworks
--------------------
Here is a list of all the component frameworks in the MPI layer of
Open MPI:
* ``bml``: BTL management layer
* ``coll``: MPI collective algorithms
* ``fbtl``: point to point file byte transfer layer: abstraction for
individual read: collective read and write operations for MPI I/O
* ``fcoll``: collective file system functions for MPI I/O
* ``fs``: file system functions for MPI I/O
* ``hook``: Generic hooks into Open MPI
* ``io``: MPI I/O
* ``mtl``: Matching transport layer, used for MPI point-to-point
messages on some types of networks
* ``op``: Back end computations for intrinsic MPI_Op operators
* ``osc``: MPI one-sided communications
* ``pml``: MPI point-to-point management layer
* ``sharedfp``: shared file pointer operations for MPI I/O
* ``topo``: MPI topology routines
* ``vprotocol``: Protocols for the "v" PML
OpenSHMEM component frameworks
------------------------------
* ``atomic``: OpenSHMEM atomic operations
* ``memheap``: OpenSHMEM memory allocators that support the
PGAS memory model
* ``scoll``: OpenSHMEM collective operations
* ``spml``: OpenSHMEM "pml-like" layer: supports one-sided,
point-to-point operations
* ``sshmem``: OpenSHMEM shared memory backing facility
Miscellaneous frameworks
------------------------
* ``allocator``: Memory allocator
* ``backtrace``: Debugging call stack backtrace support
* ``btl``: Point-to-point Byte Transfer Layer
* ``dl``: Dynamic loading library interface
* ``hwloc``: Hardware locality (hwloc) versioning support
* ``if``: OS IP interface support
* ``installdirs``: Installation directory relocation services
* ``memchecker``: Run-time memory checking
* ``memcpy``: Memory copy support
* ``memory``: Memory management hooks
* ``mpool``: Memory pooling
* ``patcher``: Symbol patcher hooks
* ``pmix``: Process management interface (exascale)
* ``rcache``: Memory registration cache
* ``reachable``: Network reachability determination
* ``shmem``: Shared memory support (NOT related to OpenSHMEM)
* ``smsc``: Shared memory single-copy support
* ``threads``: OS and userspace thread support
* ``timer``: High-resolution timers
Framework notes
---------------
Each framework typically has one or more components that are used at
run-time. For example, the ``btl`` framework is used by the MPI layer
to send bytes across different types underlying networks. The ``tcp``
``btl``, for example, sends messages across TCP-based networks; the
``ucx`` ``pml`` sends messages across InfiniBand-based networks.
MCA parameter notes
-------------------
Each component typically has some tunable parameters that can be
changed at run-time. Use the :ref:`ompi_info(1) <man1-ompi_info>`
command to check a component to see what its tunable parameters are.
For example:
.. code-block:: sh
shell$ ompi_info --param btl tcp
shows some of the parameters (and default values) for the ``tcp`` ``btl``
component (use ``--all`` or ``--level 9`` to show *all* the parameters).
Note that ``ompi_info`` (without ``--all`` or a specified level) only
shows a small number a component's MCA parameters by default. Each
MCA parameter has a "level" value from 1 to 9, corresponding to the
MPI-3 MPI_T tool interface levels. :ref:`See the LEVELS section in
the ompi_info(1) man page <man1-ompi_info-levels>` for an explanation
of the levels and how they correspond to Open MPI's code.
Here's rules of thumb to keep in mind when using Open MPI's levels:
* Levels 1-3:
* These levels should contain only a few MCA parameters.
* Generally, only put MCA parameters in these levels that matter to
users who just need to *run* Open MPI applications (and don't
know/care anything about MPI). Examples (these are not
comprehensive):
* Selection of which network interfaces to use.
* Selection of which MCA components to use.
* Selective disabling of warning messages (e.g., show warning
message XYZ unless a specific MCA parameter is set, which
disables showing that warning message).
* Enabling additional stderr logging verbosity. This allows a
user to run with this logging enabled, and then use that output
to get technical assistance.
* Levels 4-6:
* These levels should contain any other MCA parameters that are
useful to expose to end users.
* There is an expectation that "power users" will utilize these MCA
parameters |mdash| e.g., those who are trying to tune the system
and extract more performance.
* Here's some examples of MCA parameters suitable for these levels
(these are not comprehensive):
* When you could have hard-coded a constant size of a resource
(e.g., a resource pool size or buffer length), make it an MCA
parameter instead.
* When there are multiple different algorithms available for a
particular operation, code them all up and provide an MCA
parameter to let the user select between them.
* Levels 7-9:
* Put any other MCA parameters here.
* It's ok for these MCA parameters to be esoteric and only relevant
to deep magic / the internals of Open MPI.
* There is little expectation of users using these MCA parameters.
See :ref:`this section <label-running-setting-mca-param-values>` for
details on how to set MCA parameters at run time.
|
