1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
|
Shared Memory
=============
The sm BTL
----------
The ``sm`` BTL is a low-latency, high-bandwidth mechanism for
transferring data between two processes via shared memory. This BTL
can only be used between processes executing on the same node.
.. note:: Between Open MPI version 1.8.0 and 4.1.x, the shared memory
BTL was named ``vader``. As of Open MPI version 5.0.0, the
BTL has been renamed ``sm``.
.. warning:: In Open MPI version 5.0.x, the name ``vader`` is simply
an alias for the ``sm`` BTL. Similarly, all
``vader_``-prefixed MCA parameters are automatically
aliased to their corresponding ``sm_``-prefixed MCA
parameter.
This alias mechanism is a legacy transition device, and
will likely disappear in a future release of Open MPI.
/////////////////////////////////////////////////////////////////////////
Specifying the Use of sm for MPI Messages
-----------------------------------------
Typically, it is unnecessary to do so; OMPI will use the best BTL available
for each communication.
Nevertheless, you may use the MCA parameter ``btl``. You should also
specify the ``self`` BTL for communications between a process and
itself. Furthermore, if not all processes in your job will run on the
same, single node, then you also need to specify a BTL for internode
communications. For example:
.. code-block:: sh
shell$ mpirun --mca btl self,sm,tcp -n 16 ./a.out
/////////////////////////////////////////////////////////////////////////
Tuning Parameters to Improve Performance
----------------------------------------
Mostly, the default values of the MCA parameters have already
been chosen to give good performance. To improve performance further
is a little bit of an art. Sometimes, it's a matter of trading off
performance for memory.
* ``btl_sm_eager_limit``: If message data plus header information fits
within this limit, the message is sent "eagerly" |mdash| that is, a
sender attempts to write its entire message to shared buffers
without waiting for a receiver to be ready. Above this size, a
sender will only write the first part of a message, then wait for
the receiver to acknowledge its readiness before continuing. Eager
sends *can* improve performance by decoupling senders from
receivers.
* ``btl_sm_max_send_size``: Large messages are sent in fragments of
this size. Larger segments *can* lead to greater efficiencies,
though they could perhaps also inhibit pipelining between sender and
receiver.
* ``btl_sm_free_list_num``: This is the initial number of fragments on
each (eager and max) free list. The free lists can grow in response
to resource congestion, but you can increase this parameter to
pre-reserve space for more fragments.
* ``btl_sm_backing_directory``: Directory to place backing files for
shared memory communication. This directory should be on a local
filesystem such as ``/tmp`` or ``/dev/shm`` (default: (linux) ``/dev/shm``,
(others) session directory)
/////////////////////////////////////////////////////////////////////////
Shared Memory Mechanisms
------------------------
The ``sm`` BTL supports two modes of shared memory communication:
#. **Two-copy:** Otherwise known as "copy-in / copy-out", this mode is
where the sender copies data into shared memory and the receiver
copies the data out.
This mechanism is always available.
#. **Single copy:** In this mode, the sender or receiver makes a
single copy of the message data from the source buffer in one
process to the destination buffer in another process. Open MPI
supports three flavors of shared memory single-copy transfers:
* `Linux KNEM <https://knem.gitlabpages.inria.fr/>`_. This is a
standalone Linux kernel module, made specifically for HPC and MPI
libraries to enable high-performance single-copy message
transfers.
Open MPI must be able to find the KNEM header files in order to
build support for KNEM.
* `Linux XPMEM <https://github.com/hjelmn/xpmem>`_. Similar to
KNEM, this is a standalone Linux kernel module, made specifically
for HPC and MPI libraries to enable high-performance single-copy
message transfers. It is derived from the Cray XPMEM system.
Open MPI must be able to find the XPMEM header files in order to
build support for XPMEM.
* Linux Cross-Memory Attach (CMA). This mechanism is built-in to
modern versions of the Linux kernel. Although more performance
than the two-copy shared memory transfer mechanism, CMA is the
lowest performance of the single-copy mechanisms. However, CMA
is likely the most widely available because it is enabled by
default in several modern Linux distributions.
Open MPI must be built on a Linux system with a recent enough
Glibc and kernel version in order to build support for Linux CMA.
Which mechanism is used at run time depends both on how Open MPI was
built and how your system is configured. You can check to see which
single-copy mechanisms Open MPI was built with via two mechanisms:
#. At the end of running ``configure``, Open MPI emits a list of
transports for which it found relevant header files and libraries
such that it will be able to build support for them. You might see
lines like this, for example:
.. code-block:: text
Shared memory/copy in+copy out: yes
Shared memory/Linux CMA: yes
Shared memory/Linux KNEM: no
Shared memory/XPMEM: no
The above output indicates that Open MPI will be built with 2-copy
(as mentioned above, 2-copy is *always* available) and with Linux
CMA support. KNEM and XPMEM support will *not* be built.
#. After Open MPI is installed, the ``ompi_info`` command can show
which ``smsc`` (shared memory single copy) components are
available:
.. code-block:: text
shell$ ompi_info | grep smsc
MCA smsc: cma (MCA v2.1.0, API v1.0.0, Component v5.1.0)
This Open MPI installation only supports the Linux CMA single-copy
mechanism.
.. note:: As implied by the SMSC component names, none of them are
supported on macOS. macOS users will use the two-copy mechanism.
/////////////////////////////////////////////////////////////////////////
Shared Memory Mapping on the Filesystem
---------------------------------------
The default location of the file is in the ``/dev/shm`` directory. If ``/dev/shm``
does not exist on the system, the default location will be the OMPI session
directory. The path is typically something like:
``/dev/shm/sm_segment.nodename.user_id.job_id.my_node_rank``.
For example, the full path could be: ``/dev/shm/sm_segment.x.1000.23c70000.0``.
You can use the MCA parameter ``btl_sm_backing_directory`` to place the
directory in a non-default location.
.. note:: The session directory can be customized via
PRRTE using ``--prtemca prte_tmpdir_base /path/to/somewhere``.
.. note:: Even when using single-copy methods like CMA, a shared memory file is still
created for managing connection metadata.
|
