1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
|
<!DOCTYPE html>
<html class="writer-html5" lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>11.2.1. OpenFabrics Interfaces (OFI) / Libfabric support — Open MPI 5.0.8 documentation</title>
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css" />
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css" />
<!--[if lt IE 9]>
<script src="../../_static/js/html5shiv.min.js"></script>
<![endif]-->
<script data-url_root="../../" id="documentation_options" src="../../_static/documentation_options.js"></script>
<script src="../../_static/jquery.js"></script>
<script src="../../_static/underscore.js"></script>
<script src="../../_static/_sphinx_javascript_frameworks_compat.js"></script>
<script src="../../_static/doctools.js"></script>
<script src="../../_static/sphinx_highlight.js"></script>
<script src="../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../genindex.html" />
<link rel="search" title="Search" href="../../search.html" />
<link rel="next" title="11.2.2. TCP" href="tcp.html" />
<link rel="prev" title="11.2. Networking support" href="index.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../index.html" class="icon icon-home">
Open MPI
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../quickstart.html">1. Quick start</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../getting-help.html">2. Getting help</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../release-notes/index.html">3. Release notes</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../installing-open-mpi/index.html">4. Building and installing Open MPI</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../features/index.html">5. Open MPI-specific features</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../validate.html">6. Validating your installation</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../version-numbering.html">7. Version numbers and compatibility</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../mca.html">8. The Modular Component Architecture (MCA)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../building-apps/index.html">9. Building MPI applications</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../launching-apps/index.html">10. Launching MPI applications</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../index.html">11. Run-time operation and tuning MPI applications</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../environment-var.html">11.1. Environment variables set for MPI applications</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="index.html">11.2. Networking support</a><ul class="current">
<li class="toctree-l3 current"><a class="current reference internal" href="#">11.2.1. OpenFabrics Interfaces (OFI) / Libfabric support</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#what-is-ofi-libfabric">11.2.1.1. What is OFI / Libfabric?</a></li>
<li class="toctree-l4"><a class="reference internal" href="#what-are-the-libfabric-ofi-components-in-open-mpi">11.2.1.2. What are the Libfabric (OFI) components in Open MPI?</a></li>
<li class="toctree-l4"><a class="reference internal" href="#omni-path-how-can-the-multi-rail-settings-be-adjusted-if-multiple-hfi-host-fabric-interface-cards-are-installed-on-the-system">11.2.1.3. Omni-Path: How can the multi-rail settings be adjusted if multiple HFI (Host Fabric Interface) cards are installed on the system?</a></li>
<li class="toctree-l4"><a class="reference internal" href="#omni-path-what-is-multi-hfi-support-in-psm2-and-how-does-it-differ-from-multi-rail">11.2.1.4. Omni-Path: What is Multi-HFI support in PSM2 and how does it differ from multi-rail?</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="tcp.html">11.2.2. TCP</a></li>
<li class="toctree-l3"><a class="reference internal" href="shared-memory.html">11.2.3. Shared Memory</a></li>
<li class="toctree-l3"><a class="reference internal" href="ib-and-roce.html">11.2.4. InifiniBand / RoCE support</a></li>
<li class="toctree-l3"><a class="reference internal" href="iwarp.html">11.2.5. iWARP Support</a></li>
<li class="toctree-l3"><a class="reference internal" href="cuda.html">11.2.6. CUDA</a></li>
<li class="toctree-l3"><a class="reference internal" href="rocm.html">11.2.7. ROCm</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../multithreaded.html">11.3. Running multi-threaded MPI applications</a></li>
<li class="toctree-l2"><a class="reference internal" href="../dynamic-loading.html">11.4. Dynamically loading <code class="docutils literal notranslate"><span class="pre">libmpi</span></code> at runtime</a></li>
<li class="toctree-l2"><a class="reference internal" href="../fork-system-popen.html">11.5. Calling fork(), system(), or popen() in MPI processes</a></li>
<li class="toctree-l2"><a class="reference internal" href="../fault-tolerance/index.html">11.6. Fault tolerance</a></li>
<li class="toctree-l2"><a class="reference internal" href="../large-clusters/index.html">11.7. Large Clusters</a></li>
<li class="toctree-l2"><a class="reference internal" href="../affinity.html">11.8. Processor and memory affinity</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mpi-io/index.html">11.9. MPI-IO tuning options</a></li>
<li class="toctree-l2"><a class="reference internal" href="../coll-tuned.html">11.10. Tuning Collectives</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking.html">11.11. Benchmarking Open MPI applications</a></li>
<li class="toctree-l2"><a class="reference internal" href="../heterogeneity.html">11.12. Building heterogeneous MPI applications</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../app-debug/index.html">12. Debugging Open MPI Parallel Applications</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../developers/index.html">13. Developer’s guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributing.html">14. Contributing to Open MPI</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../license/index.html">15. License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../history.html">16. History of Open MPI</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../man-openmpi/index.html">17. Open MPI manual pages</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../man-openshmem/index.html">18. OpenSHMEM manual pages</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../index.html">Open MPI</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../index.html"><span class="section-number">11. </span>Run-time operation and tuning MPI applications</a></li>
<li class="breadcrumb-item"><a href="index.html"><span class="section-number">11.2. </span>Networking support</a></li>
<li class="breadcrumb-item active"><span class="section-number">11.2.1. </span>OpenFabrics Interfaces (OFI) / Libfabric support</li>
<li class="wy-breadcrumbs-aside">
<a href="../../_sources/tuning-apps/networking/ofi.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<style>
.wy-table-responsive table td,.wy-table-responsive table th{white-space:normal}
</style><div class="section" id="openfabrics-interfaces-ofi-libfabric-support">
<h1><span class="section-number">11.2.1. </span>OpenFabrics Interfaces (OFI) / Libfabric support<a class="headerlink" href="#openfabrics-interfaces-ofi-libfabric-support" title="Permalink to this heading"></a></h1>
<div class="admonition error">
<p class="admonition-title">Error</p>
<p>TODO This section needs to be converted from FAQ Q&A style
to regular documentation style.</p>
</div>
<div class="section" id="what-is-ofi-libfabric">
<h2><span class="section-number">11.2.1.1. </span>What is OFI / Libfabric?<a class="headerlink" href="#what-is-ofi-libfabric" title="Permalink to this heading"></a></h2>
<p>“OFI” stands for the <a class="reference external" href="https://libfabric.org/">OpenFabrics Interfaces</a>, which are implemented in the <code class="docutils literal notranslate"><span class="pre">libfabric</span></code>
library. These two terms are typically used interchangeably.</p>
<p>Open MPI supports many different underlying networks via Libfabric,
including (but not limited to):</p>
<ul class="simple">
<li><p>AWS EFA</p></li>
<li><p>Cisco usNIC</p></li>
<li><p>Cray uGNI</p></li>
<li><p>Cornelis Networks Omni-Path</p></li>
<li><p>HPE Slingshot 11</p></li>
</ul>
<p>In general, the OFI-based components in Open MPI will auto-select
themselves as appropriate at run time.</p>
<p>That being said, additional questions are available in this FAQ
section to provide more information about specific OFI-based network
types and support.</p>
</div>
<hr class="docutils" />
<div class="section" id="what-are-the-libfabric-ofi-components-in-open-mpi">
<h2><span class="section-number">11.2.1.2. </span>What are the Libfabric (OFI) components in Open MPI?<a class="headerlink" href="#what-are-the-libfabric-ofi-components-in-open-mpi" title="Permalink to this heading"></a></h2>
<p>Open MPI has three main components for Libfabric (a.k.a., OFI)
communications:</p>
<ol class="arabic simple">
<li><p><code class="docutils literal notranslate"><span class="pre">ofi</span></code> MTL: Available since Open MPI v1.10, this component is used
with the <code class="docutils literal notranslate"><span class="pre">cm</span></code> PML and is used for two-sided MPI communication
(e.g., <code class="docutils literal notranslate"><span class="pre">MPI_SEND</span></code> and <code class="docutils literal notranslate"><span class="pre">MPI_RECV</span></code>).</p></li>
</ol>
<blockquote>
<div><p>The <code class="docutils literal notranslate"><span class="pre">ofi</span></code> MTL requires that the Libfabric provider support
reliable datagrams with ordered tagged messaging (specifically:
<code class="docutils literal notranslate"><span class="pre">FI_EP_RDM</span></code> endpoints, <code class="docutils literal notranslate"><span class="pre">FI_TAGGED</span></code> capabilities, and
<code class="docutils literal notranslate"><span class="pre">FI_ORDER_SAS</span></code> ordering).</p>
</div></blockquote>
<ol class="arabic simple">
<li><p><code class="docutils literal notranslate"><span class="pre">ofi</span></code> BTL: Available since Open MPI v4.0.0, this component is primarily
intended for one-sided MPI communications (e.g., <code class="docutils literal notranslate"><span class="pre">MPI_PUT</span></code>). It
can also support BTL send/recv operations.
<code class="docutils literal notranslate"><span class="pre">ofi</span></code> BTL requires that the Libfabric provider support reliable
datagrams, RMA and atomic operations, and remote atomic completion
notifications (specifically: <code class="docutils literal notranslate"><span class="pre">FI_EP_RDM</span></code> endpoints, <code class="docutils literal notranslate"><span class="pre">FI_RMA</span></code>
and <code class="docutils literal notranslate"><span class="pre">FI_ATOMIC</span></code> capabilities, and <code class="docutils literal notranslate"><span class="pre">FI_DELIVERY_COMPLETE</span></code> op
flags).</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">usnic</span></code> BTL: This BTL is used exclusively with Cisco usNIC-based
networks. It will auto-select itself over the other OFI-based
components when run with Cisco usNIC-based networks.</p></li>
</ol>
<p>See each Lifabric provider man page (e.g., fi_sockets(7)) to understand which
provider will work for each of the above-listed Open MPI components. Some
providers may require to be used with one of the Libfabric utility providers;
for example, the verbs provider needs to be paired with utility provider
<code class="docutils literal notranslate"><span class="pre">ofi_rxm</span></code> to provide reliable datagram endpoint support (<code class="docutils literal notranslate"><span class="pre">verbs;ofi_rxm</span></code>).</p>
<p>Both components have MCA parameters to specify the Libfabric provider(s) that
will be included/excluded in the selection process. For example:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>shell$ mpirun --mca pml cm --mca mtl ofi --mca mtl_ofi_provider_include psm2 mpi_hello
</pre></div>
</div>
<p>In addition, each component has specific parameters for each one; see
<code class="docutils literal notranslate"><span class="pre">ompi_info</span> <span class="pre">--param</span> <span class="pre"><framework></span> <span class="pre"><component></span> <span class="pre">-level</span> <span class="pre">9</span></code> for a full
list. For example:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>shell$ ompi_info --param mtl ofi --level 9
</pre></div>
</div>
<div class="admonition important">
<p class="admonition-title">Important</p>
<p>When using the HPE CXI provider and <code class="docutils literal notranslate"><span class="pre">mpirun</span></code> as the job launcher,
it is recommended that the PRTE <code class="docutils literal notranslate"><span class="pre">ras_base_launch_orted_on_hn</span></code> MCA parameter be set to 1.
This can be done by adding <code class="docutils literal notranslate"><span class="pre">--prtemca</span> <span class="pre">ras_base_launch_orted_on_hn</span> <span class="pre">1</span></code> to the job launch
command line. This ensures that MPI processes launched on the first node of
an allocation are able to use the CXI provider.</p>
</div>
<p>For more information refer to the <a class="reference external" href="https://libfabric.org/">Libfabric web site</a>.</p>
</div>
<hr class="docutils" />
<div class="section" id="omni-path-how-can-the-multi-rail-settings-be-adjusted-if-multiple-hfi-host-fabric-interface-cards-are-installed-on-the-system">
<h2><span class="section-number">11.2.1.3. </span>Omni-Path: How can the multi-rail settings be adjusted if multiple HFI (Host Fabric Interface) cards are installed on the system?<a class="headerlink" href="#omni-path-how-can-the-multi-rail-settings-be-adjusted-if-multiple-hfi-host-fabric-interface-cards-are-installed-on-the-system" title="Permalink to this heading"></a></h2>
<p>Multi-Rail feature allows a process to use multiple HFIs to transfer a message
to improve message bandwidth. The PSM2 library handles the support for multi-rail
which is off by default. The multi-rail settings can be modified using the
following environment variables:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">PSM2_MULTIRAIL=[0,1,2]</span> <span class="pre">]</span></code>: 0=Disabled, 1=Enable across all HFIs in the
system, 2=Enable multi-rail within a NUMA node.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">PSM2_MULTIRAIL_MAP=unit:port,unit:port...</span></code></p></li>
</ul>
<p>The variables above may be included in the <code class="docutils literal notranslate"><span class="pre">mpirun</span></code> command line or in
the environment. For example:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>shell$ mpirun -mca mtl [psm2|ofi] -x PSM2_MULTIRAIL=1 -n 2 -H host1,host2 ./a.out
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>When using the OFI MTL, please ensure that the PSM2 OFI
provider is used for communication with OPA devices.</p>
</div>
</div>
<hr class="docutils" />
<div class="section" id="omni-path-what-is-multi-hfi-support-in-psm2-and-how-does-it-differ-from-multi-rail">
<h2><span class="section-number">11.2.1.4. </span>Omni-Path: What is Multi-HFI support in PSM2 and how does it differ from multi-rail?<a class="headerlink" href="#omni-path-what-is-multi-hfi-support-in-psm2-and-how-does-it-differ-from-multi-rail" title="Permalink to this heading"></a></h2>
<p>Multi-HFI support is intended to describe the use of multiple HFIs in
a system among MPI ranks local to a node in order to load-balance the
hardware resources. It differs from the Multi-Rail feature, which is
intended to allow a single process to use all HFIs in the system. For
an MPI job with multiple processes on a single node, the default PSM2
behavior depends on the affinity settings of the MPI process. The PSM2
library defaults to using the HFI (Host Fabric Interface) that is in
the same NUMA node as that of the MPI process.</p>
<p>Users can restrict access to a single HFI using the environment variable:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">HFI_UNIT=N</span></code>: valid values of N are 0,1,2 and 3</p></li>
</ul>
<p>More details can be found on the PSM2 Programmer’s Guide and the Omni-Path
Fabric Performance Tuning Guide.</p>
<p>Please see the <a class="reference external" href="https://customercenter.cornelisnetworks.com/">Cornelis Networks Customer Center</a>
for more details.</p>
</div>
</div>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="index.html" class="btn btn-neutral float-left" title="11.2. Networking support" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="tcp.html" class="btn btn-neutral float-right" title="11.2.2. TCP" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>
<hr/>
<div role="contentinfo">
<p>© Copyright 2003-2025, The Open MPI Community.
<span class="lastupdated">Last updated on 2025-05-30 16:41:43 UTC.
</span></p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
|
