1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
|
<!DOCTYPE html>
<html class="writer-html5" lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>4.12. Installation location — Open MPI 5.0.8 documentation</title>
<link rel="stylesheet" type="text/css" href="../_static/pygments.css" />
<link rel="stylesheet" type="text/css" href="../_static/css/theme.css" />
<!--[if lt IE 9]>
<script src="../_static/js/html5shiv.min.js"></script>
<![endif]-->
<script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
<script src="../_static/jquery.js"></script>
<script src="../_static/underscore.js"></script>
<script src="../_static/_sphinx_javascript_frameworks_compat.js"></script>
<script src="../_static/doctools.js"></script>
<script src="../_static/sphinx_highlight.js"></script>
<script src="../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="4.13. Installing custom components" href="custom-components.html" />
<link rel="prev" title="4.11. make targets" href="make-targets.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../index.html" class="icon icon-home">
Open MPI
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../quickstart.html">1. Quick start</a></li>
<li class="toctree-l1"><a class="reference internal" href="../getting-help.html">2. Getting help</a></li>
<li class="toctree-l1"><a class="reference internal" href="../release-notes/index.html">3. Release notes</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="index.html">4. Building and installing Open MPI</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="quickstart.html">4.1. Quick start: Installing Open MPI</a></li>
<li class="toctree-l2"><a class="reference internal" href="downloading.html">4.2. Downloading Open MPI</a></li>
<li class="toctree-l2"><a class="reference internal" href="supported-systems.html">4.3. Supported Systems</a></li>
<li class="toctree-l2"><a class="reference internal" href="definitions.html">4.4. Definitions</a></li>
<li class="toctree-l2"><a class="reference internal" href="filesystem-requirements.html">4.5. Filesystem requirements</a></li>
<li class="toctree-l2"><a class="reference internal" href="vpath-builds.html">4.6. VPATH builds</a></li>
<li class="toctree-l2"><a class="reference internal" href="compilers-and-flags.html">4.7. Specifying compilers and flags</a></li>
<li class="toctree-l2"><a class="reference internal" href="required-support-libraries.html">4.8. Required support libraries</a></li>
<li class="toctree-l2"><a class="reference internal" href="configure-cli-options/index.html">4.9. <code class="docutils literal notranslate"><span class="pre">configure</span></code> command line options</a></li>
<li class="toctree-l2"><a class="reference internal" href="configure-output-summary.html">4.10. <code class="docutils literal notranslate"><span class="pre">configure</span></code> output summary</a></li>
<li class="toctree-l2"><a class="reference internal" href="make-targets.html">4.11. <code class="docutils literal notranslate"><span class="pre">make</span></code> targets</a></li>
<li class="toctree-l2 current"><a class="current reference internal" href="#">4.12. Installation location</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#filesystem-types">4.12.1. Filesystem types</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#network-filesystem">4.12.1.1. Network filesystem</a></li>
<li class="toctree-l4"><a class="reference internal" href="#local-filesystem">4.12.1.2. Local filesystem</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#installing-over-a-prior-open-mpi-installation">4.12.2. Installing over a prior Open MPI installation</a></li>
<li class="toctree-l3"><a class="reference internal" href="#relocating-an-open-mpi-installation">4.12.3. Relocating an Open MPI installation</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#move-an-existing-open-mpi-installation-to-a-new-prefix">4.12.3.1. Move an existing Open MPI installation to a new prefix</a></li>
<li class="toctree-l4"><a class="reference internal" href="#stage-an-open-mpi-installation-in-a-temporary-location">4.12.3.2. “Stage” an Open MPI installation in a temporary location</a></li>
<li class="toctree-l4"><a class="reference internal" href="#overriding-individual-directories">4.12.3.3. Overriding individual directories</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#installing-multiple-copies-of-open-mpi">4.12.4. Installing Multiple Copies of Open MPI</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="custom-components.html">4.13. Installing custom components</a></li>
<li class="toctree-l2"><a class="reference internal" href="updating.html">4.14. Updating or upgrading an Open MPI installation</a></li>
<li class="toctree-l2"><a class="reference internal" href="packagers.html">4.15. Advice for packagers</a></li>
<li class="toctree-l2"><a class="reference internal" href="sysadmins.html">4.16. Advice for System Administrators</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../features/index.html">5. Open MPI-specific features</a></li>
<li class="toctree-l1"><a class="reference internal" href="../validate.html">6. Validating your installation</a></li>
<li class="toctree-l1"><a class="reference internal" href="../version-numbering.html">7. Version numbers and compatibility</a></li>
<li class="toctree-l1"><a class="reference internal" href="../mca.html">8. The Modular Component Architecture (MCA)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../building-apps/index.html">9. Building MPI applications</a></li>
<li class="toctree-l1"><a class="reference internal" href="../launching-apps/index.html">10. Launching MPI applications</a></li>
<li class="toctree-l1"><a class="reference internal" href="../tuning-apps/index.html">11. Run-time operation and tuning MPI applications</a></li>
<li class="toctree-l1"><a class="reference internal" href="../app-debug/index.html">12. Debugging Open MPI Parallel Applications</a></li>
<li class="toctree-l1"><a class="reference internal" href="../developers/index.html">13. Developer’s guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributing.html">14. Contributing to Open MPI</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license/index.html">15. License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../history.html">16. History of Open MPI</a></li>
<li class="toctree-l1"><a class="reference internal" href="../man-openmpi/index.html">17. Open MPI manual pages</a></li>
<li class="toctree-l1"><a class="reference internal" href="../man-openshmem/index.html">18. OpenSHMEM manual pages</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../index.html">Open MPI</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="index.html"><span class="section-number">4. </span>Building and installing Open MPI</a></li>
<li class="breadcrumb-item active"><span class="section-number">4.12. </span>Installation location</li>
<li class="wy-breadcrumbs-aside">
<a href="../_sources/installing-open-mpi/installation-location.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<style>
.wy-table-responsive table td,.wy-table-responsive table th{white-space:normal}
</style><div class="section" id="installation-location">
<span id="building-open-mpi-installation-location-label"></span><h1><span class="section-number">4.12. </span>Installation location<a class="headerlink" href="#installation-location" title="Permalink to this heading"></a></h1>
<p>A common environment to run Open MPI is in a “Beowulf”-class or
similar cluster (e.g., a bunch of 1U servers in a bunch of racks).
Simply stated, Open MPI can run on a group of servers or workstations
connected by a network. As mentioned in the
<a class="reference internal" href="../launching-apps/prerequisites.html#running-prerequisites-label"><span class="std std-ref">prerequisites section</span></a> there are
several caveats however (for example, you typically must have an
account on all the machines, you can <code class="docutils literal notranslate"><span class="pre">ssh</span></code> between the
nodes without using a password, etc.).</p>
<p>Regardless of whether Open MPI is installed on a shared / networked
filesystem or independently on each node, it is usually easiest if
Open MPI is available in the same filesystem path on every node.
For example, if you install Open MPI to <code class="docutils literal notranslate"><span class="pre">/opt/openmpi-|ompi_ver|</span></code> on
one node, ensure that it is available in <code class="docutils literal notranslate"><span class="pre">/opt/openmpi-|ompi_ver|</span></code>
on <em>all</em> nodes.</p>
<div class="admonition important">
<p class="admonition-title">Important</p>
<p>For simplicity, the Open MPI team <em>strongly</em> recommends
that you install Open MPI at the same path location on all nodes in
your cluster. This <em>greatly</em> simplifies the user experience of
running MPI jobs across multiple nodes in your cluster.</p>
<p>It is <em>possible</em> to install Open MPI in unique path locations in
the different nodes in your cluster, but it is not <em>advisable</em>.</p>
</div>
<p>This raises the question for Open MPI system administrators: where to
install the Open MPI binaries, header files, etc.? This discussion
mainly addresses this question for homogeneous clusters (i.e., where
all nodes and operating systems are the same), although elements of
this discussion apply to heterogeneous clusters as well.</p>
<div class="section" id="filesystem-types">
<h2><span class="section-number">4.12.1. </span>Filesystem types<a class="headerlink" href="#filesystem-types" title="Permalink to this heading"></a></h2>
<p>There are two common approaches.</p>
<div class="section" id="network-filesystem">
<h3><span class="section-number">4.12.1.1. </span>Network filesystem<a class="headerlink" href="#network-filesystem" title="Permalink to this heading"></a></h3>
<p>Have a common filesystem, such as NFS, between all the machines to be
used. Install Open MPI such that the installation directory is the
<em>same value</em> on each node. This will <em>greatly</em> simplify user’s shell
startup scripts (e.g., <code class="docutils literal notranslate"><span class="pre">.bashrc</span></code>, <code class="docutils literal notranslate"><span class="pre">.cshrc</span></code>, <code class="docutils literal notranslate"><span class="pre">.profile</span></code> etc.)
— the <code class="docutils literal notranslate"><span class="pre">PATH</span></code> can be set without checking which machine the
user is on. It also simplifies the system administrator’s job; when
the time comes to patch or otherwise upgrade Open MPI, only one copy
needs to be modified.</p>
<p>For example, consider a cluster of four machines: <code class="docutils literal notranslate"><span class="pre">inky</span></code>,
<code class="docutils literal notranslate"><span class="pre">blinky</span></code>, <code class="docutils literal notranslate"><span class="pre">pinky</span></code>, and <code class="docutils literal notranslate"><span class="pre">clyde</span></code>.</p>
<ul>
<li><p>Install Open MPI on <code class="docutils literal notranslate"><span class="pre">inky</span></code>’s local hard drive in the directory
<code class="docutils literal notranslate"><span class="pre">/opt/openmpi-VERSION</span></code>. The system administrator then mounts
<code class="docutils literal notranslate"><span class="pre">inky:/opt/openmpi-VERSION</span></code> on the remaining three machines, such
that <code class="docutils literal notranslate"><span class="pre">/opt/openmpi-VERSION</span></code> on all machines is effectively “the
same”. That is, the following directories all contain the Open MPI
installation:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>inky:/opt/openmpi-VERSION
blinky:/opt/openmpi-VERSION
pinky:/opt/openmpi-VERSION
clyde:/opt/openmpi-VERSION
</pre></div>
</div>
</li>
<li><p>Install Open MPI on <code class="docutils literal notranslate"><span class="pre">inky</span></code>’s local hard drive in the directory
<code class="docutils literal notranslate"><span class="pre">/usr/local/openmpi-VERSION</span></code>. The system administrator then
mounts <code class="docutils literal notranslate"><span class="pre">inky:/usr/local/openmpi-VERSION</span></code> on <em>all four</em> machines in
some other common location, such as <code class="docutils literal notranslate"><span class="pre">/opt/openmpi-VERSION</span></code> (a
symbolic link can be installed on <code class="docutils literal notranslate"><span class="pre">inky</span></code> instead of a mount point
for efficiency). This strategy is typically used for environments
where one tree is NFS exported, but another tree is typically used
for the location of actual installation. For example, the following
directories all contain the Open MPI installation:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>inky:/opt/openmpi-VERSION
blinky:/opt/openmpi-VERSION
pinky:/opt/openmpi-VERSION
clyde:/opt/openmpi-VERSION
</pre></div>
</div>
<p>Notice that there are the same four directories as the previous
example, but on <code class="docutils literal notranslate"><span class="pre">inky</span></code>, the directory is <em>actually</em> located in
<code class="docutils literal notranslate"><span class="pre">/usr/local/openmpi-VERSION</span></code>.</p>
</li>
</ul>
<p>There is a bit of a disadvantage in this approach; each of the remote
nodes have to incur NFS (or whatever filesystem is used) delays to
access the Open MPI directory tree. However, both the administration
ease and low cost (relatively speaking) of using a networked file
system usually greatly outweighs the cost. Indeed, once an MPI
application is past MPI initialization, it doesn’t use the Open MPI
binaries very much.</p>
</div>
<div class="section" id="local-filesystem">
<h3><span class="section-number">4.12.1.2. </span>Local filesystem<a class="headerlink" href="#local-filesystem" title="Permalink to this heading"></a></h3>
<p>If you are concerned with networked filesystem costs of accessing the
Open MPI binaries, you can install Open MPI on the local hard drive of
each node in your system. Again, it is <em>highly</em> advisable to install
Open MPI in the <em>same</em> directory on each node so that each user’s
<code class="docutils literal notranslate"><span class="pre">PATH</span></code> can be set to the same value, regardless of the node that a
user has logged on to.</p>
<p>This approach will save some network latency of accessing the Open MPI
binaries, but is typically only used where users are very concerned
about squeezing every single cycle out of their machines, or are
running at extreme scale where a networked filesystem may get
overwhelmed by filesystem requests for Open MPI binaries when running
very large parallel jobs.</p>
</div>
</div>
<div class="section" id="installing-over-a-prior-open-mpi-installation">
<span id="building-open-mpi-install-overwrite-label"></span><h2><span class="section-number">4.12.2. </span>Installing over a prior Open MPI installation<a class="headerlink" href="#installing-over-a-prior-open-mpi-installation" title="Permalink to this heading"></a></h2>
<div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>The Open MPI team does not recommend installing a new
version of Open MPI over an existing / older installation of Open
MPI.</p>
</div>
<p>In its default configuration, an Open MPI installation consists of
several shared libraries, header files, executables, and plugins
(dynamic shared objects — DSOs). These installation files act
together as a single entity. The specific filenames and
contents of these files are subject to change between different
versions of Open MPI.</p>
<div class="admonition important">
<p class="admonition-title">Important</p>
<p>Installing one version of Open MPI does <em>not</em> uninstall
another version.</p>
</div>
<p>If you install a new version of Open MPI over an older version, this
may not overwrite all the files from the older version. Hence, you
may end up with an incompatible muddle of files from two different
installations — which can cause problems.</p>
<p>See <a class="reference internal" href="updating.html#building-open-mpi-updating-label"><span class="std std-ref">updating Open MPI</span></a> for more
information about updating or upgrading an installation of Open MPI.</p>
</div>
<div class="section" id="relocating-an-open-mpi-installation">
<h2><span class="section-number">4.12.3. </span>Relocating an Open MPI installation<a class="headerlink" href="#relocating-an-open-mpi-installation" title="Permalink to this heading"></a></h2>
<p>It can be desirable to initially install Open MPI to one location
(e.g., <code class="docutils literal notranslate"><span class="pre">/path/to/openmpi</span></code>) and then later move it to another
location (e.g., <code class="docutils literal notranslate"><span class="pre">/opt/myproduct/bundled-openmpi-a.b.c</span></code>).</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Open MPI hard-codes some directory paths in its executables
based on installation paths specified by the <code class="docutils literal notranslate"><span class="pre">configure</span></code>
script. For example, if you configure with an installation
prefix of <code class="docutils literal notranslate"><span class="pre">/opt/openmpi/</span></code>, Open MPI encodes in its
executables that it should be able to find its help files in
<code class="docutils literal notranslate"><span class="pre">/opt/openmpi/share/openmpi</span></code>.</p>
</div>
<p>The “installdirs” functionality in Open MPI lets you change any of
these hard-coded directory paths at run time (<em>assuming</em> that you have
already adjusted your <code class="docutils literal notranslate"><span class="pre">PATH</span></code> and/or <code class="docutils literal notranslate"><span class="pre">LD_LIBRARY_PATH</span></code> environment
variables to the new location where Open MPI now resides).</p>
<p>There are three methods.</p>
<div class="section" id="move-an-existing-open-mpi-installation-to-a-new-prefix">
<span id="install-location-opal-prefix"></span><h3><span class="section-number">4.12.3.1. </span>Move an existing Open MPI installation to a new prefix<a class="headerlink" href="#move-an-existing-open-mpi-installation-to-a-new-prefix" title="Permalink to this heading"></a></h3>
<p>Set the <code class="docutils literal notranslate"><span class="pre">OPAL_PREFIX</span></code> environment variable before launching Open
MPI. For example, if Open MPI had initially been installed to
<code class="docutils literal notranslate"><span class="pre">/opt/openmpi</span></code> and the entire <code class="docutils literal notranslate"><span class="pre">openmpi</span></code> tree was later moved to
<code class="docutils literal notranslate"><span class="pre">/home/openmpi</span></code>, setting <code class="docutils literal notranslate"><span class="pre">OPAL_PREFIX</span></code> to <code class="docutils literal notranslate"><span class="pre">/home/openmpi</span></code> will
enable Open MPI to function properly.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>The <code class="docutils literal notranslate"><span class="pre">OPAL_PREFIX</span></code> mechanism relies on all installation
directories being specified as relative to the <code class="docutils literal notranslate"><span class="pre">prefix</span></code>
directory specified during <code class="docutils literal notranslate"><span class="pre">configure</span></code>.</p>
<p>For example, if Open MPI is configured the following way:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>$ ./configure --prefix=/opt/openmpi --libdir=/usr/lib ...
</pre></div>
</div>
<p>Then setting <code class="docutils literal notranslate"><span class="pre">OPAL_PREFIX</span></code> will not affect the run-time
implications of <code class="docutils literal notranslate"><span class="pre">libdir</span></code>, since <code class="docutils literal notranslate"><span class="pre">/usr/lib</span></code> is not
specified as relative to <code class="docutils literal notranslate"><span class="pre">/opt/openmpi</span></code>.</p>
<p>Instead of specifying absolute directories, you can make
them relative to other <code class="docutils literal notranslate"><span class="pre">configure</span></code>-recognized directories.
For example:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>$ ./configure --prefix=/opt/openmpi --libdir='${exec_prefix}/x86_64/lib' ...
</pre></div>
</div>
<p>Note the additional shell quoting that is likely necessary
to prevent shell variable expansion, and the additional
<code class="docutils literal notranslate"><span class="pre">${}</span></code> around <code class="docutils literal notranslate"><span class="pre">exec_prefix</span></code> that is necessary for Open MPI
to recognize that it is a special name that needs to be
expanded.</p>
<p>The directory names recognized by Open MPI are listed in the
<a class="reference internal" href="#install-location-overriding-individual-directories"><span class="std std-ref">Overriding individual directories</span></a>
section (below), without the <code class="docutils literal notranslate"><span class="pre">OPAL_</span></code> prefix, and in lower
case. For example, the <code class="docutils literal notranslate"><span class="pre">OPAL_SYSCONFDIR</span></code> environment
variable corresponds to <code class="docutils literal notranslate"><span class="pre">${sysconfdir}</span></code>.</p>
</div>
</div>
<div class="section" id="stage-an-open-mpi-installation-in-a-temporary-location">
<h3><span class="section-number">4.12.3.2. </span>“Stage” an Open MPI installation in a temporary location<a class="headerlink" href="#stage-an-open-mpi-installation-in-a-temporary-location" title="Permalink to this heading"></a></h3>
<p>When <em>creating</em> self-contained installation packages, systems such as
RPM install Open MPI into temporary locations. The package system
then bundles up everything under the temporary location into a package
that can be installed into its real location later. For example, when
<em>creating</em> an RPM that will be installed to <code class="docutils literal notranslate"><span class="pre">/opt/openmpi</span></code>, the RPM
system will transparently prepend a “destination directory” (or
“destdir”) to the installation directory. As such, Open MPI will
think that it is installed in <code class="docutils literal notranslate"><span class="pre">/opt/openmpi</span></code>, but it is actually
temporarily installed in (for example)
<code class="docutils literal notranslate"><span class="pre">/var/rpm/build.1234/opt/openmpi</span></code>. If it is necessary to <em>use</em> Open
MPI while it is installed in this staging area, the <code class="docutils literal notranslate"><span class="pre">OPAL_DESTDIR</span></code>
environment variable can be used; setting <code class="docutils literal notranslate"><span class="pre">OPAL_DESTDIR</span></code> to
<code class="docutils literal notranslate"><span class="pre">/var/rpm/build.1234</span></code> will automatically prefix every directory such
that Open MPI can function properly.</p>
</div>
<div class="section" id="overriding-individual-directories">
<span id="install-location-overriding-individual-directories"></span><h3><span class="section-number">4.12.3.3. </span>Overriding individual directories<a class="headerlink" href="#overriding-individual-directories" title="Permalink to this heading"></a></h3>
<p>Open MPI uses the GNU-specified directories (per Autoconf/Automake),
and can be overridden by setting environment variables directly
related to their common names. The list of environment variables that
can be used is:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">OPAL_PREFIX</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">OPAL_EXEC_PREFIX</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">OPAL_BINDIR</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">OPAL_SBINDIR</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">OPAL_LIBEXECDIR</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">OPAL_DATAROOTDIR</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">OPAL_DATADIR</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">OPAL_SYSCONFDIR</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">OPAL_SHAREDSTATEDIR</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">OPAL_LOCALSTATEDIR</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">OPAL_LIBDIR</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">OPAL_INCLUDEDIR</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">OPAL_INFODIR</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">OPAL_MANDIR</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">OPAL_PKGDATADIR</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">OPAL_PKGLIBDIR</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">OPAL_PKGINCLUDEDIR</span></code></p></li>
</ul>
<p>Note that not all of the directories listed above are used by Open
MPI; they are listed here in entirety for completeness.</p>
<p>Also note that several directories listed above are defined in terms
of other directories. For example, the <code class="docutils literal notranslate"><span class="pre">$bindir</span></code> is defined by
default as <code class="docutils literal notranslate"><span class="pre">$prefix/bin</span></code>. Hence, overriding the <code class="docutils literal notranslate"><span class="pre">$prefix</span></code> (via
<code class="docutils literal notranslate"><span class="pre">OPAL_PREFIX</span></code>) will automatically change the first part of the
<code class="docutils literal notranslate"><span class="pre">$bindir</span></code> (which is how method 1 described above works).
Alternatively, <code class="docutils literal notranslate"><span class="pre">OPAL_BINDIR</span></code> can be set to an absolute value that
ignores <code class="docutils literal notranslate"><span class="pre">$prefix</span></code> altogether.</p>
</div>
</div>
<div class="section" id="installing-multiple-copies-of-open-mpi">
<span id="building-open-mpi-installation-location-multiple-copies-label"></span><h2><span class="section-number">4.12.4. </span>Installing Multiple Copies of Open MPI<a class="headerlink" href="#installing-multiple-copies-of-open-mpi" title="Permalink to this heading"></a></h2>
<p>Open MPI can handle a variety of different run-time environments
(e.g., ssh, Slurm, PBS, etc.) and a variety of different
interconnection networks (e.g., ethernet, InfiniBand, etc.)
in a single installation. Specifically: because Open MPI is
fundamentally powered by a component architecture, plug-ins for all
these different run-time systems and interconnect networks can be
installed in a single installation tree. The relevant plug-ins will
only be used in the environments where they make sense.</p>
<p>Hence, there is no need to have one MPI installation for InfiniBand, one
MPI installation for ethernet, one MPI installation for PBS, one MPI
installation for <code class="docutils literal notranslate"><span class="pre">ssh</span></code>, etc. Open MPI can handle all of these in a
single installation.</p>
<p>However, there are some issues that Open MPI cannot solve. Binary
compatibility between different compilers is such an issue and may require
installation of multiple versions of Open MPI.</p>
<p>Let’s examine this on a per-language basis (be sure see the big caveat at
the end):</p>
<ul>
<li><p><em>C:</em> Most C compilers are fairly compatible, such that if you compile
Open MPI with one C library and link it to an application that was
compiled with a different C compiler, everything should “just work.”
As such, a single installation of Open MPI should work for most C MPI
applications.</p></li>
<li><p><em>C++:</em> The same is not necessarily true for C++. While Open MPI does not
currently contain any C++ code (the MPI C++ bindings were removed in a prior
release), and C++ compilers <em>should</em> produce ABI-equivalent code for C
symbols, obscure problem can sometimes arise when mixing compilers from
different suites. For example, if you compile Open MPI with the XYZ C/C++
compiler, you may need to have the XYC C++ run-time libraries
installed everywhere you want to run.</p></li>
<li><p><em>Fortran:</em> There are multiple issues with Fortran.</p>
<blockquote>
<div><ol class="arabic">
<li><p>Fortran compilers do something called “symbol mangling,” meaning that the
back-end symbols may have slightly different names than their corresponding
global variables, subroutines, and functions. There are 4 common name
mangling schemes in use by Fortran compilers. On many systems (e.g.,
Linux), Open MPI will automatically support all 4 schemes. As such, a
single Open MPI installation <em>should</em> just work with multiple different
Fortran compilers. However, on some systems, this is not possible (e.g.,
OS X), and Open MPI will only support the name mangling scheme of the
Fortran compiler that was identified during <code class="docutils literal notranslate"><span class="pre">configure</span></code>.</p></li>
<li><p>That being said, there are two notable exceptions that do <em>not</em> work
across Fortran compilers that are “different enough”:</p>
<blockquote>
<div><ol class="arabic simple">
<li><dl class="simple">
<dt>The C constants <code class="docutils literal notranslate"><span class="pre">MPI_F_STATUS_IGNORE</span></code> and <code class="docutils literal notranslate"><span class="pre">MPI_F_STATUSES_IGNORE</span></code></dt><dd><p>will only compare properly to Fortran applications that were
created with Fortran compilers that that use the same
name-mangling scheme as the Fortran compiler with which Open MPI was
configured.</p>
</dd>
</dl>
</li>
<li><dl class="simple">
<dt>Fortran compilers may have different values for the logical</dt><dd><p><code class="docutils literal notranslate"><span class="pre">.TRUE.</span></code> constant. As such, any MPI function that uses the
Fortran <code class="docutils literal notranslate"><span class="pre">LOGICAL</span></code> type may only get <code class="docutils literal notranslate"><span class="pre">.TRUE.</span></code> values back that
correspond to the the <code class="docutils literal notranslate"><span class="pre">.TRUE.</span></code> value of the Fortran compiler with which
Open MPI was configured.</p>
</dd>
</dl>
</li>
</ol>
</div></blockquote>
</li>
<li><p>Similar to C++, linking object files that Fortran language features such
as modules and/or polymorphism from different
Fortran compilers is not likely to work. The <code class="docutils literal notranslate"><span class="pre">mpi</span></code> and <code class="docutils literal notranslate"><span class="pre">mpi_f08</span></code> modules that
Open MPI creates will likely only work with the Fortran compiler
that was identified during <code class="docutils literal notranslate"><span class="pre">configure</span></code> (and used to build Open MPI).</p></li>
</ol>
</div></blockquote>
</li>
</ul>
<p>The big caveat to all of this is that Open MPI will only work with
different compilers <em>if all the datatype sizes are the same.</em> For
example, even though Open MPI supports all 4 name mangling schemes,
the size of the Fortran <code class="docutils literal notranslate"><span class="pre">LOGICAL</span></code> type may be 1 byte in some compilers
and 4 bytes in others. This will likely cause Open MPI to perform
unpredictably.</p>
<p>The bottom line is that Open MPI can support all manner of run-time
systems and interconnects in a single installation, but supporting
multiple compilers “sort of” works (i.e., is subject to trial and
error) in some cases, and definitely does not work in other cases.
There’s unfortunately little that we can do about this — it’s a
compiler compatibility issue, and one that compiler authors have
little incentive to resolve.</p>
</div>
</div>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="make-targets.html" class="btn btn-neutral float-left" title="4.11. make targets" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="custom-components.html" class="btn btn-neutral float-right" title="4.13. Installing custom components" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>
<hr/>
<div role="contentinfo">
<p>© Copyright 2003-2025, The Open MPI Community.
<span class="lastupdated">Last updated on 2025-05-30 16:41:43 UTC.
</span></p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
|
