File: slurm.html

package info (click to toggle)
openmpi 5.0.8-4
  • links: PTS, VCS
  • area: main
  • in suites:
  • size: 201,684 kB
  • sloc: ansic: 613,078; makefile: 42,353; sh: 11,194; javascript: 9,244; f90: 7,052; java: 6,404; perl: 5,179; python: 1,859; lex: 740; fortran: 61; cpp: 20; tcl: 12
file content (258 lines) | stat: -rw-r--r-- 16,801 bytes parent folder | download | duplicates (3) 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
 
<!DOCTYPE html>
<html class="writer-html5" lang="en">
<head>
  <meta charset="utf-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  <title>10.7. Launching with Slurm &mdash; Open MPI 5.0.8 documentation</title>
      <link rel="stylesheet" type="text/css" href="../_static/pygments.css" />
      <link rel="stylesheet" type="text/css" href="../_static/css/theme.css" />

  
  <!--[if lt IE 9]>
    <script src="../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
        <script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
        <script src="../_static/jquery.js"></script>
        <script src="../_static/underscore.js"></script>
        <script src="../_static/_sphinx_javascript_frameworks_compat.js"></script>
        <script src="../_static/doctools.js"></script>
        <script src="../_static/sphinx_highlight.js"></script>
    <script src="../_static/js/theme.js"></script>
    <link rel="index" title="Index" href="../genindex.html" />
    <link rel="search" title="Search" href="../search.html" />
    <link rel="next" title="10.8. Launching with LSF" href="lsf.html" />
    <link rel="prev" title="10.6. Launching with SSH" href="ssh.html" /> 
</head>

<body class="wy-body-for-nav"> 
  <div class="wy-grid-for-nav">
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search" >

          
          
          <a href="../index.html" class="icon icon-home">
            Open MPI
          </a>
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
    <input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>
        </div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../quickstart.html">1. Quick start</a></li>
<li class="toctree-l1"><a class="reference internal" href="../getting-help.html">2. Getting help</a></li>
<li class="toctree-l1"><a class="reference internal" href="../release-notes/index.html">3. Release notes</a></li>
<li class="toctree-l1"><a class="reference internal" href="../installing-open-mpi/index.html">4. Building and installing Open MPI</a></li>
<li class="toctree-l1"><a class="reference internal" href="../features/index.html">5. Open MPI-specific features</a></li>
<li class="toctree-l1"><a class="reference internal" href="../validate.html">6. Validating your installation</a></li>
<li class="toctree-l1"><a class="reference internal" href="../version-numbering.html">7. Version numbers and compatibility</a></li>
<li class="toctree-l1"><a class="reference internal" href="../mca.html">8. The Modular Component Architecture (MCA)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../building-apps/index.html">9. Building MPI applications</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="index.html">10. Launching MPI applications</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="quickstart.html">10.1. Quick start: Launching MPI applications</a></li>
<li class="toctree-l2"><a class="reference internal" href="prerequisites.html">10.2. Prerequisites</a></li>
<li class="toctree-l2"><a class="reference internal" href="pmix-and-prrte.html">10.3. The role of PMIx and PRRTE</a></li>
<li class="toctree-l2"><a class="reference internal" href="scheduling.html">10.4. Scheduling processes across hosts</a></li>
<li class="toctree-l2"><a class="reference internal" href="localhost.html">10.5. Launching only on the local node</a></li>
<li class="toctree-l2"><a class="reference internal" href="ssh.html">10.6. Launching with SSH</a></li>
<li class="toctree-l2 current"><a class="current reference internal" href="#">10.7. Launching with Slurm</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#using-mpirun">10.7.1. Using <code class="docutils literal notranslate"><span class="pre">mpirun</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#using-slurm-s-direct-launch-functionality">10.7.2. Using Slurm’s “direct launch” functionality</a></li>
<li class="toctree-l3"><a class="reference internal" href="#slurm-20-11">10.7.3. Slurm 20.11</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="lsf.html">10.8. Launching with LSF</a></li>
<li class="toctree-l2"><a class="reference internal" href="tm.html">10.9. Launching with PBS / Torque</a></li>
<li class="toctree-l2"><a class="reference internal" href="gridengine.html">10.10. Launching with Grid Engine</a></li>
<li class="toctree-l2"><a class="reference internal" href="unusual.html">10.11. Unusual jobs</a></li>
<li class="toctree-l2"><a class="reference internal" href="troubleshooting.html">10.12. Troubleshooting</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../tuning-apps/index.html">11. Run-time operation and tuning MPI applications</a></li>
<li class="toctree-l1"><a class="reference internal" href="../app-debug/index.html">12. Debugging Open MPI Parallel Applications</a></li>
<li class="toctree-l1"><a class="reference internal" href="../developers/index.html">13. Developer’s guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributing.html">14. Contributing to Open MPI</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license/index.html">15. License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../history.html">16. History of Open MPI</a></li>
<li class="toctree-l1"><a class="reference internal" href="../man-openmpi/index.html">17. Open MPI manual pages</a></li>
<li class="toctree-l1"><a class="reference internal" href="../man-openshmem/index.html">18. OpenSHMEM manual pages</a></li>
</ul>

        </div>
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../index.html">Open MPI</a>
      </nav>

      <div class="wy-nav-content">
        <div class="rst-content">
          <div role="navigation" aria-label="Page navigation">
  <ul class="wy-breadcrumbs">
      <li><a href="../index.html" class="icon icon-home" aria-label="Home"></a></li>
          <li class="breadcrumb-item"><a href="index.html"><span class="section-number">10. </span>Launching MPI applications</a></li>
      <li class="breadcrumb-item active"><span class="section-number">10.7. </span>Launching with Slurm</li>
      <li class="wy-breadcrumbs-aside">
            <a href="../_sources/launching-apps/slurm.rst.txt" rel="nofollow"> View page source</a>
      </li>
  </ul>
  <hr/>
</div>
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
             
  <style>
.wy-table-responsive table td,.wy-table-responsive table th{white-space:normal}
</style><div class="section" id="launching-with-slurm">
<h1><span class="section-number">10.7. </span>Launching with Slurm<a class="headerlink" href="#launching-with-slurm" title="Permalink to this heading"></a></h1>
<p>Open MPI supports two modes of launching parallel MPI jobs under
Slurm:</p>
<ol class="arabic simple">
<li><p>Using Open MPI’s full-features <code class="docutils literal notranslate"><span class="pre">mpirun</span></code> launcher.</p></li>
<li><p>Using Slurm’s “direct launch” capability.</p></li>
</ol>
<p>Unless there is a strong reason to use <code class="docutils literal notranslate"><span class="pre">srun</span></code> for direct launch, the
Open MPI team recommends using <code class="docutils literal notranslate"><span class="pre">mpirun</span></code> for launching under Slurm jobs.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>In versions of Open MPI prior to 5.0.x, using <code class="docutils literal notranslate"><span class="pre">srun</span></code> for
direct launch could be faster than using <code class="docutils literal notranslate"><span class="pre">mpirun</span></code>.  <strong>This is no
longer true.</strong></p>
</div>
<div class="section" id="using-mpirun">
<h2><span class="section-number">10.7.1. </span>Using <code class="docutils literal notranslate"><span class="pre">mpirun</span></code><a class="headerlink" href="#using-mpirun" title="Permalink to this heading"></a></h2>
<p>When <code class="docutils literal notranslate"><span class="pre">mpirun</span></code> is launched in a Slurm job, <code class="docutils literal notranslate"><span class="pre">mpirun</span></code> will
automatically utilize the Slurm infrastructure for launching and
controlling the individual MPI processes.
Hence, it is unnecessary to specify the <code class="docutils literal notranslate"><span class="pre">--hostfile</span></code>,
<code class="docutils literal notranslate"><span class="pre">--host</span></code>, or <code class="docutils literal notranslate"><span class="pre">-n</span></code> options to <code class="docutils literal notranslate"><span class="pre">mpirun</span></code>.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Using <code class="docutils literal notranslate"><span class="pre">mpirun</span></code> is the recommended method for launching Open
MPI jobs in Slurm jobs.</p>
<p><code class="docutils literal notranslate"><span class="pre">mpirun</span></code>’s Slurm support should always be available, regardless
of how Open MPI or Slurm was installed.</p>
</div>
<p>For example:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span><span class="c1"># Allocate a Slurm job with 4 slots</span>
shell$<span class="w"> </span>salloc<span class="w"> </span>-n<span class="w"> </span><span class="m">4</span>
salloc:<span class="w"> </span>Granted<span class="w"> </span>job<span class="w"> </span>allocation<span class="w"> </span><span class="m">1234</span>

<span class="c1"># Now run an Open MPI job on all the slots allocated by Slurm</span>
shell$<span class="w"> </span>mpirun<span class="w"> </span>mpi-hello-world
</pre></div>
</div>
<p>This will run the 4 MPI processes on the node(s) that were allocated
by Slurm.</p>
<p>Or, if submitting a script:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>shell$<span class="w"> </span>cat<span class="w"> </span>my_script.sh
<span class="c1">#!/bin/sh</span>
mpirun<span class="w"> </span>mpi-hello-world
shell$<span class="w"> </span>sbatch<span class="w"> </span>-n<span class="w"> </span><span class="m">4</span><span class="w"> </span>my_script.sh
srun:<span class="w"> </span>jobid<span class="w"> </span><span class="m">1235</span><span class="w"> </span>submitted
shell$
</pre></div>
</div>
<p>Similar to the <code class="docutils literal notranslate"><span class="pre">salloc</span></code> case, no command line options specifying
number of MPI processes were necessary, since Open MPI will obtain
that information directly from Slurm at run time.</p>
</div>
<div class="section" id="using-slurm-s-direct-launch-functionality">
<h2><span class="section-number">10.7.2. </span>Using Slurm’s “direct launch” functionality<a class="headerlink" href="#using-slurm-s-direct-launch-functionality" title="Permalink to this heading"></a></h2>
<p>Assuming that Slurm was configured with its PMIx plugin, you can use
<code class="docutils literal notranslate"><span class="pre">srun</span></code> to “direct launch” Open MPI applications without the use of
Open MPI’s <code class="docutils literal notranslate"><span class="pre">mpirun</span></code> command.</p>
<p>First, you must ensure that Slurm was built and installed with PMIx
support.  This can determined as shown below:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>shell$<span class="w"> </span>srun<span class="w"> </span>--mpi<span class="o">=</span>list
MPI<span class="w"> </span>plugin<span class="w"> </span>types<span class="w"> </span>are...
<span class="w">     </span>none
<span class="w">     </span>pmi2
<span class="w">     </span>pmix
specific<span class="w"> </span>pmix<span class="w"> </span>plugin<span class="w"> </span>versions<span class="w"> </span>available:<span class="w"> </span>pmix_v4
</pre></div>
</div>
<p>The output from <code class="docutils literal notranslate"><span class="pre">srun</span></code> may vary somewhat depending on the version of Slurm installed.
If PMIx is not present in the output, then you will not be able to use srun
to launch Open MPI applications.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>PMI-2 is not supported in Open MPI 5.0.0 and later releases.</p>
</div>
<p>Provided the Slurm installation includes the PMIx plugin, Open MPI applications
can then be launched directly via the <code class="docutils literal notranslate"><span class="pre">srun</span></code> command.  For example:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>shell$<span class="w"> </span>srun<span class="w"> </span>-N<span class="w"> </span><span class="m">4</span><span class="w"> </span>--mpi<span class="o">=</span>pmix<span class="w"> </span>mpi-hello-world
</pre></div>
</div>
<p>Or you can use <code class="docutils literal notranslate"><span class="pre">sbatch</span></code> with a script:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>shell$<span class="w"> </span>cat<span class="w"> </span>my_script.sh
<span class="c1">#!/bin/sh</span>
srun<span class="w"> </span>--mpi<span class="o">=</span>pmix<span class="w"> </span>mpi-hello-world
shell$<span class="w"> </span>sbatch<span class="w"> </span>-N<span class="w"> </span><span class="m">4</span><span class="w"> </span>my_script.sh
srun:<span class="w"> </span>jobid<span class="w"> </span><span class="m">1235</span><span class="w"> </span>submitted
shell$
</pre></div>
</div>
<p>Similar using <code class="docutils literal notranslate"><span class="pre">mpirun</span></code> inside of an <code class="docutils literal notranslate"><span class="pre">sbatch</span></code> batch script, no
<code class="docutils literal notranslate"><span class="pre">srun</span></code> command line options specifying number of processes were
necessary, because <code class="docutils literal notranslate"><span class="pre">sbatch</span></code> set all the relevant Slurm-level
parameters about number of processes, cores, partition, etc.</p>
</div>
<div class="section" id="slurm-20-11">
<h2><span class="section-number">10.7.3. </span>Slurm 20.11<a class="headerlink" href="#slurm-20-11" title="Permalink to this heading"></a></h2>
<p>There were some changes in Slurm behavior that were introduced in
Slurm 20.11.0 and subsequently reverted out in Slurm 20.11.3.</p>
<p>SchedMD (the makers of Slurm) strongly suggest that all Open MPI users
avoid using Slurm versions 20.11.0 through 20.11.2.</p>
<p>Indeed, you will likely run into problems using just about any version
of Open MPI these problematic Slurm releases.</p>
<div class="admonition important">
<p class="admonition-title">Important</p>
<p>Please either downgrade to an older version or upgrade
to a newer version of Slurm.</p>
</div>
</div>
</div>


           </div>
          </div>
          <footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
        <a href="ssh.html" class="btn btn-neutral float-left" title="10.6. Launching with SSH" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
        <a href="lsf.html" class="btn btn-neutral float-right" title="10.8. Launching with LSF" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
    </div>

  <hr/>

  <div role="contentinfo">
    <p>&#169; Copyright 2003-2025, The Open MPI Community.
      <span class="lastupdated">Last updated on 2025-05-30 16:41:43 UTC.
      </span></p>
  </div>

  Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
    <a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
    provided by <a href="https://readthedocs.org">Read the Docs</a>.
   

</footer>
        </div>
      </div>
    </section>
  </div>
  <script>
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script> 

</body>
</html>