1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
|
<!DOCTYPE html>
<html class="writer-html5" lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>8.1. Overview — PMIx Reference Run Time Environment 3.0.8 documentation</title>
<link rel="stylesheet" type="text/css" href="../_static/pygments.css" />
<link rel="stylesheet" type="text/css" href="../_static/css/theme.css" />
<!--[if lt IE 9]>
<script src="../_static/js/html5shiv.min.js"></script>
<![endif]-->
<script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
<script src="../_static/jquery.js"></script>
<script src="../_static/underscore.js"></script>
<script src="../_static/_sphinx_javascript_frameworks_compat.js"></script>
<script src="../_static/doctools.js"></script>
<script src="../_static/sphinx_highlight.js"></script>
<script src="../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="8.4. Examples" href="examples.html" />
<link rel="prev" title="8. Process placement" href="index.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../index.html" class="icon icon-home">
PMIx Reference Run Time Environment
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../quickstart.html">1. Quick start</a></li>
<li class="toctree-l1"><a class="reference internal" href="../release-notes.html">2. Release Notes</a></li>
<li class="toctree-l1"><a class="reference internal" href="../getting-help.html">3. Getting help</a></li>
<li class="toctree-l1"><a class="reference internal" href="../install.html">4. Installing PRRTE</a></li>
<li class="toctree-l1"><a class="reference internal" href="../configuration.html">5. PRRTE DVM Configuration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../how-things-work/index.html">6. How Things Work</a></li>
<li class="toctree-l1"><a class="reference internal" href="../hosts/index.html">7. Host specification</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="index.html">8. Process placement</a><ul class="current">
<li class="toctree-l2 current"><a class="current reference internal" href="#">8.1. Overview</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#quick-summary">8.1.1. Quick Summary</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#definition-of-slot">8.2. Definition of ‘slot’</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#slots-are-not-hardware-resources">8.2.1. Slots are not hardware resources</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#definition-of-processor-element">8.3. Definition of “processor element”</a></li>
<li class="toctree-l2"><a class="reference internal" href="examples.html">8.4. Examples</a></li>
<li class="toctree-l2"><a class="reference internal" href="fundamentals.html">8.5. Fundamentals</a></li>
<li class="toctree-l2"><a class="reference internal" href="limits.html">8.6. Overloading and Oversubscribing</a></li>
<li class="toctree-l2"><a class="reference internal" href="diagnostics.html">8.7. Diagnostics</a></li>
<li class="toctree-l2"><a class="reference internal" href="rankfiles.html">8.8. Rankfiles</a></li>
<li class="toctree-l2"><a class="reference internal" href="deprecated.html">8.9. Deprecated options</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../notifications.html">9. Notifications</a></li>
<li class="toctree-l1"><a class="reference internal" href="../session-directory.html">10. Session directory</a></li>
<li class="toctree-l1"><a class="reference internal" href="../developers/index.html">11. Developer’s guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributing.html">12. Contributing to PRRTE</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">13. License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../man/index.html">14. PRRTE manual pages</a></li>
<li class="toctree-l1"><a class="reference internal" href="../versions.html">15. Software Version Numbers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../news/index.html">16. News</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../index.html">PMIx Reference Run Time Environment</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="index.html"><span class="section-number">8. </span>Process placement</a></li>
<li class="breadcrumb-item active"><span class="section-number">8.1. </span>Overview</li>
<li class="wy-breadcrumbs-aside">
<a href="../_sources/placement/overview.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<style>
.wy-table-responsive table td,.wy-table-responsive table th{white-space:normal}
</style><div class="section" id="overview">
<h1><span class="section-number">8.1. </span>Overview<a class="headerlink" href="#overview" title="Permalink to this heading"></a></h1>
<p>PRRTE provides a set of three controls for assigning process
locations and ranks:</p>
<ol class="arabic simple">
<li><p>Mapping: Assigns a default location to each process</p></li>
<li><p>Ranking: Assigns a unique integer rank value to each process</p></li>
<li><p>Binding: Constrains each process to run on specific processors</p></li>
</ol>
<p>This section provides an overview of these three controls. Unless
otherwise this behavior is shared by <code class="docutils literal notranslate"><span class="pre">prun(1)</span></code> (working with a PRRTE
DVM), and <code class="docutils literal notranslate"><span class="pre">prterun(1)</span></code>. More detail about PRRTE process placement is
available in the following sections (using <code class="docutils literal notranslate"><span class="pre">--help</span>
<span class="pre">placement-<section></span></code>):</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">examples</span></code>: some examples of the interactions between mapping,
ranking, and binding options.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">fundamentals</span></code>: provides deeper insight into PRRTE’s mapping,
ranking, and binding options.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">limits</span></code>: explains the difference between <em>overloading</em> and
<em>oversubscribing</em> resources.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">diagnostics</span></code>: describes options for obtaining various diagnostic
reports that aid the user in verifying and tuning the placement for
a specific job.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">rankfiles</span></code>: explains the format and use of the rankfile mapper
for specifying arbitrary process placements.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">deprecated</span></code>: a list of deprecated options and their new
equivalents.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">all</span></code>: outputs all the placement help except for the
<code class="docutils literal notranslate"><span class="pre">deprecated</span></code> section.</p></li>
</ul>
<div class="section" id="quick-summary">
<h2><span class="section-number">8.1.1. </span>Quick Summary<a class="headerlink" href="#quick-summary" title="Permalink to this heading"></a></h2>
<p>The two binaries that most influence process layout are <code class="docutils literal notranslate"><span class="pre">prte(1)</span></code>
and <code class="docutils literal notranslate"><span class="pre">prun(1)</span></code>. The <code class="docutils literal notranslate"><span class="pre">prte(1)</span></code> process discovers the allocation,
establishes a Distributed Virtual Machine by starting a <code class="docutils literal notranslate"><span class="pre">prted(1)</span></code>
daemon on each node of the allocation, and defines the efault
mapping/ranking/binding policies for all jobs. The <code class="docutils literal notranslate"><span class="pre">prun(1)</span></code> process
defines the specific mapping/ranking/binding for a specific job. Most
of the command line controls are targeted to <code class="docutils literal notranslate"><span class="pre">prun(1)</span></code> since each job
has its own unique requirements.</p>
<p><code class="docutils literal notranslate"><span class="pre">prterun(1)</span></code> is just a wrapper around <code class="docutils literal notranslate"><span class="pre">prte(1)</span></code> for a single job
PRRTE DVM. It is doing the job of both <code class="docutils literal notranslate"><span class="pre">prte(1)</span></code> and <code class="docutils literal notranslate"><span class="pre">prun(1)</span></code>,
and, as such, accepts the sum all of their command line arguments. Any
example that uses <code class="docutils literal notranslate"><span class="pre">prun(1)</span></code> can substitute the use of <code class="docutils literal notranslate"><span class="pre">prterun(1)</span></code>
except where otherwise noted.</p>
<p>The <code class="docutils literal notranslate"><span class="pre">prte(1)</span></code> process attempts to automatically discover the nodes
in the allocation by querying supported resource managers. If a
supported resource manager is not present then <code class="docutils literal notranslate"><span class="pre">prte(1)</span></code> relies on a
hostfile provided by the user. In the absence of such a hostfile it
will run all processes on the localhost.</p>
<p>If running under a supported resource manager, the <code class="docutils literal notranslate"><span class="pre">prte(1)</span></code> process
will start the daemon processes (<code class="docutils literal notranslate"><span class="pre">prted(1)</span></code>) on the remote nodes
using the corresponding resource manager process starter. If no such
starter is available then <code class="docutils literal notranslate"><span class="pre">ssh</span></code> (or <code class="docutils literal notranslate"><span class="pre">rsh</span></code>) is used.</p>
<p>Minus user direction, PRRTE will automatically map processes in a
round-robin fashion by CPU, binding each process to its own CPU. The
type of CPU used (core vs hwthread) is determined by (in priority
order):</p>
<ul class="simple">
<li><p>user directive on the command line via the HWTCPUS qualifier to
the <code class="docutils literal notranslate"><span class="pre">--map-by</span></code> directive</p></li>
<li><p>setting the <code class="docutils literal notranslate"><span class="pre">rmaps_default_mapping_policy</span></code> MCA parameter to
include the <code class="docutils literal notranslate"><span class="pre">HWTCPUS</span></code> qualifier. This parameter sets the default
value for a PRRTE DVM — qualifiers are carried across to DVM
jobs started via <code class="docutils literal notranslate"><span class="pre">prun</span></code> unless overridden by the user’s command
line</p></li>
<li><p>defaulting to <code class="docutils literal notranslate"><span class="pre">CORE</span></code> in topologies where core CPUs are defined,
and to <code class="docutils literal notranslate"><span class="pre">hwthreads</span></code> otherwise.</p></li>
</ul>
<p>By default, the ranks are assigned in accordance with the mapping
directive — e.g., jobs that are mapped by-node will have the
process ranks assigned round-robin on a per-node basis.</p>
<p>PRRTE automatically binds processes unless directed not to do so by
the user. Minus direction, PRRTE will bind individual processes to
their own CPU within the object to which they were mapped. Should a
node become oversubscribed during the mapping process, and if
oversubscription is allowed, all subsequent processes assigned to that
node will <em>not</em> be bound.</p>
</div>
</div>
<div class="section" id="definition-of-slot">
<h1><span class="section-number">8.2. </span>Definition of ‘slot’<a class="headerlink" href="#definition-of-slot" title="Permalink to this heading"></a></h1>
<p>The term “slot” is used extensively in the rest of this documentation.
A slot is an allocation unit for a process. The number of slots on a
node indicate how many processes can potentially execute on that node.
By default, PRRTE will allow one process per slot.</p>
<p>If PRRTE is not explicitly told how many slots are available on a node
(e.g., if a hostfile is used and the number of slots is not specified
for a given node), it will determine a maximum number of slots for
that node in one of two ways:</p>
<ol class="arabic simple">
<li><p>Default behavior: By default, PRRTE will attempt to discover the
number of processor cores on the node, and use that as the number
of slots available.</p></li>
<li><p>When <code class="docutils literal notranslate"><span class="pre">--use-hwthread-cpus</span></code> is used: If <code class="docutils literal notranslate"><span class="pre">--use-hwthread-cpus</span></code> is
specified on the command line, then PRRTE will attempt to discover
the number of hardware threads on the node, and use that as the
number of slots available.</p></li>
</ol>
<p>This default behavior also occurs when specifying the <code class="docutils literal notranslate"><span class="pre">--host</span></code>
option with a single host. Thus, the command:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>shell$<span class="w"> </span>prun<span class="w"> </span>--host<span class="w"> </span>node1<span class="w"> </span>./a.out
</pre></div>
</div>
<p>launches a number of processes equal to the number of cores on node
<code class="docutils literal notranslate"><span class="pre">node1</span></code>, whereas:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>shell$<span class="w"> </span>prun<span class="w"> </span>--host<span class="w"> </span>node1<span class="w"> </span>--use-hwthread-cpus<span class="w"> </span>./a.out
</pre></div>
</div>
<p>launches a number of processes equal to the number of hardware
threads on <code class="docutils literal notranslate"><span class="pre">node1</span></code>.</p>
<p>When PRRTE applications are invoked in an environment managed by a
resource manager (e.g., inside of a Slurm job), and PRRTE was built
with appropriate support for that resource manager, then PRRTE will
be informed of the number of slots for each node by the resource
manager. For example:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>shell$<span class="w"> </span>prun<span class="w"> </span>./a.out
</pre></div>
</div>
<p>launches one process for every slot (on every node) as dictated by
the resource manager job specification.</p>
<p>Also note that the one-process-per-slot restriction can be overridden
in unmanaged environments (e.g., when using hostfiles without a
resource manager) if oversubscription is enabled (by default, it is
disabled). Most parallel applications and HPC environments do not
oversubscribe; for simplicity, the majority of this documentation
assumes that oversubscription is not enabled.</p>
<div class="section" id="slots-are-not-hardware-resources">
<h2><span class="section-number">8.2.1. </span>Slots are not hardware resources<a class="headerlink" href="#slots-are-not-hardware-resources" title="Permalink to this heading"></a></h2>
<p>Slots are frequently incorrectly conflated with hardware resources.
It is important to realize that slots are an entirely different metric
than the number (and type) of hardware resources available.</p>
<p>Here are some examples that may help illustrate the difference:</p>
<ol class="arabic">
<li><p>More processor cores than slots: Consider a resource manager job
environment that tells PRRTE that there is a single node with 20
processor cores and 2 slots available. By default, PRRTE will
only let you run up to 2 processes.</p>
<p>Meaning: you run out of slots long before you run out of processor
cores.</p>
</li>
<li><p>More slots than processor cores: Consider a hostfile with a single
node listed with a <code class="docutils literal notranslate"><span class="pre">slots=50</span></code> qualification. The node has 20
processor cores. By default, PRRTE will let you run up to 50
processes.</p>
<p>Meaning: you can run many more processes than you have processor
cores.</p>
</li>
</ol>
</div>
</div>
<div class="section" id="definition-of-processor-element">
<h1><span class="section-number">8.3. </span>Definition of “processor element”<a class="headerlink" href="#definition-of-processor-element" title="Permalink to this heading"></a></h1>
<p>By default, PRRTE defines that a “processing element” is a processor
core. However, if <code class="docutils literal notranslate"><span class="pre">--use-hwthread-cpus</span></code> is specified on the command
line, then a “processing element” is a hardware thread.</p>
</div>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="index.html" class="btn btn-neutral float-left" title="8. Process placement" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="examples.html" class="btn btn-neutral float-right" title="8.4. Examples" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>
<hr/>
<div role="contentinfo">
<p>© Copyright 2003-2025, The PRRTE Community.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
|
