1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
|
.. -*- rst -*-
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
$COPYRIGHT$
Additional copyrights may follow
$HEADER$
.. The following line is included so that Sphinx won't complain
about this file not being directly included in some toctree
Fundamentals
============
The mapping of processes to nodes can be defined not just with general
policies but also, if necessary, using arbitrary mappings that cannot
be described by a simple policy. Supported directives, given on the
command line via the ``--map-by`` option, include:
* ``SEQ``: (often accompanied by the ``file=<path>`` qualifier)
assigns one process to each node specified in the file. The
sequential file is to contain an entry for each desired process, one
per line of the file.
* ``RANKFILE``: (often accompanied by the ``file=<path>`` qualifier)
assigns one process to the node/resource specified in each entry of
the file, one per line of the file.
For example, using the hostfile below:
.. code::
$ cat myhostfile
aa slots=4
bb slots=4
cc slots=4
The command below will launch three processes, one on each of nodes
``aa``, ``bb``, and ``cc``, respectively. The slot counts don't
matter; one process is launched per line on whatever node is listed on
the line.
.. code::
$ prun --hostfile myhostfile --map-by seq ./a.out
Impact of the ranking option is best illustrated by considering the
following hostfile and test cases where each node contains two
packages (each package with two cores). Using the ``--map-by
ppr:2:package`` option, we map two processes onto each package and
utilize the ``--rank-by`` option as show below:
.. code::
$ cat myhostfile
aa
bb
.. list-table::
:header-rows: 1
* - Command
- Ranks on ``aa``
- Ranks on ``bb``
* - ``--rank-by core``
- 0 1 ! 2 3
- 4 5 ! 6 7
* - ``--rank-by package``
- 0 2 ! 1 3
- 4 6 ! 5 7
* - ``--rank-by package:SPAN``
- 0 4 ! 1 5
- 2 6 ! 3 7
Ranking by slot provides the identical result as ranking by core in
this case |mdash| a simple progression of ranks across each
node. Ranking by package does a round-robin ranking across packages
within each node until all processes have been assigned a rank, and
then progresses to the next node. Adding the ``:SPAN`` qualifier to
the ranking directive causes the ranking algorithm to treat the entire
allocation as a single entity |mdash| thus, the process ranks are
assigned across all packages before circling back around to the
beginning.
The binding operation restricts the process to a subset of the CPU
resources on the node.
The processors to be used for binding can be identified in terms of
topological groupings |mdash| e.g., binding to an l3cache will bind
each process to all processors within the scope of a single L3 cache
within their assigned location. Thus, if a process is assigned by the
mapper to a certain package, then a ``--bind-to l3cache`` directive
will cause the process to be bound to the processors that share a
single L3 cache within that package.
To help balance loads, the binding directive uses a round-robin method,
binding a process to the first available specified object type within
the object where the process was mapped. For example, consider the case
where a job is mapped to the package level, and then bound to core. Each
package will have multiple cores, so if multiple processes are mapped to
a given package, the binding algorithm will assign each process located
to a package to a unique core in a round-robin manner.
Binding can only be done to the mapped object or to a resource located
within that object.
An object is considered completely consumed when the number of
processes bound to it equals the number of CPUs within it. Unbound
processes are not considered in this computation. Additional
processes cannot be mapped to consumed objects unless the
OVERLOAD qualifier is provided via the "--bind-to" command
line option.
Default process mapping/ranking/binding policies can also be set with MCA
parameters, overridden by the command line options when provided. MCA
parameters can be set on the ``prte`` command line when starting the
DVM (or in the ``prterun`` command line for a single-execution job), but
also in a system or user ``mca-params.conf`` file or as environment
variables, as described in the MCA section below. Some examples include:
.. list-table::
:header-rows: 1
* - ``prun`` option
- MCA parameter key
- Value
* - ``--map-by core``
- ``rmaps_default_mapping_policy``
- ``core``
* - ``--map-by package``
- ``rmaps_default_mapping_policy``
- ``package``
* - ``--rank-by core``
- ``rmaps_default_ranking_policy``
- ``core``
* - ``--bind-to core``
- ``hwloc_default_binding_policy``
- ``core```
* - ``--bind-to package``
- ``hwloc_default_binding_policy``
- ``package``
* - ``--bind-to none``
- ``hwloc_default_binding_policy``
- ``none``
|
