1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
|
<!--#include virtual="header.txt"-->
<h1><a name="top">Plane distribution: <i>-m plane=plane_size</i></a></h1>
<p>The plane distribution allocates tasks in blocks of size
<i>plane_size</i> in a round-robin fashion across allocated nodes.
<p>To specify the plane distribution add to the srun command line
<i>--distribution=plane=plane_size</i> or <i>-m plane=plane_size</i>
where <i>plane_size</i> is the requested plane/block size.
<h2 id="Examples">Examples of plane distributions
<a class="slurm_link" href="#Examples"></a>
</h2>
<p>In the examples below we assume we have 21 tasks and that the
task list is: 0, 1, 2, 3, 4, ..., 19, 20.
<p>On <u>one (1)</u> node: <i>srun -N 1-1 -n 21 -m plane=4 <...></i>.
<p>Even though the user specified a <i>plane_size</i> of 4 the final plane
distribution results in a <i>plane_size</i> of 21, since all the tasks landed
on one node.
<p>
<center>
<img src="plane_ex1.gif">
<p>Figure 1: Process layout for <i>srun -N 1-1 -n 21 -m plane=4 <...></i>
</center>
<br>
<p>On <u>four (4)</u> nodes: <i>srun -N 4-4 -n 21 -m plane=4 <...></i>.
<p>The plane distribution with a <i>plane_size</i> of 4 results in the
following allocation of the task ids:
<p>
<center>
<img src="plane_ex2.gif">
<p>Figure 2: Process layout for <i>srun -N 4-4 -n 21 -m plane=4 <...> </i>
</center>
<br>
<p>On <u>four (4)</u> nodes: <i>srun -N 4-4 -n 21 -m plane=2 <...>
</i>.
<p>The plane distribution with a <i>plane_size</i> of 2 results in the
following allocation of the task ids:
<p>
<center>
<img src="plane_ex3.gif">
<p>Figure 3: Process layout for <i>srun -N 4-4 -n 21 -m plane=2 <...></i>
</center>
<h2 id="Distribution">Plane distribution and task affinity
<a class="slurm_link" href="#Distribution"></a>
</h2>
<p>The concept behind this distribution is to divide the clusters into
planes. Each plane includes a number of the lowest level of logical
processors (CPU, cores, threads depending on the architecture) on each
node. We then schedule within each plane first and then across planes.
<p>We ensure that the processes are located correctly by setting the
process affinity to the specified/appropriate logical processor. Process
affinity is available in Slurm when the task/affinity plug-in is
enabled.
<p>On a dual-processor node with quad-core processors (see figure 4)
the plane distribution results in:
<ul>
<li>One plane if the <i>plane_size=8</i>. In this case the processors are
scheduled by first filling up the nodes and then scheduled across the
nodes.</li>
<li>Eight planes if the <i>plane_size=1</i>. In this case we would always
schedule across the node first.</li>
</ul>
<p>
<center>
<img src="plane_ex4.gif">
<p>Figure 4: Quad-core dual-processor system
</center>
<br>
<p>In a multi-core/hyper-threaded environment, two planes would
provide better locality but potentially more contention for other
resources.
<p>On the other hand, four planes (scheduling across processors) would
minimize contention for cache and memory.
<h2 id="Affinity">Examples of plane distributions with process affinity enabled
<a class="slurm_link" href="#Affinity"></a>
</h2>
<p>In the examples below we assume we have 21 tasks and that the
task list is: 0, 1, 2, 3, 4, ..., 19, 20.
<p>On <u>one (1)</u> node:
<i>srun -N 1-1 -n 21 -m plane=4 --cpu-bind=core <...></i>.
Even though the user specified a <i>plane_size</i> of 4 the final plane
distribution results in a plane distribution with <i>plane_size=8</i>.
<p>
<center>
<img src="plane_ex5.gif">
<p>Figure 5: Process layout for
<i>srun -N 1-1 -n 21 -m plane=4 --cpu-bind=core <...></i>.
</center>
<br>
<p>On <u>four (4)</u> nodes:
<i>srun -N 4-4 -n 21 -m plane=4 --cpu-bind=core <...></i>.
The plane distribution with a <i>plane_size</i> of 4 results in the
following allocation of the task ids:
<p>
<center>
<img src="plane_ex6.gif" width=600>
<p>Figure 6: Process layout for
<i>srun -N 4-4 -n 21 -m plane=4 --cpu-bind=core <...></i>.
</center>
<br>
<p>On <u>four (4)</u> nodes:
<i>srun -N 4-4 -n 21 -m plane=2 --cpu-bind=core <...>
</i>. The plane distribution with a <i>plane_size</i> of 2 results in the
following allocation of the task ids:
<p>
<center>
<img src="plane_ex7.gif" width=600>
<p>Figure 7: Process layout for
<i>srun -N 4-4 -n 21 -m plane=2 --cpu-bind=core <...></i>.
</center>
<br>
<p style="text-align:center;">Last modified 22 April 2021</p>
<!--#include virtual="footer.txt"-->
|
