1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<!--Converted with LaTeX2HTML 2019.2 (Released June 5, 2019) -->
<HTML lang="EN">
<HEAD>
<TITLE>3.3 Parallelization levels</TITLE>
<META NAME="description" CONTENT="3.3 Parallelization levels">
<META NAME="keywords" CONTENT="user_guide">
<META NAME="resource-type" CONTENT="document">
<META NAME="distribution" CONTENT="global">
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=utf-8">
<META NAME="viewport" CONTENT="width=device-width, initial-scale=1.0">
<META NAME="Generator" CONTENT="LaTeX2HTML v2019.2">
<LINK REL="STYLESHEET" HREF="user_guide.css">
<LINK REL="next" HREF="node21.html">
<LINK REL="previous" HREF="node19.html">
<LINK REL="next" HREF="node21.html">
</HEAD>
<BODY >
<!--Navigation Panel-->
<A
HREF="node21.html">
<IMG WIDTH="37" HEIGHT="24" ALT="next" SRC="next.png"></A>
<A
HREF="node17.html">
<IMG WIDTH="26" HEIGHT="24" ALT="up" SRC="up.png"></A>
<A
HREF="node19.html">
<IMG WIDTH="63" HEIGHT="24" ALT="previous" SRC="prev.png"></A>
<A ID="tex2html225"
HREF="node1.html">
<IMG WIDTH="65" HEIGHT="24" ALT="contents" SRC="contents.png"></A>
<BR>
<B> Next:</B> <A
HREF="node21.html">3.4 Tricks and problems</A>
<B> Up:</B> <A
HREF="node17.html">3 Parallelism</A>
<B> Previous:</B> <A
HREF="node19.html">3.2 Running on parallel</A>
<B> <A ID="tex2html226"
HREF="node1.html">Contents</A></B>
<BR>
<BR>
<!--End of Navigation Panel-->
<!--Table of Child-Links-->
<A ID="CHILD_LINKS"><STRONG>Subsections</STRONG></A>
<UL>
<LI><UL>
<LI><A ID="tex2html227"
HREF="node20.html#SECTION00043010000000000000">3.3.0.1 About communications</A>
<LI><A ID="tex2html228"
HREF="node20.html#SECTION00043020000000000000">3.3.0.2 Choosing parameters</A>
<LI><A ID="tex2html229"
HREF="node20.html#SECTION00043030000000000000">3.3.0.3 Massively parallel calculations</A>
</UL>
<BR>
<LI><A ID="tex2html230"
HREF="node20.html#SECTION00043100000000000000">3.3.1 Understanding parallel I/O</A>
</UL>
<!--End of Table of Child-Links-->
<HR>
<H2><A ID="SECTION00043000000000000000">
3.3 Parallelization levels</A>
</H2>
<P>
In Q<SMALL>UANTUM </SMALL>ESPRESSO several MPI parallelization levels are
implemented, in which both calculations
and data structures are distributed across processors.
Processors are organized in a hierarchy of groups,
which are identified by different MPI communicators level.
The groups hierarchy is as follow:
<UL>
<LI><B>world</B>: is the group of all processors (MPI_COMM_WORLD).
</LI>
<LI><B>images</B>: Processors can then be divided into different "images", each corresponding to a
different self-consistent or linear-response
calculation, loosely coupled to others.
</LI>
<LI><B>pools</B>: each image can be subpartitioned into
"pools", each taking care of a group of k-points.
</LI>
<LI><B>bands</B>: each pool is subpartitioned into
"band groups", each taking care of a group
of Kohn-Sham orbitals (also called bands, or
wavefunctions). Especially useful for calculations
with hybrid functionals.
</LI>
<LI><B>PW</B>: orbitals in the PW basis set,
as well as charges and density in either
reciprocal or real space, are distributed
across processors.
This is usually referred to as "PW parallelization".
All linear-algebra operations on array of PW /
real-space grids are automatically and effectively parallelized.
3D FFT is used to transform electronic wave functions from
reciprocal to real space and vice versa. The 3D FFT is
parallelized by distributing planes of the 3D grid in real
space to processors (in reciprocal space, it is columns of
G-vectors that are distributed to processors).
</LI>
<LI><B>tasks</B>:
In order to allow good parallelization of the 3D FFT when
the number of processors exceeds the number of FFT planes,
FFTs on Kohn-Sham states are redistributed to
``task'' groups so that each group
can process several wavefunctions at the same time.
Alternatively, when this is not possible, a further
subdivision of FFT planes is performed.
</LI>
<LI><B>linear-algebra group</B>:
A further level of parallelization, independent on
PW or k-point parallelization, is the parallelization of
subspace diagonalization / iterative orthonormalization.
Both operations required the diagonalization of
arrays whose dimension is the number of Kohn-Sham states
(or a small multiple of it). All such arrays are distributed block-like
across the ``linear-algebra group'', a subgroup of the pool of processors,
organized in a square 2D grid. As a consequence the number of processors
in the linear-algebra group is given by <I>n</I><SUP>2</SUP>, where <I>n</I> is an integer;
<I>n</I><SUP>2</SUP> must be smaller than the number of processors in the PW group.
The diagonalization is then performed
in parallel using standard linear algebra operations.
(This diagonalization is used by, but should not be confused with,
the iterative Davidson algorithm). The preferred option is to use
ELPA and ScaLAPACK; alternative built-in algorithms are anyway available.
</LI>
</UL>
Note however that not all parallelization levels
are implemented in all codes.
<P>
<H4><A ID="SECTION00043010000000000000">
3.3.0.1 About communications</A>
</H4>
Images and pools are loosely coupled: inter-processors communication
between different images and pools is modest. Processors within each
pool are instead tightly coupled and communications are significant.
This means that fast communication hardware is needed if
your pool extends over more than a few processors on different nodes.
<P>
<H4><A ID="SECTION00043020000000000000">
3.3.0.2 Choosing parameters</A>
</H4>:
To control the number of processors in each group,
command line switches:
<TT>-nimage</TT>, <TT>-npools</TT>, <TT>-nband</TT>,
<TT>-ntg</TT>, <TT>-ndiag</TT> or <TT>-northo</TT>
(shorthands, respectively: <TT>-ni</TT>, <TT>-nk</TT>, <TT>-nb</TT>,
<TT>-nt</TT>, <TT>-nd</TT>)
are used.
As an example consider the following command line:
<PRE>
mpirun -np 4096 ./neb.x -ni 8 -nk 2 -nt 4 -nd 144 -i my.input
</PRE>
This executes a NEB calculation on 4096 processors, 8 images (points in the configuration
space in this case) at the same time, each of
which is distributed across 512 processors.
k-points are distributed across 2 pools of 256 processors each,
3D FFT is performed using 4 task groups (64 processors each, so
the 3D real-space grid is cut into 64 slices), and the diagonalization
of the subspace Hamiltonian is distributed to a square grid of 144
processors (12x12).
<P>
Default values are: <TT>-ni 1 -nk 1 -nt 1</TT> ;
<TT>nd</TT> is set to 1 if ScaLAPACK is not compiled,
it is set to the square integer smaller than or equal to the number of
processors of each pool.
<P>
<H4><A ID="SECTION00043030000000000000">
3.3.0.3 Massively parallel calculations</A>
</H4>
For very large jobs (i.e. O(1000) atoms or more) or for very long jobs,
to be run on massively parallel machines (e.g. IBM BlueGene) it is
crucial to use in an effective way all available parallelization levels:
on linear algebra (requires compilation with ELPA and/or ScaLAPACK),
on "task groups" (requires run-time option "-nt N"), and mixed
MPI-OpenMP (requires OpenMP compilation: <TT>configure</TT>–enable-openmp).
Without a judicious choice of parameters, large jobs will find a
stumbling block in either memory or CPU requirements. Note that I/O
may also become a limiting factor.
<P>
<H3><A ID="SECTION00043100000000000000">
3.3.1 Understanding parallel I/O</A>
</H3>
In parallel execution, each processor has its own slice of data
(Kohn-Sham orbitals, charge density, etc), that have to be written
to temporary files during the calculation,
or to data files at the end of the calculation.
This can be done in two different ways:
<UL>
<LI>``collected'': all slices are
collected by the code to a single processor
that writes them to disk, in a single file,
using a format that doesn't depend upon
the number of processors or their distribution.
This is the default since v.6.2 for final data.
</LI>
<LI>``portable'': as above, but data can be
copied to and read from a different machines
(this is not guaranteed with Fortran binary files).
Requires compilation with <code>-D__HDF5</code>
preprocessing option and HDF
</LI>
</UL>
There is a third format, no longer used for final
data but used for scratch and restart files:
<UL>
<LI>``distributed'': each processor
writes its own slice to disk in its internal
format to a different file.
The ``distributed'' format is fast and simple,
but the data so produced is readable only by
a job running on the same number of processors,
with the same type of parallelization, as the
job who wrote the data, and if all
files are on a file system that is visible to all
processors (i.e., you cannot use local scratch
directories: there is presently no way to ensure
that the distribution of processes across
processors will follow the same pattern
for different jobs).
</LI>
</UL>
<P>
The directory for data is specified in input variables
<TT>outdir</TT> and <TT>prefix</TT> (the former can be specified
as well in environment variable ESPRESSO_TMPDIR):
<TT>outdir/prefix.save</TT>. A copy of pseudopotential files
is also written there. If some processor cannot access the
data directory, the pseudopotential files are read instead
from the pseudopotential directory specified in input data.
Unpredictable results may follow if those files
are not the same as those in the data directory!
<P>
<EM>IMPORTANT:</EM>
Avoid I/O to network-mounted disks (via NFS) as much as you can!
Ideally the scratch directory <TT>outdir</TT> should be a modern
Parallel File System. If you do not have any, you can use local
scratch disks (i.e. each node is physically connected to a disk
and writes to it) but you may run into trouble anyway if you
need to access your files that are scattered in an unpredictable
way across disks residing on different nodes.
<P>
You can use input variable <TT>disk_io</TT> to vary the
amount of I/O done by <TT>pw.x</TT>. Since v.5.1, the dafault value is
<TT>disk_io='low'</TT>, so the code will store wavefunctions
into RAM and not on disk during the calculation. Specify
<TT>disk_io='medium'</TT> only if you have too many k-points
and you run into trouble with memory; choose <TT>disk_io='none'</TT>
if you do not need to keep final data files.
<P>
<HR>
<!--Navigation Panel-->
<A
HREF="node21.html">
<IMG WIDTH="37" HEIGHT="24" ALT="next" SRC="next.png"></A>
<A
HREF="node17.html">
<IMG WIDTH="26" HEIGHT="24" ALT="up" SRC="up.png"></A>
<A
HREF="node19.html">
<IMG WIDTH="63" HEIGHT="24" ALT="previous" SRC="prev.png"></A>
<A ID="tex2html225"
HREF="node1.html">
<IMG WIDTH="65" HEIGHT="24" ALT="contents" SRC="contents.png"></A>
<BR>
<B> Next:</B> <A
HREF="node21.html">3.4 Tricks and problems</A>
<B> Up:</B> <A
HREF="node17.html">3 Parallelism</A>
<B> Previous:</B> <A
HREF="node19.html">3.2 Running on parallel</A>
<B> <A ID="tex2html226"
HREF="node1.html">Contents</A></B>
<!--End of Navigation Panel-->
</BODY>
</HTML>
|
