1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
|
<!doctype html public "-//w3c//dtd html 4.0 transitional//en">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="GENERATOR" content="Mozilla/4.6 [en] (WinNT; I) [Netscape]">
<title> Disk Resident Arrays Operations </title>
<!-- DO NOT EDIT: this is a cleared public file -->
</head>
<body text="#000000" bgcolor="#F5F3D3" link="#0000EF" vlink="#55188A" alink="#FF0000" background="magenta">
<center>
<h3>
Disk Resident Arrays</h3></center>
Disk Resident Arrays (DRA) extend the Global Arrays (GA) programming model
to disk. The library encapsulates the details of data layout, addressing
and I/O transfer in disk arrays objects. Disk resident arrays resemble
global arrays except that they reside on the disk instead of the main memory.
The main features of this model are:
<ul>
<li>
<font size=-1>Data can be transfered (copied) between disk and global memory.</font></li>
<li>
<font size=-1>I/O operations have nonblocking interface to allow overlapping
of I/O with computations.</font></li>
<li>
<font size=-1>All I/O operations are collective.</font></li>
<li>
<font size=-1>Either whole or sections of global arrays can be transferred
between GA memory and the disk.</font></li>
<li>
<font size=-1>Reshaping and transpose operation are allowed during the
transfer.</font></li>
<li>
<font size=-1>Disk resident arrays might be temporary or persistent (saved
after program termination and available in next runs).</font></li>
<li>
<font size=-1>Persistent disk arrays can be accessed by any program executing
on arbitrary number of processors.</font></li>
<li>
<font size=-1>Distribution and internal data layout on the disk is optimized
for large data transfer.</font></li>
<li>
<font size=-1>Hints provided by the user are utilized to guide optimization
of the library performance for the specific I/O patterns.</font></li>
</ul>
DRA can take advantage of a shared file system of a collection of independent
filesystems accessible from individual computing nodes.
<br>
<hr ALIGN=LEFT SIZE=1 NOSHADE WIDTH="100%">
<p><a NAME="dra_init"></a>
<h2>
dra_init</h2>
<pre>status = dra_init(max_arrays, max_array_size, total_disk_space, max_memory)
integer max_arrays [input]
double precision max_array_size [input]
double precision total_disk_space [input]
double precision max_memory [input]</pre>
Initializes Disk Resident Array I/O subsystem.
<p><i>max_array_size </i>, <i>total_disk_space</i> and <i>max_memory</i>
are given in bytes.
<p><i>max_memory</i> specifies how much local memory per processor the
application is willing to provide to the DRA I/O subsystem for buffering.
<p>The value of -1 for any of input arguments means: "don't care ", "don't
know" , or "use defaults ".
<p>
<hr>
<p><a NAME="dra_terminate"></a>
<h2>
dra_terminate</h2>
<pre> status=dra_terminate()</pre>
Close all open disk resident arrays and shut down DRA I/O subsystem.
<p>
<hr>
<p><a NAME="dra_create"></a>
<h2>
dra_create</h2>
<pre> status = dra_create(type, dim1, dim2, name, filename, mode,rdim1,rdim2,d_a)
integer type [input] ! MA type identifier
integer dim1 [input]
integer dim2 [input]
character*(*) name [input]
character*(*) filename [input]
integer mode [input]
integer rdim1 [input]
integer rdim2 [input]
integer d_a [output] ! DRA handle</pre>
Creates a new disk resident array with specified dimensions and data
<i>type</i>
(represented by MA type handle).
<p>The string <i>filename</i> specifies name of an abstract meta-file that
will store the data on the disk. The meta-file might be implemented as
multiple files that will contain parts of the disk resident array. The
component files will have names derived from the string <i>filename</i>
according to some established scheme(s). Only one DRA object can be stored
in DRA meta-file identified by <i>filename</i> .
<p>DRA objects persist on the disk after calling <a href="#dra_close">dra_close</a>.
<a href="#dra_delete">dra_delete</a>
should be used instead of <a href="#dra_close">dra_close</a> to delete
disk array and associated meta-file on the disk.
<p>String <i>name</i> can be used for more informative (longer)names.
<p>The data in disk resident array is implicitly initialized to 0 .
<p>Access permissions (read, write, read& write) are set in <i>mode</i>
. These are set using defined in the header files DRA.fh (Fortran) and
DRA.h (C) preprocessor constants: DRA_R, DRA_W, DRA_RW.
<p>The pair [<i>rdim1</i>, <i>rdim2</i>] specifies dimensions of a typical
request. The value of -1 for either of them means "unspecified". The layout
of the data on the disk(s) is determined based on the values of these arguments.
Performance of the DRA operations depends on the dimensions (section shape)
of the requests. If data layout is optimized for column-like sections,
performance of DRA operations for row-like sections might be seriously
degraded. This is analogous to the effect of wrong loop ordering iyielding
frequent cache misses in the following example .
<pre> double precision a(1000, 1000)
do i = 1, 1000
do j = 1, 1000
a(i,j) = dfloat(i+j)
enddo
enddo
</pre>
instead of
<pre> do j = 1, 1000
do i = 1, 1000
a(i,j) = dfloat(i+j)
enddo
enddo</pre>
<hr>
<p><a NAME="dra_open"></a>
<h2>
dra_open</h2>
<pre> status = dra_open(filename, mode, d_a)
character*(*) filename [input]
integer mode [input]
integer d_a [output] ! DRA handle</pre>
Open and assign DRA handle to disk resident array stored in DRA meta-file
<i>filename.</i>
Disk resident arrays that are created with <a href="#dra_create">dra_create</a>
and saved by calling <a href="#dra_close">dra_close</a> can be later opened
and accessed by the same or different application.
<p>Attributes of the disk resident array can be found by calling <a href="#dra_inquire">dra_inquire</a>.
<p>
<hr>
<p><a NAME="dra_write"></a>
<h2>
dra_write</h2>
<pre> status = dra_write(g_a, d_a, request)
integer g_a [input] ! GA handle
integer d_a [input] ! DRA handle
integer request [output] ! request id</pre>
Write asynchronously specified global array to specified disk resident
array.
<p>The dimensions and type of arrays represented by handles <i>g_a</i>
and <i>d_a</i> must match. If dimensions don't match, <a href="#dra_write_section">dra_write_section</a>
should be used instead.
<p>The operation is by definition asynchronous but it might be implemented
as synchronous i.e., it would return only when the I/O is completed.
<p><i>request</i> can be used to <a href="#dra_probe">dra_probe</a> or
<a href="#dra_wait">dra_wait</a>
for completion of the associated operation.
<p>
<hr>
<p><a NAME="dra_write_section"></a>
<h2>
dra_write_section</h2>
<pre> status = dra_write_section(transp, g_a, gilo, gihi, gjlo, gjhi,
d_a, dilo, dihi, djlo, djhi, request)
logical transp [input] ! transpose operator
integer g_a [input] ! GA handle
integer d_a [input] ! DRA handle
integer gilo [input]
integer gihi [input]
integer gjlo [input]
integer gjhi [input]
integer dilo [input]
integer dihi [input]
integer djlo [input]
integer djhi [input]
integer request [output] ! request id</pre>
Write asynchronously specified global array section to specified disk resident
array section:
<pre> OP(g_a[ gilo:gihi, gjlo:gjhi]) --> d_a[ dilo:dihi, djlo:djhi]</pre>
where <i>OP</i> is the transpose operator (.<i>true./.false</i>.). Return
error if the two section's types or sizes mismatch. See <a href="#dra_write">dra_write</a>
specs for discussion of <i>request</i> .
<p>
<hr>
<p><a NAME="dra_read"></a>
<h2>
dra_read</h2>
<pre> status = dra_read(g_a, d_a, request)
integer g_a [input] ! GA handle
integer d_a [input] ! DRA handle
integer request [output] ! request id</pre>
Read asynchronously the specified global array from the specified disk
resident array.
<p>Dimensions and type of arrays referred to by handles <i>g_a</i> and
<i>d_a</i>
must match. If dimensions don't match, <a href="#dra_read_section">dra_read_section</a>
could be used instead.
<p>See <a href="#dra_write">dra_write</a> specs for discussion of <i>request
</i>.
<p>
<hr>
<p><a NAME="dra_read_section"></a>
<h2>
dra_read_section</h2>
<pre> status = dra_read_section(transp, g_a, gilo, gihi, gjlo, gjhi,
d_a, dilo, dihi, djlo, djhi, request)
logical transp [input] ! transpose operator
integer g_a [input] ! GA handle
integer d_a [input] ! DRA handle
integer gilo [input]
integer gihi [input]
integer gjlo [input]
integer gjhi [input]
integer dilo [input]
integer dihi [input]
integer djlo [input]
integer djhi [input]
integer request [output] ! request id</pre>
Read asynchronously specified global array section from specified disk
resident array section:
<pre> OP(d_a[ dilo:dihi, djlo:djhi]) --> g_a[ gilo:gihi, gjlo:gjhi]</pre>
where <i>OP </i>is the transpose operator (.<i>true./.false</i>.).
<p>See <a href="#dra_write">dra_write</a> specs for discussion of <i>request</i>
.
<p>
<hr>
<p><a NAME="dra_probe"></a>
<h2>
dra_probe</h2>
<pre> status = dra_probe(request, compl_status)
integer request [input] ! request id
integer compl_status [output] ! completion status</pre>
Tests for completion of <a href="#dra_write">dra_write</a>/<a href="#dra_read">read</a>
or <a href="#dra_write_section">dra_write</a>/<a href="#dra_read_section">read_section</a>
operation which set the value passed in request argument.
<p><i>compl_status .eq. </i>0 means the operation has been completed.
<p><i>compl_status .ne. </i>0 means "not done yet ".
<p>
<hr>
<p><a NAME="dra_wait"></a>
<h2>
dra_wait</h2>
<pre> status = dra_wait(request)
integer request [input] ! request id</pre>
Blocks until completion of <a href="#dra_write">dra_write</a>/<a href="#dra_read">read</a>
or <a href="#dra_write_section">dra_write</a>/<a href="#dra_read_section">read_section</a>
operation which set the value passed in <i>request</i> argument.
<p>
<hr>
<p><a NAME="dra_inquire"></a>
<h2>
dra_inquire</h2>
<pre> status = dra_inquire(d_a, type, dim1, dim2, name, filename)
integer d_a [input] ! DRA handle
integer type [output]
integer dim1 [output]
integer dim2 [output]
character*(*) name [output]
character*(*) filename [output]</pre>
Return <i>dimensions</i>, <i>type</i> , <i>name</i> of disk resident array,
and <i>filename</i> of DRA meta-file associated with <i>d_a</i> handle.
<p>
<hr>
<p><a NAME="dra_delete"></a>
<h2>
dra_delete</h2>
<pre> status = dra_delete(d_a)
integer d_a [input] ! DRA handle</pre>
Delete a disk resident array associated with <i>d_a</i> handle. Invalidate
handle. The corresponding DRA meta-file is destroyed.
<p>
<hr>
<p><a NAME="dra_close"></a>
<h2>
dra_close</h2>
<pre> status = dra_close(d_a)
integer d_a [input] ! DRA handle</pre>
Close DRA meta-file associated with <i>d_a</i> handle and deallocate data
structures corresponding to this disk array. Invalidate <i>d_a</i> handle.
The array on the disk is persistent.
<p>
<hr>
<p><a NAME="dra_flick"></a>
<h2>
dra_flick</h2>
<pre> subroutine dra_flick()</pre>
Returns control to DRA for a VERY short time to improve progress of pending
asynchronous operations.
</body>
</html>
|
