1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
|
STAGE WRITE CODE
This code can be used to perform your disk I/O after sending the data to a staging area.
It can also be used to read in your file with a certain number of processors with a new decomposition and write out the data with the new layout.
BUILD
=====
Edit the Makefile for your needs. This example is not built automatically with ADIOS. You need to install ADIOS first, then use the installation to build this code.
The read method in this code is not set by argument but is in the source. Set it for your target usage. See variable "read_method", set around line 32.
USAGE
=====
The 'job' file provides an example how to launch 3 applications within one job. Besides your own app, first you have to prepare and start the staging server (DataSpaces in the job script). Then you can run both your app and this stage writer code.
Parameters to stage write:
E.g. genarray.bp staged.bp MPI \"\" $READPROC 1 1
input Input stream path
output Output file path
method ADIOS method to write with
params Write method parameters (in quotes; comma-separated list)
<decomposition> list of numbers e.g. 32 8 4
Decomposition values in each dimension of an array
The product of these number must be less then the number
of processes. Processes whose rank is higher than the
product, will not read/write anything.
Arrays with less dimensions than the number of values,
will be decomposed with using the numbers from the beginning.
NOTES
=====
1. The job example is set for the ./genarray_stream.F90 writing application, which is not built automatically. Use the Makefile.genarray_stream to build it.
WARNING: the method used in the genarray_stream.xml should match the reading method in stage_write.c. That is, if you use a file method (e.g. MPI) to write to a file, the stage_write should use the BP reading method to be able to process it. Then you don't need to run the dataspaces server in the job either. Modify both places to use DataSpaces for staging.
2. The writing application must append new steps into the same file (name). A stream in ADIOS staging cannot consist of multiple file names but of a single "file" with multiple steps in it. E.g. we had to modify the tests/genarray/genarray.F90 to append to a file named "genarray.bp". See ./genarray_stream.F90, which is not built automatically.
3. You probably end up modifying this code. For instance, you might find that the desired decomposition cannot be expressed (e.g. you have arrays with various dimensions and you want decomposition different from them), or you want to pass the output file name within the stream dataset itself, or you want to do something with the data before writing it out.
|
