1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
|
****
HDF5
****
In ADIOS2, the default engine for reading and writing HDF5 files is called *"HDF5"*.
To use this engine, you can either specify it in your xml config file, with tag ``<engine type=HDF5>``
or, set it in client code. For example, here is how to create a hdf5 reader:
.. code-block:: c++
adios2::IO h5IO = adios.DeclareIO("SomeName");
h5IO.SetEngine("HDF5");
adios2::Engine h5Reader = h5IO.Open(filename, adios2::Mode::Read);
To read back the h5 files generated with VDS to ADIOS2, one can use the HDF5 engine. Please make sure you are using the HDF5 library that has version greater than or equal to 1.11 in ADIOS2.
The h5 file generated by ADIOS2 has two levels of groups: The top Group, ``/`` and its subgroups: ``Step0`` ... ``StepN``, where ``N`` is number of steps. All datasets belong to the subgroups.
Any other h5 file can be read back to ADIOS as well. To be consistent, when reading back to ADIOS2, we assume a default Step0, and all datasets from the original h5 file belong to that subgroup. The full path of a dataset (from the original h5 file) is used when represented in ADIOS2.
We can pass options to HDF5 API from ADIOS xml configuration. Currently we support CollectionIO (default false), and chunk specifications. The chunk specification uses space to seperate values, and by default, if a valid H5ChunkDim exists, it applies to all variables, unless H5ChunkVar is specified. Examples:
.. code-block:: xml
<parameter key="H5CollectiveMPIO" value="yes"/>
<parameter key="H5ChunkDim" value="200 200"/>
<parameter key="H5ChunkVar" value="VarName1 VarName2"/>
We suggest to read HDF5 documentation before appling these options.
After the subfile feature is introduced in HDF5 version 1.14, the ADIOS2 HDF5 engine will use subfiles as the default h5 format as it improves I/O in general (for example, see https://escholarship.org/uc/item/6fs7s3jb)
To use the subfile feature, client needs to support MPI_Init_thread with MPI_THREAD_MULTIPLE.
Useful parameters from the HDF library to tune subfiles are:
.. code-block:: xml
H5FD_SUBFILING_IOC_PER_NODE (num of subfiles per node)
set H5FD_SUBFILING_IOC_PER_NODE to 0 if the regular h5 file is preferred, before using ADIOS2 HDF5 engine.
H5FD_SUBFILING_STRIPE_SIZE
H5FD_IOC_THREAD_POOL_SIZE
|
