1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
|
# Support NetCDF-4 files
Starting from version 1.11.0, PnetCDF supports data access to HDF5-based
NetCDF-4 files. This format is also referred as the `enhanced data model`,
versus the `classic model` (in NetCDF language). The new I/O driver, called the
NetCDF-4 driver, is a wrapper of the NetCDF Library that translates PnetCDF
APIs to their corresponding NetCDF APIs, for example, `ncmpi_create` calls
`nc_create`. Through calling PnetCDF APIs, this feature allows applications to
read/write files in NetCDF-4 format.
## Enable NetCDF-4 support
PnetCDF requires a NetCDF library that is built with parallel HDF5 capability.
Example build instructions for HDF5 and NetCDF-4 are given below.
* To build parallel HDF5 (version 1.10.4 or later is recommended)
+ Source tar ball of HDF5 can be downloaded from URL:
https://www.hdfgroup.org/downloads/hdf5/source-code/
+ Build commands:
```
gzip -dc hdf5-1.10.4.tar.gz | tar -xf -
cd hdf5-1.10.4
./configure --prefix=/HDF5/install/path \
--enable-parallel=yes \
CC=mpicc FC=mpifort CXX=mpicxx
make install
```
* To build NetCDF-4 (version 4.6.2 or later is recommended)
+ Source tar ball of NetCDF-4 can be downloaded from URL:
https://github.com/Unidata/netcdf-c/releases
+ Build commands:
```
gzip -dc v4.6.2.tar.gz | tar -xf -
cd netcdf-c-4.6.2
./configure --prefix=/NetCDF4/install/path \
--enable-netcdf-4 \
CC=mpicc \
CPPFLAGS="-I/HDF5/install/path/include" \
LDFLAGS="-L/HDF5/install/path/lib"
make install
```
+ Note NetCDF-4 library built with PnetCDF feature enabled, i.e. option
`--enable-pnetcf`, is not supported.
* To build PnetCDF with NetCDF-4 support
+ Add `--with-netcdf4` option at the configure command line. Option
`--with-netcdf` can also be used to specify the installation path of
NetCDF-4 if it is not in the standard localtions. For example,
```
gzip -dc pnetcdf-1.11.0.tar.gz | tar -xf -
cd pnetcdf-1.11.0
./configure --prefix=/PnetCDF/install/path \
--with-netcdf4=/NetCDF4/install/path
```
## Accessing NetCDF-4 file
To create a NetCDF-4 file, add the flag NC_NETCDF4 into argument cmode when
calling `ncmpi_create()`. For example,
```
int cmode;
cmode = NC_CLOBBER | NC_NETCDF4;
ncmpi_create(MPI_COMM_WORLD, "testfile.nc", cmode, MPI_INFO_NULL, &ncid);
```
or
```
cmode = NC_CLOBBER | NC_NETCDF4 | NC_CLASSIC_MODEL;
ncmpi_create(MPI_COMM_WORLD, "testfile.nc", cmode, MPI_INFO_NULL, &ncid);
```
NC_NETCDF4 flag is not required when opening an existing file in NetCDF-4
format, as PnetCDF checks the file format and selects the proper I/O driver.
Users can also set the default file format to NetCDF-4 by calling API
`ncmpi_set_default_format()` with argument `NC_FORMAT_NETCDF4C` or
`NC_FORMAT_NETCDF4_CLASSIC`. For example,
```
ncmpi_set_default_format(NC_FORMAT_NETCDF4, &old_formatp);
```
or
```
ncmpi_set_default_format(NC_FORMAT_NETCDF4_CLASSIC, &old_formatp);
```
When no file format specific flag is set in argument cmode, PnetCDF will use
the default setting.
## Known Problems
Some features are not supported due to the availability of APIs different
between PnetCDF and NetCDF libraries. I/O semantics are also slightly
different.
* API families of `vard`, `varn`, and nonblocking I/O are not supported. This
is because NetCDF does not have corresponding APIs. Error code
`NC_ENOTSUPPORT` will be returned. For vard APIs, NetCDF does not have APIs
that allow accessing the file directly by an MPI derived file type. For
`varn` APIs, a potential solution is to split the request into into multiple
vara calls. However, such solution must deal with the situation when the
numbers of requests are different among processes in the collective data
mode. Supporting `varn` is thus a future work.
* Although a new file of format NC_FORMAT_NETCDF4 or NC_FORMAT_NETCDF4_CLASSIC
can be created, the I/O operations are limited to the NetCDF classic model
I/O, This is because PnetCDF APIs do not include those for operating enhanced
data objects, such as groups, compound data types, compression etc. As for
reading a NetCDF-4 file created by NetCDF-4, PnetCDF supports the classic way
of reading variables, attributes, and dimensions. Inquiry APIs for enhanced
metadata is currently not supported.
* Due to a bug in HDF5 version 1.10.2 and prior, collective I/O on record
variables with a subset of participating processes making zero-length
requests may cause an HDF5 error and the program to hang. Readers refer to
the HDF bug issue HDFFV-10501. The bug fix will appear in HDF5 1.10.4 release.
* PnetCDF allows different kinds of APIs called by different processes in a
collective I/O. For example, `ncmpi_put_vara_int_all()` is called at process
0 and `ncmpi_put_vars_float_int()` is called at process 1. However, this is
not allowed in NetCDF-4. Thus, the same kind of API must be used in a
collective call when accessing a NetCDF-4 file.
* When creating a new NetCDF-4 file, PnetCDF only allows creating 1 unlimited
dimension. When reading an existing NetCDF-4 file, the file can have more
than 1 unlimited dimension. However, in this case, API ncmpi_inq_unlimdim
will only return the dimension ID of first unlimited dimension. This behavior
conforms with NetCDF-4 library.
* The following APIs are not supported yet. An error code NC_ENOTSUPPORT will
be returned, if called.
* `ncmpi_inq_header_size`, `ncmpi_inq_header_extent`
* `ncmpi_inq_striping`
* `ncmpi_inq_varoffset`
* `ncmpi_fill_var_rec`
* `ncmpi_sync_numrecs`
* `ncmpi_flush`
* all nonblocking APIs
* vard and varn APIs
* flexible APIs (i.e. argument buftype is a constructed MPI derived data type)
* APIs `ncmpi_inq_get_size` and `ncmpi_inq_put_size` report the amount of data
that has been read or written, but excluding the I/O to the file header. They
are simply the size of data passed between PnetCDF and NetCDF, not the actual
size read from or written to the file system.
Copyright (C) 2018, Northwestern University and Argonne National Laboratory.
See COPYRIGHT notice in top-level directory.
|
