1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294
|
<!--Copyright (C) 1988-2005 by the Institute of Global Environment and Society (IGES). See file COPYRIGHT for more information.-->
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<title>How to Generate NetCDF Descriptor Files</title>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<link href="GrADS.css" rel="stylesheet" type="text/css">
<style type="text/css">
<!--
.style1 {color: #990000}
-->
</style>
</head>
<body bgcolor="e0f0ff">
<p class="item16"><b>Reading NetCDF and HDF Files with GrADS</b></p>
<p class="plaintext">Data files in the NetCDF and HDF file formats are called
self-describing files (SDF) because the data and metadata are packaged together
in the same file. GrADS can read data NetCDF and HDF formatted files, as long as the data are on a regular grid. The HDF format is very general; the GrADS interface is limited to gridded data sets that fit into the internal 5-D lon/lat/lev/time/ensemble grid space. GrADS handles HDF4 Scientific Data Sets and (as of <span class="style1">version 2.0.a7</span>) some HDF5 files. In order to read the data in SDFs, GrADS needs a certain amount of metadata in order to place the data
in the internal grid space. There are three ways
to do this: </p>
<ol>
<li class="plaintext">Use the <a href="gradcomdsdfopen.html"><code>sdfopen</code></a>
command to open the file. This requires the least amount of effort for the
user -- simply provide the file name (or an OPeNDAP URL) and GrADS does the
rest. If you use the sdfopen command to open your SDF, then all the metadata
in the file that GrADS requires must conform to the
<a href="http://ferret.wrc.noaa.gov/noaa_coop/coop_cdf_profile.html" target="_parent">COARDS
conventions</a>. The 'sdfopen' interface does not support the HDF5 format. If sdfopen doesn't work, then ...<br>
<br>
</li>
<li class="plaintext">Use the <code><a href="gradcomdxdfopen.html">xdfopen</a></code>
command to open the file. This requires a bit more effort for the user --
you must write a data descriptor file to supplement or replace the existing
metadata so that GrADS can understand it. The syntax of the descriptor file
used with xdfopen is not exactly the same as that used in a descriptor file
for gridded binary data -- see the <a href="gradcomdxdfopen.html">documentation
page</a> for further details. The xdfopen command provides access to a greater
number of SDFs, including many that do not conform to any known standard.
The 'xdfopen' interface does not support the HDF5 format. If xdfopen doesn't work, then ... <br>
<br>
</li>
<li class="plaintext">Use the <code><a href="gradcomdopen.html">open</a></code>
command to open the file. This requires the user to write a complete GrADS
descriptor file to override all the metadata in the file. Guidance for composing
a complete descriptor file for NetCDF, HDF-SDS, or HDF5 gridded data files is given below.
Please also see the reference page <a href="descriptorfile.html">Elements
of a Data Descriptor File</a>. The 'open' interface is recommended if you
are templating large numbers of data files together, the data are pre-projected
onto a non-lat/lon grid, the variables in the file have different undefined
values, or the variables in the file have been packed in a non-standard way. The 'open' interface is the only way to read HDF5 files. </li>
</ol>
<span class="item16"><b>NetCDF and HDF-SDS Descriptor File Components</b> </span>
<p class="plaintext">The data descriptor file is free format, which means the
components of each record (line of text) are blank delimited and can appear
in any order. Leading blanks at the beginning of each record are removed before
parsing. Individual records may not be more than 255 characters long. Each record
begins with a specific entry name, followed by a number of arguments or keywords,
depending on the entry.</p>
<p class="plaintext">Descriptor file entries used for NetCDF, HDF-SDS, and HDF5 files
are: </p>
<table width="700" cellpadding="2" cellspacing="5" class="plaintext">
<tr bgcolor="#b8d8dd">
<td width="76" class="plaintext">DSET</td>
<td width="609" class="plaintext">This entry points to the data file. See
the <a href="descriptorfile.html#DSET">reference page</a> for more details.</td>
</tr>
<tr bgcolor="#b8d8dd">
<td class="plaintext">DTYPE </td>
<td class="plaintext">This entry should have either the 'netcdf' or 'hdfsds' keywords. <br>
(<span class="style1">GrADS version 2.0.a7+</span>) For HDF5, use the 'hdf5_grid' keyword.</td>
</tr>
<tr bgcolor="#b8d8dd">
<td class="plaintext">TITLE</td>
<td class="plaintext">It is good general practice to include a descriptive
title in every GrADS descriptor file. </td>
</tr>
<tr bgcolor="#b8d8dd">
<td class="plaintext">UNDEF</td>
<td><p class="plaintext">This entry specifies the undefined or missing data
value. An optional second argument is the name of the attribute in the
SDF that contains the undefined value. This should be used when individual
variables in the data file have different undefined values. After data
I/O, the missing values in the grid are converted from the variable undef
to the file-wide undef (the numerical value in the first argument of the
UNDEF record). Then it appears to GrADS that all variables have the same
undef value, even if they don't in the SDF. Attribute names are case sensitive,
and it is assumed that the name is identical for all variables in the
SDF. If the name given does not match any attributes, or if no name is
given, the file-wide undef value will be used. <br>
Example: UNDEF -9.99e8 _FillValue</p></td>
</tr>
<tr bgcolor="#b8d8dd">
<td> <span class="plaintext">UNPACK</span></td>
<td >This
entry is used for data variables that are 'packed' -- i.e. non-float data
that need to be converted to float by applying the following formula: <br>
y = x * <em>scale_factor</em>
+ <em>add_offset</em><br>
Only the attribute name for the scale factor is required. If your SDF does
not have an offset attribute, the 2nd argument may be omitted, and the offset
will be assigned the default value of 0.0. Attribute names are case sensitive,
and it is assumed that the names are identical for all variables in the
netcdf or hdfsds data file. If the names given do not match any attributes,
the scale factor will be assigned a value of 1.0 and the offset will be
assigned a value of 0.0. The transformation of packed data is done after
the undef test has been applied. <br>
Examples: <br>
UNPACK scale_factor add_offset<br>
UNPACK Slope Intercept</td>
</tr>
<tr bgcolor="#b8d8dd">
<td class="plaintext"> OPTIONS</td>
<td class="plaintext">Valid keywords are 'yrev', 'zrev', 'template', and '365_day_calendar'.</td>
</tr>
<tr bgcolor="#b8d8dd">
<td class="plaintext">CACHESIZE</td>
<td class="plaintext">(<font color="#990000">GrADS version 2.0.a8+</font>)
This entry overrides the default size of the cache for reading HDF5 or NetCDF4 files. It is not relevant for other data types. It should not be necessary to set the cache size explicitly unless the data file has especially large chunks. Please see the documentation on <a href="compression.html">compression</a>. </td>
</tr>
<tr bgcolor="#b8d8dd">
<td class="plaintext">PDEF</td>
<td class="plaintext"> (<font color="#990000">GrADS version 1.9b4+</font>)
This is used when the SDF contains data on a native projection other than
lat/lon, such as a lambert conformal or polar stereographic grid. See the
<a href="pdef.html">PDEF documentation</a> for more information.</td>
</tr>
<tr bgcolor="#b8d8dd">
<td class="plaintext"> XDEF<br>
YDEF<br>
ZDEF<br>
TDEF<br>
EDEF</td>
<td class="plaintext">These entries are used to describe the coordinate dimensions
in the SDF. The syntax is the same as for binary files. See the <a href="descriptorfile.html#DSET">reference
page</a> for more details. You can use the output from ncdump with the -c
option to get information about the coordinate dimensions in the SDF. </td>
</tr>
<tr bgcolor="#b8d8dd">
<td class="plaintext"> VECTORPAIRS</td>
<td> <p class="plaintext">This entry is for explicity identifying vector component
pairs. The VECTORPAIRS entry is only necessary if the data are on a native
projection other than lat/lon (i.e. you are using <a href="pdef.html">PDEF</a>)
and if the winds have to be <a href="pdef.html#rotation">rotated</a> from
a grid-relative sense to an Earth-relative sense. (GrADS has to retrieve
both the u and v component in order to do the rotation calculation.)</p>
<p class="plaintext"> The arguments are the <i>U-component</i> and <i>V-component
</i> variable names, separated by a comma, with no spaces. More than one
pair of components may be listed; in this case, the pairs should be separated
by a space. <br>
Example:<br>
VECTORPAIRS u,v u10,v10 uflx,vflx</p></td>
</tr>
<tr bgcolor="#b8d8dd">
<td class="plaintext"> VARS<br>
through <br>
ENDVARS</td>
<td><p class="plaintext">The variable declarations in a SDF descriptor file
have a few special features, described below. It is not necessary to include
a variable declaration for all the variables in the SDF, only those you
wish to read with GrADS.</p>
<p class="plaintext">The <em>varname </em>field has the following syntax:
<br>
<em>SDF_name</em>=><em>grads_name</em> <br>
<em>SDF_name</em> must exactly match the data variable name in the SDF
-- it may contain uppercase letters and non-alpha-numeric characters.
The <em>grads_name</em> is an alias for <em>SDF_name</em> and must be
less than 16 characters, start with an alphabetic character, and cannot
contain any upper case letters or non-alpha-numeric characters. The aliasing
of variable names may be omitted (i.e., "<em>SDF_name</em>=>"
does not precede <em>grads_name</em>) if the <em>SDF_name</em> already
meets the criteria for GrADS variable names listed above. For dtype hdf5_grid, the <em>SDF_name</em> must contain the names of all the nested groups (separated by "/") to which the data set belongs (see example below).</p>
<p class="plaintext">The<em> levs</em> field is an integer that specifies
the number of vertical levels the variable contains. Variables that do
not have a Z dimension should have a <i>levs</i> value of 0. Variables
that do have a Z dimension should have a <em>levs</em> value equal to
the <em>znum</em> value specified in the ZDEF statement.</p>
<p class="plaintext">The <em>units</em> field is a comma-delimited list
of the varying dimensions of the variable. The dimensions are expressed
as x, y, z, t, and e and correspond to the five axes defined by XDEF, YDEF,
ZDEF, TDEF, and EDEF. The order of the dimensions listed in the <em>units</em>
field is important -- it must describe the shape of the variable as it
was written to the SDF data file. For NetCDf files, this information appears
in the output from ncdump next to the variable name. For HDF5 files, this information appears in the output from h5dump as the variable's dataspace.</p>
<p class="plaintext">Examples:<br>
Height=>hgt 17 t,z,y,x Geopotential
Height (m) <br>
/HDFEOS/GRIDS/ColumnAmountNO2/Data~Fields/CloudFraction=>cf 15 z,y,x Cloud Fraction <br>
</td>
</tr>
</table>
<p class="item16"><strong>Usage Notes</strong> </p>
<ol>
<li class="plaintext">The NetCDF data types that GrADS currently handles are
short, long, and float. The HDF-SDS data types that are handled are 8-bit
ints (int8 and uint8), shorts (int16 and uint16), ints (int32 and uint32)
and float. These are all converted to type float after the I/O is done.<br>
<br>
</li>
<li class="plaintext"> The <a href="gradcomdsdfopen.html">sdfopen</a>/<a href="gradcomdxdfopen.html">xdfopen</a>
interface will automatically handle the unpacking of NetCDF data if the following
conditions are met:<br>
a. The packed data type is "short" <br>
b. The constants used for the transformation are data type
"float" <br>
c. The attribute names are "scale_factor" or "slope"
and "add_offset" or "intercept"<br>
If the packed data in your SDF does not fit this description, then you must
use the <a href="gradcomdopen.html">open</a> command with a complete descriptor
file, providing the attribute names in the <a href="#unpack">UNPACK</a> entry.
In this case, the attribute data type may be short, long, float, or double.<br>
<br>
</li>
<li class="plaintext">If the data in the SDF are not floating-point numbers
and require a transformation using the attributes named in the <a href="#unpack">UNPACK</a>
entry, GrADS assumes the variable undef value corresponds to the data values
as they appear in the file, i.e., <em>before</em> they are transformed using
a scale factor and offset. Missing packed data values are assigned the file-wide
undef value and are never unpacked. <br>
<br>
</li>
<li class="plaintext">If your data file contains a variable that varies in a non-world-coordinate
dimension (e.g. histogram interval, spectral band, ensemble number) then you
can put a non-negative integer in the list of varying dimensions that will become
the array index of the extra dimension. For example:
<p class="plaintext"> VAR=>hist0 0 0,y,x
First historgram interval for VAR<br>
VAR=>hist1 0 1,y,x Second historgram
interval for VAR<br>
VAR=>hist2 0 2,y,x Third histogram interval
for VAR </p>
<p class="plaintext">Another option in this example would be to fill the unused
Z axis with the histogram intervals: </p>
<p class="plaintext"> ZDEF 3 linear 1 1<br>
... <br>
VAR=>hist 3 z,y,x VAR Histogram</p>
<p class="plaintext">In this case, it would appear to GrADS that variable 'hist'
varies in Z, but the user would have to remember that the Z levels correspond
to histogram intervals and not pressure levels. The latter technique makes it
easier to slice through the data, but is not the most accurate representation.
And if you don't have an unsued world-coordinate axis available, then you still
have a way to access all the dimensions of your data<em> </em>variable. </p>
</li>
<li class="plaintext">Some SDFs have many more than four coordinate dimensions
-- staggered longitude and latitude axes are one example. In this case, it
is likely that there will be variables defined on different grids contained
in the same SDF. GrADS can only handle one 4D grid per data file -- all the
SDF variables listed in a descriptor file must share the same coordinate axes.
Multiple descriptor files must be written to describe the varibles defined
on different grids.<br>
<br>
</li></ol>
<p class="item16"><strong>Examples</strong></p>
<ol>
<li class="plaintext"> Here is a <a href="sample.ncdump">sample output from
ncdump</a> for a file containing ocean model output. This file contains eight
coordinate dimensions and nine data variables, which are defined on different
combinations of coordinate axes. Five separate descriptor files are required
to describe all the variables: one for the velocity components <a href="sample_uv.ctl">u
and v</a>, another for velocity component <a href="sample_w.ctl">w</a>, a
third for <a href="sample_temp.ctl">potential temperature</a>, a fourth for
wind stress components <a href="sample_tau.ctl">taux and tauy</a>, and a fifth
for surface variables<a href="sample_sfc.ctl"> hflx, sflx, and eta</a>.<br>
<br>
</li>
<li class="plaintext"> The <a href="http://www.wrf-model.org/" target="_parent">Weather Research
and Forecasting (WRF) Model</a> can generate NetCDF output on non-lat/lon
grids. GrADS can read these files in their native format using a complete
descriptor file with a <a href="pdef.html">PDEF</a> entry. To extract the
arguments for the PDEF entry, you can use the global attribute values, which
describe the native grid parameters, as well as the data variables which provide
the grid point lat/lon values. The WRF model uses staggered grids, just as
the ocean model does in the example above. For the sake of clarity, this <a href="wrf.ncdump">WRF
ncdump output</a> has been edited to show only the four coordinate axes that
are relevant for the data variables used in the example descriptor files.
There are many, many more data variables and coordinate dimensions in the
actual output files. First, here is a sample descriptor file to get the native
<a href="wrfgrid.ctl">grid point longitude and latitude</a> values -- note
that no PDEF statement is included and the XDEF and YDEF statements do not
map to longitude and latitude, they are simply used as abstract grid increments.
Any one of these grid points may be used as the reference point in the PDEF
entry, this example uses grid point (1,1) with values (-125.898, 26.9628).
Finally, here is the descriptor file for <a href="wrfvars.ctl">four data variables</a>.
The WRF model is highly configurable, and also under active development, so
this example should be used only as a guideline. </li>
</ol>
</body>
</html>
|