File: SDFdescriptorfile.html

package info (click to toggle)
grads 3%3A2.2.1-4
  • links: PTS, VCS
  • area: main
  • in suites: bullseye
  • size: 17,336 kB
  • sloc: ansic: 61,642; sh: 10,612; makefile: 201; python: 3
file content (294 lines) | stat: -rw-r--r-- 18,433 bytes parent folder | download | duplicates (7)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
<!--Copyright (C) 1988-2005 by the Institute of Global Environment and Society (IGES). See file COPYRIGHT for more information.-->

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<title>How to Generate NetCDF Descriptor Files</title>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<link href="GrADS.css" rel="stylesheet" type="text/css">
<style type="text/css">
<!--
.style1 {color: #990000}
-->
</style>
</head>

<body bgcolor="e0f0ff">
<p class="item16"><b>Reading NetCDF and HDF Files with GrADS</b></p>
<p class="plaintext">Data files in the NetCDF and HDF file formats are called 
  self-describing files (SDF) because the data and metadata are packaged together 
  in the same file. GrADS can read data NetCDF and HDF formatted files, as long as the data are on a regular grid. The HDF format is very general; the GrADS interface is limited to gridded data sets that fit into the internal 5-D  lon/lat/lev/time/ensemble grid space. GrADS handles HDF4  Scientific Data Sets and (as of <span class="style1">version 2.0.a7</span>)  some HDF5 files. In order to read the data in SDFs, GrADS needs a certain amount of  metadata in order to place the data 
  in the internal grid space. There are three ways 
  to do this: </p>
<ol>
  <li class="plaintext">Use the <a href="gradcomdsdfopen.html"><code>sdfopen</code></a> 
    command to open the file. This requires the least amount of effort for the 
    user -- simply provide the file name (or an OPeNDAP URL) and GrADS does the 
    rest. If you use the sdfopen command to open your SDF, then all the metadata 
    in the file that GrADS requires must conform to the 
	<a href="http://ferret.wrc.noaa.gov/noaa_coop/coop_cdf_profile.html" target="_parent">COARDS 
    conventions</a>. The 'sdfopen' interface does not support the HDF5 format. If sdfopen doesn't work, then ...<br>
    <br>
  </li>
  <li class="plaintext">Use the <code><a href="gradcomdxdfopen.html">xdfopen</a></code> 
    command to open the file. This requires a bit more effort for the user -- 
    you must write a data descriptor file to supplement or replace the existing 
    metadata so that GrADS can understand it. The syntax of the descriptor file 
    used with xdfopen is not exactly the same as that used in a descriptor file 
    for gridded binary data -- see the <a href="gradcomdxdfopen.html">documentation 
    page</a> for further details. The xdfopen command provides access to a greater 
    number of SDFs, including many that do not conform to any known standard. 
    The 'xdfopen' interface does not support the HDF5 format. If xdfopen doesn't work, then ... <br>
    <br>
  </li>
  <li class="plaintext">Use the <code><a href="gradcomdopen.html">open</a></code> 
    command to open the file. This requires the user to write a complete GrADS 
    descriptor file to override all the metadata in the file. Guidance for composing 
    a complete descriptor file for NetCDF, HDF-SDS, or HDF5 gridded data files is given below. 
    Please also see the reference page <a href="descriptorfile.html">Elements 
    of a Data Descriptor File</a>. The 'open' interface is recommended if you 
    are templating large numbers of data files together, the data are pre-projected 
    onto a non-lat/lon grid, the variables in the file have different undefined 
    values, or the variables in the file have been packed in a non-standard way. The 'open' interface is the only way to read HDF5 files. </li>
 </ol>
 
<span class="item16"><b>NetCDF and HDF-SDS Descriptor File Components</b> </span>
<p class="plaintext">The data descriptor file is free format, which means the 
  components of each record (line of text) are blank delimited and can appear 
  in any order. Leading blanks at the beginning of each record are removed before 
  parsing. Individual records may not be more than 255 characters long. Each record 
  begins with a specific entry name, followed by a number of arguments or keywords, 
  depending on the entry.</p>
<p class="plaintext">Descriptor file entries used for NetCDF, HDF-SDS, and HDF5 files 
  are: </p>
<table width="700" cellpadding="2" cellspacing="5" class="plaintext">
  <tr bgcolor="#b8d8dd"> 
    <td width="76" class="plaintext">DSET</td>
    <td width="609" class="plaintext">This entry points to the data file. See 
      the <a href="descriptorfile.html#DSET">reference page</a> for more details.</td>
  </tr>
  <tr bgcolor="#b8d8dd"> 
    <td class="plaintext">DTYPE </td>
    <td class="plaintext">This entry should have either the 'netcdf' or 'hdfsds' keywords. <br>
      (<span class="style1">GrADS version 2.0.a7+</span>) For HDF5, use the 'hdf5_grid' keyword.</td>
   </tr>
  <tr bgcolor="#b8d8dd"> 
    <td class="plaintext">TITLE</td>
    <td class="plaintext">It is good general practice to include a descriptive 
      title in every GrADS descriptor file. </td>
  </tr>
  <tr bgcolor="#b8d8dd"> 
    <td class="plaintext">UNDEF</td>
    <td><p class="plaintext">This entry specifies the undefined or missing data 
        value. An optional second argument is the name of the attribute in the 
        SDF that contains the undefined value. This should be used when individual 
        variables in the data file have different undefined values. After data 
        I/O, the missing values in the grid are converted from the variable undef 
        to the file-wide undef (the numerical value in the first argument of the 
        UNDEF record). Then it appears to GrADS that all variables have the same 
        undef value, even if they don't in the SDF. Attribute names are case sensitive, 
        and it is assumed that the name is identical for all variables in the 
        SDF. If the name given does not match any attributes, or if no name is 
        given, the file-wide undef value will be used. <br>
        Example: UNDEF -9.99e8 _FillValue</p></td>
  </tr>
  <tr bgcolor="#b8d8dd"> 
    <td> <span class="plaintext">UNPACK</span></td>
    <td >This 
      entry is used for data variables that are 'packed' -- i.e. non-float data 
      that need to be converted to float by applying the following formula: <br> 
	  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;y = x * <em>scale_factor</em> 
      + <em>add_offset</em><br>
      Only the attribute name for the scale factor is required. If your SDF does 
      not have an offset attribute, the 2nd argument may be omitted, and the offset 
      will be assigned the default value of 0.0. Attribute names are case sensitive, 
      and it is assumed that the names are identical for all variables in the 
      netcdf or hdfsds data file. If the names given do not match any attributes, 
      the scale factor will be assigned a value of 1.0 and the offset will be 
      assigned a value of 0.0. The transformation of packed data is done after 
      the undef test has been applied. <br>
      Examples: <br>
      UNPACK scale_factor add_offset<br>
      UNPACK Slope Intercept</td>
  </tr>
  <tr bgcolor="#b8d8dd"> 
    <td class="plaintext"> OPTIONS</td>
    <td class="plaintext">Valid keywords are 'yrev', 'zrev', 'template', and '365_day_calendar'.</td>
  </tr>
  <tr bgcolor="#b8d8dd">
    <td class="plaintext">CACHESIZE</td>
    <td class="plaintext">(<font color="#990000">GrADS version 2.0.a8+</font>) 
    This entry overrides the default size of the cache for reading HDF5 or NetCDF4 files. It is not relevant for other data types. It should not be necessary to set the cache size explicitly unless the data file has especially large chunks. Please see the documentation on <a href="compression.html">compression</a>. </td>
  </tr>
  <tr bgcolor="#b8d8dd"> 
    <td class="plaintext">PDEF</td>
    <td class="plaintext"> (<font color="#990000">GrADS version 1.9b4+</font>) 
      This is used when the SDF contains data on a native projection other than 
      lat/lon, such as a lambert conformal or polar stereographic grid. See the 
      <a href="pdef.html">PDEF documentation</a> for more information.</td>
  </tr>
  <tr bgcolor="#b8d8dd"> 
    <td class="plaintext"> XDEF<br>
      YDEF<br>
      ZDEF<br>
      TDEF<br>
      EDEF</td>
    <td class="plaintext">These entries are used to describe the coordinate dimensions 
      in the SDF. The syntax is the same as for binary files. See the <a href="descriptorfile.html#DSET">reference 
      page</a> for more details. You can use the output from ncdump with the -c 
      option to get information about the coordinate dimensions in the SDF. </td>
  </tr>
  <tr bgcolor="#b8d8dd"> 
    <td class="plaintext"> VECTORPAIRS</td>
    <td> <p class="plaintext">This entry is for explicity identifying vector component 
        pairs. The VECTORPAIRS entry is only necessary if the data are on a native 
        projection other than lat/lon (i.e. you are using <a href="pdef.html">PDEF</a>) 
        and if the winds have to be <a href="pdef.html#rotation">rotated</a> from 
        a grid-relative sense to an Earth-relative sense. (GrADS has to retrieve 
        both the u and v component in order to do the rotation calculation.)</p>
      <p class="plaintext"> The arguments are the <i>U-component</i> and <i>V-component 
        </i> variable names, separated by a comma, with no spaces. More than one 
        pair of components may be listed; in this case, the pairs should be separated 
        by a space. <br>
        Example:<br>
        VECTORPAIRS &nbsp;u,v &nbsp;u10,v10 &nbsp;uflx,vflx</p></td>
  </tr>
  <tr bgcolor="#b8d8dd"> 
    <td class="plaintext"> VARS<br>
      through <br>
      ENDVARS</td>
    <td><p class="plaintext">The variable declarations in a SDF descriptor file 
        have a few special features, described below. It is not necessary to include 
        a variable declaration for all the variables in the SDF, only those you 
        wish to read with GrADS.</p>
      <p class="plaintext">The <em>varname </em>field has the following syntax: 
        <br>
        &nbsp;&nbsp;&nbsp;&nbsp;<em>SDF_name</em>=&gt;<em>grads_name</em> <br>
        <em>SDF_name</em> must exactly match the data variable name in the SDF 
        -- it may contain uppercase letters and non-alpha-numeric characters. 
        The <em>grads_name</em> is an alias for <em>SDF_name</em> and must be 
        less than 16 characters, start with an alphabetic character, and cannot 
        contain any upper case letters or non-alpha-numeric characters. The aliasing 
        of variable names may be omitted (i.e., &quot;<em>SDF_name</em>=&gt;&quot; 
        does not precede <em>grads_name</em>) if the <em>SDF_name</em> already 
        meets the criteria for GrADS variable names listed above. For dtype hdf5_grid, the <em>SDF_name</em> must contain the names of all the nested groups (separated by &quot;/&quot;) to which the data set belongs (see example below).</p>
      <p class="plaintext">The<em> levs</em> field is an integer that specifies 
        the number of vertical levels the variable contains. Variables that do 
        not have a Z dimension should have a <i>levs</i> value of 0. Variables 
        that do have a Z dimension should have a <em>levs</em> value equal to 
        the <em>znum</em> value specified in the ZDEF statement.</p>
      <p class="plaintext">The <em>units</em> field is a comma-delimited list 
        of the varying dimensions of the variable. The dimensions are expressed 
        as x, y, z, t, and e and correspond to the five axes defined by XDEF, YDEF, 
        ZDEF, TDEF, and EDEF. The order of the dimensions listed in the <em>units</em> 
        field is important -- it must describe the shape of the variable as it 
        was written to the SDF data file. For NetCDf files, this information appears 
        in the output from ncdump next to the variable name. For HDF5 files, this information appears in the output from h5dump as the variable's dataspace.</p>
      <p class="plaintext">Examples:<br>
        Height=&gt;hgt&nbsp;&nbsp; 17&nbsp;&nbsp; t,z,y,x&nbsp;&nbsp; Geopotential 
        Height (m) <br>
        /HDFEOS/GRIDS/ColumnAmountNO2/Data~Fields/CloudFraction=&gt;cf &nbsp;15 &nbsp;z,y,x &nbsp;Cloud Fraction <br>
    </td>
  </tr>
</table>

    
<p class="item16"><strong>Usage Notes</strong> </p>
<ol>
  <li class="plaintext">The NetCDF data types that GrADS currently handles are 
    short, long, and float. The HDF-SDS data types that are handled are 8-bit 
    ints (int8 and uint8), shorts (int16 and uint16), ints (int32 and uint32) 
    and float. These are all converted to type float after the I/O is done.<br>
    <br>
  </li>
  <li class="plaintext"> The <a href="gradcomdsdfopen.html">sdfopen</a>/<a href="gradcomdxdfopen.html">xdfopen</a> 
    interface will automatically handle the unpacking of NetCDF data if the following 
    conditions are met:<br>
    &nbsp;&nbsp;&nbsp;a. The packed data type is &quot;short&quot; <br>
    &nbsp;&nbsp;&nbsp;b. The constants used for the transformation are data type 
    &quot;float&quot; <br>
    &nbsp;&nbsp;&nbsp;c. The attribute names are &quot;scale_factor&quot; or &quot;slope&quot; 
    and &quot;add_offset&quot; or &quot;intercept&quot;<br>
    If the packed data in your SDF does not fit this description, then you must 
    use the <a href="gradcomdopen.html">open</a> command with a complete descriptor 
    file, providing the attribute names in the <a href="#unpack">UNPACK</a> entry. 
    In this case, the attribute data type may be short, long, float, or double.<br>
    <br>
  </li>
  <li class="plaintext">If the data in the SDF are not floating-point numbers 
    and require a transformation using the attributes named in the <a href="#unpack">UNPACK</a> 
    entry, GrADS assumes the variable undef value corresponds to the data values 
    as they appear in the file, i.e., <em>before</em> they are transformed using 
    a scale factor and offset. Missing packed data values are assigned the file-wide 
    undef value and are never unpacked. <br>
    <br>
  </li>
 
  <li class="plaintext">If your data file contains a variable that varies in a non-world-coordinate 
  dimension (e.g. histogram interval, spectral band, ensemble number) then you 
  can put a non-negative integer in the list of varying dimensions that will become 
  the array index of the extra dimension. For example: 
<p class="plaintext"> VAR=&gt;hist0&nbsp;&nbsp; 0&nbsp;&nbsp; 0,y,x&nbsp;&nbsp; 
  First historgram interval for VAR<br>
  VAR=&gt;hist1&nbsp;&nbsp; 0&nbsp;&nbsp; 1,y,x&nbsp;&nbsp; Second historgram 
  interval for VAR<br>
  VAR=&gt;hist2&nbsp;&nbsp; 0&nbsp;&nbsp; 2,y,x&nbsp;&nbsp; Third histogram interval 
  for VAR </p>
<p class="plaintext">Another option in this example would be to fill the unused 
  Z axis with the histogram intervals: </p>
<p class="plaintext"> ZDEF 3 linear 1 1<br>
  ... <br>
  VAR=&gt;hist&nbsp;&nbsp; 3&nbsp;&nbsp; z,y,x&nbsp;&nbsp; VAR Histogram</p>
<p class="plaintext">In this case, it would appear to GrADS that variable 'hist' 
  varies in Z, but the user would have to remember that the Z levels correspond 
  to histogram intervals and not pressure levels. The latter technique makes it 
  easier to slice through the data, but is not the most accurate representation. 
  And if you don't have an unsued world-coordinate axis available, then you still 
  have a way to access all the dimensions of your data<em> </em>variable. </p>
</li>
 <li class="plaintext">Some SDFs have many more than four coordinate dimensions 
    -- staggered longitude and latitude axes are one example. In this case, it 
    is likely that there will be variables defined on different grids contained 
    in the same SDF. GrADS can only handle one 4D grid per data file -- all the 
    SDF variables listed in a descriptor file must share the same coordinate axes. 
    Multiple descriptor files must be written to describe the varibles defined 
    on different grids.<br>
    <br>
  </li></ol>

<p class="item16"><strong>Examples</strong></p>
<ol>
  <li class="plaintext"> Here is a <a href="sample.ncdump">sample output from 
    ncdump</a> for a file containing ocean model output. This file contains eight 
    coordinate dimensions and nine data variables, which are defined on different 
    combinations of coordinate axes. Five separate descriptor files are required 
    to describe all the variables: one for the velocity components <a href="sample_uv.ctl">u 
    and v</a>, another for velocity component <a href="sample_w.ctl">w</a>, a 
    third for <a href="sample_temp.ctl">potential temperature</a>, a fourth for 
    wind stress components <a href="sample_tau.ctl">taux and tauy</a>, and a fifth 
    for surface variables<a href="sample_sfc.ctl"> hflx, sflx, and eta</a>.<br>
    <br>
  </li>
  <li class="plaintext"> The <a href="http://www.wrf-model.org/" target="_parent">Weather Research 
    and Forecasting (WRF) Model</a> can generate NetCDF output on non-lat/lon 
    grids. GrADS can read these files in their native format using a complete 
    descriptor file with a <a href="pdef.html">PDEF</a> entry. To extract the 
    arguments for the PDEF entry, you can use the global attribute values, which 
    describe the native grid parameters, as well as the data variables which provide 
    the grid point lat/lon values. The WRF model uses staggered grids, just as 
    the ocean model does in the example above. For the sake of clarity, this <a href="wrf.ncdump">WRF 
    ncdump output</a> has been edited to show only the four coordinate axes that 
    are relevant for the data variables used in the example descriptor files. 
    There are many, many more data variables and coordinate dimensions in the 
    actual output files. First, here is a sample descriptor file to get the native 
    <a href="wrfgrid.ctl">grid point longitude and latitude</a> values -- note 
    that no PDEF statement is included and the XDEF and YDEF statements do not 
    map to longitude and latitude, they are simply used as abstract grid increments. 
    Any one of these grid points may be used as the reference point in the PDEF 
    entry, this example uses grid point (1,1) with values (-125.898, 26.9628). 
    Finally, here is the descriptor file for <a href="wrfvars.ctl">four data variables</a>. 
    The WRF model is highly configurable, and also under active development, so 
    this example should be used only as a guideline. </li>
</ol>
</body>
</html>