1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
|
Definition of HDF-EOS
Doug Ilg and Larry Klein
Abstract
HDF-EOS is a software library designed to support EOS-specific
data structures, namely Grid, Point, and Swath. The new data structures
are constructed from standard HDF data objects, using EOS conventions,
through the use of a software library [1,4]. A key feature of HDF-EOS files
is that instrument-independent services, such as subsetting by geolocation,
can be applied to the files across a wide variety of data products. The
library is extensible and new data structures can be added.
This document provides a basic specification of an HDF-EOS file, a
generalized description of file contents and a discussion of certain
limitations of the format.
Definitions
HDF-EOS is a software library that is an extension of National Center
for Supercomputing Applications (NCSA) HDF. The library supports the
construction of new data structures: Grid, Point and Swath.
HDF-EOS File:
An HDF-EOS file is any valid HDF file (i.e., any file created by the
NCSA HDF library [1]), that contains a family of global attributes called
"coremetadata.X", where ".X" is a sequence number beginning at 0 and running
as high as 9. Optional data objects which may appear in an HDF-EOS file
include another family of global attributes called "archivemetadata.X" and
any number of Point, Swath, and/or Grid data structures. The existence of
Point, Swath, or Grid structures in an HDF-EOS file implies the existence
of another family of global attributes called "StructMetadata.X", which is
automatically created, populated, and maintained by the HDF-EOS library.
Note that all attribute names are case sensitive.
Core Metadata:
Core metadata represent information to which system-wide services, such as
searching, will be applied. These metadata, which are defined in DID 311 [3],
are contained in the "coremetadata.X" family of global attributes within an
HDF-EOS file. The syntax of these metadata is compliant with the Object
Description Language (ODL) [4]. Tools for formatting, accessing and writing
core metadata are provided in the ECS Science Data Processing (SDP) Toolkit [2].
Archive Metadata:
Archive metadata represent information that, by definition, will not be
searchable. It contains whatever information the file creator considers
necessary to be in file, but to which system-wide services will not be
applied. These metadata are also accessed via SDP Toolkit [2] calls and
are written in ODL [4] syntax into the "archivemetadata.X" family of global
attribtes.
Structural Metadata:
Structural metadata describe the contents and structure of an HDF-EOS file.
These metadata are present in the file only if the HDF-EOS library has been
invoked to create a Grid, Point, or Swath structure. These metadata are stored
in the "StructMetadata.X" family of global attributes and created and maintained
by the HDF-EOS library. These metadata are not intended to be directly accessed
by data producers or users. Therefore, all access to these metadata should be
via appropriate function calls in the HDF-EOS library.
General Limitations
1.) External element implementations of HDF or HDF-EOS files are not supported
by the ECS.
2.) There is an absolute limit of two gigabytes on the size of a single HDF or
HDF-EOS file [1].
3.) Individual HDF and HDF-EOS character attributes are limited to 64 kilobytes
in length. Hence the potential need for "families" of attributes [1].
4.) The HDF-EOS library does not provide tools for creating, populating, or
maintaining the "coremetadata" or "archivemetadata" families of attributes.
Those tools appear in the SDP Toolkit [2, 3].
5.) Any user-supplied names are case sensitive and may contain up to 64
characters. Commas (","), forward slashes ("/"), and colons (":") are the
only invalid characters in names.
Point Structure
Point structures are implemented as a hierarchy of Vgroups containing several
Vdatas. All Vgroups and Vdatas that are part of any Point structure carry the
class "POINT". Each level of data within a Point structure is implemented as
a single Vdata, with each data field being a named field in the Vdata. Hence,
the following limitations apply:
1.) The reserved field names for special purpose geolocation fields are
"Longitude", "Latitude", "Colatitude", and "Time" (case sensitive). These
fields are subject to the following requirements:
Field Name Data Type Format
Longitude float32 or float64 Decimal degrees on the range [-180.0, 180.0]
Latitude float32 or float64 Decimal degrees on the range [-90.0, 90.0]
Colatitude float32 or float64 Decimal degrees on the range [0.0, 180.0]
Time float64 TAI93 (seconds until(-)/since(+) midnight,
1/1/93)
These fields may be one- or two-dimensional.
2.) Fields may only be one-dimensional.
3.) Up to 8 levels may exist in a Point structure.
Swath Structure
Swath structures are implemented as a hierarchy of Vgroups containing a number
of Vdatas and/or SDSs. All Vgroups and Vdatas that are part of any Swath
structure carry the class "SWATH".
Each one-dimensional field is implemented as a named field within its own Vdata.
One-dimensional fields that are the same length, and are specifically allowed
by the calling program, are merged into "communal" Vdatas, with each data field
occupying one field in the Vdata.
Each multi-dimensional field is implemented as an SDS. Three-dimensional fields
which share the same dimensionality, dimension sizes, and data type and which
are specifically allowed by the calling program are merged into communal SDSs
with three dimensions. Two-dimensional arrays are merged as if they were
three-dimensional arrays with a first dimension of size 1. No merging is
performed on fields with more than three dimensions, on fields with an unlimited
dimension, or on compressed fields.
Accordingly, the following limitations apply:
1.) The reserved field names for special purpose geolocation fields are
"Longitude", "Latitude", "Colatitude", and "Time" (case sensitive). These
fields are subject to the following requirements:
Field Name Data Type Format
Longitude float32 or float64 Decimal degrees on the range [-180.0, 180.0]
Latitude float32 or float64 Decimal degrees on the range [-90.0, 90.0
Colatitude float32 or float64 Decimal degrees on the range [0.0, 180.0]
Time float64 TAI93 (seconds until(-)/since(+) midnight,
1/1/93)
These fields may be one- or two-dimensional.
2.) Non-reserved fields may have up to 8 dimensions.
3.) An "unlimited" dimension must be the first dimension (in C-order).
4.) For all multi-dimensional fields in scan- or profile-oriented Swaths, the
dimension representing the "along track" dimension must precede the dimension
representing the scan or profile dimension(s).
5.) Compression is selectable at the field level within a Swath. All
HDF-supported
compression methods are available through the HDF-EOS library. Specifying
compression on a field prevents merging.
Grid Structure
Grid structures are implemented as a hierarchy of Vgroups containing several
SDSs.
All Vgroups that are part of any Grid structure carry the class "GRID". Each
data field within a Grid structure is implemented as a single SDS. Merging is
done the same way for Grid data fields as for multi-dimensional Swath
data fields. Therefore, the following limitations apply:
1.) Fields may have from 2 to 8 dimensions.
2.) Compression is selectable at the field level within a Grid. All
HDF5-supported compression methods are available through the HDF-EOS5
library. Specifying compression on a field prevents merging.
References
1.) Introduction to HDF5; HDF5 User's Guide, Version 1.4.5,
August, 2007; HDF5 Reference Manual, for HDF5 Version 1.6.6, August, 2007;
See the The HDF Group site http://hdf.ncsa.uiuc.edu/HDF5/doc/index.html
for documents on " HDF5 - A New Generation of HDF, The Hierarchical Data
Format"
2.) SDP Toolkit Users Guide for the ECS Project , October, 2007 [333-EMD-001].
Documentation and software are available at http://edhs1.gsfc.nasa.gov/
3.) Release B SDPS Database Design and Database Schema Specifications ,
May 1996 [311-CD-008-001]; Science Data Processing Segment (SDPS) Database
Design and Database Schema Specifications for the ECS Project, December 1995,
[311-CD-002-004].
4.) Object Descriptor Language reference: Planetary Data Sytems Standard Reference,
Version 3.2, July 1995, JPL, California.
For more specific information on the use of the HDF-EOS5 library, please
refer to the HDF-EOS5 User's Guide for the ECS Project , October , 2007
[175-EMD-001, Rev 05].
Referral:
Refer comments to:
Abe Taaheri
EMD Project
Abe_Taaheri@raytheon.com
301-925-0783
|
