1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
|
Aggregation special cases
=============================
As we have already seen in this `page <https://fsspec.github.io/kerchunk/test_example.html#multi-file-jsons>`_,
that the main purpose of ``kerchunk`` it to generate references, to view whole archive
of files like GRIB2, NetCDF etc, allowing us for direct access to the data. In
this part of the documentation, we will see some other efficient ways of
combining references.
GRIB Aggregations
-----------------
This reference aggregation method of GRIB files, developed by `Camus Energy <https://www.camus.energy>`_,
and it functions if accompanying ``.idx`` files are present. It involves creating a reference index
for every GRIB message across the files that we want to aggregate.
**But this procedure has some certain restrictions:**
- GRIB files must paired with their ``.idx`` files
- The ``.idx`` file must be of *text* type.
- Only specialised for time-series data, where GRIB files
have *identical* structure.
- Each horizon(forecast time) must be indexed separately.
Utilizing this method can significantly reduce the time required to combine
references, cutting it down to a fraction of the previous duration. The original
idea was showcased in this `talk <https://discourse.pangeo.io/t/pangeo-showcase-optimizations-for-kerchunk-aggregation-and-zarr-i-o-at-scale-for-machine-learning/4074>`_.
It follows a three step approach.
**Three step approach:**
1. Extract and persist metadata directly from a few arbitrary grib
files for a given product such as HRRR SUBH, GEFS, GFS etc.
2. Use the metadata mapping to build an index table of every grib
message from the ``.idx`` files
3. Combine the index data with the metadata to build any FMRC
slice (Horizon, RunTime, ValidTime, BestAvailable)
*How is it faster*
The ``.idx`` file otherwise known as an *index* file contains the key
metadata of the messages in the GRIB files. These metadata include `index`, `offset`, `datetime`,
`variable` and `forecast time` for their respective messages. This metadata
will be used to index every GRIB message. By following this approach, we only have to
``scan_grib`` a single GRIB file, not the whole archive.
Building the index of a time horizon, first requires a single one-to-one mapping of GRIB/Zarr
metadata to the attributes in the idx file. Only constraint is the mapping needs to be
made from a single GRIB file, belonging to the *same time horizon*. The indexing process
primarily involves the `pandas <https://pandas.pydata.org/>`_ library. To confirm this,
see this `notebook <https://gist.github.com/Anu-Ra-g/efa01ad1c274c1bd1c14ee01666caa77>`_.
After indexing a single time horizon, you can combine this index with indexes of
other time horizon and store it.
.. note::
The index in ``.idx`` file indexes the GRIB messages where as the ``k_index``
(kerchunk index), index the variables
in those messages.
The table mentioned below is a *k_index* made from a single GRIB file.
.. list-table:: k_index for a single GRIB file
:header-rows: 1
:widths: 5 10 15 10 20 15 10 20 20 30 10 10 10
* -
- varname
- typeOfLevel
- stepType
- name
- step
- level
- time
- valid_time
- uri
- offset
- length
- inline_value
* - 0
- gh
- isobaricInhPa
- instant
- Geopotential height
- 0 days 06:00:00
- 0.0
- 2017-01-01 06:00:00
- 2017-01-01 12:00:00
- s3://noaa-gefs-pds/gefs.20170101/06/gec00.t06z...
- 0
- 47493
- None
* - 1
- t
- isobaricInhPa
- instant
- Temperature
- 0 days 06:00:00
- 0.0
- 2017-01-01 06:00:00
- 2017-01-01 12:00:00
- s3://noaa-gefs-pds/gefs.20170101/06/gec00.t06z...
- 47493
- 19438
- None
* - 2
- r
- isobaricInhPa
- instant
- Relative humidity
- 0 days 06:00:00
- 0.0
- 2017-01-01 06:00:00
- 2017-01-01 12:00:00
- s3://noaa-gefs-pds/gefs.20170101/06/gec00.t06z...
- 66931
- 10835
- None
* - 3
- u
- isobaricInhPa
- instant
- U component of wind
- 0 days 06:00:00
- 0.0
- 2017-01-01 06:00:00
- 2017-01-01 12:00:00
- s3://noaa-gefs-pds/gefs.20170101/06/gec00.t06z...
- 77766
- 22625
- None
* - 4
- v
- isobaricInhPa
- instant
- V component of wind
- 0 days 06:00:00
- 0.0
- 2017-01-01 06:00:00
- 2017-01-01 12:00:00
- s3://noaa-gefs-pds/gefs.20170101/06/gec00.t06z...
- 100391
- 20488
- None
*What now*
After creating the k_index as per the desired duration, we will use the ``DataTree`` model
from the `xarray-datatree <https://xarray-datatree.readthedocs.io/en/latest/>`_ to view a
part(desired variables) or the whole of the aggregation, using the k_index. Below is a
tree model made from an aggregation of GRIB files produced from **GEFS** model hosted
in AWS S3 bucket.
.. code-block:: bash
DataTree('None', parent=None)
├── DataTree('prmsl')
│ │ Dimensions: ()
│ │ Data variables:
│ │ *empty*
│ │ Attributes:
│ │ name: Pressure reduced to MSL
│ └── DataTree('instant')
│ │ Dimensions: ()
│ │ Data variables:
│ │ *empty*
│ │ Attributes:
│ │ stepType: instant
│ └── DataTree('meanSea')
│ Dimensions: (latitude: 181, longitude: 360, time: 1, step: 1,
│ model_horizons: 1, valid_times: 237)
│ Coordinates:
│ * latitude (latitude) float64 1kB 90.0 89.0 88.0 87.0 ... -88.0 -89.0 -90.0
│ * longitude (longitude) float64 3kB 0.0 1.0 2.0 3.0 ... 357.0 358.0 359.0
│ meanSea float64 8B ...
│ number (time, step) int64 8B ...
│ step (model_horizons, valid_times) timedelta64[ns] 2kB ...
│ time (model_horizons, valid_times) datetime64[ns] 2kB ...
│ valid_time (model_horizons, valid_times) datetime64[ns] 2kB ...
│ Dimensions without coordinates: model_horizons, valid_times
│ Data variables:
│ prmsl (model_horizons, valid_times, latitude, longitude) float64 124MB ...
│ Attributes:
│ typeOfLevel: meanSea
└── DataTree('ulwrf')
│ Dimensions: ()
│ Data variables:
│ *empty*
│ Attributes:
│ name: Upward long-wave radiation flux
└── DataTree('avg')
│ Dimensions: ()
│ Data variables:
│ *empty*
│ Attributes:
│ stepType: avg
└── DataTree('nominalTop')
Dimensions: (latitude: 181, longitude: 360, time: 1, step: 1,
model_horizons: 1, valid_times: 237)
Coordinates:
* latitude (latitude) float64 1kB 90.0 89.0 88.0 87.0 ... -88.0 -89.0 -90.0
* longitude (longitude) float64 3kB 0.0 1.0 2.0 3.0 ... 357.0 358.0 359.0
nominalTop float64 8B ...
number (time, step) int64 8B ...
step (model_horizons, valid_times) timedelta64[ns] 2kB ...
time (model_horizons, valid_times) datetime64[ns] 2kB ...
valid_time (model_horizons, valid_times) datetime64[ns] 2kB ...
Dimensions without coordinates: model_horizons, valid_times
Data variables:
ulwrf (model_horizons, valid_times, latitude, longitude) float64 124MB ...
Attributes:
typeOfLevel: nominalTop
.. tip::
For a full tutorial on this workflow, refer this `kerchunk cookbook <https://projectpythia.org/kerchunk-cookbook/README.html>`_
in `Project Pythia <https://projectpythia.org/>`_.
|
