1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
|
``drs_tool`` command-line interface
===================================
The ``drs_tool`` command can be used to invoke functions from the
:mod:`drslib.drs_tree` API. The tool is designed to help ESGF
datanode managers to
1. Prepare incoming data for publication, placing files in the DRS
directory structure.
2. Manage multiple versions of publication-level datasets to minimise
disk usage.
3. Deduce the CMIP5 product into which data should be published
according to the rules definied in the CMIP5 experiment definition.
Concepts
--------
DRS root
The root of the published DRS directory tree. This is synonimous
with the directory representing the DRS activity component.
Incoming directory
The directory scanned for data files to be added to
the DRS directory tree. This directory is scanned recursively to
find NetCDF files in the DRS filename encoding. It defaults to
``<drs-root>/output`` for compatibility with the output structure of
CMOR2. However, the directory names under the incoming directory
are not taken into account when deducing the DRS components of
files.
DRS pattern
Each invocation of ``drs_tool`` is given a set of DRS component
values that define the portion of the DRS space on which it acts.
Usage
-----
Usage: drs_tool [command] [options] [drs-pattern]
command:
======= ====================================================================
list list publication-level datasets
todo show file operations pending for the next version
upgrade make changes to the selected datasets to upgrade to the next version
mapfile make a mapfile of the selected dataset
history list all versions of the selected dataset
init initialise CMIP5 product detection data
======= ====================================================================
drs-pattern:
A dataset identifier in '.'-separated notation. Use the '%' to indicate unknown DRS components, e.g. ``cmip5.%.MPI-M`` will match any product.
Options:
-h, --help show this help message and exit
-R ROOT, --root=ROOT Root directory of the DRS tree
-I INCOMING, --incoming=INCOMING
Incoming directory for DRS files. Defaults to
<root>/output
-a ACTIVITY, --activity=ACTIVITY
Set DRS attribute activity for dataset discovery
-p PRODUCT, --product=PRODUCT
Set DRS attribute product for dataset discovery
-i INSTITUTE, --institute=INSTITUTE
Set DRS attribute institute for dataset discovery
-m MODEL, --model=MODEL
Set DRS attribute model for dataset discovery
-e EXPERIMENT, --experiment=EXPERIMENT
Set DRS attribute experiment for dataset discovery
-f FREQUENCY, --frequency=FREQUENCY
Set DRS attribute frequency for dataset discovery
-r REALM, --realm=REALM
Set DRS attribute realm for dataset discovery
-v VERSION, --version=VERSION
Force version upgrades to this version
-P FILE, --profile=FILE
Profile the script exectuion into FILE
--detect-product Automatically detect the DRS product of incoming data
-j JSON_DRS, --json-drs=JSON_DRS
Use the JSON output from the `ceda-cc` quality control tool
to define the incoming set of files and their associated DRS terms.
An Example
----------
This example walks through using ``drs_tool`` to prepare datasets for
publication. It uses some dummy data files based on test runs of the
Met Office Hadley Centre HadGEM2-ES model. You can repeat the example
from within the drslib source distribution directory.
First generate a dummy set of incoming data files. In this example we
assume all incoming files are in a single flat directory, however
``drs_tool`` will recursively travers the incoming directory to find
all DRS-compliant NetCDF files.
.. code-block:: bash
$ mkdir mohc_eg
$ python test/gen_drs.py test/mohc_delivery.ls mohc_eg/incoming
$ ls mohc_eg/incoming/ | head
baresoilFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_201512-204011.nc
baresoilFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc
baresoilFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_206512-209011.nc
baresoilFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_209012-209911.nc
c3PftFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_201512-204011.nc
c3PftFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc
c3PftFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_206512-209011.nc
c3PftFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_209012-209911.nc
c4PftFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_201512-204011.nc
c4PftFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc
$ ls mohc_eg/incoming/ | wc -l
494
We now have about 500 dummy NetCDF files in ``mohc_eg/incoming``. You
can ask ``drs_tool`` to list which publication-level datasets these
files would be put in using the ``drs_tool list`` subcommand. For
this to work ``drs_tool`` requires 2 DRS components not decidable from
the filenames: activity and product [*]_. ``drs_tool list`` will list
all publication-level datasets with the criteria given, including
those that would be created by processing the incoming directory.
.. [*] later versions of drslib will be able to decide the product
component from other components and by inspecting the NetCDF.
.. code-block:: bash
$ drs_tool list -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1
==============================================================================
DRS Tree at mohc_eg/
------------------------------------------------------------------------------
cmip5.output1.MOHC.HadGEM2-ES.rcp45.3hr.atmos.3hr.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.3hr.land.3hr.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrLev.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrPlev.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.day.atmos.day.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.day.land.day.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.atmos.Amon.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.land.Lmon.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.landIce.LImon.r1i1p1 *
==============================================================================
The asterisk against each dataset_id indicates there are files in the
incoming directory to add to the dataset. In this case all datasets
are empty.
We can restrict ``drs_tool list`` output by using a dataset_id
wildcard. For instance to select only datasets in the ``atmos`` realm:
.. code-block:: bash
$ drs_tool list -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1.%.%.%.%.atmos
==============================================================================
DRS Tree at mohc_eg/
------------------------------------------------------------------------------
cmip5.output1.MOHC.HadGEM2-ES.rcp45.3hr.atmos.3hr.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrLev.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrPlev.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.day.atmos.day.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.atmos.Amon.r1i1p1 *
==============================================================================
The same effect can be achieved with individual component options:
.. code-block:: bash
$ drs_tool list -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1 --realm=atmos
==============================================================================
DRS Tree at mohc_eg/
------------------------------------------------------------------------------
cmip5.output1.MOHC.HadGEM2-ES.rcp45.3hr.atmos.3hr.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrLev.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrPlev.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.day.atmos.day.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.atmos.Amon.r1i1p1 *
==============================================================================
Now we will focus on a single dataset in the ``aerosol`` realm and
show how to move files into the DRS directory structure ready for
publication. We can check what filesystem commands will be done using
the ``drs_tool todo`` subcommand.
.. code-block:: bash
$ drs_tool list -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1 --realm=aerosol
==============================================================================
DRS Tree at mohc_eg/
------------------------------------------------------------------------------
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1 *
==============================================================================
$ drs_tool todo -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1 --realm=aerosol | head
==============================================================================
DRS Tree at mohc_eg/
------------------------------------------------------------------------------
Publisher Tree cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1 todo for version 20100927
------------------------------------------------------------------------------
mv mohc_eg/incoming/emidust_aero_HadGEM2-ES_rcp45_r1i1p1_206512-209011.nc /home/spascoe/git/esgf-drslib/mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/files/emidust_20100927/emidust_aero_HadGEM2-ES_rcp45_r1i1p1_206512-209011.nc
ln -s /home/spascoe/git/esgf-drslib/mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/files/emidust_20100927/emidust_aero_HadGEM2-ES_rcp45_r1i1p1_206512-209011.nc /home/spascoe/git/esgf-drslib/mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/emidust/emidust_aero_HadGEM2-ES_rcp45_r1i1p1_206512-209011.nc
mv mohc_eg/incoming/reffclwtop_aero_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc /home/spascoe/git/esgf-drslib/mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/files/reffclwtop_20100927/reffclwtop_aero_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc
ln -s /home/spascoe/git/esgf-drslib/mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/files/reffclwtop_20100927/reffclwtop_aero_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc /home/spascoe/git/esgf-drslib/mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/reffclwtop/reffclwtop_aero_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc
mv mohc_eg/incoming/dryso2_aero_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc /home/spascoe/git/esgf-drslib/mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/files/dryso2_20100927/dryso2_aero_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc
You can see here that drslib will move files into datestamped
directories under ``<dataset-dir>/files`` then symbolically link them
into the DRS directory structure. To do the actual moving use
``drs_tool upgrade``. Then use ``drs_tool list`` to view the result.
.. code-block:: bash
$ drs_tool upgrade -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1 --realm=aerosol
==============================================================================
DRS Tree at mohc_eg/
------------------------------------------------------------------------------
Upgrading cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1 to version 20100927 ... done
==============================================================================
$ drs_tool list -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1
==============================================================================
DRS Tree at mohc_eg/
------------------------------------------------------------------------------
cmip5.output1.MOHC.HadGEM2-ES.rcp45.3hr.atmos.3hr.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.3hr.land.3hr.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrLev.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrPlev.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.day.atmos.day.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.day.land.day.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1.v20100927 -
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.atmos.Amon.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.land.Lmon.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.landIce.LImon.r1i1p1 *
==============================================================================
Using ``drs_tool``'s criteria options you can upgrade multiple datasets in one command:
.. code-block:: bash
$ drs_tool upgrade -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1 --realm=atmos --frequency=6hr
==============================================================================
DRS Tree at mohc_eg/
------------------------------------------------------------------------------
Upgrading cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrLev.r1i1p1 to version 20100927 ... done
Upgrading cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrPlev.r1i1p1 to version 20100927 ... done
==============================================================================
$ drs_tool list -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1
==============================================================================
DRS Tree at mohc_eg/
------------------------------------------------------------------------------
cmip5.output1.MOHC.HadGEM2-ES.rcp45.3hr.atmos.3hr.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.3hr.land.3hr.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrLev.r1i1p1.v20100927 -
cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrPlev.r1i1p1.v20100927 -
cmip5.output1.MOHC.HadGEM2-ES.rcp45.day.atmos.day.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.day.land.day.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1.v20100927 -
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.atmos.Amon.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.land.Lmon.r1i1p1 *
cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.landIce.LImon.r1i1p1 *
==============================================================================
Finally you need to send publish the datasets with ``esgpublish``. To make this easier ``drs_tool`` can create a mapfile of a dataset:
.. code-block:: bash
$ drs_tool mapfile -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1 --realm=aerosol >rcp45.mon.aerosol.map
$ head rcp45.mon.aerosol.map
mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/loadsoa/loadsoa_aero_HadGEM2-ES_rcp45_r1i1p1_206512-209011.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/loadsoa/loadsoa_aero_HadGEM2-ES_rcp45_r1i1p1_209012-209911.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/loadsoa/loadsoa_aero_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/loadsoa/loadsoa_aero_HadGEM2-ES_rcp45_r1i1p1_201512-204011.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/loadbc/loadbc_aero_HadGEM2-ES_rcp45_r1i1p1_201512-204011.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/loadbc/loadbc_aero_HadGEM2-ES_rcp45_r1i1p1_206512-209011.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/loadbc/loadbc_aero_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/loadbc/loadbc_aero_HadGEM2-ES_rcp45_r1i1p1_209012-209911.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/wetbc/wetbc_aero_HadGEM2-ES_rcp45_r1i1p1_209012-209911.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/wetbc/wetbc_aero_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
Some further examples of usage can be found in the doctest file
``test/test_command.txt``.
|
