1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
|
======================================================
Satpy internal workings: having a look under the hood
======================================================
Querying and identifying data arrays
====================================
DataQuery
---------
The loading of data in Satpy is usually done through giving the name or the wavelength of the data arrays we are interested
in. This way, the highest, most calibrated data arrays is often returned.
However, in some cases, we need more control over the loading of the data arrays. The way to accomplish this is to load
data arrays using queries, eg::
scn.load([DataQuery(name='channel1', resolution=400)]
Here a data array with name `channel1` and of resolution `400` will be loaded if available.
Note that None is not a valid value, and keys having a value set to None will simply be ignored.
If one wants to use wildcards to query data, just provide `'*'`, eg::
scn.load([DataQuery(name='channel1', resolution=400, calibration='*')]
Alternatively, one can provide a list as parameter to query data, like this::
scn.load([DataQuery(name='channel1', resolution=[400, 800])]
DataID
------
Satpy stores loaded data arrays in a special dictionary (`DatasetDict`) inside scene objects.
In order to identify each data array uniquely, Satpy is assigning an ID to each data array, which is then used as the key in
the scene object. These IDs are of type `DataID` and are immutable. They are not supposed to be used by regular users and should only be
created in special circumstances. Satpy should take care of creating and assigning these automatically. They are also stored in the
`attrs` of each data array as `_satpy_id`.
Default and custom metadata keys
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
One thing however that the user has control over is which metadata keys are relevant to which datasets. Satpy provides two default sets
of metadata key (or ID keys), one for regular imager bands, and the other for composites.
The first one contains: name, wavelength, resolution, calibration, modifiers.
The second one contains: name, resolution.
As an example here is the definition of the first one in yaml:
.. code-block:: yaml
data_identification_keys:
name:
required: true
wavelength:
type: !!python/name:satpy.dataset.dataid.WavelengthRange
resolution:
calibration:
enum:
- reflectance
- brightness_temperature
- radiance
- counts
transitive: true
modifiers:
required: true
default: []
type: !!python/name:satpy.dataset.ModifierTuple
To create a new set, the user can provide indications in the relevant yaml file.
It has to be provided in header of the reader configuration file, under the `reader`
section, as `data_identification_keys`. Each key under this is the name of relevant
metadata key that will used to find relevant information in the attributes of the data
arrays. Under each of this, a few options are available:
- `required`: if the item is required, False by default
- `type`: the type to use. More on this further down.
- `enum`: if the item has to be limited to a finite number of options, an enum can be used.
Be sure to place the options in the order of preference, with the most desirable option on top.
- `default`: the default value to assign to the item if nothing (or None) is provided. If this
option isn't provided, the key will simply be omitted if it is not present in the attrs or if it
is None. It will be passed to the type's `convert` method if available.
- `transitive`: whether the key is to be passed when looking for dependencies of composites/modifiers.
Here for example, a composite that has in a given calibration type will pass this calibration
type requirement to its dependencies.
If the definition of the metadata keys need to be done in python rather than in a yaml file, it will
be a dictionary very similar to the yaml code. Here is the same example as above in python:
.. code-block:: python
from satpy.dataset.dataid import WavelengthRange, ModifierTuple
id_keys_config = {'name': {
'required': True,
},
'wavelength': {
'type': WavelengthRange,
},
'resolution': None,
'calibration': {
'enum': [
'reflectance',
'brightness_temperature',
'radiance',
'counts'
],
'transitive': True,
},
'modifiers': {
'required': True,
'default': ModifierTuple(),
'type': ModifierTuple,
},
}
Types
~~~~~
Types are classes that implement a type to be used as value for metadata in the `DataID`. They have
to implement a few methods:
- a `convert` class method that returns it's argument as an instance of the class
- `__hash__`, `__eq__` and `__ne__` methods
- a `distance` method the tells how "far" an instance of this class is from it's argument.
An example of such a class is the :class:`WavelengthRange <satpy.dataset.dataid.WavelengthRange>` class.
Through its implementation, it allows us to use the wavelength in a query to find out which of the
DataID in a list which has its central wavelength closest to that query for example.
DataID and DataQuery interactions
=================================
Different DataIDs and DataQuerys can have different metadata items defined. As such
we define equality between different instances of these classes, and across the classes
as equality between the sorted key/value pairs shared between the instances.
If a DataQuery has one or more values set to `'*'`, the corresponding key/value pair will be omitted from the comparison.
Instances sharing no keys will no be equal.
Breaking changes from DatasetIDs
================================
- The way to access values from the DataID and DataQuery is through getitem: `my_dataid['resolution']`
- For checking if a dataset is loaded, use `'mydataset' in scene`, as `'mydataset' in scene.keys()` will always return `False`:
the `DatasetDict` instance only supports `DataID` as key type.
Creating DataID for tests
=========================
Sometimes, it is useful to create `DataID` instances for testing purposes. For these cases, the `satpy.tests.utils` module
now has a `make_dsid` function that can be used just for this::
from satpy.tests.utils import make_dataid
did = make_dataid(name='camembert', modifiers=('runny',))
|
