1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
|
.. HARP documentation master file, created by
sphinx-quickstart on Wed Oct 14 15:17:12 2015.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
HARP manual
===========
Contents
--------
.. toctree::
:maxdepth: 2
install
conventions/index
algorithms/index
operations
ingestions/index
libharp
idl
matlab
python
tools
What is HARP?
-------------
HARP is a toolkit for reading, processing and inter-comparing satellite remote
sensing data, model data, in-situ data, and ground based remote sensing data.
The toolkit is composed of:
- A set of :ref:`command line tools <command-line-tools>`
- A library of analysis functions
The main goal of HARP is to assist in the inter-comparison of data sets. By
appropriately chaining calls to HARP command line tools one can pre-process
data sets such that two datasets that need to be compared end up having the
same temporal/spatial grid, same data format/structure, and same physical unit.
At the end of the toolchain you will have a set of data files that can be
directly compared in e.g. Python, IDL or MATLAB.
In order for the HARP command line tools to handle each others output the
toolkit uses its own :ref:`data format conventions <data-formats>` for
intermediate files. The use of this harmonized formatting means that satellite
products need to be converted first to HARP compliant files. Once in this
format, further processing can be performed.
.. _data-formats:
Data formats
------------
In order for HARP command line tools to handle each others output the toolkit
makes uses of its own :doc:`data format conventions <conventions/index>`.
There is one standard for representing and storing measurement data, which
only prescribes a specific set of constraints to the data format and is still
flexible enough to allow data storage using either netCDF, HDF4, or HDF5 and
inclusion of metadata from other standards such as netCDF-CF.
The other data format convention is a standard for storing information on
collocations, the :doc:`Collocation Result File <conventions/collocation_result>`,
which is based on CSV.
Both data formats are described in more detail in the :doc:`Data formats
<conventions/index>` section of the documentation.
Algorithms
----------
HARP can perform a wide variety of operations on the data. A description of
the algorithms as used in HARP can be found in the :doc:`Algorithms
<algorithms/index>` section of the documentation.
Operations
----------
Several HARP functions and tools can take a sequence of operations, such as
filters or regridding operations, which are then performed on a product.
A description of the operations expressions is provided in the :doc:`Operations
<operations>` section of the documentation.
C library
---------
Common parts that are used by the various HARP command line tools are gathered
in a single :doc:`HARP C library <libharp>`. This library can be used to build
custom applications or libraries that work with HARP compliant data products.
IDL interface
----------------
The :doc:`HARP IDL interface <idl>` provides a set of functions to import
and export HARP products, and to ingest non-HARP products of a type supported by
HARP from IDL.
MATLAB interface
----------------
The :doc:`HARP MATLAB interface <matlab>` provides a set of functions to import
and export HARP products, and to ingest non-HARP products of a type supported by
HARP from MATLAB.
Python interface
----------------
The :doc:`HARP Python interface <python>` provides a set of functions to import
and export HARP products, and to ingest non-HARP products of a type supported by
HARP from Python. The Python interface depends on the ``_cffi_backend`` module,
which is part of the C foreign function interface (cffi) package. This package
must be installed in order to be able to use the Python interface. See the `cffi
documentation`_ for details on how to install the cffi package.
.. _cffi documentation: http://cffi.readthedocs.org/en/latest/installation.html
.. _command-line-tools:
Command line tools
------------------
The HARP command line tools perform various operations on HARP compliant data
products. All command line tools except ``harpconvert`` expect one or more HARP
data products as input. The ``harpconvert`` tool can be used to convert non-HARP
products into HARP products.
The available command line tools are described in more detail in the
:doc:`Command line tools <tools>` section of the documentation.
A typical inter-comparison
--------------------------
Typical sequence when comparing two datasets of measurements:
harpconvert ...
Convert all products that do not already follow the HARP format conventions
to a common data format.
Filter/convert the data in each file: remove unneeded
parameters/measurements, add derived physical parameters, perform unit
conversion, regrid dimensions, etc.
harpcollocate matchup ...
Find the measurement pairs that match within the specified collocation
criteria and save the result into a collocation result file.
harpcollocate resample ...
Apply resampling to the collocation result file.
harpconvert -a 'collocate-left(<csvfile>)' ...
Apply the collocation filter for each file in the primary dataset
harpconvert -a 'collocate-right(<csvfile>)' ...
Apply the collocation filter for each file in the secondary dataset
Input files can be verified with
harpcheck ...
and after each stage, the contents of each file can be inspected from the
command line using:
harpdump ...
|
