1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
|
<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>HARP manual — HARP 1.5 documentation</title>
<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="_static/custom.css" type="text/css" />
<link rel="index" title="Index" href="genindex.html" />
<link rel="next" title="Installation" href="install.html" />
<script src="_static/js/modernizr.min.js"></script>
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search">
<a href="#" class="icon icon-home"> HARP
</a>
<div class="version">
1.5
</div>
</div>
<div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
<ul>
<li class="toctree-l1"><a class="reference internal" href="install.html">Installation</a></li>
<li class="toctree-l1"><a class="reference internal" href="conventions/index.html">Conventions</a></li>
<li class="toctree-l1"><a class="reference internal" href="algorithms/index.html">Algorithms</a></li>
<li class="toctree-l1"><a class="reference internal" href="operations.html">Operations</a></li>
<li class="toctree-l1"><a class="reference internal" href="ingestions/index.html">Ingestion definitions</a></li>
<li class="toctree-l1"><a class="reference internal" href="libharp.html">C library</a></li>
<li class="toctree-l1"><a class="reference internal" href="idl.html">IDL interface</a></li>
<li class="toctree-l1"><a class="reference internal" href="matlab.html">MATLAB interface</a></li>
<li class="toctree-l1"><a class="reference internal" href="python.html">Python interface</a></li>
<li class="toctree-l1"><a class="reference internal" href="tools.html">Command line tools</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
<nav class="wy-nav-top" aria-label="top navigation">
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="#">HARP</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li><a href="#">Docs</a> »</li>
<li>HARP manual</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<div class="section" id="harp-manual">
<h1>HARP manual</h1>
<div class="section" id="contents">
<h2>Contents</h2>
<div class="toctree-wrapper compound">
<ul>
<li class="toctree-l1"><a class="reference internal" href="install.html">Installation</a><ul>
<li class="toctree-l2"><a class="reference internal" href="install.html#supported-platforms">Supported platforms</a></li>
<li class="toctree-l2"><a class="reference internal" href="install.html#what-you-will-need">What you will need</a></li>
<li class="toctree-l2"><a class="reference internal" href="install.html#using-github">Using GitHub</a></li>
<li class="toctree-l2"><a class="reference internal" href="install.html#building-the-source-package-unix">Building the source package (Unix)</a></li>
<li class="toctree-l2"><a class="reference internal" href="install.html#installing-the-binary-package-windows">Installing the binary package (Windows)</a></li>
<li class="toctree-l2"><a class="reference internal" href="install.html#building-the-source-package-windows">Building the source package (Windows)</a></li>
<li class="toctree-l2"><a class="reference internal" href="install.html#documentation-location">Documentation location</a></li>
<li class="toctree-l2"><a class="reference internal" href="install.html#feedback">Feedback</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="conventions/index.html">Conventions</a><ul>
<li class="toctree-l2"><a class="reference internal" href="conventions/datatypes.html">Data types</a></li>
<li class="toctree-l2"><a class="reference internal" href="conventions/dimensions.html">Dimensions</a></li>
<li class="toctree-l2"><a class="reference internal" href="conventions/global_attributes.html">Global attributes</a></li>
<li class="toctree-l2"><a class="reference internal" href="conventions/variable_attributes.html">Variable attributes</a></li>
<li class="toctree-l2"><a class="reference internal" href="conventions/variables.html">Variables</a></li>
<li class="toctree-l2"><a class="reference internal" href="conventions/variable_names.html">Variable names</a></li>
<li class="toctree-l2"><a class="reference internal" href="conventions/categorical_variables.html">Categorical Variables</a></li>
<li class="toctree-l2"><a class="reference internal" href="conventions/datetime.html">Datetime values</a></li>
<li class="toctree-l2"><a class="reference internal" href="conventions/filenames.html">File naming</a></li>
<li class="toctree-l2"><a class="reference internal" href="conventions/netcdf3.html">netCDF-3</a></li>
<li class="toctree-l2"><a class="reference internal" href="conventions/hdf4.html">HDF4</a></li>
<li class="toctree-l2"><a class="reference internal" href="conventions/hdf5.html">HDF5/netCDF-4</a></li>
<li class="toctree-l2"><a class="reference internal" href="conventions/compatibility.html">Compatibility with other standards</a></li>
<li class="toctree-l2"><a class="reference internal" href="conventions/collocation_result.html">Collocation result file</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="algorithms/index.html">Algorithms</a><ul>
<li class="toctree-l2"><a class="reference internal" href="algorithms/definitions.html">Definitions</a></li>
<li class="toctree-l2"><a class="reference internal" href="algorithms/formula.html">Common formula</a></li>
<li class="toctree-l2"><a class="reference internal" href="algorithms/index.html#variable-conversions">Variable Conversions</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="operations.html">Operations</a><ul>
<li class="toctree-l2"><a class="reference internal" href="operations.html#comparison-filter">Comparison filter</a></li>
<li class="toctree-l2"><a class="reference internal" href="operations.html#membership-test-filter">Membership test filter</a></li>
<li class="toctree-l2"><a class="reference internal" href="operations.html#function-call">Function call</a></li>
<li class="toctree-l2"><a class="reference internal" href="operations.html#collocation-result-file">Collocation result file</a></li>
<li class="toctree-l2"><a class="reference internal" href="operations.html#area-mask-file">Area mask file</a></li>
<li class="toctree-l2"><a class="reference internal" href="operations.html#examples">Examples</a></li>
<li class="toctree-l2"><a class="reference internal" href="operations.html#formal-definition">Formal definition</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="ingestions/index.html">Ingestion definitions</a><ul>
<li class="toctree-l2"><a class="reference internal" href="ingestions/index.html#ace-products">ACE products</a></li>
<li class="toctree-l2"><a class="reference internal" href="ingestions/index.html#aeolus-products">AEOLUS products</a></li>
<li class="toctree-l2"><a class="reference internal" href="ingestions/index.html#aerosol-cci-products">Aerosol CCI products</a></li>
<li class="toctree-l2"><a class="reference internal" href="ingestions/index.html#calipso-lidar-products">CALIPSO LIDAR products</a></li>
<li class="toctree-l2"><a class="reference internal" href="ingestions/index.html#cloudnet-products">CLOUDNET products</a></li>
<li class="toctree-l2"><a class="reference internal" href="ingestions/index.html#cloud-cci-products">Cloud CCI products</a></li>
<li class="toctree-l2"><a class="reference internal" href="ingestions/index.html#earlinet-products">EARLINET products</a></li>
<li class="toctree-l2"><a class="reference internal" href="ingestions/index.html#ecmwf-grib-products">ECMWF GRIB products</a></li>
<li class="toctree-l2"><a class="reference internal" href="ingestions/index.html#geoms-products">GEOMS products</a></li>
<li class="toctree-l2"><a class="reference internal" href="ingestions/index.html#gome-products">GOME products</a></li>
<li class="toctree-l2"><a class="reference internal" href="ingestions/index.html#gome-2-products">GOME-2 products</a></li>
<li class="toctree-l2"><a class="reference internal" href="ingestions/index.html#gomos-products">GOMOS products</a></li>
<li class="toctree-l2"><a class="reference internal" href="ingestions/index.html#gosat-fts-products">GOSAT FTS products</a></li>
<li class="toctree-l2"><a class="reference internal" href="ingestions/index.html#green-house-gases-cci-products">Green House Gases CCI products</a></li>
<li class="toctree-l2"><a class="reference internal" href="ingestions/index.html#hirdls-products">HIRDLS products</a></li>
<li class="toctree-l2"><a class="reference internal" href="ingestions/index.html#iasi-products">IASI products</a></li>
<li class="toctree-l2"><a class="reference internal" href="ingestions/index.html#mipas-products">MIPAS products</a></li>
<li class="toctree-l2"><a class="reference internal" href="ingestions/index.html#mls-products">MLS products</a></li>
<li class="toctree-l2"><a class="reference internal" href="ingestions/index.html#npp-products">NPP products</a></li>
<li class="toctree-l2"><a class="reference internal" href="ingestions/index.html#omi-products">OMI products</a></li>
<li class="toctree-l2"><a class="reference internal" href="ingestions/index.html#osiris-products">OSIRIS products</a></li>
<li class="toctree-l2"><a class="reference internal" href="ingestions/index.html#ozone-cci-products">Ozone CCI products</a></li>
<li class="toctree-l2"><a class="reference internal" href="ingestions/index.html#qa4ecv-products">QA4ECV products</a></li>
<li class="toctree-l2"><a class="reference internal" href="ingestions/index.html#sciamachy-products">SCIAMACHY products</a></li>
<li class="toctree-l2"><a class="reference internal" href="ingestions/index.html#smr-products">SMR products</a></li>
<li class="toctree-l2"><a class="reference internal" href="ingestions/index.html#sentinel-5p-products">Sentinel-5P products</a></li>
<li class="toctree-l2"><a class="reference internal" href="ingestions/index.html#temis-products">TEMIS products</a></li>
<li class="toctree-l2"><a class="reference internal" href="ingestions/index.html#tes-products">TES products</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="libharp.html">C library</a><ul>
<li class="toctree-l2"><a class="reference internal" href="libharp.html#introduction">Introduction</a></li>
<li class="toctree-l2"><a class="reference internal" href="libharp.html#modules">Modules</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="idl.html">IDL interface</a><ul>
<li class="toctree-l2"><a class="reference internal" href="idl.html#dimension-types">Dimension types</a></li>
<li class="toctree-l2"><a class="reference internal" href="idl.html#data-types">Data types</a></li>
<li class="toctree-l2"><a class="reference internal" href="idl.html#unicode">Unicode</a></li>
<li class="toctree-l2"><a class="reference internal" href="idl.html#examples">Examples</a></li>
<li class="toctree-l2"><a class="reference internal" href="idl.html#api-reference">API reference</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="matlab.html">MATLAB interface</a><ul>
<li class="toctree-l2"><a class="reference internal" href="matlab.html#dimension-types">Dimension types</a></li>
<li class="toctree-l2"><a class="reference internal" href="matlab.html#data-types">Data types</a></li>
<li class="toctree-l2"><a class="reference internal" href="matlab.html#unicode">Unicode</a></li>
<li class="toctree-l2"><a class="reference internal" href="matlab.html#examples">Examples</a></li>
<li class="toctree-l2"><a class="reference internal" href="matlab.html#api-reference">API reference</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="python.html">Python interface</a><ul>
<li class="toctree-l2"><a class="reference internal" href="python.html#dimension-types">Dimension types</a></li>
<li class="toctree-l2"><a class="reference internal" href="python.html#data-types">Data types</a></li>
<li class="toctree-l2"><a class="reference internal" href="python.html#unicode">Unicode</a></li>
<li class="toctree-l2"><a class="reference internal" href="python.html#examples">Examples</a></li>
<li class="toctree-l2"><a class="reference internal" href="python.html#api-reference">API reference</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="tools.html">Command line tools</a><ul>
<li class="toctree-l2"><a class="reference internal" href="harpcheck.html">harpcheck</a></li>
<li class="toctree-l2"><a class="reference internal" href="harpcollocate.html">harpcollocate</a></li>
<li class="toctree-l2"><a class="reference internal" href="harpconvert.html">harpconvert</a></li>
<li class="toctree-l2"><a class="reference internal" href="harpdump.html">harpdump</a></li>
<li class="toctree-l2"><a class="reference internal" href="harpmerge.html">harpmerge</a></li>
</ul>
</li>
</ul>
</div>
</div>
<div class="section" id="what-is-harp">
<h2>What is HARP?</h2>
<p>HARP is a toolkit for reading, processing and inter-comparing satellite remote
sensing data, model data, in-situ data, and ground based remote sensing data.
The toolkit is composed of:</p>
<ul class="simple">
<li>A set of <a class="reference internal" href="#command-line-tools"><span class="std std-ref">command line tools</span></a></li>
<li>A library of analysis functions</li>
</ul>
<p>The main goal of HARP is to assist in the inter-comparison of data sets. By
appropriately chaining calls to HARP command line tools one can pre-process
data sets such that two datasets that need to be compared end up having the
same temporal/spatial grid, same data format/structure, and same physical unit.
At the end of the toolchain you will have a set of data files that can be
directly compared in e.g. Python, IDL or MATLAB.</p>
<p>In order for the HARP command line tools to handle each others output the
toolkit uses its own <a class="reference internal" href="#data-formats"><span class="std std-ref">data format conventions</span></a> for
intermediate files. The use of this harmonized formatting means that satellite
products need to be converted first to HARP compliant files. Once in this
format, further processing can be performed.</p>
</div>
<div class="section" id="data-formats">
<span id="id1"></span><h2>Data formats</h2>
<p>In order for HARP command line tools to handle each others output the toolkit
makes uses of its own <a class="reference internal" href="conventions/index.html"><span class="doc">data format conventions</span></a>.</p>
<p>There is one standard for representing and storing measurement data, which
only prescribes a specific set of constraints to the data format and is still
flexible enough to allow data storage using either netCDF, HDF4, or HDF5 and
inclusion of metadata from other standards such as netCDF-CF.</p>
<p>The other data format convention is a standard for storing information on
collocations, the <a class="reference internal" href="conventions/collocation_result.html"><span class="doc">Collocation Result File</span></a>,
which is based on CSV.</p>
<p>Both data formats are described in more detail in the <a class="reference internal" href="conventions/index.html"><span class="doc">Data formats</span></a> section of the documentation.</p>
</div>
<div class="section" id="algorithms">
<h2>Algorithms</h2>
<p>HARP can perform a wide variety of operations on the data. A description of
the algorithms as used in HARP can be found in the <a class="reference internal" href="algorithms/index.html"><span class="doc">Algorithms</span></a> section of the documentation.</p>
</div>
<div class="section" id="operations">
<h2>Operations</h2>
<p>Several HARP functions and tools can take a sequence of operations, such as
filters or regridding operations, which are then performed on a product.
A description of the operations expressions is provided in the <a class="reference internal" href="operations.html"><span class="doc">Operations</span></a> section of the documentation.</p>
</div>
<div class="section" id="c-library">
<h2>C library</h2>
<p>Common parts that are used by the various HARP command line tools are gathered
in a single <a class="reference internal" href="libharp.html"><span class="doc">HARP C library</span></a>. This library can be used to build
custom applications or libraries that work with HARP compliant data products.</p>
</div>
<div class="section" id="idl-interface">
<h2>IDL interface</h2>
<p>The <a class="reference internal" href="idl.html"><span class="doc">HARP IDL interface</span></a> provides a set of functions to import
and export HARP products, and to ingest non-HARP products of a type supported by
HARP from IDL.</p>
</div>
<div class="section" id="matlab-interface">
<h2>MATLAB interface</h2>
<p>The <a class="reference internal" href="matlab.html"><span class="doc">HARP MATLAB interface</span></a> provides a set of functions to import
and export HARP products, and to ingest non-HARP products of a type supported by
HARP from MATLAB.</p>
</div>
<div class="section" id="python-interface">
<h2>Python interface</h2>
<p>The <a class="reference internal" href="python.html"><span class="doc">HARP Python interface</span></a> provides a set of functions to import
and export HARP products, and to ingest non-HARP products of a type supported by
HARP from Python. The Python interface depends on the <code class="docutils literal notranslate"><span class="pre">_cffi_backend</span></code> module,
which is part of the C foreign function interface (cffi) package. This package
must be installed in order to be able to use the Python interface. See the <a class="reference external" href="http://cffi.readthedocs.org/en/latest/installation.html">cffi
documentation</a> for details on how to install the cffi package.</p>
</div>
<div class="section" id="command-line-tools">
<span id="id2"></span><h2>Command line tools</h2>
<p>The HARP command line tools perform various operations on HARP compliant data
products. All command line tools except <code class="docutils literal notranslate"><span class="pre">harpconvert</span></code> expect one or more HARP
data products as input. The <code class="docutils literal notranslate"><span class="pre">harpconvert</span></code> tool can be used to convert non-HARP
products into HARP products.</p>
<p>The available command line tools are described in more detail in the
<a class="reference internal" href="tools.html"><span class="doc">Command line tools</span></a> section of the documentation.</p>
</div>
<div class="section" id="a-typical-inter-comparison">
<h2>A typical inter-comparison</h2>
<p>Typical sequence when comparing two datasets of measurements:</p>
<dl class="docutils">
<dt>harpconvert …</dt>
<dd>Convert all products that do not already follow the HARP format conventions
to a common data format.
Filter/convert the data in each file: remove unneeded
parameters/measurements, add derived physical parameters, perform unit
conversion, regrid dimensions, etc.</dd>
<dt>harpcollocate matchup …</dt>
<dd>Find the measurement pairs that match within the specified collocation
criteria and save the result into a collocation result file.</dd>
<dt>harpcollocate resample …</dt>
<dd>Apply resampling to the collocation result file.</dd>
<dt>harpconvert -a ‘collocate-left(<csvfile>)’ …</dt>
<dd>Apply the collocation filter for each file in the primary dataset</dd>
<dt>harpconvert -a ‘collocate-right(<csvfile>)’ …</dt>
<dd>Apply the collocation filter for each file in the secondary dataset</dd>
</dl>
<p>Input files can be verified with</p>
<blockquote>
<div>harpcheck …</div></blockquote>
<p>and after each stage, the contents of each file can be inspected from the
command line using:</p>
<blockquote>
<div>harpdump …</div></blockquote>
</div>
</div>
</div>
</div>
<footer>
<div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
<a href="install.html" class="btn btn-neutral float-right" title="Installation" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a>
</div>
<hr/>
<div role="contentinfo">
<p>
© Copyright 2015-2018 S&T, The Netherlands
</p>
</div>
</footer>
</div>
</div>
</section>
</div>
<script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<script async="async" type="text/javascript" src="/usr/share/javascript/mathjax/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
<script type="text/javascript" src="_static/js/theme.js"></script>
<script type="text/javascript">
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
|
