1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
|
# pyodc
[](https://pypi.org/project/pyodc/)
[](https://github.com/ecmwf/pyodc/actions/workflows/ci.yml)
[](https://pyodc.readthedocs.io/en/latest/?badge=latest)
[](https://github.com/psf/black)
[](https://github.com/ecmwf/pyodc/blob/develop/LICENSE)
A Python interface to `odc` for encoding/decoding ODB\-2 files.
The package contains two different implementations of the same library:
* `pyodc` is a pure-python encoder and decoder for ODB\-2 data, which encodes data from, and decodes it into pandas data frames
* `codc` is an implementation of the same API as `pyodc` that depends on the ECMWF `odc` library, and comes with _much_ better performance.
Both libraries are be installed by running `pip install pyodc`, and since version 1.6.0, a pre-built wheel version of `odc` will be automatically installed so that `codc` can be used without any additional steps.
[Documentation] [Changelog]
## Dependencies
### Required
* Python 3.x
### Optional
* [odc]
* [pytest]
* [pandoc]
* [Jupyter Notebook]
For `codc` to work, the `odc` library must be compiled and installed on the system and made available to Python. Typically this happens automatically as described above through the dependency on `odclib` which bundles a precompiled version of `odc` as a wheel. If some some reason this doesn't work, there are multiple other ways to make the library visible to pyodc:
* It can be installed as a system library.
* The installation prefix can be passed in the `odc_DIR` or `ODC_DIR` environment variables.
* The library directory can be included in `LD_LIBRARY_PATH.
## Installation
```sh
pip install pyodc
```
Check if the module was installed correctly:
```sh
python
>>> import pyodc as odc # pure python
>>> import codc as odc # faster
```
## Usage
An introductory Jupyter Notebook with helpful usage examples is provided in the root of this repository:
```sh
git clone git@github.com:ecmwf/pyodc.git
cd pyodc
jupyter notebook Introduction.ipynb
```
Note that **codc is not thread safe** so care should be taken when using it with dask. You can set dask to use processses rather than threads by doing:
```
with dask.config.set(scheduler='processes'):
dask.compute(...)
```
## Development
### Run Unit Tests
To run the unit tests, make sure that the `pytest` module is installed first:
```sh
python -m pytest
```
### Run Unit Tests across multiple python versions with Tox
Tox is a useful tool to quickly run pytest across multiple python versions by managing a set of python environments for you. A tox.ini file is provided that targets python3.8 - 3.12. Note that this will also install older versions of libraries like numpy which helps to catch incompatibilities with older versions of those libraries too.
To run tox, [install it](https://tox.wiki/), modify the `ODC_HOME = ../build` line in tox.ini to point to a build of odc, this will be reused for all the tests. Then run
```sh
tox
```
The first run will take a while for it to install all the environments but after that it's very fast.
### Build Documentation
To build the documentation locally, please install the Python dependencies first:
```sh
cd docs
pip install -r requirements.txt
make html
```
The built HTML documentation will be available under the `docs/_build/html/index.html` path.
## License
This software is licensed under the terms of the Apache Licence Version 2.0 which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
In applying this licence, ECMWF does not waive the privileges and immunities granted to it by virtue of its status as an intergovernmental organisation nor does it submit to any jurisdiction.
[Documentation]: https://pyodc.readthedocs.io/en/latest/
[Changelog]: ./CHANGELOG.md
[odc]: https://github.com/ecmwf/odc
[pytest]: https://pytest.org
[pandoc]: https://pandoc.org/
[Jupyter Notebook]: https://jupyter.org
|
