1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
|
Metadata-Version: 1.1
Name: Stetl
Version: 1.0.9
Summary: Stetl provides transformation for spatial data
Home-page: http://github.com/justb4/stetl
Author: Just van den Broecke
Author-email: justb4@gmail.com
License: GNU GPL v3
Description: # Stetl - Streaming ETL
Stetl, streaming ETL, pronounced "staedl", is a lightweight ETL-framework for the conversion of rich (as GML)
geospatial data conversion. Stetl is Open Source (GNU GPL v3).
## Documentation
The main website and documentation can be found on http://stetl.org (or http://stetl.readthedocs.org).
Read a 5-minute introduction here: http://www.slideshare.net/justb4/5-minute-intro-to-setl and a longer presentation
here: http://www.slideshare.net/justb4/stetl-foss4g20131024v1.
Stetl was presented at several occasions like the
FOSS4G 2013 in Nottingham http://2013.foss4g.org.
## Origins
Stetl originated in the INSPIRE-FOSS project (www.inspire-foss.org)
and was created by Just van den Broecke. Since Stetl evolved into a wider use like
transforming Dutch GML-based datasets such as IMGEO/BGT (Large Scale Topography)
and IMKAD/BRK (Cadastral Data) it has now a repo of its own.
## Design
Stetl basically glues together existing parsing and transformation tools like GDAL/OGR (ogr2ogr) and XSLT.
By using native tools like libxml and libxslt (via Python lxml) Stetl is speed-optimized.
Stetl has a similar design as Spring (Java) and other modern frameworks based on IoC (Inversion of Control, http://en.wikipedia.org/wiki/Inversion_of_Control).
A configuration file (in Python config format) specifies your chain of ETL steps.
This chain is formed by a series of Python modules/objects and their parameters. These are
symbolically specified in the config file. You just invoke etl.py the main program with a config file.
The config file specifies the input modules (e.g. PostGIS), transformers (e.g. XSLT) and outputs (e.g. a GML file or even
WFS-T a geospatial protocol to publish GML to a server).
Stetl has been proven to handle 10's of millions of objects without any memory issues.
This is achieved through a technique called "streaming and splitting".
For example: using the OgrPostgisInput module an GML stream can be generated from the database.
A component called the GmlSplitter can split this stream into managable chunks (like 20000 features)
and feed this upstream into the ETL chain.
## Examples
Stetl has been found in particularly useful for INSPIRE-related transformations and other complex GML-related ETL.
See examples under the examples dir.
Another example in http://code.google.com/p/inspire-foss/source/browse/trunk/etl/NL.Kadaster/Addresses
(Dutch Addresses (BAG) to INSPIRE Addresses)
## Contributing
Anyone and everyone is welcome to contribute. Please take a moment to
review the [guidelines for contributing](CONTRIBUTING.md).
* [Bug reports](CONTRIBUTING.md#bugs)
* [Feature requests](CONTRIBUTING.md#features)
* [Pull requests](CONTRIBUTING.md#pull-requests)
## Finally
The word "stetl" is also an alternative writing for "shtetl":
http://en.wikipedia.org/wiki/Stetl : "...Material things were neither disdained nor
extremely praised in the shtetl. Learning and education were the ultimate measures of worth in the eyes of the community,
while money was secondary to status..."
Changes
=======
v1.0.9 - 17 june 2016
---------------------
See https://github.com/justb4/stetl/issues?q=milestone%3A%22Version+1.0.9%22+is%3Aclosed
Highlights:
- Substitutable config options in properties file (-a arg)
- Docker support via Stetl Docker image
- Generic ogr2ogr Input Component enhancement
v1.0.8 - 2 july 2015
--------------------
See https://github.com/justb4/stetl/issues?q=milestone%3A%22Version+1.0.8%22
- generic OgrOutput component
- Apache Log File Input
v1.0.7 - 24 nov 2014
--------------------
- start of Stetl Format conversion Filter
- generic OgrInput component
- SQLite Input component
- BAG to INSPIRE Addresses example with Jinja2 Templating Filter
- Jinja2 Filter refinements
v1.0.6 - 5 sept 2014
--------------------
- allow spatial_extent in OGROutput Top10NL example
- httpinput more robust
- Packet: new type 'record', as Python dict structure
- httpinput: ApacheDirInput, input data from Apache index listing
- dboutput: PostgresInsertOutput, insert single record into Postgres
- component: add before_/after_invoke and after_chain_invoke() for intercepting
- filters: start of Python Templating filters: simple string and Jinja2 templating
- Packet: new type 'struct' basically a free form dict, the result of reading CSV
- input: CSV file input
- new examples: 9_string_templating and 10_jinja2_templating
- start stetl --doc option to print class configuration info
- config: start of adding meta attribute config info via class vars of type Attr
v1.0.5 - 19 feb 2014
--------------------
- cater for strange lxml parse error: https://bugs.launchpad.net/lxml/+bug/1185701
- more Dutch BGT (large scale topo) examples by thijsbrentjens
v1.0.4 - 23 sept 2013
---------------------
- more documention
- Dutch BGT (Basis Registratie Grootschalige Topografie) example
- Ordnance Survey Mastermap example
- strip XML namespaces option to XmlElementStreamerFileInput
v1.0.1 v1.0.3 - aug/sept 2013
-----------------------------
Minor changes to enable distribution.
v1.0.0 - june 2013
------------------
- First version
- Add to Python Package Index (#3).
Credits
=======
Stetl is written by:
* Just van den Broecke (http://www.justobjects.nl)
This project would not be possible without the great work of Frank Warmerdam and other
GDAL/OGR developers (http://gdal.org).
Plus the people that brought Python, PostGIS (like Paul Ramsey), lxml and the libs
like GEOS, Proj, libxml2 and libxslt.
We are mainly standing on the shoulders of these giants.
Keywords: etl xsl gdal gis vector feature data
Platform: UNKNOWN
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 2
Classifier: Topic :: Scientific/Engineering :: GIS
|
