1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
|
Notes on the MontagePy Package
Layout
------
The top level directory for the package is "MontagePy". See the diagram
below.
In it is another subdirectory of the same name ("MontagePy") which will
contain the files that make up the package distributed to the Python Package
Index (PyPI). This doubly nested structure is normal for Python packages.
The outer top level directory name is immaterial to the build but it would
normally be silly to use something different from the package name.
The "MontagePy" subdirectory initially contains "__init__.py", a
file that tell Python that this is a package. It can also contain the
package description (available using help()) and setup code, though we
don't need the latter. The other files necessary to make that package will
copied in from elsewhere or built by Cython as described below.
This top level directory mainly contains the build instructions ("setup.py").
This is a simple Python script that provides metadata about the package.
|
| Note: The descriptive metadata can include the following parameters:
|
| name name of the package
| version version of this release
| author package author’s name
| author_email email address of the package author
| maintainer package maintainer’s name short string
| maintainer_email email address of the package maintainer
| url home page for the package
| description short, summary description of the package
| long_description longer description of the package
| download_url location where the package may be downloaded
| classifiers a list of classifiers
| platforms a list of platforms
| license license for the package
|
For MontagePy, this metadata also includes the C library and Cython wrapper
dependencies. It has a couple of active bits: it checks for the existence
of the Montage/lib and Montage/MontageLib directories; and it generates the
"extension module" parameter by running cythonize() on the .pyx files (below) and
list of libraries to create the linkage between python and the Montage C
libraries.
The files needed to build the Cython wrapper are "parse.py" and in the "templates"
directory. Basically, parse.py looks through the Montage source directories for
a set of JSON files we have created that describe each module and creates a
Python 'functions' structure with the information. Then it uses the templates
to create the following files in the "MontagePy" subdirectory:
templates/template_main.pyx -> MontagePy/main.pyx
templates/template.pyx -> MontagePy/_wrappers.pyx
templates/template.pxd -> MontagePy/wrappers.pxd
main.pyx -- .pyx files are compiled by Cython to .c files, containing
the code of a Python extension module. The .c file is compiled by a C compiler
to a .so file which can be import -ed directly into a Python session.
_wrappers.pyx -- Contains the code describing the calling syntax for each
Montage module. This conde is converted by Cython into C that can be called
by Python. Here is the snippet for the Montage "mAdd" function.
wrappers.pxd -- In addition to .pyx source files, Cython uses .pxd files
which work like C header files – they contain Cython declarations which are
only meant for inclusion by Cython modules. A pxd file is imported into a
pyx module by using the cimport keyword. In our case, the declarations
involve the names and calling parameters for each Montage C library module.
The creation of the C code associated with the above (main.c and _wrappers.c)
and the compiled shared-object files from these (the .so files are what actually
gets distributed) is done by cythonize() commands in setup.py.
The package layout looks like this:
MontagePy
|
| setup.py (Package setup script.
| Besides defining the package
| this also runs the cythonize()
| function that populates the
| MontagePy directory.)
|
| parse.py (Run manually before distributing
| to create the pyx/pxd files from
| the templates. Scans MontageLib
| for JSON module definitions.)
|
| validate_json.py (A tool used during development
| to find and check the validity
| of the module JSON files in
| MontageLib directories.)
|
| lib (This directory contains all the
| support library .o files and was
| populated by the MontageLib
| Makefile.)
|
|
| MontagePy |
| | __init__.py (Package initialization file. In
| | our case this only contains the
| | package description text.)
| |
| | wrappers.pxd (The first three files here
| | main.pyx are generated before distribution
| | _wrappers.pyx using a "parse.py" utility.)
| |
| | main.c (The C versions are generated
| | _wrappers.c from the above using cythonize()
| | during setup.)
| |
| | main.so (As are the compiled library
| | _wrappers.so versions. Only these files are
| | distributed, the above five are
| | not.)
| |
| | FreeSans.tff (The mViewer module required a
| font file and we include one rather
| than trying to find it on the user
| machine.)
|
| templates |
| | template.pxd (These templates are used
| | template_main.pyx by the "parse.py" script
| | template.pyx to generate the pyx/pxd
| files above.)
|
| dist (This directory gets created and
filled when setup.py is run. The
resulting .whl files get uplaoded
to PyPI for public consumption.)
Build
-----
Building the MontagePy package consists of running "python parse.py" to build
the Cython wrapper code and "python setup.py build_wheel --inplace", or variants
on this command. Our setup code requires Python packages "Cython" and "jinja2"
which need to be pip-installed beforehand. The top-level directory incudes a
"make.sh" file with the commands to produce a clean package and upload it. This
script removes previous attempts from the package index and uninstalls the package
locally in preparation for the new attempt. It also has code to tweak one of
the functions in the cythonized _wrappers.pyx file: We needed to include a
reference in the Python part of the code there to our included font file.
But first we describe below how we built the package and our test "package index"
setup and finally the process of getting it into PyPI.
. . . . . .
Ultimately, we want to upload this package to PyPI for others to use but since
our initial trial and error phase will involve a lot of false starts we don't
really want that to be too public. So instead we create a package index service
of our own, which is reasonably simple as there are a number of simple server
packages out there. Here's the one we set up:
|
| On desktop Mac (local machine "mosaic"), in directory: /Users/jcg/PyPIserver
|
| pip install pypiserver
| rehash
| mkdir packages
| pypi-server -p 9022 -P htpasswd.txt packages &
|
| This can be accessed through pip but also viewed through a
| browser at
|
| http://mosaic:9022
|
| Once we have a package (on exodev in Montage directory) we
| can upload it to this server using setup.py. (On ours, the
| password is "p....s".)
|
. . . . . .
The processes of building the package, uploading it to a package index,
and installing it locally (though we don't do the last this way) are all
handled by "setup.py". This file is a python script but one that mainly
calls the setuptools.setup() method. This method is given all the package
metadata (e.g. "author", "version") and does all the work of compiling
building distribution files, installing and uploading to a package index
for others to get.
You activate these steps with commands like
python setup.py build
python setup.py bdist_wheel upload -r http://mosaic.ipac.caltech.edu:9022
In the last command we uploaded it to our local package index server using
the "-r" flag.
To actually install the package from our index we use the standard 'pip'
command (with the alternate index again called out):
pip install -i http://mosaic.ipac.caltech.edu:9022 MontagePy
However, since we ran the package index server without secure HTTPS we
need to authorize the client pip to use it. This is done by adding pip
configuration (the ~/.pip/pip.conf file):
[global]
extra-index-url = http://mosaic.ipac.caltech.edu:9022
[install]
trusted-host = mosaic.ipac.caltech.edu
Use
---
The Montage package is called MontagePy but due to the Cython machinations
we need to refer to modules used as coming from "MontagePy.main":
from MontagePy.main import *
After that, we can use it just like any normal Python module, e.g.:
help("MontagePy")
help(mHdr)
mHdr("M51", 0.5, 0.5, "region.hdr")
|
