1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
|
Developer Guide
###############
Do not hang around in Pythran code base without your developer guide! It is
the compass that will guide you in the code jungle!
Disclaimer
----------
This document is a never ending work-in-progress draft. Please contribute!
Configuration
-------------
Pythran can be configured with a rc file. An example is found in `pythran/pythran.cfg`.
Look at it! To customize it::
$> cp pythran/pythran.cfg ~/.pythranrc
In particular, you may want to add ``-g -O0`` to the ``cxxflags``.
Coding Style
------------
All Python code must be conform to the PEP 8, and the ``flake8`` command must not
yield any message when run on our database. Additionally, avoid backslashes,
and try to make your code as concise as possible.
$> flake8 pythran/*.py pythran/*/*.py --exclude="pythran/tests/test*.py,__init__.py"
C++ code use spaces (no tabs) and a tab width of 4.
File Hierarchy
--------------
Listing the top level directory yields the following entries:
setup.py
The files that describes what gets installed, that holds ``PyPI`` entries
and such.
docs/
If you're reading this document, you know what it's all about! ``MANUAL``
is the user documentation and ``DEVGUIDE`` is the developer documentation.
Use ``make`` from this directory to produce the static website.
LICENSE
Boring but important stuff.
MANIFEST.in
Describe additional stuff to package there.
README.rst
Quick introduction and description of _pythran_.
pythran/
The source of all things.
pythran/tests/
The source of all issues.
pythran/pythonic/
Where C++ back-end lies.
Validation
----------
``pythran`` uses the ``unittest`` module and the `pytest
<http://pytest.org/latest/>`_ package to manage test cases.
All requirements are listed in ``pythran/tests/requirements.txt``.
The whole validation suite is run through the command::
$> python -m pytest pythran/tests
To run it faster we use the ``pytest`` extension `xdist
<https://pypi.org/project/pytest-xdist/>`_, the test suite will run using all
available cores. Otherwise it might run **very** slowly, something like four
hours on a decent laptop :'(.
Note that it is possible to use the ``pytest`` module to pass a subset of the
test suite::
$> pytest -n 8 pythran/tests/test_list.py
runs all the tests found in ``pythran/tests/test_list.py``.
There are two kinds of tests in ``pythran``:
1. unit tests that test a specific feature of the implementation. Such tests
are listed as method of a class deriving from ``test_env.TestEnv`` and must
call the ``run_test(function_to_translate, *effective_parameters,
**name_to_signature)`` method [1]_. It translates ``function_to_translate``
into a native function using the type annotations given in the
``name_to_signature`` dictionary, runs both the python and the native
version with ``effective_parameters`` as arguments and asserts the results
are the same.
.. [1] See examples in ``pythran/tests/test_base.py`` for more details.
2. test cases that are just plain python modules to be converted in native
module by ``pythran``. It is used to test complex situations, codes or
benchmarks found on the web etc. They are just translated, not run. These
test cases lie in ``pythran/tests/cases/`` and are listed in
``pythran/tests/test_cases.py``.
C++ runtime
-----------
The C++ code generated by ``pythran`` relies on a specific back-end,
``pythonic``. It is a set of headers that mimics Python's intrinsics and
collections behavior in C++. It lies in ``pythran/pythonic/``. There is one
directory per module, e.g. ``pythran/pythonic/numpy`` for the ``numpy`` module,
and one file per function, e.g. ``pythran/pythonic/numpy/ones.hpp`` for the
``numpy.ones`` function. Type definitions are stored in the seperate
``pythran/pythonic/types`` directory, one header per type. Each function header
must be ``#includ``-able independently, i.e. it itself includes all the type
and function definition it needs. This helps keeping compilation time low.
All Pythran functions and types live in the ``pythonic`` namespace. Each extra
module defines a new namespace, like ``pythonic::math`` or
``pythonic::random``, and each type is defined in the ``pythonic::types``
namespace. The ``DECLARE_FUNCTOR`` and ``DEFINE_FUNCTOR`` macros from
``pythonic/utils/functor.hpp`` is commonly used to convert functions into
functors and put them into the mandatory ``functor`` namespace.
The pythonic runtime can be used without Python support, so it is important to
protect all Python-specific stuff inside ``ENABLE_PYTHON_MODULE`` guard.
All methods are represented by functions in Pythran. The associated
pseudo-modules are prefixed and suffixed by a double underscore ``__``, as in
``pythran/pythonic/__list__``.
Benchmarking and Testing
------------------------
Stand-alone algorithms are put into ``pythran/tests/cases``. They must be valid
Pythran input (including spec annotations). To be taken into account by the
validation suite, they must be listed in ``pythran/tests/test_cases.py``. To be
taken into account by the benchmarking suite, they must have a line starting
with the ``#runas`` directive. Check ``pythran/tests/matmul.py`` for a complete
example.
To run the benchmark suite, one can rely on::
$> python setup.py bench --mode=<mode>
where *<mode>* is one among:
python
Uses the interpreter used to run ``setup.py``.
pythran
Uses the Pythran compiler.
pythran+omp
Uses the Pythran compiler in OpenMP mode.
All measurements are made using the ``timeit`` module. The number of iterations
is customizable through the ``--nb-iter`` switch.
How to
------
:Add support for a new module:
1. Provide its C++ implementation in ``pythran/pythonic++/<mymodule>``.
``pythran/pythonic++/math/*.hpp`` and
``pythran/pythonic++/__list__/*.hpp`` are good example to referer to.
2. Provide its description in ``pythran/tables.py``. Each function, method
or variable must be listed there with the appropriate description.
3. Provide its test suite in ``pythran/tests/`` under the name
``test_my_module.py``. One test case per function, method or variable
is great.
:Add a new analysis:
1. Subclass one of ``ModuleAnalysis``, ``FunctionAnalysis`` or ``NodeAnalysis``.
2. List analysis required by yours in the parent constructor, they will be built automatically and stored in the attribute with the corresponding uncameled name.
3. Write your analysis as a regular ``ast.NodeVisitor``. The analysis result must be stored in ``self.result``.
4. Use it either from another pass's constructor, or through the ``passmanager.gather`` function.
:Push changes into the holy trunk:
1. Use the ``github`` interface and the pull/push requests features
2. Make your dev available on the web and asks for a merge on the IRC
channel ``#pythran`` on ``irc.oftc.net``
(via your browser: https://webchat.oftc.net)
|
