1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
|
Distributing
============
Distributing Python packages is nontrivial - especially for a package with
complex build requirements like Scipy - and subject to change. For an up-to-date
overview of recommended tools and techniques, see the `Python Packaging User
Guide`_. This document discusses some of the main issues and considerations for
Scipy.
Dependencies
------------
Dependencies are things that a user has to install in order to use (or
build/test) a package. They usually cause trouble, especially if they're not
optional. Scipy tries to keep its dependencies to a minimum; currently they
are:
*Unconditional run-time dependencies:*
- Numpy_
*Conditional run-time dependencies:*
- nose_ (to run the test suite)
- asv_ (to run the benchmarks)
- matplotlib_ (for some functions that can produce plots)
- Pillow_ (for image loading/saving)
- scikits.umfpack_ (optionally used in ``sparse.linalg``)
- mpmath_ (for more extended tests in ``special``)
*Unconditional build-time dependencies:*
- Numpy_
- A BLAS and LAPACK implementation (reference BLAS/LAPACK, ATLAS, OpenBLAS,
MKL, Accelerate are all known to work)
- (for development versions) Cython_
*Conditional build-time dependencies:*
- setuptools_
- wheel_ (``python setup.py bdist_wheel``)
- Sphinx_ (docs)
- matplotlib_ (docs)
- LaTeX (pdf docs)
- Pillow_ (docs)
Furthermore of course one needs C, C++ and Fortran compilers to build Scipy,
but those we don't consider to be dependencies and are therefore not discussed
here. For details, see http://scipy.org/scipylib/building/index.html.
When a package provides useful functionality and it's proposed as a new
dependency, consider also if it makes sense to vendor (i.e. ship a copy of it with
scipy) the package instead. For example, six_ and decorator_ are vendored in
``scipy._lib``.
The only dependency that is reported to pip_ is Numpy_, see
``install_requires`` in Scipy's main ``setup.py``. The other dependencies
aren't needed for Scipy to function correctly, and the one unconditional build
dependency that pip_ knows how to install (Cython_) we prefer to treat like a
compiler rather than a Python package that pip_ is allowed to upgrade.
Issues with dependency handling
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
There are some serious issues with how Python packaging tools handle
dependencies reported by projects. Because Scipy gets regular bug reports
about this, we go in a bit of detail here.
Scipy only reports its dependency on Numpy via ``install_requires`` if Numpy
isn't installed at all on a system. This will only change when there are
either 32-bit and 64-bit Windows wheels for Numpy on PyPI or when
``pip upgrade`` becomes available (with sane behavior, unlike ``pip install
-U``, see `this PR
<https://github.com/pypa/pip/pull/3194>`_). For more details, see
`this summary <http://article.gmane.org/gmane.comp.python.distutils.devel/24218>`_.
The situation with ``setup_requires`` is even worse; pip_ doesn't handle that
keyword at all, while ``setuptools`` has issues (here's a `current one
<https://bitbucket.org/pypa/setuptools/issues/391>`_) and invokes
``easy_install`` which comes with its own set of problems (note that Scipy doesn't
support ``easy_install`` at all anymore; issues specific to it will be closed
as "wontfix").
Supported Python and Numpy versions
-----------------------------------
The Python_ versions that Scipy supports are listed in the list of PyPI
classifiers in ``setup.py``, and mentioned in the release notes for each
release. All newly released Python versions will be supported as soon as
possible. The general policy on dropping support for a Python version is that
(a) usage of that version has to be quite low (say <5% of users) and (b) the
version isn't included in an active long-term support release of one of the
main Linux distributions anymore. Scipy typically follows Numpy, which has a
similar policy. The final decision on dropping support is always taken on the
scipy-dev mailing list.
The lowest supported Numpy_ version for a Scipy version is mentioned in the
release notes and is encoded in ``scipy/__init__.py`` and the
``install_requires`` field of ``setup.py``. Typically the latest Scipy release
supports 3 or 4 minor versions of Numpy. That may become more if the frequency
of Numpy releases increases (it's about 1x/year at the time of writing).
Support for a particular Numpy version is typically dropped if (a) that Numpy
version is several years old, and (b) the maintenance cost of keeping support
is starting to outweigh the benefits. The final decision on dropping support
is always taken on the scipy-dev mailing list.
Supported versions of optional dependencies and compilers is less clearly
documented, and also isn't tested well or at all by Scipy's Continuous
Integration setup. Issues regarding this are dealt with as they come up in the
issue tracker or mailing list.
Building binary installers
--------------------------
.. note::
This section is only about building Scipy binary installers to *distribute*.
For info on building Scipy on the same machine as where it will be used, see
`here <http://scipy.org/scipylib/building/index.html>`_.
There are a number of things to take into consideration when building binaries
and distributing them on `PyPI`_ or elsewhere.
**General**
- A binary is specific for a single Python version (because different Python
versions aren't ABI-compatible, at least up to Python 3.4).
- Build against the lowest Numpy version that you need to support, then it will
work for all Numpy versions with the same major version number (Numpy does
maintain backwards ABI compatibility).
**Windows**
- For 64-bit Windows installers built with a free toolchain, use the method
documented at https://github.com/numpy/numpy/wiki/Mingw-static-toolchain.
That method will likely be used for Scipy itself once it's clear that the
maintenance of that toolchain is sustainable long-term. See the MingwPy_
project and `this thread
<http://article.gmane.org/gmane.comp.python.numeric.general/61727>`_ for
details.
- The other way to produce 64-bit Windows installers is with ``icc``, ``ifort``
plus ``MKL`` (or ``MSVC`` instead of ``icc``). For Intel toolchain
instructions see
`here <https://software.intel.com/en-us/articles/numpyscipy-with-intel-mkl>`_
and for (partial) MSVC instructions see
`here <https://github.com/numpy/numpy/wiki/Building-with-MSVC>`_.
- Older Scipy releases contained a .exe "superpack" installer. Those contain
3 complete builds (no SSE, SSE2, SSE3), and were built with
https://github.com/numpy/numpy-vendor. That build setup is known to not work
well anymore and is no longer supported. It used g77 instead of gfortran,
due to complex DLL distribution issues (see `gh-2829
<https://github.com/scipy/scipy/issues/2829>`_). Because the toolchain is no
longer supported, g77 support isn't needed anymore and Scipy can now include
Fortran 90/95 code.
**OS X**
- To produce OS X wheels that work with various Python versions (from
python.org, Homebrew, MacPython), use the build method provided by
https://github.com/MacPython/scipy-wheels.
- DMG installers for the Python from python.org on OS X can still be produced
by ``tools/scipy-macosx-installer/``. Scipy doesn't distribute those
installers anymore though, now that there are binary wheels on PyPi.
**Linux**
Besides PyPi not allowing Linux wheels (which is about to change with `PEP 513
<https://www.python.org/dev/peps/pep-0513>`_), there are no specific issues with
building binaries. To build a set of wheels for a Linux distribution and
providing them in a Wheelhouse_, look at the wheel_ and Wheelhouse_ docs. A
Wheelhouse for wheels compatible with TravisCI is http://wheels.scipy.org.
.. _Numpy: http://numpy.org
.. _Python: http://python.org
.. _nose: http://nose.readthedocs.org
.. _asv: http://asv.readthedocs.org
.. _matplotlib: http://matplotlib.org
.. _Pillow: http://pillow.readthedocs.org
.. _scikits.umfpack: https://pypi.python.org/pypi/scikit-umfpack
.. _mpmath: http://mpmath.org
.. _Cython: http://cython.org
.. _setuptools: https://bitbucket.org/pypa/setuptools
.. _wheel: wheel.readthedocs.org
.. _pip: http://pip-installer.org
.. _PyPI: http://pypi.python.org/pypi/scipy
.. _Python Packaging User Guide: https://packaging.python.org
.. _Wheelhouse: https://pypi.python.org/pypi/Wheelhouse
.. _MingwPy: https://mingwpy.github.io
|
