1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286
|
===============================
Building From Source on Windows
===============================
.. contents::
:local:
Overview
--------
Compared to OSX and Linux, building NumPy and SciPy on Windows is more difficult,
largely due to the lack of compatible, open-source libraries like LAPACK_ or
ATLAS_ that are necessary to build both libraries and have them perform
relatively well. It is not possible to just call a one-liner on the command
prompt as you would on other platforms via ``sudo apt-get install`` machinery.
This document describes one option to build OpenBLAS and SciPy from source that
was validated for scipy 1.0.0. However, in light of all the work currently being
done, **do not expect** these instructions to be accurate in the long-run and be
sure to check up on any of the open source projects mentioned for the most up-to-date
information. For more information on all of these projects, the Mingwpy_ website
is an excellent source of in-depth information than this document will provide.
.. _Mingwpy: http://mingwpy.github.io/
.. _ATLAS: http://math-atlas.sourceforge.net/
.. _OpenBLAS: https://github.com/xianyi/OpenBLAS
.. _LAPACK: http://www.netlib.org/lapack/
Building the Released SciPy
---------------------------
This section provides the step-by-step process to build the released scipy. If you want
to build completely from source, you should estimate at least three hours to build all
libraries and compile SciPy. Feel free to stop and inspect any step at any time, but
for this section, we'll just mention the steps without providing an in-depth explanation
for the reasons behind them. If you have further questions about what we're doing, more
in-depth documentation is provided in the sections below. Also, please make sure to read
this section before proceeding, as the presence or absence of error messages in general
is not a good indication of whether you've completed a step correctly. Each step creates
particular files, and what ultimately matters is whether you have built the required files
rather than whether error messages appeared in your terminal.
Building OpenBLAS
=================
First, we need to install the software required to build OpenBLAS_, which is the BLAS_
library that we're going to use. Because the software to build OpenBLAS is different than
that required to build SciPy and because OpenBLAS takes a long time to build, we're going
to start building OpenBLAS first and then explain what to do next while the OpenBLAS build
is running. **Alternatively, if you'd rather download a pre-built OpenBLAS, download the
one of the** `pre-built zip files`_ **and skip to the Installing OpenBLAS section below.**
Otherwise, install MSYS2 using `these instructions`_ including the `pacman` update
instructions. Occasionally during the updates the terminal might ask you to close the
terminal but then might refuse to be closed and hang. If this happens you can kill it via
Task Manager and continue with the instructions. Make sure to install the correct
architecture for the SciPy that you want to build (eg. 32 or 64 bit). Now, you have three
options for opening a terminal which are MSYS2, MINGW (32 or 64 bit). After updating all
the packages, now we are ready to install some more package bundles that we will need.
Open a MSYS2 terminal and type the following depending on the architecture of your
choice; run the following for a 32-bit build
.. code:: shell
pacman -S --needed base-devel mingw-w64-i686-toolchain mingw-w64-i686-cmake git
and for 64-bit
.. code:: shell
pacman -S --needed base-devel mingw-w64-x86_64-toolchain mingw-w64-x86_64-cmake git
It will prompt to whether install everything in these packages and you can simply accept
all via hitting enter key at each step.
We should be aware of the fact that these tools also install Python2, very similar to
a virtual environment, which is only usable within an MSYS2 terminal and we are **not**
going to use it at any point.
If you already have a GitHub repository folder where you keep your own repos, it is better
to use that location to keep things nice and tidy since we are going to clone yet another
repository to obtain the source code, hence
.. code:: shell
cd /c/<wherever the GitHub repo folder is>/GitHub
You don't necessarily need to build in that particular location, but it should be somewhere
convenient. To make sure that we're ready to build, type the following in the terminal:
.. code:: shell
make
gfortran
gcc
These commands should give errors as we have not provided any arguments to them.
However an error also implies that they are accessible on the path. Now clone
the repository required to build OpenBLAS:
.. code:: shell
git clone https://github.com/matthew-brett/build-openblas.git
cd build-openblas
git submodule update --init --recursive
If any of these commands fail, you're not ready to build. Go back and make sure that MSYS2
is installed correctly and has the required packages enabled. Now, let's set some
environment variables. In the MSYS2 terminal, type the following.
.. code:: shell
export OPENBLAS_COMMIT=5f998ef
export OPENBLAS_ROOT="C:\\opt"
export BUILD_BITS=64
Please check these variables' purpose for a moment. More specifically, make sure that
you have read/write access to the path that :code:`OPENBLAS_ROOT` points to. The output of the
OpenBLAS build will be collected in this folder. Make sure that the :code:`OPENBLAS_COMMIT`
points to the correct OpenBLAS commit that you want to build in the cloned repo. In the
future, :code:`build_openblas` repository might get updated and you might want to get those
updates by changing the commit. Make sure that the architecture is correctly set to either
32 or 64 bit. And after you've made sure of that, start the OpenBLAS build with:
.. code:: shell
./build_openblas.sh
Building OpenBLAS is challenging. The build may fail with an error after a few
hours but may also fail silently and produce an incorrect binary. Please, if you
have any issues, `report them`_ so that we can save the next person's time.
While you're waiting on OpenBLAS to finish building, go ahead and install `build tools`_
from Microsoft, since these take a while to install and you'll need them later.
After the :code:`build_openblas.sh` script has completed (probably with an error), there
should be an :code:`openblas.a` file somewhere on your system. If :code:`OPENBLAS_ROOT` was
set to :code:`C:\\opt`, then you might see a line like this in the MSYS2 terminal:
.. code:: shell
Copying the static library to /c/opt/64/lib
Installing OpenBLAS
===================
If you see that line, then you might have OpenBLAS correctly built, even if other failures
occurred. Look in that folder for :code:`openblas.a`. If you find a file called something
like :code:`libopenblas_5f998ef_gcc7_2_0.a`, just rename it to :code:`openblas.a` and continue.
If the file isn't there, then poke around and try to find the file elsewhere in
:code:`OPENBLAS_ROOT`. If you don't have that file, you'll probably need to find out what
happened and then build OpenBLAS again. But if you have that file, we'll assume that you've
completed this step correctly. Proceeding on that assumption, let's build SciPy.
**Before continuing, make sure that you don't have other copies of either**
:code:`openblas.lib` **or** :code:`libopenblas.lib` **on your computer elsewhere.
Multiple copies could result in later build errors that will be difficult to debug.
You may verifiy that the openblas library was correctly picked up by looking for
the following in your build log:**
.. code:: shell
FOUND:
libraries = ['openblas']
library_dirs = ['C:\opt\64\lib']
language = c
define_macros = [('HAVE_CBLAS', None)]
Building SciPy
==============
Once you have built OpenBLAS, it's time to build SciPy. Before continuing make sure to
install the following software for building on the latest Python version. For building
on other Python versions, see the WindowsCompilers_ page.
1) Install Microsoft Visual Studio 2015 or 2017 Community Edition (use the `build tools`_
from Microsoft)
2) Finally, install Python from https://python.org/ (make sure to check the box to install
pip)
After you've installed the required software, open an MSYS2 terminal, change to a good
location to build, and clone SciPy.
.. code:: shell
cd C:\Users\MyUser\Downloads
git clone https://github.com/scipy/scipy.git
cd scipy
Now we need to copy the :code:`openblas.a` file that we've built earlier to the correct
location. If your Python is installed somewhere like the following:
.. code:: shell
C:\Users\<user name>\AppData\Local\Programs\Python\Python36\python.exe
Then you'll need to put the :code:`openblas.a` file somewhere like the following:
.. code:: shell
C:\Users\<user name>\AppData\Local\Programs\Python\Python36\Lib
Adjust the location accordingly based on where :code:`python.exe` is located. Now for a
sanity check. Type the following and press enter.
.. code:: shell
gfortran
If you see an error with the above command, :code:`gfortran` is not correctly installed.
Go back to the "Building OpenBLAS" section and make sure that you have installed the correct
tools.
Now install the dependencies that we need to build and test SciPy. **It's important that you
specify the full path to the native Python interpreter so that the built-in MSYS2 Python will
not be used. Attempting to build with the MSYS2 Python will not work correctly.**
**18 October 2017** Until NumPy 1.14 is officially released, we have to work with the latest
development version of the NumPy repository. See the NumPy documentation....
.. code:: shell
/c/Users/<user name>/AppData/Local/Programs/Python/Python36/python.exe \
-m pip install numpy cython pytest pytest-xdist pytest-faulthandler
Please note that this is a simpler procedure than what is used for the official binaries.
**Your binaries will only work with the latest NumPy (v1.14.0dev and higher)**. For
building against older Numpy versions, see `Building Against an Older Numpy Version`_.
Make sure that you are in the same directory where ``setup.py`` is (you should be if you
have not changed directories):
.. code:: shell
ls setup.py
Assuming that you have set up everything correctly, you should be ready to build. Run
the following commands:
.. code:: shell
/c/Users/<user name>/AppData/Local/Programs/Python/Python36/python.exe \
-m pip wheel -v -v -v .
/c/Users/<user name>/AppData/Local/Programs/Python/Python36/python.exe \
runtests.py --mode full
Congratulatations, you've built SciPy!
.. _BLAS: https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms
.. _OpenBLAS: https://github.com/xianyi/OpenBLAS
.. _`these instructions`: https://github.com/msys2/msys2/wiki/MSYS2-installation
.. _`build tools`: https://www.visualstudio.com/downloads/#build-tools-for-visual-studio-2017
.. _`report them`: https://github.com/scipy/scipy/issues/new
.. _`pre-built zip files`: https://3f23b170c54c2533c070-1c8a9b3114517dc5fe17b7c3f8c63a43.ssl.cf2.rackcdn.com/
.. _WindowsCompilers: https://wiki.python.org/moin/WindowsCompilers
Building Against an Older Numpy Version
---------------------------------------
If you want to build SciPy to work with an older numpy version, then you will need
to replace the Numpy "distutils" folder with the folder from the latest numpy.
The following powershell snipped can upgrade Numpy distutils while retaining an older
Numpy ABI_.
.. code:: shell
$NumpyDir = $((python -c 'import os; import numpy; print(os.path.dirname(numpy.__file__))') | Out-String).Trim()
rm -r -Force "$NumpyDir\distutils"
$tmpdir = New-TemporaryFile | %{ rm $_; mkdir $_ }
git clone -q --depth=1 -b master https://github.com/numpy/numpy.git $tmpdir
mv $tmpdir\numpy\distutils $NumpyDir
.. _ABI: https://en.wikipedia.org/wiki/Application_binary_interface
Additional Resources
--------------------
As discussed in the overview, this document is not meant to provide extremely detailed explanations on how to build
NumPy and SciPy on Windows. This is largely because currently, there is no single superior way to do so
and because the process for building these libraries on Windows is under development. It is likely that any
information will go out of date relatively soon. If you wish to receive more assistance, please reach out to the NumPy
and SciPy mailing lists, which can be found `here <http://www.scipy.org/scipylib/mailing-lists.html>`__. There are many
developers out there, working on this issue right now, and they would certainly be happy to help you out! Google is also
a good resource, as there are many people out there who use NumPy and SciPy on Windows, so it would not be surprising if
your question or problem has already been addressed.
|