1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
|
.. _devpy-test:
===========================
Running SciPy Tests Locally
===========================
Basic test writing and execution from within the Python interpreter is
documented in the
:doc:`NumPy/SciPy testing guidelines <numpy:reference/testing>`. This page
includes information about running tests from the command line using SciPy's
``dev.py`` command line tool. *Note: Before beginning, ensure that* |pytest|_
*is installed.*
.. note::
The ``dev.py`` interface is self-documenting, in the sense that everything on
this page and more (including usage examples for each command) can be
accessed with ``python dev.py --help`` and for individual commands like
``python dev.py <command-name> --help``. In this case, you can check
``python dev.py test --help``.
To run all tests, navigate to the root SciPy directory at the command
line and execute
::
python dev.py test
This builds SciPy (or updates an existing build) and runs the tests.
To run tests on a particular submodule, such as ``optimize``, use the
``--submodule`` option:
::
python dev.py test -s optimize
To run a particular test module, use the Pytest syntax of ``--test`` (or
``-t``)::
python dev.py test -t scipy.<module>.tests.<test_file>
Example for |test-linprog|_ file tests, run:
::
python dev.py test -t scipy.optimize.tests.test_linprog
To run a test class:
::
python dev.py test -t scipy.<module>.tests.<test_file>::<TestClass>
Example for ``TestLinprogRSCommon`` class from ``test_linprog.py``:
::
python dev.py test -t scipy.optimize.tests.test_linprog::TestLinprogRSCommon
To run a particular test:
::
python dev.py test -t scipy.<module>.tests.<test_file>::<test_name>
Example for ``test_unknown_solvers_and_options`` from ``test_linprog.py``:
::
python dev.py test -t scipy.optimize.tests.test_linprog::test_unknown_solvers_and_options
For tests within a class, you need to specify the class name and the test name:
::
python dev.py test -t scipy.<module>.tests.<test_file>::<TestClass>::<test_name>
Example:
::
python dev.py test -t scipy.optimize.tests.test_linprog::TestLinprogRSCommon::test_nontrivial_problem_with_guess
Other useful options include:
- ``-v`` or ``--verbose``, which activates the verbose option for more
detailed output.
- ``-b`` or ``--array-api-backend`` *backend* to include alternative
array backends in array-api-compatible tests. See :ref:`dev-arrayapi`
for details.
- ``--coverage`` to generate a test coverage report in
``scipy/build/coverage/index.html``. *Note:* |pytest-cov|_ *must be
installed.*
- ``-n`` or ``--no-build`` to prevent SciPy from updating the build
before testing
- ``-j`` or ``--parallel`` *n* to engage *n* cores when building SciPy;
e.g. \ ``python dev.py test -j 4`` engages four cores. As of `#10172`_
this also runs the tests on four cores if |pytest-xdist|_ is installed.
- ``-m full`` or ``--mode full`` to run the "full" test suite, including
tests marked ``slow`` (e.g. with ``@pytest.mark.slow``). Note that this
does not *run* tests marked ``xslow``; see Tips below.
- ``--`` to send remaining command line arguments to ``pytest`` instead of
``dev.py test``. For instance, while ``-n`` sent to ``pytest.py`` activates
the ``--no-build`` option, ``-n`` sent to ``pytest`` runs the tests on
multiple cores; e.g. \ ``python dev.py test -- -n 4`` runs tests using
four cores. *Note:* |pytest-xdist|_ *must be installed for testing on
multiple cores.* Common command line arguments for ``pytest`` include:
- ``--durations=m`` to display durations of the slowest ``m`` tests. Use
``--durations=0`` together with ``--durations-min=x`` to display
durations of all tests with durations that exceed ``x`` seconds.
- ``--fail-slow=x`` to cause test to fail if they exceed ``x`` seconds.
(*Note*: |pytest-fail-slow|_ must be installed.)
- ``--timeout=x`` to halt all test execution if any test time exceeds
``x`` seconds. (*Note*: |pytest-timeout|_ must be installed.)
For much more information about ``pytest``, see the ``pytest``
`documentation <https://docs.pytest.org/en/latest/usage.html>`_.
Tips:
-----
If you built SciPy from source but are having trouble running tests
after a change to the codebase, try deleting the ``scipy/build``
directory. This forces ``dev.py`` to completely rebuild SciPy before
performing tests.
There is an additional level of very slow tests (several minutes),
which are disabled even when calling ``python dev.py test -m full``.
They can be enabled by setting the environment variable ``SCIPY_XSLOW=1``
before running the test suite.
By default, tests that use ``Hypothesis`` run with the ``deterministic``
profile defined in ``scipy/scipy/conftest.py``. This profile includes the
Hypothesis setting ``derandomize=True`` so the same examples are used until
Hypothesis, Python, or the test function are updated. To better use
Hypothesis' abilities to find counterexamples, select the ``nondeterministic``
profile by setting the environment variable
``SCIPY_HYPOTHESIS_PROFILE=nondeterministic`` before running the test suite.
The number of examples that are run can be configured by editing the selected
configuration, e.g. adding ``max_examples=100_000``.
.. |pytest-cov| replace:: ``pytest-cov``
.. _pytest-cov: https://pypi.org/project/pytest-cov/
.. _#10172: https://github.com/scipy/scipy/pull/10172
.. |pytest-xdist| replace:: ``pytest-xdist``
.. _pytest-xdist: https://pypi.org/project/pytest-xdist/
.. |pytest-fail-slow| replace:: ``pytest-fail-slow``
.. _pytest-fail-slow: https://github.com/jwodder/pytest-fail-slow
.. |pytest-timeout| replace:: ``pytest-timeout``
.. _pytest-timeout: https://github.com/pytest-dev/pytest-timeout
.. |pytest| replace:: ``pytest``
.. _pytest: https://docs.pytest.org/en/latest/
.. |test-linprog| replace:: ``scipy/optimize/tests/test_linprog.py``
.. _test-linprog: https://github.com/scipy/scipy/blob/main/scipy/optimize/tests/test_linprog.py
.. |Hypothesis| replace:: ``Hypothesis``
.. _Hypothesis: https://hypothesis.readthedocs.io/en/latest/
|
