1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
|
Testing Suite
========================================================================================
|coverage|
.. include:: ../README.rst
:start-after: begin_badges
:end-before: end_badges
Exhale uses `pytest <https://docs.pytest.org/en/latest/>`_ for its testing suite. To
run the tests, Exhale uses `tox <https://tox.wiki/en/latest/>`_.
.. code-block:: console
$ cd /path/to/exhale
$ pip install tox
$ tox
.. note::
See ``tox.ini`` in the root of the repository. You only need to install ``tox`` to
be able to run the tests, which will internally download the additional dependencies
such as ``pytest``. The tests will be run in a virtual environment.
Running Specific Tests
----------------------------------------------------------------------------------------
By default, ``tox`` will run the python tests and linting checks with
`flake8 <https://flake8.pycqa.org/en/latest/>`_. More formally, the default environment
list is defined as:
.. code-block:: ini
[testenv]
envlist = py, flake8
This means that the version of python the tests are run with are the interpreter that
you installed ``tox`` for. To run a specific test:
``tox -e py``
Run the python unit tests.
``tox -e flake8``
Run the lint tests.
``tox -e docs``
Build the sphinx documentation using the *html* builder (in nitpicky mode). You can
view the generated html website with ``open .tox/docs/tmp/html/index.html``.
``tox -e linkcheck``
Build the sphinx documentation using the *linkcheck* builder.
.. tip::
If you need to debug a test case to discern why it is failing, the vibrantly colorful
`ipdb <https://pypi.org/project/ipdb/>`_ debugger is already installed in the
environment running the tests. It is nearly identical to ``pdb``, and although it
is designed for IPython, it works just as well in "regular" code. Just like with
``pdb``, set a trace in the test case you are debugging:
.. code-block:: py
import ipdb
ipdb.set_trace()
In order to be able to achieve the break-point, you would launch the tests with
.. code-block:: console
$ tox -e py -- -s
The ``--`` signals that the command line arguments for ``tox`` are complete, and
everything afterward should be forwarded to the specified environment (in this case
to ``pytest`` since we launched ``-e py``). The ``-s`` switch for ``pytest`` enables
the debugger to stop the test, without this flag the test will fail at the
break-point.
The same goes for other test cases, if you need to specify a flag to ``flake8`` for
some reason you would run ``tox -e flake8 -- --ignore XYYY`` where ``XYYY`` is the
linter check you want to ignore.
Writing Project Test Cases
----------------------------------------------------------------------------------------
``exhale`` provides tools to build tests that need to generate output files (= ``*.rst``
files) from actual test projects. To create such tests:
1. Create a new test-case folder in ``testing/projects/{new project name}``.
2. Inherit from :class:`testing.base.ExhaleTestCase` in a module you create
in the ``testing/tests`` folder, setting the ``test_project`` class-level variable
to ``{new project name}``
3. Add some ``test_{something}`` methods. :class:`testing.base.ExhaleTestCase` will
work its magic to apply the necessary pytest fixtures to all these methods so that
the configuration is applied, ``sphinx`` is bootstrapped and all ``*.rst`` files
generated by ``exhale`` are in the specified output directory at the start of your
test. Note that the ``sphinx`` application is made available as ``self.app``.
4. In case you need to run tests with a configuration differing from the default
configuration, you can use the :func:`testing.decorators.confoverrides` decorator
to tweak the configuration for a test method or the whole test class::
@confoverrides(conf_var1=conv_val1, conf_var2=conf_val2)
class MyTestCase(ExhaleTestCase):
test_project = 'my_project'
or::
@confoverrides(conf_var1=conv_val1, conf_var2=conf_val2)
def test_something(self):
...
Note that ``@confoverrides`` applied to a test method are combined with
``@confoverrides`` applied to test classes and have precedence.
5. In case you don't want ``exhale`` to generate any files and just need to test the
configuration, you can use the :func:`testing.decorators.no_run` on a test
method or test class::
@no_run
class MyTestCase(ExhaleTestCase):
test_project = 'my_project'
or::
@no_run
def test_something_else(self):
...
For more examples, just have a look at the existing tests in the ``testing/tests``
folder.
Full Testing Suite Documentation
----------------------------------------------------------------------------------------
..
A **HUGE** thank you to Thomas Khyn for his thorough help figuring out how to
coordinate ``pytest``, ``sphinx``, metaclasses, and more -- this testing framework
would never have come to fruition without his
`epic pull request <https://github.com/svenevs/exhale/pull/31>`_.
.. automodule:: testing
:members:
.. toctree::
:maxdepth: 2
testing/base
testing/conftest
testing/decorators
testing/fixtures
testing/hierarchies
testing/projects
testing/utils
testing/tests
|
