1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
|
TAP Producers
=============
tappy integrates with ``unittest`` based test cases to produce TAP output.
The producers come in three varieties:
support with only the standard library,
support for `nose <https://nose.readthedocs.io/en/latest/>`_,
and support for `pytest <http://pytest.org/latest/>`_.
* ``TAPTestRunner`` - This subclass of ``unittest.TextTestRunner`` provides all
the functionality of ``TextTestRunner`` and generates TAP files or streams.
* tappy for **nose** - tappy provides a plugin (simply called ``TAP``)
for the **nose** testing tool.
* tappy for **pytest** - tappy provides a plugin called ``tap``
for the **pytest** testing tool.
* tappy as the test runner - tappy can run like ``python -m unittest``.
Run your test suite with ``python -m tap``.
By default, the producers will create one TAP file for each ``TestCase``
executed by the test suite.
The files will use the name of the test case class with a ``.tap``
extension. For example:
.. code-block:: python
class TestFoo(unittest.TestCase):
def test_identity(self):
"""Test numeric equality as an example."""
self.assertTrue(1 == 1)
The class will create a file named ``TestFoo.tap`` containing the following.
.. code-block:: tap
# TAP results for TestFoo
ok 1 - Test numeric equality as an example.
1..1
The producers also have streaming modes which bypass the default runner
output and write TAP to the output stream instead of files. This is useful
for piping output directly to tools that read TAP natively.
.. code-block:: tap
$ nosetests --with-tap --tap-stream tap.tests.test_parser
# TAP results for TestParser
ok 1 - test_after_hash_is_not_description (tap.tests.test_parser.TestParser)
ok 2 - The parser extracts a bail out line.
ok 3 - The parser extracts a diagnostic line.
ok 4 - The TAP spec dictates that anything less than 13 is an error.
ok 5 - test_finds_description (tap.tests.test_parser.TestParser)
ok 6 - The parser extracts a not ok line.
ok 7 - The parser extracts a test number.
ok 8 - The parser extracts an ok line.
ok 9 - The parser extracts a plan line.
ok 10 - The parser extracts a plan line containing a SKIP.
1..10
.. image:: images/stream.gif
Examples
--------
The ``TAPTestRunner`` works like the ``TextTestRunner``. To use the runner,
load test cases using the ``TestLoader`` and pass the tests to the run method.
The sample below is the test runner used with tappy's own tests.
.. literalinclude:: ../tap/tests/run.py
:lines: 3-
Running tappy with **nose** is as straightforward as enabling the plugin
when calling ``nosetests``.
.. code-block:: console
$ nosetests --with-tap
...............
----------------------------------------------------------------------
Ran 15 tests in 0.020s
OK
The **pytest** plugin is automatically activated for **pytest**
when tappy is installed.
Because it is automatically activated,
**pytest** users should specify an output style.
.. code-block:: console
$ py.test --tap-files
=========================== test session starts ============================
platform linux2 -- Python 2.7.6 -- py-1.4.30 -- pytest-2.7.2
rootdir: /home/matt/tappy, inifile:
plugins: tap.py
collected 94 items
tests/test_adapter.py .....
tests/test_directive.py ......
tests/test_line.py ......
tests/test_loader.py ......
tests/test_main.py .
tests/test_nose_plugin.py ......
tests/test_parser.py ................
tests/test_pytest_plugin.py .........
tests/test_result.py .......
tests/test_rules.py ........
tests/test_runner.py .......
tests/test_tracker.py .................
======================== 94 passed in 0.24 seconds =========================
The configuration options for each TAP tool are listed
in the following sections.
TAPTestRunner
-------------
You can configure the ``TAPTestRunner`` from a set of class or instance
methods.
* ``set_stream`` - Enable streaming mode to send TAP output directly to
the output stream. Use the ``set_stream`` instance method.
.. code-block:: python
runner = TAPTestRunner()
runner.set_stream(True)
* ``set_outdir`` - The ``TAPTestRunner`` gives the user the ability to
set the output directory. Use the ``set_outdir`` class method.
.. code-block:: python
TAPTestRunner.set_outdir('/my/output/directory')
* ``set_combined`` - TAP results can be directed into a single output file.
Use the ``set_combined`` class method to store the results in
``testresults.tap``.
.. code-block:: python
TAPTestRunner.set_combined(True)
* ``set_format`` - Use the ``set_format`` class method to change the
format of result lines. ``{method_name}`` and ``{short_description}``
are available options.
.. code-block:: python
TAPTestRunner.set_format('{method_name}: {short_description}')
* ``set_header`` - Turn off or on the test case header output.
The default is ``True`` (ie, the header is displayed.)
Use the ``set_header`` instance method.
.. code-block:: python
runner = TAPTestRunner()
runner.set_header(False)
nose TAP Plugin
---------------
.. note::
To use this plugin, install it with ``pip install nose-tap``.
The **nose** TAP plugin is configured from command line flags.
* ``--with-tap`` - This flag is required to enable the plugin.
* ``--tap-stream`` - Enable streaming mode to send TAP output directly to
the output stream.
* ``--tap-combined`` - Store test results in a single output file
in ``testresults.tap``.
* ``--tap-outdir`` - The **nose** TAP plugin also supports an optional
output directory when you don't want to store the ``.tap`` files
wherever you executed ``nosetests``.
Use ``--tap-outdir`` followed by a directory path to store the files
in a different place. The directory will be created if it does not exist.
* ``--tap-format`` - Provide a different format for the result lines.
``{method_name}`` and ``{short_description}`` are available options.
For example, ``'{method_name}: {short_description}'``.
pytest TAP Plugin
-----------------
.. note::
To use this plugin, install it with ``pip install pytest-tap``.
The **pytest** TAP plugin is configured from command line flags.
Since **pytest** automatically activates the TAP plugin,
the plugin does nothing by default.
Users must enable a TAP output mode
(via ``--tap-stream|files|combined``)
or the plugin will take no action.
* ``--tap-stream`` - Enable streaming mode to send TAP output directly to
the output stream.
* ``--tap-files`` - Store test results in individual test files.
One test file is created for each test case.
* ``--tap-combined`` - Store test results in a single output file
in ``testresults.tap``.
* ``--tap-outdir`` - The **pytest** TAP plugin also supports an optional
output directory when you don't want to store the ``.tap`` files
wherever you executed ``py.test``.
Use ``--tap-outdir`` followed by a directory path to store the files
in a different place. The directory will be created if it does not exist.
Python and TAP
--------------
The TAP specification is open-ended
on certain topics.
This section clarifies how tappy interprets these topics.
The specification indicates that a test line represents a "test point"
without explicitly defining "test point."
tappy assumes that each test line is **per test method**.
TAP producers in other languages may output test lines **per assertion**,
but the unit of work in the Python ecosystem is the test method
(i.e. ``unittest``, nose, and pytest all report per method by default).
tappy does not permit setting the plan.
Instead, the plan is a count of the number of test methods executed.
Python test runners execute all test methods in a suite,
regardless of any errors encountered.
Thus, the test method count should be an accurate measure for the plan.
|
