1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
|
pytest-console-scripts
======================
[](https://pypi.org/project/pytest-console-scripts/)
[](https://github.com/kvas-it/pytest-console-scripts/blob/master/LICENSE)
[](https://github.com/kvas-it/pytest-console-scripts/actions)
[](https://codecov.io/gh/kvas-it/pytest-console-scripts)
[](https://github.com/kvas-it/pytest-console-scripts/issues)
[](https://github.com/kvas-it/pytest-console-scripts/pulls)
[](https://github.com/kvas-it/pytest-console-scripts/blob/master/CHANGELOG.md)
Pytest-console-scripts is a [pytest][1] plugin for running python scripts from
within tests. It's quite similar to `subprocess.run()`, but it also has an
in-process mode, where the scripts are executed by the interpreter that's
running `pytest` (using some amount of sandboxing).
In-process mode significantly reduces the run time of the test suites that
run many external scripts. This is speeds up development. In the CI environment
subprocess mode can be used to make sure the scripts also work (and behave the
same) when run by a fresh interpreter.
Requirements
------------
- Python 3.8+, or PyPy3,
- Pytest 4.0 or newer.
Installation
------------
You can install "pytest-console-scripts" via [pip][2] from [PyPI][3]:
```sh
$ pip install pytest-console-scripts
```
Normally you would add it as a test dependency in `tox.ini` (see [tox
documentation][9]).
Usage
-----
This plugin will run scripts that are installed via `console_scripts` entry
point in `setup.py`, python files in current directory (or anywhere else, if
given the path), and Python scripts anywhere else in the path. It will also run
executables that are not Python scripts, but only in subprocess mode (there's
no benefit in using `pytest-console-scripts` for this, you should just use
`subprocess.run`).
Here's an example with `console_scripts` entry point. Imagine we have a python
package `foo` with the following `setup.py`:
```py
setup(
name='foo',
version='0.0.1',
py_modules=['foo'],
entry_points={
'console_scripts': ['foobar=foo:bar']
},
)
```
We could use pytest-console-scripts to test the `foobar` script:
```py
def test_foo_bar(script_runner):
result = script_runner.run(['foobar', '--version'])
assert result.returncode == 0
assert result.stdout == '3.2.1\n'
assert result.stderr == ''
script_runner.run('foobar --version', shell=True, check=True)
```
This would use the `script_runner` fixture provided by the plugin to
run the script and capture its output.
The arguments of `script_runner.run` are the command name of the script and
any command line arguments that should be passed to it. Additionally the
following keyword arguments can be used:
- `cwd` - set the working directory of the script under test.
- `env` - a dictionary with environment variables to use instead of the current
environment.
- `stdin` - a file-like object that will be piped to standard input of the
script.
- `check` - raises an exception if `returncode != 0`, defaults to False.
- `shell` - mimic shell execution, this should work well for simple cases,
defaults to False.
Type-hinting is also supported.
You may type-hint the fixture with the following code:
```py
from pytest_console_scripts import ScriptRunner
def test_foo_bar(script_runner: ScriptRunner) -> None:
...
```
Configuring script execution mode
---------------------------------
In the example above the `foobar` script would run in in-process mode (which is
the default). This is fast and good for quick iteration during development.
After we're happy with the functionality, it's time to run the script in
subprocess mode to simulate real invocation more closely. There are several
ways to do this. We can configure it via pytest configuration (for example in
`tox.ini`):
```ini
[pytest]
script_launch_mode = subprocess
```
We can give a command line option to pytest (this will override the
configuration file):
```sh
$ pytest --script-launch-mode=subprocess test_foobar.py
```
We can also mark individual tests to run in a specific mode:
```py
@pytest.mark.script_launch_mode('subprocess')
def test_foobar(script_runner):
...
```
Between these three methods the marking of the tests has priority before the
command line option that in turn overrides the configuration setting. All three
can take three possible values: "inprocess", "subprocess", and "both" (which
will cause the test to be run twice: in in-process and in subprocess modes).
Interaction with mocking
------------------------
It is possible to mock objects and functions inside of console scripts when
they are run using `pytest-console-scripts` but only in inprocess mode. When
the script is run in subprocess mode, it is executed by a separate Python
interpreter and the test can't mock anything inside of it.
Another limitation of mocking is that with simple Python scripts that are not
installed via [`console_scripts` entry point][14] mocking of objects inside of
the main script will not work. The reason for that is that when we run
`myscript.py` with `$ python myscript.py` the script gets imported into
`__main__` namespace instead of `myscript` namespace. Our patching of
`myscript.myfunction` will have no effect on what the code in `__main__`
namespace sees when it's calling `myfunction` defined in the same file.
See [this stackoverflow answer](https://stackoverflow.com/a/66693954/1595738)
for some ideas of how to get around this.
Suppressing the printing of script run results
----------------------------------------------
When tests involving `pytest-console-scripts` fail, it tends to be quite
useful to see the output of the scripts that were executed in them. We try
to be helpful and print it out just before returning the result from
`script_runner.run()`. Normally PyTest [captures][12] all the output during a
test run and it's not shown to the user unless some tests fail. This is exactly
what we want.
However, in some cases it might be useful to disable the output capturing and
PyTest provides [ways to do it][13]. When capturing is disabled, all test run
results will be printed out and this might make it harder to inspect the other
output of the tests. To deal with this, `pytest-console-scripts` has an option
to disable the printing of script run results:
```sh
$ pytest --hide-run-results test_foobar.py
```
It's also possible to disable it just for one script run:
```py
result = script_runner.run('foobar', print_result=False)
```
When printing of script run results is disabled, script output won't be
visible even when the test fails. Unfortunately there's no automatic way to
print it only if the test fails because by the time a script run completes we
don't know whether the test will fail or not. It's possible to do it manually
from the test by using:
```py
result.print()
```
This, combined with `--hide-run-results` or `print_result=False` can be used to
only print interesting run results when capturing is off.
Package installation and testing during development
---------------------------------------------------
Since `pytest-console-scripts` relies on the scripts being located in the path,
it can only run the console scripts from packages that have been installed (if
you are interested in working on removing this limitation, take a look at [this
ticket](https://github.com/kvas-it/pytest-console-scripts/issues/34) and in
particular [this comment](https://github.com/kvas-it/pytest-console-scripts/issues/34#issuecomment-649497564)).
If you want to run the tests quickly during development, the additional
installation step would add a significant overhead and slow you down.
There's a way around this: install your package in [development mode][10] using
`pip install -e .`. If you use [tox][9], you can take one of its
existing virtualenvs (they live in `.tox/`). Otherwise create a
[virtualenv][11] just for development, activate it and run `python setup.py
develop` to install your package in development mode. You will need to
re-install every time you add a new console script, but otherwise all the
changes to your code will be immediately picked up by the tests.
Contributing
------------
Contributions are very welcome. Tests can be run with `tox`, please ensure
the coverage at least stays the same before you submit a pull request.
License
-------
Distributed under the terms of the [MIT][8] license, "pytest-console-scripts"
is free and open source software.
Issues
------
If you encounter any problems, please [file an issue][7] along with a detailed
description.
----
Pytest-console-scripts was initially generated with [Cookiecutter][4] along
with [@hackebrot][5]'s [Cookiecutter-pytest-plugin][6] template.
[1]: https://github.com/pytest-dev/pytest
[2]: https://pypi.python.org/pypi/pip/
[3]: https://pypi.python.org/pypi
[4]: https://github.com/audreyr/cookiecutter
[5]: https://github.com/hackebrot
[6]: https://github.com/pytest-dev/cookiecutter-pytest-plugin
[7]: https://github.com/kvas-it/pytest-console-scripts/issues
[8]: http://opensource.org/licenses/MIT
[9]: https://tox.readthedocs.org/en/latest/
[10]: https://setuptools.pypa.io/en/latest/userguide/development_mode.html
[11]: https://docs.python.org/3/library/venv.html
[12]: https://docs.pytest.org/en/stable/capture.html
[13]: https://docs.pytest.org/en/stable/capture.html#setting-capturing-methods-or-disabling-capturing
[14]: https://python-packaging.readthedocs.io/en/latest/command-line-scripts.html#the-console-scripts-entry-point
|
