1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
|
Comparing past runs
===================
Before comparing different runs it's ideal to make your tests as consistent as possible, see :doc:`faq` for more details.
`pytest-benchmark` has support for storing stats and data for the previous runs.
To store a run just add ``--benchmark-autosave`` or ``--benchmark-save=some-name`` to your pytest arguments. All the files are
saved in a path like ``.benchmarks/Linux-CPython-3.4-64bit``.
* ``--benchmark-autosave`` saves a file like ``0001_c9cca5de6a4c7eb2_20150815_215724.json`` where:
* ``0001`` is an automatically incremented id, much like how django migrations have a number.
* ``c9cca5de6a4c7eb2`` is the commit id (if you use Git or Mercurial)
* ``20150815_215724`` is the current time
You should add ``--benchmark-autosave`` to ``addopts`` in you pytest configuration so you dont have to specify it all
the time.
* ``--benchmark-save=foobar`` works similarly, but saves a file like ``0001_foobar.json``. It's there in case you want to
give specific name to the run.
After you have saved your first run you can compare against it with ``--benchmark-compare=0001``. You will get an additional
row for each test in the result table, showing the differences.
You can also make the suite fail with ``--benchmark-compare-fail=<stat>:<num>%`` or ``--benchmark-compare-fail=<stat>:<num>``.
Examples:
* ``--benchmark-compare-fail=min:5%`` will make the suite fail if ``Min`` is 5% slower for any test.
* ``--benchmark-compare-fail=mean:0.001`` will make the suite fail if ``Mean`` is 0.001 seconds slower for any test.
Comparing outside of pytest
---------------------------
There is a convenience CLI for listing/comparing past runs: ``pytest-benchmark`` (:ref:`comparison-cli`).
Example::
pytest-benchmark compare 0001 0002
Plotting
--------
.. note::
To use plotting you need to ``pip install pygal pygaljs`` or ``pip install pytest-benchmark[histogram]``.
You can also get a nice plot with ``--benchmark-histogram``. The result is a modified Tukey box and whisker plot where the
outliers (the small bullets) are ``Min`` and ``Max``. Note that if you do not supply a name for the plot it is recommended
that ``--benchmark-histogram`` is the last option passed.
Example output:
.. image:: screenshot-histogram.png
|
