========================
Writing doctest examples
========================

Doctest support is provided in Sphinx by the extension
`sphinx.ext.doctest`_, and its documentation is one
useful resurce for working with the pyparsing doctests.

.. _sphinx.ext.doctest: https://www.sphinx-doc.org/en/master/usage/extensions/doctest.html


Types of doctests
=================

There are two basic forms of doctest, and both are used extensively
in the Pyparsing documentation. Which one to use for a given example
is a decision that needs to be made when writing it, but there are
some factors that usually make the correct choice an obvious one.

Doctest type 1: ``testcode`` / ``testoutput`` blocks
----------------------------------------------------

The first form involves one or potentially two separate code blocks.
The ``testcode`` block contains all of the input code in the form of
a standard Python script. This can optionally be paired with a
second ``testoutput`` block, which if present will contain the output
for the preceding ``testcode`` block.

An example of a ``testcode`` / ``testoutput`` pair, from the docstring
for ``ParserElement.__add__``:


.. code-block:: rst

    Example:

    .. testcode::

        greet = Word(alphas) + "," + Word(alphas) + "!"
        hello = "Hello, World!"
        print(hello, "->", greet.parse_string(hello))

    prints:

    .. testoutput::

        Hello, World! -> ['Hello', ',', 'World', '!']

Examples written like this will be formatted in the rendered HTML/Latex/etc.
documentation **exactly** as if they'd been written as normal code blocks.
There is no visible difference between the code above and this code without
doctest support:

.. code-block:: rst

    Example::

        greet = Word(alphas) + "," + Word(alphas) + "!"
        hello = "Hello, World!"
        print(hello, "->", greet.parse_string(hello))

    prints::

        Hello, World! -> ['Hello', ',', 'World', '!']

However, the advantage to writing doctests is that when ``make doctest``
is run from the ``docs/`` directory, the doctest extension will execute
each ``testcode`` block, and verify that its output exactly matches the
``testoutput`` block (if present).

Any deviations will be displayed in "ndiff" format. This enhancement
to the standard unified diff will (sometimes) indicate where in each
line the differences occur. (The character-difference highlighting is
frustratingly inconsistent. But at worst ndiff is equivalent to unified
diff, so it's still worth using.)

Testing examples with doctest allows the code used to demonstrate the
pyparsing API to be verified against the *actual* API as it's currently
implemented, and ensures that examples stay current and relevant.

Not all ``testcode`` blocks need a corresponding ``testoutput`` — if a
``testcode`` block is included on its own, the code inside the block will
still be executed, but its output won't be verified. This can be useful
when displaying code that doesn't require demonstration of its output
(or doesn't output anything), as the extension will still verify that
the code can be run without error.

It's also possible to include a *hidden* ``testoutput`` block, which will
beverified against the preceding ``testcode`` but won't be displayed in the
documentation. To hide a ``testoutput`` block (or a ``testcode`` block,
for that matter), add the ``:hide:`` option as an argument to the
directive, i.e.:

.. code-block:: rst

    .. testoutput::
        :hide:

        """Output that won't be shown, but will be verified against the
        preceding testcode block."""

Doctest type 2: ``doctest`` interactive blocks
----------------------------------------------

The second type of doctest is a ``doctest`` block, which takes the form of
an interactive Python REPL session in standard format (using ``>>>`` and
``...`` markers for input lines).

With these tests, output is interleaved with the code, which can be much
easier to follow when there are multiple lines producing output. If an
example would contains multiple ``print()`` calls, rather than first
displaying all of the code in a ``testcode`` block, then all of the
output in a ``testoutput`` block, consider using a ``doctest`` session
so that the reader can follow along each step as it occurs.

A typical ``doctest`` example can be found in the ``ParserElement.ignore``
docstring:

.. code-block:: rst

    Example:

    .. doctest::

        >>> patt = Word(alphas)[...]
        >>> print(patt.parse_string('ablaj /* comment */ lskjd'))
        ['ablaj']

        >>> patt = Word(alphas)[...].ignore(c_style_comment)
        >>> print(patt.parse_string('ablaj /* comment */ lskjd'))
        ['ablaj', 'lskjd']


Setup code for doctest blocks
=============================

The doctest extension is configured with extensive setup code
which is run before each test block. It can be viewed in the
:download:`docs/conf.py <../docs/conf.py>` file — look for the
``doctest_global_setup`` variable near the end of the file.

The setup code is intended to make any useful symbols available
to the tests without them having to be included in each and every
doctest block. If additional modules are needed, feel free to add
them to the global setup. When writing doctests, Pyparsing classes
can be invoked directly, or as members of the ``pp`` alias namespace.
Either way, the definition of those symbols can be assumed without
explicitly importing/defining them.

When using symbols from other aliased namespaces, however, it's a
good idea to establish the alias for the reader at the start of the
example code. Even though these are both defined in the global setup,
showing the establishing lines before referencing ``ppc`` or ``ppu``
in an example makes that example clearer:

.. code-block: py

    ppc = pp.pyparsing_common
    ppu = pp.pyparsing_unicode

However, because those symbols *are* provided by default, they don't
need to be explicitly established for **every** example. Feel free
to omit them after the first use, when writing multiple examples for
a given class or function.

Documenting exceptions
======================

Code that will trigger an exception can be both demonstrated and
verified using doctests (of either type), although when a ``testoutput``
block will demonstrate an exception it should be the only output in
that block — doctest does not support mixing regular output and
exceptions.

Both the ``IGNORE_EXCEPTION_DETAIL`` and ``ELLIPSIS`` doctest options
are enabled by default, which make demonstrating exceptions far more
convenient. Ignoring exception detail means that the full traceback
for an exception can be omitted, as well as the fully-qualified name
of the exception class. As long as the ``Traceback...`` line and the
exception class name match, the doctest will pass. (The exception
message is also verified by default, but read on for more about that.)

This example code, from the ``ParserElement.set_name`` docstring, will
actually output a long traceback, followed by an exception of type
``pyparsing.exceptions.ParseException``. But because the ignore-detail
option is enabled, the doctest will pass with this abbreviated form:

.. code-block:: rst

    .. doctest::

        >>> integer = Word(nums)
        >>> integer.parse_string("ABC")
        Traceback (most recent call last):
        ParseException: Expected W:(0-9) (at char 0), (line:1, col:1)

Relaxing doctest output validation
==================================

For even more flexibility in demonstrating output, the ``ELLIPSIS``
option (enabled by default) means that parts of the output can be
replaced with an ellipsis (three periods, ``...``) which will validate
against any output.

This is an extremely useful tool when the exact output of the code is
unpredictable (for example, when messages include line and column
numbers, or variable data like the current date or a directory path).
The code above could also be written like this, and it would still
pass the doctest:

.. code-block:: rst

    .. doctest::

        >>> integer = Word(nums)
        >>> integer.parse_string("ABC")
        Traceback (most recent call last):
        ParseException: Expected W:(0-9) ...

While this is necessary in some situations, it shouldn't be overused.
The more precisely a doctest validates the output of its example,
the more useful it is, so think twice before employing an ellipsis in
doctest output.

Normalizing whitespace checks
=============================

Another method of relaxing doctest checks that doesn't impact the
test's ability to validate output is the ``NORMALIZE_WHITESPACE``
option. This option isn't enabled by default, but can be turned on
for any doctest block with a directive argument:

.. code-block::  rst

    .. testoutput::
        :options: +NORMALIZE_WHITESPACE

(Note the preceding ``+`` sign, which adds the option to the default
set instead of replacing the default options.)

With normalization activated, any combination of spaces, tabs, and
newlines will compare equal to any other combination.

One advantage this has is permitting long messages to be wrapped
over several lines in the example output. In this example from the
``Keyword`` class docstring, the exception message at the end would
normally be printed as one long line. To make the example readable
without excessive horizontal scrolling, ``NORMALIZE_WHITESPACE``
allows the example output to be broken into multiple lines:

.. code-block:: rst

    .. doctest::
        :options: +NORMALIZE_WHITESPACE

        >>> Keyword("start").parse_string("start")
        ParseResults(['start'], {})
        >>> Keyword("start").parse_string("starting")
        Traceback (most recent call last):
        ParseException: Expected Keyword 'start',
        keyword was immediately followed by keyword character,
        found 'ing'  (at char 5), (line:1, col:6)

Doctests in the Pyparsing codebase
==================================

While the preceding is generally applicable to doctests in any
codebase, there are some issues specific to Pyparsing doctests that
you should be aware of.

``run_tests()`` output
----------------------

There is one scenario in the pyparsing documentation where the
``NORMALIZE_WHITESPACE`` option *must* be used.

When the example code uses the ``ParserElement.run_tests()`` method,
the output will consist of test strings and matches potentially
separated by two blank lines each. (Unless each test is preceded
by a comment, then there will be only one blank line.)

Since ReStructuredText will collapse multiple blank lines in embedded
code, the only way to get the ``run_tests`` output to validate against
the example is to enable ``NORMALIZE_WHITESPACE`` and collapse the
multiple blank lines in the expected output, as well.

Also, "any whitespace compares equal" doesn't mean that *no*
whitespace will be accepted, so the beginning of the ``testoutput``
block MUST include an extra blank line at the start, in order
to match the leading 2 (or 1) blank lines in the output.

So, a valid ``run_tests`` output block consists of the ``testoutput``
directive, the ``:options: +NORMALIZE_WHITESPACE`` argument, then
**TWO blank lines** followed by the output to be verified. This
example, from the ``ParserElement.run_tests`` docstring itself,
demonstrates the required format:

.. code-block:: rst
    :linenos:
    :emphasize-lines: 17,21,22,26,27

    Failing example:

    .. testcode::

        number_expr = pyparsing_common.number.copy()
        result = number_expr.run_tests('''
            100Z
            3.14.159
            ''', failure_tests=True)
        print("Success" if result[0] else "Failed!")

    prints:

    .. testoutput::
        :options: +NORMALIZE_WHITESPACE


        100Z
        100Z
        ^
        ParseException: Expected end of text, found 'Z' ...

        3.14.159
        3.14.159
            ^
        ParseException: Expected end of text, found '.' ...
        FAIL: Expected end of text, found '.' ...
        Success

Note in particular:

- The extra blank line (line 17) before the first line of output, which
  is required to match the *two* blank lines in the actual output.

- Only one blank line (line 22) separating the two tests' output.
  The real output will again contain two blank lines.

- The use of ellipses to abbreviate the expected output (lines 21, 26, 27).

- Exception messages mixed with normal output.

  In this case that presents no problems, because ``run_tests()`` catches
  any exceptions generated and prints their messages as normal output.
  Doctest has no restrictions on normal output, only when the exception
  is raised and a traceback is triggered.

  By the same token, ``IGNORE_EXCEPTION_DETAIL`` is not applicable here
  (there are no exceptions in the expected string, only regular output),
  so the normal string-matching rules apply when comparing expected output
  to actual output.

Two final notes about failing doctests
--------------------------------------

There are two things to watch out for, when attempting to address
doctest failures during a ``make doctest`` run.

Code location references are not useful
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Due to the uncommon structure of the pyparsing namespace (with the
symbols from all of the package's files imported into the top-level
``pyparsing`` namespace, and documented there rather than at their
"home" locations where they're defined), the doctest output for
failing test will not display the correct source location for the
code. Every failing test will be preceded by a reference similar to:

.. code-block::

    File "../pyparsing/core.py", line ?, in default

However, this will be followed by a listing of the code that
produced the failing test. So as long as we write examples
which are not too generic and are sufficiently distinct from
each other (which is good practice anyway), it should be easy
enough to find the failing code.

Diffs on failing tests will include *ALL* differences
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When ``doctest`` displays the NDIFF-format differences between the
expected output and the actual output, it will indicate **EVERY**
difference between them — even the differences that would otherwise
be ignored. The ``IGNORE_TRACEBACK_DETAILS``, ``ELLIPSIS``, and
``NORMALIZE_WHITESPACE`` options do not apply when NDIFF is generating
the comparison ouput for a failed test.

What this means is that, even though the NDIFF flags an ellipsized
section of text as a difference from the actual output, or marks a
difference where an output line has been split into two when the
``NORMALIZE_WHITESPACE`` option is enabled, those differences WILL be
ignored when the doctest is in a passing state. It's important to
focus on the differences that *wouldn't* otherwise be ignored, and
just trust that correcting those differences will result in a passing
test.

For example, consider this failing test:

.. code-block:: shell-session
    :linenos:
    :emphasize-lines: 20-23

    $ make doctest
    ...
    File "../pyparsing/core.py", line ?, in default
    Failed example:
        data_word = Word(alphas)
        label = data_word + FollowedBy(':')

        attr_expr = (
            label + Suppress(':')
            + OneOrMore(data_word, stop_on=label
        ).set_parse_action(' '.join))

        print(attr_expr.parse_string("color: RED"))

        text = "shape: SQUARE posn: upper left color: light blue texture: burlap"

        # print attributes as plain groups
        print(attr_expr[1, ...].parse_string(text))
    Differences (ndiff with -expected +actual):
        - ['color', "RED"]
        ?           ^   ^
        + ['color', 'RED']
        ?           ^   ^
        + ['shape', 'SQUARE', 'posn', 'upper left', 'color', 'light blue', 'texture', 'burlap']
        - ['shape', 'SQUARE',
        - ... 'texture', 'burlap']
    **********************************************************************
    1 item had failures:
       1 of 208 in default
    208 tests in 1 item.
    207 passed and 1 failed.
    ***Test Failed*** 1 failure.

The **only** significant difference is the highlighted one: The wrong
quotes used around the word ``"RED"`` in the expected output. Once that's
changed to ``'RED'``, the doctest will pass. The remaining diff line(s),
where the expected output uses an ellipsis and is split over two lines
(with ``NORMALIZE_WHITESPACE`` enabled), will not fail despite being
shown as differing from the actual output. (Technically it *does* differ,
after all. The configuration simply ignores that difference.)