pyramid_tm
==========

.. _overview: 

Overview
--------

``pyramid_tm`` is a package which allows :term:`Pyramid` requests to join
the active :term:`transaction` as provided by the Python `transaction 
<https://pypi.org/project/transaction/>`_ package. (See the `documentation
for the transaction package 
<https://transaction.readthedocs.io/en/latest/>`_ for an
explanation of what "joining the active transaction" means).

Installation
------------

Install using pip, e.g. (within a virtualenv)::

  $ pip install pyramid_tm

Setup
-----

Once ``pyramid_tm`` is installed, you must use the ``config.include``
mechanism to include it into your Pyramid project's configuration.  In your
Pyramid project's ``__init__.py``:

.. code-block:: python
   :linenos:

   config = Configurator(.....)
   config.include('pyramid_tm')

Or use the ``pyramid.includes`` configuration setting in your ``.ini`` file:

.. code-block:: ini
   :linenos:

   [app:myapp]
   pyramid.includes = pyramid_tm

After the package is included, whenever a new request enters the application,
a new transaction is associated with that request.

.. note::

   When the ``repoze.tm`` or ``repoze.tm2`` middleware is in the WSGI
   pipeline, ``pyramid_tm`` becomes inactive.

:term:`transaction` Usage
-------------------------

At the beginning of a request a new :term:`transaction` is started
using the ``request.tm.begin()`` function.  Once the request has
finished all of its works (ie views have finished running), a few checks
are tested:

  1) Did a transaction.doom() cause the transaction to become "doomed"?
     if so, ``request.tm.abort()``.

  2) Did an exception occur in the underlying code? if so,
     ``request.tm.abort()``

  3) If the ``tm.commit_veto`` configuration setting was used, did
     the commit veto callback, called with the response generated by the
     application, return a result that evaluates to ``True``? if so,
     ``request.tm.abort()``.

If none of these checks calls ``request.tm.abort()`` then the transaction is
instead committed using ``request.tm.commit()``.

By itself, this :term:`transaction` machinery doesn't do much.  It is up to
third-party code to *join* the active transaction to benefit.  See
`repoze.filesafe <https://pypi.org/project/repoze.filesafe/>`_ for an
example of how files creation can be committed or rolled back based on
:term:`transaction` and the `pyramid_mailer
<https://docs.pylonsproject.org/projects/pyramid_mailer/en/latest/>`_ package to see
how you can prevent emails from being sent until a transaction succeeds.
ZODB database connections are automatically joined to the transaction, as
well as SQLAlchemy connections which are configured with
``zope.sqlalchemy.register(session)`` from the `zope.sqlalchemy
<https://pypi.org/project/zope.sqlalchemy/>`_ package.

Savepoints
----------

When using sessions / data managers joined to the transaction,
it's important to synchronize changes across those managers. This means that
it's usually incorrect to use your backend's session lifecycle functions
directly such as ``sqlalchemy.orm.Session.begin_nested``. Instead, synchronize
a savepoint across all joined data managers via
``sp = request.tm.savepoint()``. The savepoint can be rolled back via
``sp.rollback()``. For example:

.. code-block:: python

    def my_view(request):
        sp = request.tm.savepoint()
        try:
            page = WikiPage()
            page.id = 5  # maybe the id 5 violates a unique constraint
            request.dbsession.add(page)
            request.dbsession.flush()
        except sqlalchemy.exc.IntegrityError:
            # page already exists!
            sp.rollback()
        # continue with or without the data added in the try-clause
        ...

.. note::

    Not every data manager supports savepoints and as such some changes
    may not be able to be rolled back.

.. _error_handling:

Error Handling
--------------

``pyramid_tm`` is positioned **OVER** the ``EXCVIEW`` tween. The implication
of this is that the transaction may still be open and alive during the
execution of your exception views. **This is not guaranteed**. If you write
an exception view that expects an open transaction then you should declare
your intent using the ``tm_active=True`` view predicate otherwise it may be
executed later in the pipeline after the transaction has already been
completed. For example:

.. code-block:: python

    from pyramid.view import exception_view_config

    log = __import__('logging').getLogger(__name__)

    @exception_view_config(Exception, tm_active=True)
    def transactional_error_view(exc, request):
        # depending on your AuthenticationPolicy the authenticated
        # userid likely requires a lookup in your database which would
        # require an active transaction
        if request.authenticated_userid is not None:
            log.exception('authenticated user caused an exception')
        else:
            log.exception('unknown user caused an exception')
        response = request.response
        response.status_code = 500
        return response

    @exception_view_config(Exception)
    def default_error_view(exc, request):
        log.exception('unknown user caused an exception')
        response = request.response
        response.status_code = 500
        return response

In the above example, ``transactional_error_view`` will be invoked only
when an exception occurs during the ``pyramid_tm`` lifecycle. Otherwise,
``default_error_view`` will be invoked as a fallback.

The transaction created and completed by ``pyramid_tm`` should be used for
operations directly related to processing the request. Very often it is
desirable to perform operations on the database and other backends in a failure
scenario. This should be done using a separate transaction / connection,
possibly in autocommit mode. **Do not** use ``request.tm`` and
``request.dbsession`` and such for these cases as the work added to that
transaction is expected to be aborted upon any failures.

Retries
-------

``pyramid_tm`` ships with support for `pyramid_retry <https://docs.pylonsproject.org/projects/pyramid-retry/en/latest/>`_ which is an
execution policy that will retry requests when they fail with exceptions
marked as retryable. By default, retrying is turned off. In order to turn it
on you must update your app's configuration:

.. code-block:: python

    from pyramid.config import Configurator

    def main(global_config, **settings):
        config = Configurator(settings=settings)
        config.include('pyramid_retry')
        config.include('pyramid_tm')

Finally, ensure that your application's settings have ``retry.attempts``
set to a value greater than ``1``.

When the transaction manager calls the downstream handler, if the handler
raises a :term:`retryable` exception, ``pyramid_tm`` will mark the exception
as retryable by ``pyramid_retry``. The execution policy will detect a
retryable error and create a new copy of the request with new state.

Retryable exceptions include ``ZODB.POSException.ConflictError``, and
certain exceptions raised by various data managers, such as
``psycopg2.extensions.TransactionRollbackError``, ``cx_Oracle.DatabaseError``
where the exception's code is 8877.  Any exception which inherits from
``transaction.interfaces.TransientError`` will be marked as retryable.

Read more about retrying requests in the `pyramid_retry documentation <https://docs.pylonsproject.org/projects/pyramid-retry/en/latest/>`_.

Custom Transaction Managers
---------------------------

By default ``pyramid_tm`` will use the threadlocal ``transaction.manager``
to associate one transaction manager per thread. If you wish to override this
and provide your own transaction manager you can create your own manager hook
that will return the manager it should use.

.. code-block:: python
   :linenos:

   import transaction

   def manager_hook(request):
       return transaction.TransactionManager(explicit=True)

To enable this hook, add it as the ``tm.manager_hook`` setting in your app.

.. code-block:: python
   :linenos:

   from pyramid.config import Configurator

   def app(global_conf, **settings):
       settings['tm.manager_hook'] = manager_hook
       config = Configurator(settings=settings)
       config.include('pyramid_tm')
       # ...

This specific example, using an explicit mode non-threadlocal manager, is
highly recommended and is shipped as :func:`pyramid_tm.explicit_manager`.
Simply set ``tm.manager_hook = pyramid_tm.explicit_manager`` in your settings
to enable it.

The current transaction manager being used for any particular request can
always be accessed on the request as ``request.tm`` so long as it is accessed
while the ``pyramid_tm`` tween is active. If you try to access ``request.tm``
outside of the tween or during a request in which ``pyramid_tm`` was disabled,
``request.tm`` will raise an ``AttributeError``.

.. note::

    It is recommended to use a custom transaction manager with
    ``explicit=True``, as in the example above, instead of the threadlocal
    ``transaction.manager`` to give greater control over the transaction's
    lifecycle and to weed out potential bugs in your application. For example,
    you may have some parts of your app that access the manager after it has
    already been committed. This will open an implicit transaction that is
    never committed, and will even hang around until a subsequent request
    aborts the implicit transaction. Instead, if you set ``explicit=True``,
    any code affecting the manager outside of the lifecycle of the transaction
    will cause an error and will be noticed quickly.

Adding an Activation Hook
-------------------------

It may not always be desirable to have every request managed by the
transaction manager automatically. It is possible to configure ``pyramid_tm``
with an "activate" hook. The callback function receives the request. It
can then examine it and return ``False`` if the transaction manager should
be disabled for that request.

.. code-block:: python
   :linenos:

   def activate_hook(request):
       if request.path_info.startswith('/long-poll'):
           # Allow the long-poll class to manage its own connections to avoid
           # long-lived transactions.
           return False
       return True

To enable this hook, add it as the ``tm.activate_hook`` setting in your app.

.. code-block:: python
   :linenos:

   from pyramid.config import Configurator

   def app(global_conf, **settings):
       settings['tm.activate_hook'] = activate_hook
       config = Configurator(settings=settings)
       config.include('pyramid_tm')
       # ...

Or via PasteDeploy:

.. code-block:: ini
   :linenos:

   [app:myapp]
   tm.activate_hook = myapp.activate_hook

In either configuration the value for ``tm.activate_hook`` is a
:term:`dotted Python name`.

Adding a Commit Veto Hook
-------------------------

It is possible to configure ``pyramid_tm`` with a "commit veto" hook.  The
commit veto hook receives the request and the response.  It can examine both
of them, and return ``True`` if the transaction should be vetoed.  If the
transaction is vetoed, it will be aborted instead of committed.  By default,
``pyramid_tm`` does not configure a commit veto into the system; you must do
it explicitly.

:mod:`pyramid_tm` contains a :func:`pyramid_tm.default_commit_veto` that is
suitable for use when you want to abort when the response's status code
indicates non-success or if you'd like to signal that the transaction should
be aborted or committed using a response header.  The default commit veto
vetoes a commit if the status code starts with ``4`` or ``5`` or there is a
``X-Tm`` response header with a value that does not equal ``commit``.

.. code-block:: python
   :linenos:

   def default_commit_veto(request, response):
       xtm = response.headers.get('x-tm')
       if xtm is not None:
           return xtm != 'commit'
       return response.status.startswith(('4', '5'))

If you'd like to use this commit veto in your system, you can do it via
Python:

.. code-block:: python
   :linenos:

   from pyramid.config import Configurator

   def app(global_conf, **settings):
       settings['tm.commit_veto'] = 'pyramid_tm.default_commit_veto'
       config = Configurator(settings=settings)
       config.include('pyramid_tm')
       # ...

Or via PasteDeploy:

.. code-block:: ini
   :linenos:

   [app:myapp]
   tm.commit_veto = pyramid_tm.default_commit_veto

If you'd like to use a different "commit veto" callback, create a function
with the same signature (``request``, ``response``) and return value
(``True`` or ``False``), then pass a ``tm.commit_veto`` key/value
pair in your settings which points at the Python dotted name of this commit
veto.

Via Python:

.. code-block:: python
   :linenos:

   from pyramid.config import Configurator

   def app(global_conf, settings):
       settings['tm.commit_veto'] = 'my.package.commit_veto'
       config = Configurator(settings=settings)
       config.include('pyramid_tm')

Via PasteDeploy:

.. code-block:: ini
   :linenos:

   [app:myapp]
   tm.commit_veto = my.package.commit_veto

In the PasteDeploy example, the path is a :term:`dotted Python name`, where
the dots separate module and package names, and the colon separates a module
from its contents.  In the above example, the code would be implemented as a
"commit_veto" function which lives in the "package" submodule of the "my"
package.

View Predicates
---------------

``pyramid_tm`` registers a view predicate named ``tm_active`` which accepts
a value of ``True`` or ``False``. This can be useful for declaring intent
when defining exception views that require access to the transaction controlled
by ``pyramid_tm``. For specific examples,  see :ref:`error_handling`.

If the request is manually completed via ``request.tm.abort()`` or
``request.tm.commit()``, this predicate may be incorrect depending on the
specific transaction manager being used. After completing a transaction
controlled by the transaction manager in explicit mode it is necessary to
invoke ``request.tm.begin()`` to start a new one or any subsequent uses of
the transaction manager will fail.

Explicit Tween Configuration
----------------------------

Note that the transaction manager is a Pyramid "tween", and it can be used in
the explicit tween list if its implicit position in the tween chain is
incorrect (see the output of ``ptweens``)::

   [app:myapp]
   pyramid.tweens = someothertween
                    pyramid_tm.tm_tween_factory
                    pyramid.tweens.excview_tween_factory

It usually belongs directly above the
"pyramid.tweens.excview_tween_factory" entry in the `` ptweens``
output, and will attempt to sort there by default as the result of having
``config.include('pyramid_tm')`` invoked.

Avoid Accessing the Authentication Policy
-----------------------------------------

By default the tween will access
:attr:`pyramid.request.Request.authenticated_userid` in order to annotate
the transaction with information about the user. This can be turned off
by setting the ini option ``tm.annotate_user = false``.

Testing
-------

You can partially disable or override ``pyramid_tm`` in your test suite.
This can be helpful if you want to handle transactions externally - allowing you to rollback or keep them open across multiple requests.

1. Tell ``pyramid_tm`` that something else is handling transactions by setting ``tm.active`` in the WSGI environ.

2. Provide your own transaction manager to the app to override ``request.tm`` by setting ``tm.manager`` to your own object.

.. code-block:: python
    :linenos:

    import pytest
    import transaction
    from webtest import TestApp

    @pytest.fixture
    def testapp():
        app = ...
        tm = transaction.TransactionManager(explicit=True)
        tm.begin()
        tm.doom()  # ensure no one can call tm.commit() manually

        testapp = TestApp(app, extra_environ={
            'tm.active': True,    # disable pyramid_tm
            'tm.manager': tm,    # pass in our own tm for the app to use
        })

        yield testapp

        tm.abort()

More Information
----------------

.. toctree::
   :maxdepth: 1

   api.rst
   glossary.rst
   changes.rst

Reporting Bugs / Development Versions
-------------------------------------

Visit https://github.com/Pylons/pyramid_tm to download development or
tagged versions.

Visit https://github.com/Pylons/pyramid_tm/issues to report bugs.

Indices and tables
------------------

* :ref:`glossary`
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`