How you implemented your Python decorator is wrong
==================================================

The rest of the Python community is currently doing lots of navel gazing
over the issue of Python 3 adoption and the whole unicode/bytes divide. I
am so over that and gave up caring when my will to work on WSGI stuff was
leached away by how long it took to get the WSGI specification updated for
Python 3.

Instead my current favourite gripe these days is about how people implement
Python decorators. Unfortunately, it appears to be a favourite topic for
blogging by Python developers. It is like how when WSGI was all the rage
and everyone wanted to write their own WSGI server or framework. Now it is
like a rite of passage that one must blog about how to implement Python
decorators as a way of showing that you understand Python. As such, I get
lots of opportunity to grumble and wince. If only they did truly understand
what they were describing and what problems exist with the approach they
use.

So what is my gripe then. My gripe is that although one can write a very
simple decorator using a function closure, the scope it can be used in is
usually limited. The most basic pattern for implementing a Python decorator
also breaks various stuff related to introspection.

Now most people will say who cares, it does the job I want to do and I
don't have time to care whether it is correct in all situations.

As people will know from when I did care more about WSGI, I am a pedantic
arse though and when one does something, I like to see it done correctly.

Besides my overly obsessive personal trait, it actually does also affect me
in my day job as well. This is because I write tools which are dependent
upon being able to introspect into code and I need the results I get back
to be correct. If they aren't, then the data I generate becomes useless as
information can get grouped against the wrong thing.

As well as using introspection, I also do lots of evil stuff with monkey
patching. As it happens, monkey patching and the function wrappers one
applies aren't much different to decorators, it is just how they get
applied which is different. Because though monkey patching entails going in
and modifying other peoples code when they were not expecting it, or
designing for it, means that when you do go in and wrap a function that how
you do it is very important. If you do not do it correctly then you can
crash the users application or inadvertently change how it runs.

The first thing that is vitally important is preserving introspection for
the wrapped function. Another not so obvious thing though is that you need
to ensure that you do not mess with how the execution model for the Python
object model works.

Now I can in my own function wrappers that are used when performing monkey
patching ensure that these two requirements are met so as to ensure that
the function wrapper is transparent, but it can all fall in a heap when one
needs to monkey patch functions which already have other decorators
applied.

So when you implement a Python decorator and do it poorly it can affect me
and what I want to do. If I have to subsequently work around when you do it
wrong, I get somewhat annoyed and grumpy as more often than not that
entails a lot of pain.

To cover everything there is to know about what is wrong with your typical
Python decorators and wrapping of functions, plus how to fix it, will take
a lot of explaining, so one blog post isn't going to be enough. See this
blog post therefore as just part one of an extended discussion.

For this first instalment I will simply go through the various ways in
which your typical Python decorator can cause problems.

Basics of a Python decorator
----------------------------

Everyone should know what the Python decorator syntax is.

```python
@function_wrapper
def function():
    pass
```

The `@` annotation to denote the application of a decorator was only
added in Python 2.4. It is actually though only fancy syntactic sugar. It
is actually equivalent to writing:

```python
def function():
    pass

function = function_wrapper(function)
```

and what you would have done prior to Python 2.4.

The decorator syntax is therefore just a short hand way of being able to
apply a wrapper around an existing function, or otherwise modify the
existing function in place, while the definition of the function is being
setup.

What is referred to as monkey patching achieves pretty much the same
outcome, the difference being that when monkey patching the wrapper isn't
being applied at the time the definition of the function is being setup,
but is applied retrospectively from a different context after the fact.

Anatomy of a function wrapper
-----------------------------

Although I mentioned using function closures to implement a decorator, to
understand how the more generic case of a function wrapper works it is more
illustrative to show how to implement it using a class.

```python
class function_wrapper(object):
    def __init__(self, wrapped):
        self.wrapped = wrapped
    def __call__(self, *args, **kwargs):
        return self.wrapped(*args, **kwargs)

@function_wrapper
def function():
    pass
```

The class instance in this example is initialised with and records the
original function object. When the now wrapped function is called, it is
actually the `__call__()` method of the wrapper object which is invoked.
This in turn would then call the original wrapped function.

Simply passing through the call to the wrapper alone isn't particularly
useful, so normally you would actually want to do some work either before
or after the wrapped function is called. Or you may want to modify the
input arguments or the result as they pass through the wrapper. This is
just a matter of modifying the `__call__()` method appropriately to do
what you want.

Using a class to implement the wrapper for a decorator isn't actually that
popular. Instead a function closure is more often used. In this case a
nested function is used as the wrapper and it is that which is returned by
the decorator function. When the now wrapped function is called, the nested
function is actually being called. This in turn would again then call the
original wrapped function.

```python
def function_wrapper(wrapped):
    def _wrapper(*args, **kwargs):
        return wrapped(*args, **kwargs)
    return _wrapper

@function_wrapper
def function():
    pass
```

In this example the nested function doesn't actually get passed the
original wrapped function explicitly. But it will still have access to it
via the arguments given to the outer function call. This does away with the
need to create a class to hold what was the wrapped function and thus why
it is convenient and generally more popular.

Introspecting a function
------------------------

Now when we talk about functions, we expect them to specify properties
which describe them as well as document what they do. These include the
`__name__` and `__doc__` attributes. When we use a wrapper though, this
no longer works as we expect as in the case of using a function closure,
the details of the nested function are returned.

```python
def function_wrapper(wrapped):
    def _wrapper(*args, **kwargs):
        return wrapped(*args, **kwargs)
    return _wrapper

@function_wrapper
def function():
    pass
```

```pycon
>>> print(function.__name__)
_wrapper
```

If we use a class to implement the wrapper, as class instances do not
normally have a `__name__` attribute, attempting to access the name of
the function will actually result in an AttributeError exception.

```python
class function_wrapper(object):
    def __init__(self, wrapped):
        self.wrapped = wrapped
    def __call__(self, *args, **kwargs):
        return self.wrapped(*args, **kwargs)

@function_wrapper
def function():
    pass
```

```pycon
>>> print(function.__name__)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'function_wrapper' object has no attribute '__name__'
```

The solution here when using a function closure is to copy the attributes
of interest from the wrapped function to the nested wrapper function. This
will then result in the function name and documentation strings being
correct.

```python
def function_wrapper(wrapped):
    def _wrapper(*args, **kwargs):
        return wrapped(*args, **kwargs)
    _wrapper.__name__ = wrapped.__name__
    _wrapper.__doc__ = wrapped.__doc__
    return _wrapper

@function_wrapper
def function():
    pass

>>> print(function.__name__)
function
```

Needing to manually copy the attributes is laborious, and would need to be
updated if any further special attributes were added which needed to be
copied. For example, we should also copy the `__module__` attribute, and
in Python 3 the `__qualname__` and `__annotations__` attributes were
added. To aid in getting this right, the Python standard library provides
the `functools.wraps()` decorator which does this task for you.s

```python
import functools

def function_wrapper(wrapped):
    @functools.wraps(wrapped)
    def _wrapper(*args, **kwargs):
        return wrapped(*args, **kwargs)
    return _wrapper

@function_wrapper
def function():
    pass
```

```pycon
>>> print(function.__name__)
function
```

If using a class to implement the wrapper, instead of the
`functools.wraps()` decorator, we would use the
`functools.update_wrapper()` function.

```python
import functools

class function_wrapper(object):
    def __init__(self, wrapped):
        self.wrapped = wrapped
        functools.update_wrapper(self, wrapped)
    def __call__(self, *args, **kwargs):
        return self.wrapped(*args, **kwargs)
```

So we might have a solution to ensuring the function name and any
documentation string is correct in the form of `functools.wraps()`, but
actually we don't and this will not always work as I will show below.

Now what if we want to query the argument specification for a function.
This also fails and instead of returning the argument specification for the
wrapped function, it returns that of the wrapper. In the case of using a
function closure, this is the nested function. The decorator is therefore
not signature preserving.

```python
import inspect

def function_wrapper(wrapped): ...

@function_wrapper
def function(arg1, arg2): pass
```

```pycon
>>> print(inspect.getargspec(function))
ArgSpec(args=[], varargs='args', keywords='kwargs', defaults=None)
```

A worse situation again occurs with the class based wrapper. This time we
get an exception complaining that the wrapped function isn't actually a
function. As a result it isn't possible to derive an argument specification
at all, even though the wrapped function is actually still callable.

```python
class function_wrapper(object): ...

@function_wrapper
def function(arg1, arg2): pass
```

```pycon
>>> print(inspect.getargspec(function))
Traceback (most recent call last):
  File "...", line XXX, in <module>
    print(inspect.getargspec(function))
  File ".../inspect.py", line 813, in getargspec
    raise TypeError('{!r} is not a Python function'.format(func))
TypeError: <__main__.function_wrapper object at 0x107e0ac90> is not a Python function
```

Another example of introspection one can do is to use
`inspect.getsource()` to get back the source code related to a function.
This also will fail, with it giving the source code for the nested wrapper
function in the case of a function closure and again failing outright with
an exception in the case of the class based wrapper.

Wrapping class methods
----------------------

Now, as well as normal functions, decorators can also be applied to methods
of classes. Python even includes a couple of special decorators called
`@classmethod` and `@staticmethod` for converting normal instance
methods into these special method types. Methods of classes do provide a
number of potential problems though.

```python
class Class(object):

    @function_wrapper
    def method(self):
        pass

    @classmethod
    def cmethod(cls):
        pass

    @staticmethod
    def smethod():
        pass
```

The first is that even if using `functools.wraps()` or
`functools.update_wrapper()` in your decorator, when the decorator is
applied around ``@classmethod` or `@staticmethod`, it can fail with an
exception. This is because the wrappers created by these, do not have some
of the attributes being copied.

```python
class Class(object):
    @function_wrapper
    @classmethod
    def cmethod(cls):
        pass
```

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in Class
  File "<stdin>", line 2, in wrapper
  File ".../functools.py", line 33, in update_wrapper
    setattr(wrapper, attr, getattr(wrapped, attr))
AttributeError: 'classmethod' object has no attribute '__module__'
```

As it happens, this is a Python 2 bug and it is fixed in Python 3 by
ignoring missing attributes.

Even when we run it under Python 3, we still hit trouble though. This is
because both wrapper types assume that the wrapped function is directly
callable. This need not actually be the case. A wrapped function can
actually be what is called a descriptor, meaning that in order to get back
a callable, the descriptor has to be correctly bound to the instance first.

```python
class Class(object):
    @function_wrapper
    @classmethod
    def cmethod(cls):
        pass

>>> Class.cmethod()
Traceback (most recent call last):
  File "classmethod.py", line 15, in <module>
    Class.cmethod()
  File "classmethod.py", line 6, in _wrapper
    return wrapped(*args, **kwargs)
TypeError: 'classmethod' object is not callable
```

Simple does not imply correctness
---------------------------------

So although the usual way that people implement decorators is simple, that
doesn't mean they are necessarily correct and will always work.

The issues highlighted so far are:

* Preservation of function `__name__` and `__doc__`.
* Preservation of function argument specification.
* Preservation of ability to get function source code.
* Ability to apply decorators on top of other decorators that are implemented as descriptors.

The `functools.wraps()` function is given as a solution to the first but
doesn't always work, at least in Python 2. It doesn't help at all though
with preserving the introspection of a functions argument specification and
ability to get the source code for a function.

Even if one could solve the introspection problem, the simple decorator
implementation that is generally offered up as the way to do things, breaks
the execution model for the Python object model, not honouring the
descriptor protocol of anything which is wrapped by the decorator.

Third party packages do exist which try and solve these issues, such as the
decorator module available on PyPi. This module in particular though only
helps with the first two and still has potential issues with how it works
that may cause problems when trying to dynamically apply function wrappers
via monkey patching.

This doesn't mean these problems aren't solvable, and solvable in a way
that doesn't sacrifice performance. In my search at least, I could not
actually find any one who has described a comprehensive solution or offered
up a package which performs all the required magic so you don't have to
worry about it yourself.

This blog post is therefore the first step in me explaining how it can be
all made to work. I have stated the problems to be solved and in subsequent
posts I will explain how they can be solved and what extra capabilities
that gives you which enables the ability to write even more magic
decorators than what is possible now with traditional ways that decorators
have been implemented.

So stay tuned for the next instalment. Hopefully I can keep the momentum up
and keep them coming. Pester me if I don't.