1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
|
The @synchronized decorator as context manager
==============================================
This is the eight post in my series of blog posts about Python decorators
and how I believe they are generally poorly implemented. It follows on from
the previous post titled [The missing @synchronized
decorator](07-the-missing-synchronized-decorator.md), with the very first
post in the series being [How you implemented your Python decorator is
wrong](01-how-you-implemented-your-python-decorator-is-wrong.md).
In the previous post I described how we could use our new universal
decorator pattern to implement a better @synchronized decorator for Python.
The intent in doing this was to come up with a better approximation of the
equivalent synchronization mechanisms in Java.
Of the two synchronization mechanisms provided by Java, synchronized
methods and synchronized statements, we have however so far only
implemented an equivalent to synchronized methods.
In this post I will describe how we can take our `@synchronized` decorator
and extend it to also be used as a context manager, thus providing an an
equivalent of synchronized statements in Java.
The original @synchronized decorator
------------------------------------
The implementation of our `@synchronized` decorator so far is:
```python
@decorator
def synchronized(wrapped, instance, args, kwargs):
if instance is None:
owner = wrapped
else:
owner = instance
lock = vars(owner).get('_synchronized_lock', None)
if lock is None:
meta_lock = vars(synchronized).setdefault(
'_synchronized_meta_lock', threading.Lock())
with meta_lock:
lock = vars(owner).get('_synchronized_lock', None)
if lock is None:
lock = threading.RLock()
setattr(owner, '_synchronized_lock', lock)
with lock:
return wrapped(*args, **kwargs)
```
By determining whether the decorator is being used to wrap a normal
function, a method bound to a class instance or a class, and with the
decorator changing behaviour as a result, we are able to use the one
decorator implementation in a number of scenarios.
```python
@synchronized # lock bound to function1
def function1():
pass
@synchronized # lock bound to function2
def function2():
pass
@synchronized # lock bound to Class
class Class(object):
@synchronized # lock bound to instance of Class
def function_im(self):
pass
@synchronized # lock bound to Class
@classmethod
def function_cm(cls):
pass
@synchronized # lock bound to function_sm
@staticmethod
def function_sm():
pass
```
What we now want to do is modify the decorator to also allow:
```python
class Object(object):
@synchronized
def function_im_1(self):
pass
def function_im_2(self):
with synchronized(self):
pass
```
That is, as well as being able to be used as a decorator, we enable it to
be used as a context manager in conjunction with the `with` statement. By
doing this it then provides the ability to only acquire the corresponding
lock for a selected number of statements within a function rather than the
whole function.
For the case of acquiring the same lock used by instance methods, we would
pass the `self` argument or instance object into `synchronized` when
used as a context manager. It could instead also be passed the class type
if needing to synchronize with class methods.
The function wrapper as context manager
---------------------------------------
Right now with how the decorator is implemented, when we use
`synchronized` as a function call, it will return an instance of our
function wrapper class.
```pycon
>>> synchronized(None)
<__main__.function_wrapper object at 0x107b7ea10>
```
This function wrapper does not implement the `__enter__()` and
`__exit__()` functions that are required for an object to be used as a
context manager. Since the function wrapper type is our own class, all we
need to do though is create a derived version of the class and use that
instead. Because though the creation of that function wrapper is bound up
within the definition of `@decorator`, we need to bypass `@decorator`
and use the function wrapper directly.
The first step therefore is to rewrite our `@synchronized` decorator so
it doesn't use `@decorator`.
```python
def synchronized(wrapped):
def _synchronized_lock(owner):
lock = vars(owner).get('_synchronized_lock', None)
if lock is None:
meta_lock = vars(synchronized).setdefault(
'_synchronized_meta_lock', threading.Lock())
with meta_lock:
lock = vars(owner).get('_synchronized_lock', None)
if lock is None:
lock = threading.RLock()
setattr(owner, '_synchronized_lock', lock)
return lock
def _synchronized_wrapper(wrapped, instance, args, kwargs):
with _synchronized_lock(instance or wrapped):
return wrapped(*args, **kwargs)
return function_wrapper(wrapped, _synchronized_wrapper)
```
This works the same as our original implementation but we now have access
to the point where the function wrapper was created. With that being the
case, we can now create a class which derives from the function wrapper and
adds the required methods to satisfy the context manager protocol.
```python
def synchronized(wrapped):
def _synchronized_lock(owner):
lock = vars(owner).get('_synchronized_lock', None)
if lock is None:
meta_lock = vars(synchronized).setdefault(
'_synchronized_meta_lock', threading.Lock())
with meta_lock:
lock = vars(owner).get('_synchronized_lock', None)
if lock is None:
lock = threading.RLock()
setattr(owner, '_synchronized_lock', lock)
return lock
def _synchronized_wrapper(wrapped, instance, args, kwargs):
with _synchronized_lock(instance or wrapped):
return wrapped(*args, **kwargs)
class _synchronized_function_wrapper(function_wrapper):
def __enter__(self):
self._lock = _synchronized_lock(self.wrapped)
self._lock.acquire()
return self._lock
def __exit__(self, *args):
self._lock.release()
return _synchronized_function_wrapper(wrapped, _synchronized_wrapper)
```
We now have two scenarios for what can happen.
In the case of `synchronized` being used as a decorator still, our new
derived function wrapper will wrap the function or method it was applied
to. When that function or class method is called, the existing
`__call__()` method of the function wrapper base class will be called and
the decorator semantics will apply with the wrapper called to acquire the
appropriate lock and call the wrapped function.
In the case where is was instead used for the purposes of a context
manager, our new derived function wrapper would actually be wrapping the
class instance or class type. Nothing is being called though and instead
the `with` statement will trigger the execution of the `__enter__()`
and `__exit__()` methods to acquire the appropriate lock around the
context of the statements to be executed.
So all nice and neat and not even that complicated compared to previous
attempts at doing the same thing which were referenced in the prior post on
this topic.
It isn't just about @decorator
------------------------------
Now one of the things that this hopefully shows is that although
`@decorator` can be used to create your own custom decorators, this isn't
always the most appropriate way to go. The separate existence of the
function wrapper gives a great deal of flexibility in what one can do as
far as wrapping objects to modify the behaviour. One can also drop down and
use the object proxy directly in certain circumstances.
All together these provide a general purpose set of tools for doing any
sort of wrapping or monkey patching and not just for use in decorators. I
will now start to shift the focus of this series of blog posts to look more
at the more general area of wrapping and monkey patching.
Before I do get into that, in my next post I will first talk about the
performance implications implicit in using the function wrapper which has
been described when compared to the more traditional way of using function
closures to implement decorators. This will include overhead comparisons
where the complete object proxy and function wrapper classes are
implemented as a Python C extension for added performance.
|
