1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
|
<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="Docutils 0.4: http://docutils.sourceforge.net/" />
<title>processing package reference</title>
<link rel="stylesheet" href="html4css1.css" type="text/css" />
</head>
<body>
<div class="header">
<a class="reference" href="intro.html">Prev</a> <a class="reference" href="index.html">Up</a> <a class="reference" href="process-objects.html">Next</a>
<hr class="header"/>
</div>
<div class="document" id="processing-package-reference">
<h1 class="title">processing package reference</h1>
<p>The <tt class="docutils literal"><span class="pre">processing</span></tt> package mostly replicates the API of the <tt class="docutils literal"><span class="pre">threading</span></tt>
module.</p>
<div class="section">
<h1><a id="classes-and-exceptions" name="classes-and-exceptions">Classes and exceptions</a></h1>
<blockquote>
<dl class="docutils">
<dt><strong>class</strong> <tt class="docutils literal"><span class="pre">Process(group=None,</span> <span class="pre">target=None,</span> <span class="pre">name=None,</span> <span class="pre">args=(),</span> <span class="pre">kwargs={})</span></tt></dt>
<dd><p class="first">An analogue of <tt class="docutils literal"><span class="pre">threading.Thread</span></tt>.</p>
<p class="last">See <a class="reference" href="process-objects.html">Process objects</a>.</p>
</dd>
<dt><strong>exception</strong> <tt class="docutils literal"><span class="pre">BufferTooShort</span></tt></dt>
<dd><p class="first">Exception raised by the <tt class="docutils literal"><span class="pre">recvBytesInto()</span></tt> method of a
<a class="reference" href="connection-objects.html">connection object</a>
when the supplied buffer object is too small for the message
read.</p>
<p class="last">If <tt class="docutils literal"><span class="pre">e</span></tt> is an instance of <tt class="docutils literal"><span class="pre">BufferTooShort</span></tt> then <tt class="docutils literal"><span class="pre">e.args[0]</span></tt>
will give the message as a byte string.</p>
</dd>
</dl>
</blockquote>
</div>
<div class="section">
<h1><a id="pipes-and-queues" name="pipes-and-queues">Pipes and Queues</a></h1>
<p>When using multiple processes one generally uses message passing for
communication between processes and avoids having to use any
synchronization primitives like locks.</p>
<p>For passing messages one can use a pipe (for a connection between two
processes) or a queue (which allows multiple producers and consumers).</p>
<p>Note that one can also create a shared queue by using a manager object
-- see <a class="reference" href="#managers">Managers</a>.</p>
<p>For an example of the usage of queues for interprocess communication
see <a class="reference" href="../examples/ex_workers.py">ex_workers.py</a>.</p>
<blockquote>
<dl class="docutils">
<dt><tt class="docutils literal"><span class="pre">Pipe(duplex=True)</span></tt></dt>
<dd><p class="first">Returns a pair <tt class="docutils literal"><span class="pre">(conn1,</span> <span class="pre">conn2)</span></tt> of connection objects
representing the ends of a pipe.</p>
<p>If <tt class="docutils literal"><span class="pre">duplex</span></tt> is true then the pipe is two way; otherwise
<tt class="docutils literal"><span class="pre">conn1</span></tt> can only be used for receiving messages and <tt class="docutils literal"><span class="pre">conn2</span></tt>
can only be used for sending messages.</p>
<p class="last">See <a class="reference" href="connection-objects.html">Connection objects</a>.</p>
</dd>
<dt><tt class="docutils literal"><span class="pre">Queue(maxsize=0)</span></tt></dt>
<dd><p class="first">Returns a process shared queue object. The usual <tt class="docutils literal"><span class="pre">Empty</span></tt> and
<tt class="docutils literal"><span class="pre">Full</span></tt> exceptions from the standard library's <tt class="docutils literal"><span class="pre">Queue</span></tt> module
are raised to signal timeouts.</p>
<p class="last">See <a class="reference" href="queue-objects.html">Queue objects</a>.</p>
</dd>
</dl>
</blockquote>
</div>
<div class="section">
<h1><a id="synchronization-primitives" name="synchronization-primitives">Synchronization primitives</a></h1>
<p>Generally synchronization primitives are not as necessary in a
multiprocess program as they are in a mulithreaded program. See the
documentation for the standard library's <tt class="docutils literal"><span class="pre">threading</span></tt> module.</p>
<p>Note that one can also create synchronization primitves by using a
manager object -- see <a class="reference" href="#managers">Managers</a>.</p>
<blockquote>
<dl class="docutils">
<dt><tt class="docutils literal"><span class="pre">BoundedSemaphore(value=1)</span></tt></dt>
<dd><p class="first">Returns a bounded semaphore object: a clone of
<tt class="docutils literal"><span class="pre">threading.BoundedSemaphore</span></tt>.</p>
<p class="last">(On Mac OSX this is indistiguishable from <tt class="docutils literal"><span class="pre">Semaphore()</span></tt>
because <tt class="docutils literal"><span class="pre">sem_getvalue()</span></tt> is not implemented on that platform).</p>
</dd>
<dt><tt class="docutils literal"><span class="pre">Condition(lock=None)</span></tt></dt>
<dd><p class="first">Returns a condition variable: a clone of <tt class="docutils literal"><span class="pre">threading.Condition</span></tt>.</p>
<p class="last">If <tt class="docutils literal"><span class="pre">lock</span></tt> is specified then it should be a <tt class="docutils literal"><span class="pre">Lock</span></tt> or <tt class="docutils literal"><span class="pre">RLock</span></tt>
object from <tt class="docutils literal"><span class="pre">processing</span></tt>.</p>
</dd>
<dt><tt class="docutils literal"><span class="pre">Event()</span></tt></dt>
<dd>Returns an event object: a clone of <tt class="docutils literal"><span class="pre">threading.Event</span></tt>.</dd>
<dt><tt class="docutils literal"><span class="pre">Lock()</span></tt></dt>
<dd>Returns a non-recursive lock object: a clone of <tt class="docutils literal"><span class="pre">threading.Lock</span></tt>.</dd>
<dt><tt class="docutils literal"><span class="pre">RLock()</span></tt></dt>
<dd>Returns a recursive lock object: a clone of <tt class="docutils literal"><span class="pre">threading.RLock</span></tt>.</dd>
<dt><tt class="docutils literal"><span class="pre">Semaphore(value=1)</span></tt></dt>
<dd>Returns a bounded semaphore object: a clone of
<tt class="docutils literal"><span class="pre">threading.Semaphore</span></tt>.</dd>
</dl>
</blockquote>
<div class="admonition-acquiring-with-a-timeout admonition">
<p class="first admonition-title">Acquiring with a timeout</p>
<p class="last">The <tt class="docutils literal"><span class="pre">acquire()</span></tt> method of <tt class="docutils literal"><span class="pre">BoundedSemaphore</span></tt>, <tt class="docutils literal"><span class="pre">Lock</span></tt>, <tt class="docutils literal"><span class="pre">RLock</span></tt> and
<tt class="docutils literal"><span class="pre">Semaphore</span></tt> has a timeout parameter not supported by the
equivalents in <tt class="docutils literal"><span class="pre">threading</span></tt>. The signature is <tt class="docutils literal"><span class="pre">acquire(block=True,</span>
<span class="pre">timeout=None)</span></tt> with keyword parameters being acceptable. If
<tt class="docutils literal"><span class="pre">block</span></tt> is true and <tt class="docutils literal"><span class="pre">timeout</span></tt> is not <tt class="docutils literal"><span class="pre">None</span></tt> then it specifies a
timeout in seconds. If <tt class="docutils literal"><span class="pre">block</span></tt> is false then <tt class="docutils literal"><span class="pre">timeout</span></tt> is ignored.</p>
</div>
<div class="admonition-interrupting-the-main-thread admonition">
<p class="first admonition-title">Interrupting the main thread</p>
<p>If the SIGINT signal generated by Ctrl-C arrives while the main
thread is blocked by a call to <tt class="docutils literal"><span class="pre">BoundedSemaphore.acquire()</span></tt>,
<tt class="docutils literal"><span class="pre">Lock.acquire()</span></tt>, <tt class="docutils literal"><span class="pre">RLock.acquire()</span></tt>, <tt class="docutils literal"><span class="pre">Semaphore.acquire()</span></tt>,
<tt class="docutils literal"><span class="pre">Condition.acquire()</span></tt> or <tt class="docutils literal"><span class="pre">Condition.wait()</span></tt> then the call will be
immediately interrupted and <tt class="docutils literal"><span class="pre">KeyboardInterrupt</span></tt> will be raised.</p>
<p class="last">This differs from the behaviour of <tt class="docutils literal"><span class="pre">threading</span></tt> where SIGINT will be
ignored while the equivalent blocking calls are in progress.</p>
</div>
</div>
<div class="section">
<h1><a id="shared-objects" name="shared-objects">Shared Objects</a></h1>
<p>It is possible to create shared objects using shared memory which can
be inherited by child processes.</p>
<blockquote>
<dl class="docutils">
<dt><tt class="docutils literal"><span class="pre">Value(typecode_or_type,</span> <span class="pre">*args,</span> <span class="pre">**,</span> <span class="pre">lock=True)</span></tt></dt>
<dd><p class="first">Returns a ctypes object allocated from shared memory. By
default the return value is actually a synchronized wrapper
for the object.</p>
<p><tt class="docutils literal"><span class="pre">typecode_or_type</span></tt> determines the type of the returned object:
it is either a ctypes type or a one character typecode of the
kind used by the <tt class="docutils literal"><span class="pre">array</span></tt> module. <tt class="docutils literal"><span class="pre">*args</span></tt> is passed on to the
constructor for the type.</p>
<p>If <tt class="docutils literal"><span class="pre">lock</span></tt> is true (the default) then a new lock object is
created to synchronize access to the value. If <tt class="docutils literal"><span class="pre">lock</span></tt> is a
<tt class="docutils literal"><span class="pre">Lock</span></tt> or <tt class="docutils literal"><span class="pre">RLock</span></tt> object then that will be used to synchronize
access to the value. If <tt class="docutils literal"><span class="pre">lock</span></tt> is false then access to the
returned object will not be automatically protected by a lock,
so it will not necessarily be "process-safe".</p>
<p class="last">Note that <tt class="docutils literal"><span class="pre">lock</span></tt> is a keyword only argument.</p>
</dd>
<dt><tt class="docutils literal"><span class="pre">Array(typecode_or_type,</span> <span class="pre">size_or_initializer,</span> <span class="pre">**,</span> <span class="pre">lock=True)</span></tt></dt>
<dd><p class="first">Returns a ctypes array allocated from shared memory. By
default the return value is actually a synchronized wrapper
for the array.</p>
<p><tt class="docutils literal"><span class="pre">typecode_or_type</span></tt> determines the type of the elements of the
returned array: it is either a ctypes type or a one character
typecode of the kind used by the <tt class="docutils literal"><span class="pre">array</span></tt> module. If
<tt class="docutils literal"><span class="pre">size_or_initializer</span></tt> is an integer then it determines the
length of the array, and the array will be initially zeroed.
Otherwise <tt class="docutils literal"><span class="pre">size_or_initializer</span></tt> is a sequence which is used to
initialize the array and whose length determines the length of
the array.</p>
<p>If <tt class="docutils literal"><span class="pre">lock</span></tt> is true (the default) then a new lock object is
created to synchronize access to the value. If <tt class="docutils literal"><span class="pre">lock</span></tt> is a
<tt class="docutils literal"><span class="pre">Lock</span></tt> or <tt class="docutils literal"><span class="pre">RLock</span></tt> object then that will be used to synchronize
access to the value. If <tt class="docutils literal"><span class="pre">lock</span></tt> is false then access to the
returned object will not be automatically protected by a lock,
so it will not necessarily be "process-safe".</p>
<p>Note that <tt class="docutils literal"><span class="pre">lock</span></tt> is a keyword only argument.</p>
<p class="last">Note that an array of <tt class="docutils literal"><span class="pre">ctypes.c_char</span></tt> has <tt class="docutils literal"><span class="pre">value</span></tt> and
<tt class="docutils literal"><span class="pre">rawvalue</span></tt> attributes which allow one to use it to store and
retrieve strings -- see the documentation for ctypes in the
standard library.</p>
</dd>
</dl>
</blockquote>
<p>See also <a class="reference" href="sharedctypes.html">sharedctypes</a>.</p>
</div>
<div class="section">
<h1><a id="managers" name="managers">Managers</a></h1>
<p>Managers provide a way to create data which can be shared between
different processes.</p>
<blockquote>
<dl class="docutils">
<dt><tt class="docutils literal"><span class="pre">Manager()</span></tt></dt>
<dd><p class="first">Returns a started <tt class="docutils literal"><span class="pre">SyncManager</span></tt> object which can be
used for sharing objects between processes. The returned
manager object corresponds to a spawned child process and has
methods which will create shared objects and return
corresponding proxies.</p>
<p>The methods for creating shared objects are</p>
<blockquote>
<tt class="docutils literal"><span class="pre">list()</span></tt>, <tt class="docutils literal"><span class="pre">dict()</span></tt>, <tt class="docutils literal"><span class="pre">Namespace()</span></tt>, <tt class="docutils literal"><span class="pre">Value()</span></tt>,
<tt class="docutils literal"><span class="pre">Array()</span></tt>, <tt class="docutils literal"><span class="pre">Lock()</span></tt>, <tt class="docutils literal"><span class="pre">RLock()</span></tt>, <tt class="docutils literal"><span class="pre">Semaphore()</span></tt>,
<tt class="docutils literal"><span class="pre">BoundedSemaphore()</span></tt>, <tt class="docutils literal"><span class="pre">Condition()</span></tt>, <tt class="docutils literal"><span class="pre">Event()</span></tt>, <tt class="docutils literal"><span class="pre">Queue()</span></tt>.</blockquote>
<p class="last">See <a class="reference" href="manager-objects.html#sync-manager">SyncManager</a>.</p>
</dd>
</dl>
</blockquote>
<p>It is possible to create managers which support other types -- see
<a class="reference" href="manager-objects.html#customized-managers">Customized managers</a>.</p>
</div>
<div class="section">
<h1><a id="process-pools" name="process-pools">Process Pools</a></h1>
<p>One can create a pool of processes which will carry out tasks
submitted to it.</p>
<blockquote>
<dl class="docutils">
<dt><tt class="docutils literal"><span class="pre">Pool(processes=None,</span> <span class="pre">initializer=None,</span> <span class="pre">initargs=())</span></tt></dt>
<dd><p class="first">Returns a process pool object which controls a pool of worker
processes to which jobs can be submitted.</p>
<p>It supports asynchronous results with timeouts and
callbacks and has a parallel map implementation.</p>
<p><tt class="docutils literal"><span class="pre">processes</span></tt> is the number of worker processes to use. If
<tt class="docutils literal"><span class="pre">processes</span></tt> is <tt class="docutils literal"><span class="pre">None</span></tt> then the number returned by <tt class="docutils literal"><span class="pre">cpuCount()</span></tt>
is used. If <tt class="docutils literal"><span class="pre">initializer</span></tt> is not <tt class="docutils literal"><span class="pre">None</span></tt> then each worker
process will call <tt class="docutils literal"><span class="pre">initializer(*initargs)</span></tt> when it starts.</p>
<p class="last">See <a class="reference" href="pool-objects.html">Pool objects</a>.</p>
</dd>
</dl>
</blockquote>
</div>
<div class="section">
<h1><a id="logging" name="logging">Logging</a></h1>
<p>Some support for logging is available. Note, however, that the
<tt class="docutils literal"><span class="pre">logging</span></tt> package does not use process shared locks so it is possible
(depending on the handler type) for messages from different processes
to get mixed up.</p>
<blockquote>
<dl class="docutils">
<dt><tt class="docutils literal"><span class="pre">enableLogging(level,</span> <span class="pre">HandlerType=None,</span> <span class="pre">handlerArgs=(),</span> <span class="pre">format=None)</span></tt></dt>
<dd><p class="first">Enables logging and sets the debug level used by the package's
logger to <tt class="docutils literal"><span class="pre">level</span></tt>. See documentation for the <tt class="docutils literal"><span class="pre">logging</span></tt> module
in the standard library.</p>
<p>If <tt class="docutils literal"><span class="pre">HandlerType</span></tt> is specified then a handler is created using
<tt class="docutils literal"><span class="pre">HandlerType(*handlerArgs)</span></tt> and this will be used by the
logger -- any previous handlers will be discarded. If
<tt class="docutils literal"><span class="pre">format</span></tt> is specified then this will be used for the handler;
otherwise <tt class="docutils literal"><span class="pre">format</span></tt> defaults to
<tt class="docutils literal"><span class="pre">'[%(levelname)s/%(processName)s]</span> <span class="pre">%(message)s'</span></tt>. (The logger
used by <tt class="docutils literal"><span class="pre">processing</span></tt> allows use of the non-standard
<tt class="docutils literal"><span class="pre">'%(processName)s'</span></tt> format.)</p>
<p>If <tt class="docutils literal"><span class="pre">HandlerType</span></tt> is not specified and the logger has no
handlers then a default one is created which prints to
<tt class="docutils literal"><span class="pre">sys.stderr</span></tt>.</p>
<p class="last"><em>Note</em>: on Windows a child process does not directly inherit
its parent's logger; instead it will automatically call
<tt class="docutils literal"><span class="pre">enableLogging()</span></tt> with the same arguments which were used when
its parent process last called <tt class="docutils literal"><span class="pre">enableLogging()</span></tt> (if it ever
did).</p>
</dd>
<dt><tt class="docutils literal"><span class="pre">getLogger()</span></tt></dt>
<dd>Returns the logger used by <tt class="docutils literal"><span class="pre">processing</span></tt>. If <tt class="docutils literal"><span class="pre">enableLogging()</span></tt>
has not yet been called then <tt class="docutils literal"><span class="pre">None</span></tt> is returned.</dd>
</dl>
</blockquote>
<p>Below is an example session with logging turned on:</p>
<pre class="literal-block">
>>> import processing, logging
>>> processing.enableLogging(level=logging.INFO)
>>> processing.getLogger().warning('doomed')
[WARNING/MainProcess] doomed
>>> m = processing.Manager()
[INFO/SyncManager-1] child process calling self.run()
[INFO/SyncManager-1] manager bound to '\\\\.\\pipe\\pyc-2776-0-lj0tfa'
>>> del m
[INFO/MainProcess] sending shutdown message to manager
[INFO/SyncManager-1] manager received shutdown message
[INFO/SyncManager-1] manager exiting with exitcode 0
</pre>
</div>
<div class="section">
<h1><a id="miscellaneous" name="miscellaneous">Miscellaneous</a></h1>
<blockquote>
<dl class="docutils">
<dt><tt class="docutils literal"><span class="pre">activeChildren()</span></tt></dt>
<dd><p class="first">Return list of all live children of the current process.</p>
<p class="last">Calling this has the side affect of "joining" any processes
which have already finished.</p>
</dd>
<dt><tt class="docutils literal"><span class="pre">cpuCount()</span></tt></dt>
<dd>Returns the number of CPUs in the system. May raise
<tt class="docutils literal"><span class="pre">NotImplementedError</span></tt>.</dd>
<dt><tt class="docutils literal"><span class="pre">currentProcess()</span></tt></dt>
<dd><p class="first">An analogue of <tt class="docutils literal"><span class="pre">threading.currentThread()</span></tt>.</p>
<p class="last">Returns the object corresponding to the current process.</p>
</dd>
<dt><tt class="docutils literal"><span class="pre">freezeSupport()</span></tt></dt>
<dd><p class="first">Adds support for when a program which uses the <tt class="docutils literal"><span class="pre">processing</span></tt>
package has been frozen to produce a Windows executable. (Has
been tested with <tt class="docutils literal"><span class="pre">py2exe</span></tt>, <tt class="docutils literal"><span class="pre">PyInstaller</span></tt> and <tt class="docutils literal"><span class="pre">cx_Freeze</span></tt>.)</p>
<p>One needs to call this function straight after the <tt class="docutils literal"><span class="pre">if</span> <span class="pre">__name__</span>
<span class="pre">==</span> <span class="pre">'__main__'</span></tt> line of the main module. For example</p>
<pre class="literal-block">
from processing import Process, freezeSupport
def f():
print 'hello world!'
if __name__ == '__main__':
freezeSupport()
Process(target=f).start()
</pre>
<p>If the <tt class="docutils literal"><span class="pre">freezeSupport()</span></tt> line is missed out then trying to run
the frozen executable will raise <tt class="docutils literal"><span class="pre">RuntimeError</span></tt>.</p>
<p class="last">If the module is being run normally by the python interpreter
then <tt class="docutils literal"><span class="pre">freezeSupport()</span></tt> has no effect.</p>
</dd>
</dl>
</blockquote>
<div class="note">
<p class="first admonition-title">Note</p>
<ul class="last simple">
<li>The <tt class="docutils literal"><span class="pre">processing.dummy</span></tt> package replicates the API of <tt class="docutils literal"><span class="pre">processing</span></tt>
but is no more than a wrapper around the <tt class="docutils literal"><span class="pre">threading</span></tt> module.</li>
<li><tt class="docutils literal"><span class="pre">processing</span></tt> contains no analogues of <tt class="docutils literal"><span class="pre">activeCount</span></tt>,
<tt class="docutils literal"><span class="pre">enumerate</span></tt>, <tt class="docutils literal"><span class="pre">settrace</span></tt>, <tt class="docutils literal"><span class="pre">setprofile</span></tt>, <tt class="docutils literal"><span class="pre">Timer</span></tt>, or
<tt class="docutils literal"><span class="pre">local</span></tt> from the <tt class="docutils literal"><span class="pre">threading</span></tt> module.</li>
</ul>
</div>
</div>
<div class="section">
<h1><a id="subsections" name="subsections">Subsections</a></h1>
<ul class="simple">
<li><a class="reference" href="process-objects.html">Process objects</a></li>
<li><a class="reference" href="queue-objects.html">Queue objects</a></li>
<li><a class="reference" href="connection-objects.html">Connection objects</a></li>
<li><a class="reference" href="manager-objects.html">Manager objects</a></li>
<li><a class="reference" href="proxy-objects.html">Proxy objects</a></li>
<li><a class="reference" href="pool-objects.html">Pool objects</a></li>
<li><a class="reference" href="sharedctypes.html">Shared ctypes object</a></li>
<li><a class="reference" href="connection-ref.html">Listeners and Clients</a></li>
</ul>
</div>
</div>
<div class="footer">
<hr class="footer" />
<a class="reference" href="intro.html">Prev</a> <a class="reference" href="index.html">Up</a> <a class="reference" href="process-objects.html">Next</a>
</div>
</body>
</html>
|
