.. include:: header.txt ============================== processing package reference ============================== The `processing` package mostly replicates the API of the `threading` module. Classes and exceptions ---------------------- **class** `Process(group=None, target=None, name=None, args=(), kwargs={})` An analogue of `threading.Thread`. See `Process objects`_. **exception** `BufferTooShort` Exception raised by the `recvBytesInto()` method of a `connection object `_ when the supplied buffer object is too small for the message read. If `e` is an instance of `BufferTooShort` then `e.args[0]` will give the message as a byte string. Pipes and Queues ---------------- When using multiple processes one generally uses message passing for communication between processes and avoids having to use any synchronization primitives like locks. For passing messages one can use a pipe (for a connection between two processes) or a queue (which allows multiple producers and consumers). Note that one can also create a shared queue by using a manager object -- see `Managers`_. For an example of the usage of queues for interprocess communication see `ex_workers.py <../examples/ex_workers.py>`_. `Pipe(duplex=True)` Returns a pair `(conn1, conn2)` of connection objects representing the ends of a pipe. If `duplex` is true then the pipe is two way; otherwise `conn1` can only be used for receiving messages and `conn2` can only be used for sending messages. See `Connection objects `_. `Queue(maxsize=0)` Returns a process shared queue object. The usual `Empty` and `Full` exceptions from the standard library's `Queue` module are raised to signal timeouts. See `Queue objects `_. Synchronization primitives -------------------------- Generally synchronization primitives are not as necessary in a multiprocess program as they are in a mulithreaded program. See the documentation for the standard library's `threading` module. Note that one can also create synchronization primitves by using a manager object -- see `Managers`_. `BoundedSemaphore(value=1)` Returns a bounded semaphore object: a clone of `threading.BoundedSemaphore`. (On Mac OSX this is indistiguishable from `Semaphore()` because `sem_getvalue()` is not implemented on that platform). `Condition(lock=None)` Returns a condition variable: a clone of `threading.Condition`. If `lock` is specified then it should be a `Lock` or `RLock` object from `processing`. `Event()` Returns an event object: a clone of `threading.Event`. `Lock()` Returns a non-recursive lock object: a clone of `threading.Lock`. `RLock()` Returns a recursive lock object: a clone of `threading.RLock`. `Semaphore(value=1)` Returns a bounded semaphore object: a clone of `threading.Semaphore`. .. admonition:: Acquiring with a timeout The `acquire()` method of `BoundedSemaphore`, `Lock`, `RLock` and `Semaphore` has a timeout parameter not supported by the equivalents in `threading`. The signature is `acquire(block=True, timeout=None)` with keyword parameters being acceptable. If `block` is true and `timeout` is not `None` then it specifies a timeout in seconds. If `block` is false then `timeout` is ignored. .. admonition:: Interrupting the main thread If the SIGINT signal generated by Ctrl-C arrives while the main thread is blocked by a call to `BoundedSemaphore.acquire()`, `Lock.acquire()`, `RLock.acquire()`, `Semaphore.acquire()`, `Condition.acquire()` or `Condition.wait()` then the call will be immediately interrupted and `KeyboardInterrupt` will be raised. This differs from the behaviour of `threading` where SIGINT will be ignored while the equivalent blocking calls are in progress. Shared Objects -------------- It is possible to create shared objects using shared memory which can be inherited by child processes. `Value(typecode_or_type, *args, **, lock=True)` Returns a ctypes object allocated from shared memory. By default the return value is actually a synchronized wrapper for the object. `typecode_or_type` determines the type of the returned object: it is either a ctypes type or a one character typecode of the kind used by the `array` module. `*args` is passed on to the constructor for the type. If `lock` is true (the default) then a new lock object is created to synchronize access to the value. If `lock` is a `Lock` or `RLock` object then that will be used to synchronize access to the value. If `lock` is false then access to the returned object will not be automatically protected by a lock, so it will not necessarily be "process-safe". Note that `lock` is a keyword only argument. `Array(typecode_or_type, size_or_initializer, **, lock=True)` Returns a ctypes array allocated from shared memory. By default the return value is actually a synchronized wrapper for the array. `typecode_or_type` determines the type of the elements of the returned array: it is either a ctypes type or a one character typecode of the kind used by the `array` module. If `size_or_initializer` is an integer then it determines the length of the array, and the array will be initially zeroed. Otherwise `size_or_initializer` is a sequence which is used to initialize the array and whose length determines the length of the array. If `lock` is true (the default) then a new lock object is created to synchronize access to the value. If `lock` is a `Lock` or `RLock` object then that will be used to synchronize access to the value. If `lock` is false then access to the returned object will not be automatically protected by a lock, so it will not necessarily be "process-safe". Note that `lock` is a keyword only argument. Note that an array of `ctypes.c_char` has `value` and `rawvalue` attributes which allow one to use it to store and retrieve strings -- see the documentation for ctypes in the standard library. See also `sharedctypes `_. Managers -------- Managers provide a way to create data which can be shared between different processes. `Manager()` Returns a started `SyncManager` object which can be used for sharing objects between processes. The returned manager object corresponds to a spawned child process and has methods which will create shared objects and return corresponding proxies. The methods for creating shared objects are `list()`, `dict()`, `Namespace()`, `Value()`, `Array()`, `Lock()`, `RLock()`, `Semaphore()`, `BoundedSemaphore()`, `Condition()`, `Event()`, `Queue()`. See `SyncManager `_. It is possible to create managers which support other types -- see `Customized managers `_. Process Pools ------------- One can create a pool of processes which will carry out tasks submitted to it. `Pool(processes=None, initializer=None, initargs=())` Returns a process pool object which controls a pool of worker processes to which jobs can be submitted. It supports asynchronous results with timeouts and callbacks and has a parallel map implementation. `processes` is the number of worker processes to use. If `processes` is `None` then the number returned by `cpuCount()` is used. If `initializer` is not `None` then each worker process will call `initializer(*initargs)` when it starts. See `Pool objects `_. Logging ------- Some support for logging is available. Note, however, that the `logging` package does not use process shared locks so it is possible (depending on the handler type) for messages from different processes to get mixed up. `enableLogging(level, HandlerType=None, handlerArgs=(), format=None)` Enables logging and sets the debug level used by the package's logger to `level`. See documentation for the `logging` module in the standard library. If `HandlerType` is specified then a handler is created using `HandlerType(*handlerArgs)` and this will be used by the logger -- any previous handlers will be discarded. If `format` is specified then this will be used for the handler; otherwise `format` defaults to `'[%(levelname)s/%(processName)s] %(message)s'`. (The logger used by `processing` allows use of the non-standard `'%(processName)s'` format.) If `HandlerType` is not specified and the logger has no handlers then a default one is created which prints to `sys.stderr`. *Note*: on Windows a child process does not directly inherit its parent's logger; instead it will automatically call `enableLogging()` with the same arguments which were used when its parent process last called `enableLogging()` (if it ever did). `getLogger()` Returns the logger used by `processing`. If `enableLogging()` has not yet been called then `None` is returned. Below is an example session with logging turned on:: >>> import processing, logging >>> processing.enableLogging(level=logging.INFO) >>> processing.getLogger().warning('doomed') [WARNING/MainProcess] doomed >>> m = processing.Manager() [INFO/SyncManager-1] child process calling self.run() [INFO/SyncManager-1] manager bound to '\\\\.\\pipe\\pyc-2776-0-lj0tfa' >>> del m [INFO/MainProcess] sending shutdown message to manager [INFO/SyncManager-1] manager received shutdown message [INFO/SyncManager-1] manager exiting with exitcode 0 Miscellaneous ------------- `activeChildren()` Return list of all live children of the current process. Calling this has the side affect of "joining" any processes which have already finished. `cpuCount()` Returns the number of CPUs in the system. May raise `NotImplementedError`. `currentProcess()` An analogue of `threading.currentThread()`. Returns the object corresponding to the current process. `freezeSupport()` Adds support for when a program which uses the `processing` package has been frozen to produce a Windows executable. (Has been tested with `py2exe`, `PyInstaller` and `cx_Freeze`.) One needs to call this function straight after the `if __name__ == '__main__'` line of the main module. For example :: from processing import Process, freezeSupport def f(): print 'hello world!' if __name__ == '__main__': freezeSupport() Process(target=f).start() If the `freezeSupport()` line is missed out then trying to run the frozen executable will raise `RuntimeError`. If the module is being run normally by the python interpreter then `freezeSupport()` has no effect. .. note:: * The `processing.dummy` package replicates the API of `processing` but is no more than a wrapper around the `threading` module. * `processing` contains no analogues of `activeCount`, `enumerate`, `settrace`, `setprofile`, `Timer`, or `local` from the `threading` module. Subsections ----------- + `Process objects `_ + `Queue objects `_ + `Connection objects `_ + `Manager objects `_ + `Proxy objects `_ + `Pool objects `_ + `Shared ctypes object `_ + `Listeners and Clients `_ .. _Prev: intro.html .. _Up: index.html .. _Next: process-objects.html