.. include:: header.txt

==============================
 processing package reference 
==============================

The `processing` package mostly replicates the API of the `threading`
module.


Classes and exceptions
----------------------

    **class** `Process(group=None, target=None, name=None, args=(), kwargs={})`
        An analogue of `threading.Thread`.

        See `Process objects`_.

    **exception** `BufferTooShort`
        Exception raised by the `recvBytesInto()` method of a
        `connection object <connection-objects.html>`_
        when the supplied buffer object is too small for the message
        read.

        If `e` is an instance of `BufferTooShort` then `e.args[0]`
        will give the message as a byte string.


Pipes and Queues
----------------

When using multiple processes one generally uses message passing for
communication between processes and avoids having to use any
synchronization primitives like locks.  

For passing messages one can use a pipe (for a connection between two
processes) or a queue (which allows multiple producers and consumers).

Note that one can also create a shared queue by using a manager object
-- see `Managers`_.

For an example of the usage of queues for interprocess communication
see `ex_workers.py <../examples/ex_workers.py>`_.

    `Pipe(duplex=True)`
        Returns a pair `(conn1, conn2)` of connection objects
        representing the ends of a pipe.  

        If `duplex` is true then the pipe is two way; otherwise
        `conn1` can only be used for receiving messages and `conn2`
        can only be used for sending messages.
        
        See `Connection objects <connection-objects.html>`_.

    `Queue(maxsize=0)`
        Returns a process shared queue object.  The usual `Empty` and
        `Full` exceptions from the standard library's `Queue` module
        are raised to signal timeouts.

        See `Queue objects <queue-objects.html>`_.

     
Synchronization primitives
--------------------------

Generally synchronization primitives are not as necessary in a
multiprocess program as they are in a mulithreaded program.  See the
documentation for the standard library's `threading` module.

Note that one can also create synchronization primitves by using a
manager object -- see `Managers`_.

    `BoundedSemaphore(value=1)`
        Returns a bounded semaphore object: a clone of
        `threading.BoundedSemaphore`.  

        (On Mac OSX this is indistiguishable from `Semaphore()`
        because `sem_getvalue()` is not implemented on that platform).

    `Condition(lock=None)`
        Returns a condition variable: a clone of `threading.Condition`.
        
        If `lock` is specified then it should be a `Lock` or `RLock`
        object from `processing`.
        
    `Event()`
        Returns an event object: a clone of `threading.Event`.

    `Lock()` 
        Returns a non-recursive lock object: a clone of `threading.Lock`.
                        
    `RLock()`
        Returns a recursive lock object: a clone of `threading.RLock`.
        
    `Semaphore(value=1)`
        Returns a bounded semaphore object: a clone of
        `threading.Semaphore`.


.. admonition:: Acquiring with a timeout

   The `acquire()` method of `BoundedSemaphore`, `Lock`, `RLock` and
   `Semaphore` has a timeout parameter not supported by the
   equivalents in `threading`.  The signature is `acquire(block=True,
   timeout=None)` with keyword parameters being acceptable.  If
   `block` is true and `timeout` is not `None` then it specifies a
   timeout in seconds.  If `block` is false then `timeout` is ignored.

.. admonition:: Interrupting the main thread

   If the SIGINT signal generated by Ctrl-C arrives while the main
   thread is blocked by a call to `BoundedSemaphore.acquire()`,
   `Lock.acquire()`, `RLock.acquire()`, `Semaphore.acquire()`,
   `Condition.acquire()` or `Condition.wait()` then the call will be
   immediately interrupted and `KeyboardInterrupt` will be raised.

   This differs from the behaviour of `threading` where SIGINT will be
   ignored while the equivalent blocking calls are in progress.



Shared Objects
--------------

It is possible to create shared objects using shared memory which can
be inherited by child processes.

    `Value(typecode_or_type, *args, **, lock=True)`
        Returns a ctypes object allocated from shared memory.  By
        default the return value is actually a synchronized wrapper
        for the object.

        `typecode_or_type` determines the type of the returned object:
        it is either a ctypes type or a one character typecode of the
        kind used by the `array` module.  `*args` is passed on to the
        constructor for the type.

        If `lock` is true (the default) then a new lock object is
        created to synchronize access to the value.  If `lock` is a
        `Lock` or `RLock` object then that will be used to synchronize
        access to the value.  If `lock` is false then access to the
        returned object will not be automatically protected by a lock,
        so it will not necessarily be "process-safe".

        Note that `lock` is a keyword only argument.

    `Array(typecode_or_type, size_or_initializer, **, lock=True)`
        Returns a ctypes array allocated from shared memory.  By
        default the return value is actually a synchronized wrapper
        for the array.

        `typecode_or_type` determines the type of the elements of the
        returned array: it is either a ctypes type or a one character
        typecode of the kind used by the `array` module.  If
        `size_or_initializer` is an integer then it determines the
        length of the array, and the array will be initially zeroed.
        Otherwise `size_or_initializer` is a sequence which is used to
        initialize the array and whose length determines the length of
        the array.

        If `lock` is true (the default) then a new lock object is
        created to synchronize access to the value.  If `lock` is a
        `Lock` or `RLock` object then that will be used to synchronize
        access to the value.  If `lock` is false then access to the
        returned object will not be automatically protected by a lock,
        so it will not necessarily be "process-safe".

        Note that `lock` is a keyword only argument.

        Note that an array of `ctypes.c_char` has `value` and
        `rawvalue` attributes which allow one to use it to store and
        retrieve strings -- see the documentation for ctypes in the
        standard library.

See also `sharedctypes <sharedctypes.html>`_.


Managers
--------

Managers provide a way to create data which can be shared between
different processes.

    `Manager()`
        Returns a started `SyncManager` object which can be
        used for sharing objects between processes.  The returned
        manager object corresponds to a spawned child process and has
        methods which will create shared objects and return
        corresponding proxies.

        The methods for creating shared objects are

            `list()`, `dict()`, `Namespace()`, `Value()`,
            `Array()`, `Lock()`, `RLock()`, `Semaphore()`,
            `BoundedSemaphore()`, `Condition()`, `Event()`, `Queue()`.
            
        See `SyncManager <manager-objects.html#sync-manager>`_.

It is possible to create managers which support other types -- see
`Customized managers <manager-objects.html#customized-managers>`_.


Process Pools
-------------

One can create a pool of processes which will carry out tasks
submitted to it.

    `Pool(processes=None, initializer=None, initargs=())`
        Returns a process pool object which controls a pool of worker
        processes to which jobs can be submitted.

        It supports asynchronous results with timeouts and
        callbacks and has a parallel map implementation.

        `processes` is the number of worker processes to use.  If
        `processes` is `None` then the number returned by `cpuCount()`
        is used.  If `initializer` is not `None` then each worker
        process will call `initializer(*initargs)` when it starts.

        See `Pool objects <pool-objects.html>`_.
        

Logging
-------

Some support for logging is available.  Note, however, that the
`logging` package does not use process shared locks so it is possible
(depending on the handler type) for messages from different processes
to get mixed up.

    `enableLogging(level, HandlerType=None, handlerArgs=(), format=None)`
        Enables logging and sets the debug level used by the package's
        logger to `level`.  See documentation for the `logging` module
        in the standard library.

        If `HandlerType` is specified then a handler is created using
        `HandlerType(*handlerArgs)` and this will be used by the
        logger -- any previous handlers will be discarded.  If
        `format` is specified then this will be used for the handler;
        otherwise `format` defaults to
        `'[%(levelname)s/%(processName)s] %(message)s'`.  (The logger
        used by `processing` allows use of the non-standard
        `'%(processName)s'` format.)

        If `HandlerType` is not specified and the logger has no
        handlers then a default one is created which prints to
        `sys.stderr`.

        *Note*: on Windows a child process does not directly inherit
        its parent's logger; instead it will automatically call
        `enableLogging()` with the same arguments which were used when
        its parent process last called `enableLogging()` (if it ever
        did).

    `getLogger()`
        Returns the logger used by `processing`.  If `enableLogging()`
        has not yet been called then `None` is returned.

Below is an example session with logging turned on::

    >>> import processing, logging
    >>> processing.enableLogging(level=logging.INFO)
    >>> processing.getLogger().warning('doomed')
    [WARNING/MainProcess] doomed
    >>> m = processing.Manager()
    [INFO/SyncManager-1] child process calling self.run()
    [INFO/SyncManager-1] manager bound to '\\\\.\\pipe\\pyc-2776-0-lj0tfa'
    >>> del m
    [INFO/MainProcess] sending shutdown message to manager
    [INFO/SyncManager-1] manager received shutdown message
    [INFO/SyncManager-1] manager exiting with exitcode 0


Miscellaneous
-------------

    `activeChildren()`
        Return list of all live children of the current process.
        
        Calling this has the side affect of "joining" any processes
        which have already finished.

    `cpuCount()`
        Returns the number of CPUs in the system.  May raise
        `NotImplementedError`.

    `currentProcess()`
        An analogue of `threading.currentThread()`.

        Returns the object corresponding to the current process.

    `freezeSupport()`
        Adds support for when a program which uses the `processing`
        package has been frozen to produce a Windows executable.  (Has
        been tested with `py2exe`, `PyInstaller` and `cx_Freeze`.)

        One needs to call this function straight after the `if __name__
        == '__main__'` line of the main module.  For example ::

            from processing import Process, freezeSupport

            def f():
                print 'hello world!'

            if __name__ == '__main__':
                freezeSupport()
                Process(target=f).start()

        If the `freezeSupport()` line is missed out then trying to run
        the frozen executable will raise `RuntimeError`.

        If the module is being run normally by the python interpreter
        then `freezeSupport()` has no effect.
        

.. note::
   * The `processing.dummy` package replicates the API of `processing`
     but is no more than a wrapper around the `threading` module.
     
   * `processing` contains no analogues of `activeCount`,
     `enumerate`, `settrace`, `setprofile`, `Timer`, or
     `local` from the `threading` module.


Subsections
-----------

+ `Process objects <process-objects.html>`_
+ `Queue objects <queue-objects.html>`_
+ `Connection objects <connection-objects.html>`_
+ `Manager objects <manager-objects.html>`_
+ `Proxy objects <proxy-objects.html>`_
+ `Pool objects <pool-objects.html>`_
+ `Shared ctypes object <sharedctypes.html>`_
+ `Listeners and Clients <connection-ref.html>`_

.. _Prev: intro.html
.. _Up: index.html
.. _Next: process-objects.html