.. default-role:: literal ========================== Changelog for processing ========================== Changes in 0.52 --------------- * On versions 0.50 and 0.51 Mac OSX `Lock.release()` would fail with `OSError(errno.ENOSYS, "[Errno 78] Function not implemented")`. This appears to be because on Mac OSX `sem_getvalue()` has not been implemented. Now `sem_getvalue()` is no longer needed. Unfortunately, however, on Mac OSX `BoundedSemaphore()` will not raise `ValueError` if it exceeds its initial value. * Some changes to the code for the reduction/rebuilding of connection and socket objects so that things work the same on Windows and Unix. This should fix a couple of bugs. * The code has been changed to consistently use "camelCase" for methods and (non-factory) functions. In the few cases where this has meant a change to the documented API, the old name has been retained as an alias. Changes in 0.51 --------------- * In 0.50 `processing.Value()` and `processing.sharedctypes.Value()` were related but had different signatures, which was rather confusing. Now `processing.sharedctypes.Value()` has been renamed `processing.sharedctypes.RawValue()` and `processing.sharedctypes.Value()` is the same as `processing.Value()`. * In version 0.50 `sendfd()` and `recvfd()` apparently did not work on 64bit Linux. This has been fixed by reverting to using the CMSG_* macros as was done in 0.40. However, this means that systems without all the necessary CMSG_* macros (such as Solaris 8) will have to disable compilation of `sendfd()` and `recvfd()` by setting `macros['HAVE_FD_TRANSFRER'] = 0` in `setup.py`. * Fixed an authentication error when using a "remote" manager created using `BaseManager.from_address()`. * Fixed a couple of bugs which only affected Python 2.4. Changes in 0.50 --------------- * `ctypes` is now a prerequisite if you want to use shared memory -- with Python 2.4 you will need to install it separately. * `LocalManager()` has been removed. * Added `processing.Value()` and `processing.Array()` which are similar to `LocalManager.SharedValue()` and `LocalManager.SharedArray()`. * In the `sharedctypes` module `new_value()` and `new_array()` have been renamed `Value()` and `Array()`. * `Process.stop()`, `Process.getStoppable()` and `Process.setStoppable()` have been removed. Use `Process.terminate()` instead. * `procesing.Lock` now matches `threading.Lock` behaviour more closely: now a thread can release a lock it does not own, and now when a thread tries acquiring a lock it already owns a deadlock results instead of an exception. * On Windows when the main thread is blocking on a method of `Lock`, `RLock`, `Semaphore`, `BoundedSemaphore`, `Condition` it will no longer ignore Ctrl-C. (The same was already true on Unix.) This differs from the behaviour of the equivalent objects in `threading` which will completely ignore Ctrl-C. * The `test` sub-package has been replaced by lots of unit tests in a `tests` sub-package. Some of the old test files have been moved over to a new `examples` sub-package. * On Windows it is now possible for a non-console python program (i.e. one using `pythonw.exe` instead of `python.exe`) to use `processing`. Previously an exception was raised when `subprocess.py` tried to duplicate stdin, stdout, stderr. * Proxy objects should now be thread safe -- they now use thread local storage. * Trying to transfer shared resources such as locks, queues etc between processes over a pipe or queue will now raise `RuntimeError` with a message saying that the object should only be shared between processes using inheritance. Previously, this worked unreliably on Windows but would fail with an unexplained `AssertionError` on Unix. * The names of some of the macros used for compiling the extension have changed. See `INSTALL.txt` and `setup.py`. * A few changes which (hopefully) make compilation possible on Solaris. * Lots of refactoring of the code. * Fixed reference leaks so that unit tests pass with "regrtest -R::" (at least on Linux). Changes in 0.40 --------------- * Removed `SimpleQueue` and `PosixQueue` types. Just use `Queue` instead. * Previously if you forgot to use the :: if __name__ == '__main__': freezeSupport() ... idiom on Windows then processes could be created recursively bringing the computer to its knees. Now `RuntimeError` will be raised instead. * Some refactoring of the code. * A Unix specific bug meant that a child process might fail to start a feeder thread for a queue if its parent process had already started its own feeder thread. Fixed. Changes in 0.39 --------------- * One can now create one-way pipes by doing `reader, writer = Pipe(duplex=False)`. * Rewrote code for managing shared memory maps. * Added a `sharedctypes` module for creating `ctypes` objects allocated from shared memory. On Python 2.4 this requires the installation of `ctypes`. `ctypes` objects are not protected by any locks so you will need to synchronize access to them (such as by using a lock). However they can be much faster to access than equivalent objects allocated using a `LocalManager`. * Rearranged documentation. * Previously the C extension caused a segfault on 64 bit machines with Python 2.5 because it used `int` instead of `Py_ssize_t` in certain places. This is now fixed. Thanks to Alexy Khrabrov for the report. * A fix for `Pool.terminate()`. * A fix for cleanup behaviour of `Queue`. Changes in 0.38 --------------- * Have revamped the queue types. Now the queue types are `Queue`, `SimpleQueue` and (on systems which support it) `PosixQueue`. Now `Queue` should behave just like Python's normal `Queue.Queue` class except that `qsize()`, `task_done()` and `join()` are not implemented. In particular, if no maximum size was specified when the queue was created then `put()` will always succeed without blocking. A `SimpleQueue` instance is really just a pipe protected by a couple of locks. It has `get()`, `put()` and `empty()` methods but does not not support timeouts or non-blocking. `BufferedPipeQueue()` and `PipeQueue()` remain as deprecated aliases of `Queue()` but `BufferedPosixQueue()` has been removed. (Not sure if we really need to keep `PosixQueue()`...) * Previously the `Pool.shutdown()` method was a little dodgy -- it could block indefinitely if `map()` or `imap*()` were used and did not try to terminate workers while they were doing a task. Now there are three new methods `close()`, `terminate()` and `join()` -- `shutdown()` is retained as a deprecated alias of `terminate()`. Thanks to Gerald John M. Manipon for feature request/suggested patch to `shutdown()`. * `Pool.imap()` and `Pool.imap_unordered()` has gained a `chunksize` argument which allows the iterable to be submitted to the pool in chunks. Choosing `chunksize` appropriately makes `Pool.imap()` almost as fast as `Pool.map()` even for long iterables and cheap functions. * Previously on Windows when the cleanup code for a `LocalManager` attempts to unlink the name of the file which backs the shared memory map an exception is raised if a child process still exists which has a handle open for that mmap. This is likely to happen if a daemon process inherits a `LocalManager` instance. Now the parent process will remember the filename and attempt to unlink the file name again once all the child processes have been joined or terminated. Reported by Paul Rudin. * `types.MethodType` is registered with `copy_reg` so now instance methods and class methods should be picklable. (Unfortunately there is no obvious way of supporting the pickling of staticmethods since they are not marked with the class in which they were defined.) This means that on Windows it is now possible to use an instance method or class method as the target callable of a Process object. * On Windows `reduction.fromfd()` now returns true instances of `_socket.socket`, so there is no more need for the `_processing.falsesocket` type. Changes in 0.37 --------------- * Updated metadata and documentation because the project is now hosted at `developer.berlios.de/projects/pyprocessing`. * The `Pool.join()` method has been removed. `Pool.shutdown()` will now join the worker processes automatically. * A pool object no longer participates in a reference cycle so `Pool.shutdown()` should get called as soon as its reference count falls to zero. * On Windows if `enableLogging()` was used at module scope then the logger used by a child process would often get two copies of the same handler. To fix this, now specifiying a handler type in `enableLogging()` will cause any previous handlers used by the logger to be discarded. Changes in 0.36 --------------- * In recent versions on Unix the finalizers in a manager process were never given a chance to run before `os._exit()` was called, so old unlinked AF_UNIX sockets could accumulate in '/tmp'. Fixed. * The shutting down of managers has been cleaned up. * In previous versions on Windows trying to acquire a lock owned by a different thread of the current process would raise an exception. Fixed. * In previous versions on Windows trying to use an event object for synchronization between two threads of the same process was likely to raise an exception. (This was caused by the bug described above.) Fixed. * Previously the arguments to `processing.Semaphore()` and `processing.BoundedSemaphore()` did not have any defaults. The defaults should be 1 to match `threading`. Fixed. * It should now be possible for a Windows Service created by using `pywin32` to spawn processes using the `processing` package. Note that `pywin32` apparently has a bug meaning that `Py_Finalize()` is never called when the service exits so functions registered with `atexit` never get a chance to run. Therefore it is advisable to explicitly call `sys.exitfunc()` or `atexit._run_exitfuncs()` at the end of `ServiceFramework.DoSvcRun()`. Otherwise child processes are liable to survive the service when it is stopped. Thanks to Charlie Hull for the report. * Added `getLogger()` and `enableLogging()` to support logging. Changes in 0.35 --------------- * By default processes are no longer be stoppable using the `stop()` method: one must call `setStoppable(True)` before `start()` in order to use the `stop()` method. (Note that `terminate()` will work regardless of whether the process is marked as being "stoppable".) The reason for this is that on Windows getting `stop()` to work involves starting a new console for the child process and installing a signal handler for the `SIGBREAK` signal. This unfortunately means that Ctrl-Break cannot not be used to kill all processes of the program. * Added `setStoppable()` and `getStoppable()` methods -- see above. * Added `BufferedQueue`/`BufferedPipeQueue`/`BufferedPosixQueue`. Putting an object on a buffered queue will always succeed without blocking (just like with `Queue.Queue` if no maximum size is specified). This makes them potentially safer than the normal queue types provided by `processing` which have finite capacity and may cause deadlocks if they fill. `test/test_worker.py` has been updated to use `BufferedQueue` for the task queue instead of explicitly spawning a thread to feed tasks to the queue without risking a deadlock. * Now when the NO_SEM_TIMED macro is set polling will be used to get around the lack of `sem_timedwait()`. This means that `Condition.wait()` and `Queue.get()` should now work with timeouts on Mac OS X. * Added a `callback` argument to `Pool.apply_async()`. * Added `test/test_httpserverpool.py` which runs a pool of http servers which share a single listening socket. * Previously on Windows the process object was passed to the child process on the commandline (after pickling and hex encoding it). This caused errors when the pickled string was too large. Now if the pickled string is large then it will be passed to the child over a pipe or socket. * Fixed bug in the iterator returned by `Pool.imap()`. * Fixed bug in `Condition.__repr__()`. * Fixed a handle/file descriptor leak when sockets or connections are unpickled. Changes in 0.34 --------------- * Although version 0.33 the C extension would compile on Mac OSX trying to import it failed with "undefined symbol: _sem_timedwait". Unfortunately the `ImportError` exception was silently swallowed. This is now fixed by using the `NO_SEM_TIMED` macro. Unfortunately this means that some methods like `Condition.wait()` and `Queue.get()` will not work with timeouts on Mac OS X. If you really need to be able to use timeouts then you can always use the equivalent objects created with a manager. Thanks to Doug Hellmann for report and testing. * Added a `terminate()` method to process objects which is more forceful than `stop()`. * Fixed bug in the cleanup function registered with `atexit` which on Windows could cause a process which is shutting down to deadlock waiting for a manager to exit. Thanks to Dominique Wahli for report and testing. * Added `test/test_workers.py` which gives an example of how to create a collection of worker processes which execute tasks from one queue and return results on another. * Added `processing.Pool()` which returns a process pool object. This allows one to execute functions asynchronously. It also has a parallel implementation of the `map()` builtin. This is still *experimental* and undocumented --- see `test/test_pool.py` for example usage. Changes in 0.33 --------------- * Added a `recvbytes_into()` method for receiving byte data into objects with the writable buffer interface. Also renamed the `_recv_string()` and `_send_string()` methods of connection objects to `recvbytes()` and `sendbytes()`. * Some optimizations for the transferring of large blocks of data using connection objects. * On Unix `os.sysconf()` is now used by default to determine whether to compile in support for posix semaphores or posix message queues. By using the `NO_SEM_TIMED` and `NO_MQ_TIMED` macros (see `INSTALL.txt`) it should now also be possible to compile in (partial) semaphore or queue support on Unix systems which lack the timeout functions `sem_timedwait()` or `mq_timedreceive()` and `mq_timesend()`. * `gettimeofday()` is now used instead of `clock_gettime()` making compilation of the C extension (hopefully) possible on Mac OSX. No modificaton of `setup.py` should be necessary. Thanks to Michele Bertoldi for report and proposed patch. * `cpuCount()` function added which returns the number of CPUs in the system. * Bugfixes to `PosixQueue` class. Changes in 0.32 --------------- * Refactored and simplified `_nonforking` module -- info about `sys.modules` of parent process is no longer passed on to child process. Also `pkgutil` is no longer used. * Allocated space from an mmap used by `LocalManager` will now be recycled. * Better tests for `LocalManager`. * Fixed bug in `managers.py` concerning refcounting of shared objects. Bug affects the case where the callable used to create a shared object does not return a unique object each time it is called. Thanks to Alexey Akimov for the report. * Added a `freezeSupport()` function. Calling this at the appropriate point in the main module is necessary when freezing a multiprocess program to produce a Windows executable. (Has been tested with `py2exe`, `PyInstaller` and `cx_Freeze`.) Changes in 0.31 --------------- * Fixed one line bug in `localmanager.py` which caused shared memory maps not to be resized properly. * Added tests for shared values/structs/arrays to `test/test_processing`. Changes in 0.30 ---------------- * Process objects now support the complete API of thread objects. In particular `isAlive()`, `isDaemon()`, `setDaemon()` have been added and `join()` now supports the `timeout` paramater. There are also new methods `stop()`, `getPid()` and `getExitCode()`. * Implemented synchronization primitives based on the Windows mutexes and semaphores and posix named semaphores. * Added support for sharing simple objects between processes by using a shared memory map and the `struct` or `array` modules. * An `activeChildren()` function has been added to `processing` which returns a list of the child processes which are still alive. * A `Pipe()` function has been added which returns a pair of connection objects representing the ends of a duplex connection over which picklable objects can be sent. * socket objects etc are now picklable and can be transferred between processes. (Requires compilation of the `_processing` extension.) * Subclasses of `managers.BaseManager` no longer automatically spawn a child process when an instance is created: the `start()` method must be called explicitly. * On Windows child processes are now spawned using `subprocess`. * On Windows the Python 2.5 version of `pkgutil` is now used for loading modules by the `_nonforking` module. On Python 2.4 this version of `pkgutil` (which uses the standard Python licence) is included in `processing.compat`. * The arguments to the functions in `processing.connection` have changed slightly. * Connection objects now have a `poll()` method which tests whether there is any data available for reading. * The `test/py2exedemo` folder shows how to get `py2exe` to create a Windows executable from a program using the `processing` package. * More tests. * Bugfixes. * Rearrangement of various stuff. Changes in 0.21 --------------- * By default a proxy is now only able to access those methods of its referent which have been explicitly exposed. * The `connection` sub-package now supports digest authentication. * Process objects are now given randomly generated 'inheritable' authentication keys. * A manager process will now only accept connections from processes using the same authentication key. * Previously `get_module()` from `_nonforking.py` was seriously messed up (though it generally worked). It is a lot saner now. * Python 2.4 or higher is now required. Changes in 0.20 --------------- * The `doc` folder contains HTML documentation. * `test` is now a subpackage. Running `processing.test.main()` will run test scripts using both processes and threads. * `nonforking.py` has been renamed `_nonforking.py`. `manager.py` has been renamed `manager.py`. `connection.py` has become a sub-package `connection` * `Listener` and `Client` have been removed from `processing`, but still exist in `processing.connection`. * The package is now *probably* compatible with versions of Python earlier than 2.4. * `set` is no longer a type supported by the default manager type. * Many more changes. Changes in 0.12 --------------- * Fixed bug where the arguments to `processing.Manager()` were passed on to `processing.manager.DefaultManager()` in the wrong order. * `processing.dummy` is now a subpackage of `processing` instead of a module. * Rearranged package so that the `test` folder, `README.txt` and `CHANGES.txt` are copied when the package is installed. Changes in 0.11 --------------- * Fixed bug on windows when the full path of `nonforking.py` contains a space. * On unix there is no longer a need to make the arguments to the constructor of `Process` be picklable or for and instance of a subclass of `Process` to be picklable when you call the start method. * On unix proxies which a child process inherits from its parent can be used by the child without any problem, so there is no longer a need to pass them as arguments to `Process`. (This will never be possible on windows.)