1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
|
Concurrency
===========
Daemonic processes
------------------
A child process can be *daemonic* or *non-daemonic*. When a process exits, it attempts to terminate
all of its daemonic child processes (SIGTERM) while it will wait for non-daemonic child processes to finish.
An interrupt (SIGINT) is propagated to both daemonic and non-daemonic child processes.
A daemonic process is not allowed to create child processes. Otherwise a daemonic process would leave its children
orphaned if it gets terminated when its parent process exits.
Create daemonic child processes
* `multiprocessing.Pool` (cannot be used in a daemonic process)
* `billiard.Pool` (can be used in a daemonic process)
Create non-daemonic child processes
* `concurrent.futures.ProcessPoolExecutor`
Create non-daemonic child processes by default (can be daemonic if requested)
* `multiprocessing.Process`
* `billiard.Process`
This can be verified with
.. code:: python
python -m pypushflow.tests.concurrent.check_daemonic
Pool types
----------
Pypushflow supports these pools for concurrent execution of workflow tasks
* gevent: pool of greenlets from `gevent`
* thread: pool of threads from the `concurrent.futures` module
* process: pool of non-daemonic processes from the `concurrent.futures` module
* ndprocess: pool of non-daemonic processes from the `concurrent.futures` module
* multiprocessing: pool of daemonic processes from the `multiprocessing` module
* ndmultiprocessing: pool of non-daemonic processes from the `multiprocessing` module
* billiard: pool of daemonic processes from the `billiard` library
The pool type `process` and `ndprocess` are the same, but `ndprocess` uses explicit
non-daemonic processes like `ndmultiprocessing`.
In a gevent-patched environment the default pool type is *gevent* else it is *process*.
Workflow tasks
--------------
* simple: no subprocess
* subprocess: the `subprocess` module
* cfpool: process pool from the `concurrent.futures` module
* mppool: process pool from the `multiprocessing` module
* mpprocess: process from the `multiprocessing` module
* bpool: process pool from the `billiard` library
* bprocess: process from the `billiard` library
Pool-task compatibility
-----------------------
Workflow tasks that can be used in each pool type (no gevent monkey patching)
* gevent: -
* thread: simple, subprocess, cfpool, mppool, mpprocess, bpool, bprocess
* process: simple, subprocess, cfpool, mppool, mpprocess, bpool (hangs sometimes), bprocess
* ndprocess: simple, subprocess, cfpool, mppool, mpprocess, bpool (hangs sometimes), bprocess
* multiprocessing: simple, subprocess, bpool (hangs sometimes), bprocess
* ndmultiprocessing: simple, subprocess, cfpool, mppool, mpprocess, bpool (hangs sometimes), bprocess
* billiard: simple, subprocess, bpool (hangs sometimes), bprocess
Workflow tasks that can be used in each pool type (with gevent monkey patching)
* gevent: simple, subprocess, cfpool, mpprocess, bprocess
* thread: simple, subprocess, cfpool, mpprocess, bprocess
* process: simple, subprocess, cfpool (not spawn), mpprocess (not spawn), bprocess (not spawn)
* ndprocess: simple, subprocess, cfpool (not spawn), mpprocess (not spawn), bprocess (not spawn)
* multiprocessing: -
* ndmultiprocessing: -
* billiard: -
|
