1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
|
Using subprocesses
==================
.. py:currentmodule:: anyio
AnyIO allows you to run arbitrary executables in subprocesses, either as a one-shot call
or by opening a process handle for you that gives you more control over the subprocess.
You can either give the command as a string, in which case it is passed to your default
shell (equivalent to ``shell=True`` in :func:`subprocess.run`), or as a sequence of
strings (``shell=False``) in which case the executable is the first item in the sequence
and the rest are arguments passed to it.
Running one-shot commands
-------------------------
To run an external command with one call, use :func:`~run_process`::
from anyio import run_process, run
async def main():
result = await run_process('ps')
print(result.stdout.decode())
run(main)
The snippet above runs the ``ps`` command within a shell. To run it directly::
from anyio import run_process, run
async def main():
result = await run_process(['ps'])
print(result.stdout.decode())
run(main)
Working with processes
----------------------
When you have more complex requirements for your interaction with subprocesses, you can
launch one with :func:`~open_process`::
from anyio import open_process, run
from anyio.streams.text import TextReceiveStream
async def main():
async with await open_process(['ps']) as process:
async for text in TextReceiveStream(process.stdout):
print(text)
run(main)
See the API documentation of :class:`~.abc.Process` for more information.
.. _RunInProcess:
Running functions in worker processes
-------------------------------------
When you need to run CPU intensive code, worker processes are better than threads
because, with the exception of the experimental free-threaded builds of Python 3.13 and
later, current implementations of Python cannot run Python code in multiple threads at
once.
Exceptions to this rule are:
#. Blocking I/O operations
#. C extension code that explicitly releases the Global Interpreter Lock
#. :doc:`Subinterpreter workers <subinterpreters>`
(experimental; available on Python 3.13 and later)
If the code you wish to run does not belong in this category, it's best to use worker
processes instead in order to take advantage of multiple CPU cores.
This is done by using :func:`.to_process.run_sync`::
import time
from anyio import run, to_process
def cpu_intensive_function(arg1, arg2):
time.sleep(1)
return arg1 + arg2
async def main():
result = await to_process.run_sync(cpu_intensive_function, 'Hello, ', 'world!')
print(result)
# This check is important when the application uses to_process.run_sync()
if __name__ == '__main__':
run(main)
Technical details
*****************
There are some limitations regarding the arguments and return values passed:
* the arguments must be pickleable (using the highest available protocol)
* the return value must be pickleable (using the highest available protocol)
* the target callable must be importable (lambdas and inner functions won't work)
Other considerations:
* Even ``cancellable=False`` runs can be cancelled before the request has been sent to
the worker process
* If a cancellable call is cancelled during execution on the worker process, the worker
process will be killed
* The worker process imports the parent's ``__main__`` module, so guarding for any
import time side effects using ``if __name__ == '__main__':`` is required to avoid
infinite recursion
* ``sys.stdin`` and ``sys.stdout``, ``sys.stderr`` are redirected to ``/dev/null`` so
:func:`print` and :func:`input` won't work
* Worker processes terminate after 5 minutes of inactivity, or when the event loop is
finished
* On asyncio, either :func:`asyncio.run` or :func:`anyio.run` must be used for proper
cleanup to happen
* Multiprocessing-style synchronization primitives are currently not available
|
