1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
|
The ``process_utils`` Module
=============================
This module provides functions for doing process management.
These are the main sections of this module:
- `Asynchronous Process Utilities`_
- `Synchronous Process Utilities`_
- `Utility Functions`_
Asynchronous Process Utilities
------------------------------
There is a function and class which can be used together with your custom `asyncio <https://docs.python.org/3/library/asyncio.html>`_ run loop.
The :py:func:`osrf_pycommon.process_utils.async_execute_process` function is a `coroutine <https://docs.python.org/3/library/asyncio-task.html#coroutines>`_ which allows you to run a process and get the output back bit by bit in real-time, either with stdout and stderr separated or combined.
This function also allows you to emulate the terminal using a pty simply by toggling a flag in the parameters.
Along side this coroutine is a `Protocol <https://docs.python.org/3/library/asyncio-protocol.html#protocols>`_ class, :py:class:`osrf_pycommon.process_utils.AsyncSubprocessProtocol`, from which you can inherit in order to customize how the yielded output is handled.
Because this coroutine is built on the ``asyncio`` framework's subprocess functions, it is portable and should behave the same on all major OS's. (including on Windows where an IOCP implementation is used)
.. autofunction:: osrf_pycommon.process_utils.async_execute_process
.. autoclass:: osrf_pycommon.process_utils.AsyncSubprocessProtocol
:members:
In addtion to these functions, there is a utility function for getting the correct ``asyncio`` event loop:
.. autofunction:: osrf_pycommon.process_utils.get_loop
Treatment of File Descriptors
-----------------------------
Like Python 3.4's ``subprocess.Popen`` (and newer versions), all of the ``process_utils`` functions do not close `inheritable <https://docs.python.org/3.4/library/os.html#fd-inheritance>` file descriptors before starting subprocesses.
This is equivalent to passing ``close_fds=False`` to ``subprocess.Popen`` on all Python versions.
For historical context, in Python 3.2, the ``subprocess.Popen`` default for the ``close_fds`` option changed from ``False`` to ``True`` so that file descriptors opened by the parent process were closed before spawning the child process.
In Python 3.4, `PEP 0446 <https://www.python.org/dev/peps/pep-0446/>`_ additionally made it so even when ``close_fds=False`` file descriptors which are `non-inheritable <https://docs.python.org/3.4/library/os.html#fd-inheritance>`_ are still closed before spawning the subprocess.
If you want to be able to pass file descriptors to subprocesses in Python 3.4 or higher, you will need to make sure they are `inheritable <https://docs.python.org/3.4/library/os.html#fd-inheritance>`.
Synchronous Process Utilities
-----------------------------
For synchronous execution and output capture of subprocess, there are two functions:
- :py:func:`osrf_pycommon.process_utils.execute_process`
- :py:func:`osrf_pycommon.process_utils.execute_process_split`
These functions are not yet using the ``asyncio`` framework as a back-end and therefore on Windows will not stream the data from the subprocess as it does on Unix machines.
Instead data will not be yielded until the subprocess is finished and all output is buffered (the normal warnings about long running programs with lots of output apply).
The streaming of output does not work on Windows because on Windows the :py:func:`select.select` method only works on sockets and not file-like objects which are used with subprocess pipes.
``asyncio`` implements Windows subprocess support by implementing a Proactor event loop based on Window's IOCP API.
One future option will be to implement this synchronous style method using IOCP in this module, but another option is to just make synchronous the asynchronous calls, but there are issues with that as well.
In the mean time, if you need streaming of output in both Windows and Unix, use the asynchronous calls.
.. autofunction:: osrf_pycommon.process_utils.execute_process
Availability: Unix (streaming), Windows (blocking)
.. autofunction:: osrf_pycommon.process_utils.execute_process_split
Availability: Unix (streaming), Windows (blocking)
Utility Functions
-----------------
Currently there is only one utility function, a Python implementation of the ``which`` shell command.
.. autofunction:: osrf_pycommon.process_utils.which
|
