1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276
|
# Blockbuster
Blockbuster is a Python package designed to detect and prevent blocking calls within an asynchronous event loop.
It is particularly useful when executing tests to ensure that your asynchronous code does not inadvertently call blocking operations,
which can lead to performance bottlenecks and unpredictable behavior.
In Python, the asynchronous event loop allows for concurrent execution of tasks without the need for multiple threads or processes.
This is achieved by running tasks cooperatively, where tasks yield control back to the event loop when they are waiting for I/O operations or other long-running tasks to complete.
However, blocking calls, such as file I/O operations or certain networking operations, can halt the entire event loop, preventing other tasks from running.
This can lead to increased latency and reduced performance, defeating the purpose of using asynchronous programming.
The difficulty with blocking calls is that they are not always obvious, especially when working with third-party libraries or legacy code.
This is where Blockbuster comes in: it helps you identify and eliminate blocking calls in your codebase during testing, ensuring that your asynchronous code runs smoothly and efficiently.
It does this by wrapping common blocking functions and raising an exception when they are called within an asynchronous context.
Notes:
- Blockbuster currently only detects `asyncio` event loops.
- Blockbuster is tested only with CPython. It may work with other Python implementations if it's possible to monkey-patch the functions with `setattr`.
## Installation
The package is named `blockbuster`.
For instance with `pip`:
```bash
pip install blockbuster
```
It is recommended to constrain the version of Blockbuster.
Blockbuster doesn't strictly follow semver. Breaking changes such as new rules added may be introduced between minor versions, but not between patch versions.
So it is recommended to constrain the Blockbuster version on the minor version.
For instance, with `uv`:
```bash
uv add "blockbuster>=1.5.5,<1.6"
```
## Using BlockBuster
### Manually
To activate BlockBuster manually, create an instance of the `BlockBuster` class and call the `activate()` method:
```python
from blockbuster import BlockBuster
blockbuster = BlockBuster()
blockbuster.activate()
```
Once activated, BlockBuster will raise a `BlockingError` exception whenever a blocking call is detected within an `asyncio` event loop.
To deactivate BlockBuster, call the `deactivate()` method:
```python
blockbuster.deactivate()
```
### Using the context manager
BlockBuster can also be activated using a context manager, which automatically activates and deactivates the checks within the `with` block:
```python
from blockbuster import blockbuster_ctx
with blockbuster_ctx():
# Your test code here
```
### Usage with Pytest
Blockbuster is intended to be used with testing frameworks like `pytest` to catch blocking calls.
Here's how you can integrate Blockbuster into your `pytest` test suite:
```python
import pytest
import time
from blockbuster import BlockBuster, BlockingError, blockbuster_ctx
from typing import Iterator
@pytest.fixture(autouse=True)
def blockbuster() -> Iterator[BlockBuster]:
with blockbuster_ctx() as bb:
yield bb
async def test_time_sleep() -> None:
with pytest.raises(BlockingError, match="sleep"):
time.sleep(1) # This should raise a BlockingError
```
By using the `blockbuster_ctx` context manager, Blockbuster is automatically activated for every test, and blocking calls will raise a `BlockingError`.
## How it works
Blockbuster works by wrapping common blocking functions from various modules (e.g., `os`, `socket`, `time`) and replacing them with versions that check if they are being called from within an `asyncio` event loop.
If such a call is detected, Blockbuster raises a `BlockingError` to indicate that a blocking operation is being performed inappropriately.
Blockbuster supports by default the following functions and modules:
- **Time Functions**:
- `time.sleep`
- **OS Functions**:
- `os.getcwd`
- `os.statvfs`
- `os.sendfile`
- `os.rename`
- `os.remove`
- `os.unlink`
- `os.mkdir`
- `os.rmdir`
- `os.link`
- `os.symlink`
- `os.readlink`
- `os.listdir`
- `os.scandir`
- `os.access`
- `os.stat`
- `os.replace`
- `os.read`
- `os.write`
- **OS path Functions**:
- `os.path.ismount`
- `os.path.samestat`
- `os.path.sameopenfile`
- `os.path.islink`
- `os.path.abspath`
- **IO Functions**:
- `io.BufferedReader.read`
- `io.BufferedWriter.write`
- `io.BufferedRandom.read`
- `io.BufferedRandom.write`
- `io.TextIOWrapper.read`
- `io.TextIOWrapper.write`
- **Socket Functions**:
- `socket.socket.connect`
- `socket.socket.accept`
- `socket.socket.send`
- `socket.socket.sendall`
- `socket.socket.sendto`
- `socket.socket.recv`
- `socket.socket.recv_into`
- `socket.socket.recvfrom`
- `socket.socket.recvfrom_into`
- `socket.socket.recvmsg`
- `ssl.SSLSocket.write`
- `ssl.SSLSocket.send`
- `ssl.SSLSocket.read`
- `ssl.SSLSocket.recv`
- **SQLite Functions**:
- `sqlite3.Cursor.execute`
- `sqlite3.Cursor.executemany`
- `sqlite3.Cursor.executescript`
- `sqlite3.Cursor.fetchone`
- `sqlite3.Cursor.fetchmany`
- `sqlite3.Cursor.fetchall`
- `sqlite3.Connection.execute`
- `sqlite3.Connection.executemany`
- `sqlite3.Connection.executescript`
- `sqlite3.Connection.commit`
- `sqlite3.Connection.rollback`
- **Thread lock Functions**:
- `threading.Lock.acquire`
- `threading.Lock.acquire_lock`
- **Built-in Functions**:
- `input`
Some exceptions to the rules are already in place:
- Importing modules does blocking calls as it interacts with the file system. Since this operation is cached and very hard to avoid, it is excluded from the detection.
- Blocking calls done by the `pydevd` debugger.
- Blocking calls done by the `pytest` framework.
## Customizing Blockbuster
### Adding custom rules
Blockbuster is not a silver bullet and may not catch all blocking calls.
In particular, it will not catch blocking calls that are done by third-party libraries that do blocking calls in C extensions.
For these third-party libraries, you can declare your own custom rules to Blockbuster to catch these blocking calls.
Eg.:
```python
from blockbuster import BlockBuster, BlockBusterFunction
import mymodule
blockbuster = BlockBuster()
blockbuster.functions["my_module.my_function"] = BlockBusterFunction(my_module, "my_function")
blockbuster.activate()
```
Note: if blockbuster has already been activated, you will need to activate the custom rule yourself.
```python
from blockbuster import blockbuster_ctx, BlockBusterFunction
import mymodule
with blockbuster_ctx() as blockbuster:
blockbuster.functions["my_module.my_function"] = BlockBusterFunction(my_module, "my_function")
blockbuster.functions["my_module.my_function"].activate()
```
### Allowing blocking calls in specific contexts
You can customize Blockbuster to allow blocking calls in specific functions by using the `can_block_in` method of the `BlockBusterFunction` class.
This method allows you to specify exceptions for particular files and functions where blocking calls are allowed.
```python
from blockbuster import BlockBuster
blockbuster = BlockBuster()
blockbuster.activate()
blockbuster.functions["os.stat"].can_block_in("specific_file.py", {"allowed_function"})
```
### Deactivating specific checks
If you need to deactivate specific checks, you can directly call the `deactivate` method on the corresponding `BlockBusterFunction` instance:
```python
from blockbuster import BlockBuster
blockbuster = BlockBuster()
blockbuster.activate()
blockbuster.functions["socket.socket.connect"].deactivate()
```
## Contributing
Contributions are welcome! If you encounter any issues or have suggestions for improvements, please open an issue on the GitHub repository.
### Development Setup
Blockbuster uses `uv` to manage its development environment.
See the [uv documentation](https://docs.astral.sh/uv/getting-started/installation/) for more informationand how to install it.
To install the required dependencies, run the following command:
```bash
uv sync
```
### Running Tests
Tests are written using `pytest`.
To run the tests, use the following command:
```bash
uv run pytest
```
### Code Formatting
Code formatting is done using `ruff`. To format the code, run the following command:
```bash
uv run ruff format
```
### Code Linting
Code linting is done using `ruff`.
To lint the code, fixing any issues that can be automatically fixed, run the following command:
```bash
uv run ruff check --fix
```
## License
This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.
## Acknowledgments
Blockbuster uses the [forbiddenfruit](https://clarete.li/forbiddenfruit/) library to monkey-patch CPython immutable builtin functions and methods.
Blockbuster was greatly inspired by the [BlockHound](https://github.com/reactor/BlockHound) library for Java, which serves a similar purpose of detecting blocking calls in JVM reactive applications.
|