1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
|
# Project architecture
This document describes how the project is architectured, both regarding boilerplate and actual code. We start by giving an overview of the project's contents:
```python exec="1" session="filetree"
from fnmatch import fnmatch
from pathlib import Path
exclude = {"dist", "*cache*", ".devbox", ".hypothesis", ".pdm*", ".coverage*", "profile.*", ".gitpod*"}
no_recurse = {".venv*", "site", "htmlcov", ".git", "fixtures"}
descriptions = {
".github": "GitHub workflows, issue templates and other configuration.",
".venv": "The default virtual environment (git-ignored). See [`make setup`][command-setup] command.",
".venvs": "The virtual environments for all supported Python versions (git-ignored). See [`make setup`][command-setup] command.",
".vscode": "The configuration for VSCode (git-ignored). See [`make vscode`][command-vscode] command.",
"docs": "Documentation sources (Markdown pages). See [`make docs`][task-docs] task.",
"docs/.overrides": "Customization of [Material for MkDocs](https://squidfunk.github.io/mkdocs-material/)' templates.",
"docs/reference/api": "Python API reference, injected with [mkdocstrings](https://mkdocstrings.github.io/).",
"config": "Contains our tooling configuration. See [Scripts, configuration](#scripts-configuration).",
"htmlcov": "HTML report for Python code coverage (git-ignored), integrated in the [Coverage report](../coverage/) page. See [`make coverage`][task-coverage] task.",
"scripts": "Our different scripts. See [Scripts, configuration](#scripts-configuration).",
"site": "Documentation site, built with `make run mkdocs build` (git-ignored).",
"src": "The source of our Python package(s). See [Sources](#sources) and [Program structure](#program-structure).",
"src/_griffe": "Our internal API, hidden from users. See [Program structure](#program-structure).",
"src/griffe": "Our public API, exposed to users. See [Program structure](#program-structure).",
"tests": "Our test suite. See [Tests](#tests).",
".copier-answers.yml": "The answers file generated by [Copier](https://copier.readthedocs.io/en/stable/). See [Boilerplate](#boilerplate).",
"devdeps.txt": "Our development dependencies specification. See [`make setup`][command-setup] command.",
"duties.py": "Our project tasks, written with [duty](https://pawamoy.github.io/duty). See [Tasks][tasks].",
".envrc": "The environment configuration, automatically sourced by [direnv](https://direnv.net/). See [commands](../commands/).",
"Makefile": "A dummy makefile, only there for auto-completion. See [commands](../commands/).",
"mkdocs.yml": "The build configuration for our docs. See [`make docs`][task-docs] task.",
"pyproject.toml": "The project metadata and production dependencies.",
}
def exptree(path):
files = []
dirs = []
for node in Path(path).iterdir():
if any(fnmatch(node.name, pattern) for pattern in exclude):
continue
if node.is_dir():
dirs.append(node)
else:
files.append(node)
annotated = []
recurse = set()
print("```tree")
for directory in sorted(dirs):
if not any(fnmatch(directory.name, pattern) for pattern in no_recurse):
recurse.add(directory)
annotated.append(directory)
print(f"{directory.name}/ # ({len(annotated)})!")
elif str(directory) in descriptions:
annotated.append(directory)
print(f"{directory.name}/ # ({len(annotated)})!")
else:
print(f"{directory.name}/")
for file in sorted(files):
if str(file) in descriptions:
annotated.append(file)
print(f"{file.name} # ({len(annotated)})!")
else:
print(file.name)
print("```\n")
for index, node in enumerate(annotated, 1):
print(f"{index}. {descriptions.get(str(node), '')}\n")
if node.is_dir() and node in recurse:
print(' ```python exec="1" session="filetree" idprefix=""')
print(f' exptree("{node}")')
print(" ```\n")
```
```python exec="1" session="filetree" idprefix=""
exptree(".")
```
## Boilerplate
This project's skeleton (the file-tree shown above) is actually generated from a [Copier](https://copier.readthedocs.io/en/stable/) template called [copier-uv](https://pawamoy.github.io/copier-uv/). When generating the project, Copier asks a series of questions (configured by the template itself), and the answers are used to render the file and directory names, as well as the file contents. Copier also records answers in the `.copier-answers.yml` file, allowing to update the project with latest changes from the template while reusing previous answers.
To update the project (in order to apply latest changes from the template), we use the following command:
```bash
copier update --trust --skip-answered
```
## Scripts, configuration
We have a few scripts that let us manage the various maintenance aspects for this project. The entry-point is the `make` script located in the `scripts` folder. It doesn't need any dependency to be installed to run. See [Management commands](commands.md) for more information.
The `make` script can also invoke what we call "[tasks][]". Tasks need our development dependencies to be installed to run. These tasks are written in the `duties.py` file, and the development dependencies are listed in `devdeps.txt`.
The tools used in tasks have their configuration files stored in the `config` folder, to unclutter the root of the repository. The tasks take care of calling the tools with the right options to locate their respective configuration files.
## Sources
Sources are located in the `src` folder, following the [src-layout](https://packaging.python.org/en/latest/discussions/src-layout-vs-flat-layout/). We use [PDM-Backend](https://backend.pdm-project.org/) to build source and wheel distributions, and configure it in `pyproject.toml` to search for packages in the `src` folder.
## Tests
Our test suite is located in the `tests` folder. It is located outside of the sources as to not pollute distributions (it would be very wrong to publish a `tests` package as part of our distributions, since this name is extremely common), or worse, the public API. The `tests` folder is however included in our source distributions (`.tar.gz`), alongside most of our metadata and configuration files. Check out `pyproject.toml` to get the full list of files included in our source distributions.
The test suite is based on [pytest](https://docs.pytest.org/en/8.2.x/). Test modules reflect our internal API structure, and except for a few test modules that test specific aspects of our API, each test module tests the logic from the corresponding module in the internal API. For example, `test_finder.py` tests code of the `_griffe.finder` internal module, while `test_functions` tests our ability to extract correct information from function signatures, statically. The general rule of thumb when writing new tests is to mirror the internal API. If a test touches to many aspects of the loading process, it can be added to the `test_loader` test module.
## Program structure
Griffe is composed of two packages:
- `_griffe`, which is our internal API, hidden from users
- `griffe`, which is our public API, exposed to users
When installing the `griffe` distribution from PyPI.org (or any other index where it is published), both the `_griffe` and `griffe` packages are installed. Users then import `griffe` directly, or import objects from it. The top-level `griffe/__init__.py` module exposes all the public API, by importing the internal objects from various submodules of `_griffe`.
We'll be honest: our code organization is not the most elegant, but it works :shrug: Have a look at the following module dependency graph, which will basically tell you nothing except that we have a lot of inter-module dependencies. Arrows read as "imports from". The code base is generally pleasant to work with though.
```python exec="true" html="true"
from pydeps import cli, colors, dot, py2depgraph
from pydeps.pydeps import depgraph_to_dotsrc
from pydeps.target import Target
cli.verbose = cli._not_verbose
options = cli.parse_args(["src/griffe", "--noshow", "--reverse"])
colors.START_COLOR = 128
target = Target(options["fname"])
with target.chdir_work():
dep_graph = py2depgraph.py2dep(target, **options)
dot_src = depgraph_to_dotsrc(target, dep_graph, **options)
svg = dot.call_graphviz_dot(dot_src, "svg").decode()
svg = "".join(svg.splitlines()[6:])
svg = svg.replace('fill="white"', 'fill="transparent"')
print(f'<div class="interactiveSVG pydeps">{svg}</div>')
```
<small><i>You can zoom and pan all diagrams on this page with mouse inputs.</i></small>
The following sections are generated automatically by iterating on the modules of our public and internal APIs respectively, and extracting the comment blocks at the top of each module. The comment blocks are addressed to readers of the code (maintainers, contributors), while module docstrings are addressed to users of the API. Module docstrings in our internal API are never written, because our [module layout][module-layout] is hidden, and therefore modules aren't part of the public API, so it doesn't make much sense to write "user documentation" in them.
```python exec="1" session="comment_blocks"
--8<-- "scripts/gen_structure_docs.py"
```
### CLI entrypoint
```python exec="1" idprefix="entrypoint-" session="comment_blocks"
render_entrypoint(heading_level=4)
```
### Public API
```python exec="1" idprefix="public-" session="comment_blocks"
render_public_api(heading_level=4)
```
### Internal API
```python exec="1" idprefix="internal-" session="comment_blocks"
render_internal_api(heading_level=4)
```
<style>
.interactiveSVG svg {
min-height: 200px;
}
.graph > polygon {
fill-opacity: 0.0;
}
/* pydeps dependency graph. */
[data-md-color-scheme="default"] .pydeps .edge > path,
[data-md-color-scheme="default"] .pydeps .edge > polygon {
stroke: black;
}
[data-md-color-scheme="slate"] .pydeps .edge > path,
[data-md-color-scheme="slate"] .pydeps .edge > polygon {
stroke: white;
}
/* Code2Flow call graphs. */
[data-md-color-scheme="default"] .code2flow .cluster > polygon {
stroke: black;
}
[data-md-color-scheme="default"] .code2flow .cluster > text {
fill: black;
}
[data-md-color-scheme="slate"] .code2flow .cluster > polygon {
stroke: white;
}
[data-md-color-scheme="slate"] .code2flow .cluster > text {
fill: white;
}
</style>
<script>
document.addEventListener("DOMContentLoaded", function(){
const divs = document.getElementsByClassName("interactiveSVG");
for (let i = 0; i < divs.length; i++) {
if (!divs[i].firstElementChild.id) {
divs[i].firstElementChild.id = `interactiveSVG-${i}`
}
svgPanZoom(`#${divs[i].firstElementChild.id}`, {});
}
});
</script>
|
