1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
|
# Generated C++ code for CS API
The code in `Quotient/csapi`, and `Quotient/application-service`, although stored in Git, is
actually generated from the official Matrix Client-Server API definition files. If you're unhappy
with something in there and want to improve that, you have to understand the way these files are
produced and setup some additional tooling. The shortest possible implementation of the procedure
described below can be found in .github/workflows/ci.yml (regeneration of those files is tested
in CI) - see the tasks "Get CS API definitions; clone and build GTAD" and "Regenerate API code".
## Why generate the code at all?
Because otherwise we have to do monkey business of writing boilerplate code,
with the same patterns, types etc., literally, for every single API endpoint,
and one of libQuotient authors got fed up with it at some point in time.
By then about 15 job classes have been written; the whole API is about 100
endpoints, with new ones added and some existing changed in each Matrix protocol
version. Other considerations can be found in
[this talk about API description languages](https://youtu.be/W5TmRozH-rg)
that also briefly touches on GTAD - the tool written for the purpose.
## Prerequisites for CS API code generation
1. Get the source code of GTAD and its dependencies. Since version 0.7,
libQuotient includes GTAD as a submodule so you can get everything you need
by updating gtad/gtad submodule in libQuotient sources:
`git submodule update --init --recursive gtad/gtad`.
You can also just clone GTAD sources to keep them separate from libQuotient:
`git clone --recursive https://github.com/quotient-im/gtad.git`
2. Configure and build GTAD: same as libQuotient, it uses CMake so this should be very
straightforward (if not - you're probably not quite ready for this stuff anyway). It's only
tested on Linux but doesn't use anything Linux-specific; Windows, in particular, should be
just fine.
3. Get Matrix CS API definitions from a matrix-spec repo. Although the official repo is at
https://github.com/matrix-org/matrix-spec.git`, it is not recommended to use its main branch
directly because the official Matrix repo doesn't check whether a particular change breaks code
generation or changes the generated API (it's a completely separate project after all). For that
reason, a soft fork of the official definitions is kept at
https://github.com/quotient-im/matrix-spec.git - that guarantees buildability of the generated
code and also has certain tweaks to make the generated API better. This repo closely follows
the official one (but maybe not its freshest commit). And of course you can fork it and use your
own repository if you need to change the API definition.
4. If you plan to submit a PR with the generated code to libQuotient or just would like it to be
properly formatted, you should either ensure you have clang-format (version 16 at least) in your
PATH or pass `-DCLANG_FORMAT=<path>` to CMake, as mentioned in the next section.
## Generating CS API contents
1. Pass additional configuration to CMake when configuring libQuotient:
`-DMATRIX_SPEC_PATH=/path/to/matrix-spec/ -DGTAD_PATH=/path/to/gtad`.
Note that `MATRIX_SPEC_PATH` should lead to the _repo_ (not the API folder)
while `GTAD_PATH` should have the path to GTAD _binary_. If you need
to specify where your clang-format is (see the previous section) add
`-DCLANG_FORMAT=/path/to/clang-format` to the line above. If everything's
right, the detected locations will be mentioned in CMake output and
an additional build target called `update-api` will be configured.
2. Generate the code: `cmake --build <your build dir> --target update-api`. Building this target
will create (overwriting without warning) source files in `Quotient/csapi`,
`Quotient/application-service` for all YAML files it can find in
`/path/to/matrix-spec/data/api/client-server` and their dependencies.
## Changing generated code
See the more detailed description of what GTAD is and how it works in
the documentation on GTAD in its source repo. Only parts specific for
libQuotient are described here.
GTAD uses the following three kinds of sources:
1. OpenAPI files. Each file is treated as a separate source (unlike
swagger-codegen, you do _not_ need to have a single file for the whole API).
2. A configuration file, in Quotient case it's `gtad/gtad.yaml` - common for
all OpenAPI files GTAD is invoked on.
3. Source code template files: `gtad/*.mustache` - are also common.
The Mustache files have a templated (not in C++ sense) definition of a network
job class derived from BaseJob; if necessary, data structure definitions used
by this job are put before the job class. Bigger Mustache files look a bit
daunting for a newcomer; and the only known highlighter that can handle
the combination of Mustache (originally a web templating language) and C++ can
be found in CLion IDE. Fortunately, all our Mustache files are reasonably
concise and well-formatted these days.
To simplify things some reusable Mustache blocks are defined in `gtad.yaml` -
see its `mustache:` section. Adventurous souls that would like to figure out
what's going on in these files should speak up in the Quotient room -
I (Kitsune) will be very glad to navigate you.
The `types` map in `gtad.yaml` defines a mapping from OpenAPI types to C++/Qt.
It uses the following type attributes aside from pretty obvious `imports:`:
* `avoidCopy` - this attribute defines whether a const ref should be used
instead of a value. For basic types like int this is obviously unnecessary;
but compound types like `QVector` should rather be taken by reference when
possible.
* `moveOnly` - some types are not copyable at all and must be moved instead
(an obvious example is any structure that uses, directly or indirectly,
`std::unique_ptr<>`).
* `useOptional` - wrap types that have no value with "null" semantics (i.e. number types and
custom-defined data structures) into `std::optional`.
* `omittedValue` - an alternative for `useOptional`, just provide a value used
for an omitted parameter. This is used for bool parameters which normally are
considered false if omitted (or they have an explicit default value, passed
in the "official" GTAD's `defaultValue` variable).
* `initializer` - this is a _partial_ (see GTAD and Mustache documentation for
explanations but basically it's a variable that is a Mustache template itself)
that specifies how exactly a default value should be passed to the parameter.
E.g., the default value for a `QString` parameter is transformed as `u"{{defaultValue}}"_s`.
Instead of relying on the event structure definition in the OpenAPI files,
`gtad.yaml` uses pointers to libQuotient's event structures: `EventPtr`,
`RoomEventPtr` and `StateEventPtr`. Respectively, arrays of events, when
encountered in OpenAPI definitions, are converted to `Events`, `RoomEvents`
and `StateEvents` containers. When there's no way to figure the type from
the definition, an opaque `QJsonObject` is used, leaving the conversion
to the library and/or client code.
## Submitting API changes
Getting the API changes upstream requires coordination across a few Matrix
projects (the API is a contract between the client and the server, after all).
The recommended sequence is different, mainly depending on on whether or not
you have to write a
[Matrix Spec Change aka MSC](https://matrix.org/docs/spec/proposals). Simply
speaking, if your changes don't break compatibility with existing Matrix
ecosystem (e.g. it's a documentation fix, or a fix in the API definition
to align with the real use), or if you implement an existing approved MSC,
you don't need to submit an MSC and it's a matter of two PRs: one to
the official repo with the spec text and API definitions, that resides at
`https://github.com/matrix-org/matrix-spec` and one to libQuotient.
If your changes require an MSC (e.g. you add a new API call or change
an existing one beyond minor adjustments):
1. Submit an MSC before submitting changes to the API definition files and
libQuotient. See the link about MSCs above on what it should and should not
have and how it should be submitted.
2. The MSC gets reviewed by the Spec Core Team. This can be a lengthy process
but it's necessary for the Matrix ecosystem integrity.
3. When your MSC has at least some approvals (not necessarily a complete
acceptance but at least some approvals should be there) the MSC process
strongly recommends to show an implementation in existing projects. In
the case of Client-Server API that usually means a homeserver and a client
application. Submit PRs to the projects you took for that including
libQuotient, referring to your MSC; for API definition files, use
`https://github.com/quotient-im/matrix-spec` (the fork) instead of
[the official repo](`https://github.com/matrix-org/matrix-spec`), as
the official repo only accepts PRs on approved MSCs. You will have to show
that your implementation is actually working to get your MSC approved.
4. Once the MSC is accepted, you can officially submit your changes in API
definitions as a "spec PR" to the official repo.
In any case, when submitting a PR for libQuotient, please make sure generated files are committed
separately from non-generated ones (no need to make two PRs; just separate the files in different
commits). This helps in situations when the same `gtad.yaml` or Mustache templates are
cherry-picked between branches or back/forward-ported.
|
