1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
|
# FlatBuffers and WAMP
## WAMP payload transparency mode
### Motivation
**tldr:**
* performance (passthrough of app payloads, general zero copy)
* support end-to-end encryption
* carrying proprietory binary payloads (MQTT)
* strict static typing of interfaces
---
WAMP supports both positional and keyword based request arguments and returns.
For example, the WAMP `CALL` message allows for the following three alternative forms:
1. `[CALL, Request|id, Options|dict, Procedure|uri]`
2. `[CALL, Request|id, Options|dict, Procedure|uri, Arguments|list]`
3. `[CALL, Request|id, Options|dict, Procedure|uri, Arguments|list, ArgumentsKw|dict]`
The actual application payload hence can take **three variants of app payload (XX)**:
1. `-`
2. `Arguments|list`
3. `Arguments|list, ArgumentsKw|dict`
This pattern repeats across **all** WAMP messages that can carry application payload, namely the following 7 WAMP messages:
* `PUBLISH`
* `EVENT`
* `CALL`
* `INVOCATION`
* `YIELD`
* `RESULT`
* `ERROR`
> Note: should the proposed new WAMP messages `EVENT_RECEIVED` and `SUBSCRIBER_RECEIVED` be introduced,
these also carry application payload, and would follow the same approach.
The approach taken (**XX**) allows for a number of useful features:
1. flexible support of popular dynamically typed serializers, namely: JSON, MsgPack, CBOR and UBJSON
2. allow arbitrary adhoc extensibility (as the router basically does not care about new app payloads)
3. transparantly translate the *application payload* between serialization formats used by different clients connected at the same time.
4. support optional router side application payload validation: both static, and dynamic (calling into user supplied payload validators)
However, a number of downsides have become apparent as well:
1. resource consumption: serialization/deserialization can eat significant chunks of CPU, and produce GC pressure
2. legacy (MQTT) and proprietory payloads that should simply be transported "as is" (passthrough, without ever touching)
3. as apps and systems get larger and more complex, the dynamic typing flexibility turns into a major problem: **the internal and external interfaces and APIs in a microservices based application must be relied upon and their evolution actively managed**
The latter does not mean an "either or" question. You can have important base APIs and external interfaces defined rigorously, using static, strict typing discipline, while at the same time have other parts of your system evolve more freely, basically allowing weakly and dynamically typed data exchange - for limited areas.
---
### Payload Transparency Mode
**Payload Transparancy Mode (PTM)** adds a 4th application payload variant to above **XX**
4. `[CALL, Request|id, Options|dict, Procedure|uri, Payload|binary]`
where the actual application payload takes this form:
4. `Payload|binary`
**PTM** can be used on a *per message basis*.
If PTM is used, then the following two attributes MUST be present in the `Details|dict` or `Options|dict` of the respective WAMP message:
* `payload|string`: Application payload type:
* `"plain"`: Plain WAMP application payload.
* `"cryptobox"`: Encrypted WAMP application payload. This is using WAMP-cryptobox (Curve25519 / Cryptobox).
* `"opaque"`: Raw pass-through of app payload, uninterpreted in any way.
* `serializer|string`: Application payload serializer type.
* `"transport"`: Use same (dynamic) serializer for the app payload as on the transport. This will be one of JSON, MSGPACK, CBOR or UBJSON.
* `"json"`: Use JSON serializer (for dynamically typed app payload).
* `"msgpack"`: Use MsgPack serializer (for dynamically typed app payload).
* `"cbor"`: Use CBOR serializer (for dynamically typed app payload).
* `"ubjson"`: Use UBJSON serializer (for dynamically typed app payload).
* `"opaque"`: Raw pass-through of app payload, uninterpreted in any way.
* `"flatbuffers"`: Explicit use of FlatBuffers also for (statically typed) payload.
* `key|string`: When using end-to-end encryption (WAMP-cryptobox), the public key to which the payload is encrypted
---
#### Payload PLAIN
When PTM is in use, and `Details.payload=="plain"`, the original application payload, one of
* `-`
* `Arguments|list`
* `Arguments|list, ArgumentsKw|dict`
is serialized according to the (dynamically typed) serializer specified in `serializer`, one of
* `"transport"`
* `"msgpack"`
* `"json"`
* `"cbor"`
* `"ubjson"`
> Note that serializers `"opaque"` and `"flatbuffers"` are illegal for payload `"plain"`.
---
#### Payload CRYPTOBOX
Write me.
---
#### Payload OPAQUE
Write me.
---
## FlatBuffers
FlatBuffers is a zero-copy serialization format open-sourced by Google in 2014 under the Apache 2 license.
Supported operating systems include:
* Android
* Linux
* MacOS X
* Windows
Supported programming languages include:
* C++
* C#
* C
* Go
* Java
* JavaScript
* PHP
* Python
---
## Building
1. Get [cmake](https://cmake.org/)
2. Follow [Building with CMake](https://github.com/google/flatbuffers/blob/master/docs/source/Building.md#building-with-cmake)
Essentially:
```console
git clone https://github.com/google/flatbuffers.git
cd flatbuffers
cmake -G "Unix Makefiles"
make -j4
./flatc --version
```
---
### flatc patches
The following patches are required currently for flatc: [this PR](https://github.com/google/flatbuffers/pull/4713) adds pieces missing for reflection in the flatc compiler.
---
## Usage
This folder contains a [Makefile](Makefile) that allows to compile all WAMP FlatBuffers schemata and generate binding code in various languages using
```console
make clean build cloc
```
Here is example output (a total of 30k LOC generated from 500 LOC .fbs):
```console
cpy2714_1) oberstet@thinkpad-t430s:~/scm/wamp-proto/wamp-proto/flatbuffers$ make clean build cloc
rm -rf ./_build
/home/oberstet/scm/xbr/flatbuffers/flatc -o ./_build/schema/ --binary --schema --bfbs-comments --bfbs-builtin-attrs *.fbs
/home/oberstet/scm/xbr/flatbuffers/flatc -o ./_build/cpp/ --cpp *.fbs
/home/oberstet/scm/xbr/flatbuffers/flatc -o ./_build/csharp/ --csharp *.fbs
/home/oberstet/scm/xbr/flatbuffers/flatc -o ./_build/go/ --go *.fbs
/home/oberstet/scm/xbr/flatbuffers/flatc -o ./_build/java/ --java *.fbs
/home/oberstet/scm/xbr/flatbuffers/flatc -o ./_build/js/ --js *.fbs
/home/oberstet/scm/xbr/flatbuffers/flatc -o ./_build/php/ --php *.fbs
/home/oberstet/scm/xbr/flatbuffers/flatc -o ./_build/python/ --python *.fbs
/home/oberstet/scm/xbr/flatbuffers/flatc -o ./_build/ts/ --ts *.fbs
cloc --read-lang-def=cloc.def --exclude-dir=_build .
11 text files.
11 unique files.
2 files ignored.
github.com/AlDanial/cloc v 1.76 T=0.01 s (774.6 files/s, 113640.1 lines/s)
---------------------------------------------------------------------------------------
Language files blank comment code
---------------------------------------------------------------------------------------
FlatBuffers 7 314 372 535
Markdown 1 64 0 119
make 1 20 8 30
Windows Module Definition 1 0 0 5
---------------------------------------------------------------------------------------
SUM: 10 398 380 689
---------------------------------------------------------------------------------------
cloc ./_build
331 text files.
331 unique files.
2 files ignored.
github.com/AlDanial/cloc v 1.76 T=0.35 s (942.1 files/s, 132452.9 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
C/C++ Header 7 405 7 5629
PHP 62 961 3206 5006
Go 62 826 62 4815
TypeScript 7 832 3430 3428
JavaScript 7 896 3493 3177
Java 62 439 62 2825
C# 62 477 186 2800
Python 62 616 454 2504
-------------------------------------------------------------------------------
SUM: 331 5452 10900 30184
-------------------------------------------------------------------------------
```
---
## Notes
* FlatBuffers [currently lacks](https://github.com/google/flatbuffers/issues/4237) syntax highlighting on GitHub.
---
## References
* [FlatBuffers Homepage](https://google.github.io/flatbuffers/)
* [FlatBuffers Source](https://github.com/google/flatbuffers)
* [flatcc - a FlatBuffers Compiler and Library in C for C](https://github.com/dvidelabs/flatcc)
---
|
