1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
|
# Tarantool Wire protocol
* **Status**: In progress
* **Start date**: 04-04-2018
* **Authors**: Vladislav Shpilevoy @Gerold103 v.shpilevoy@tarantool.org, Konstantin Osipov @kostja kostja@tarantool.org, Alexey Gadzhiev @alg1973 alg1973@gmail.com
* **Issues**: [#2677](https://github.com/tarantool/tarantool/issues/2677), [#2620](https://github.com/tarantool/tarantool/issues/2620), [#2618](https://github.com/tarantool/tarantool/issues/2618)
## Summary
Tarantool wire protocol is a convention for encoding and sending execution results of SQL, Lua and C stored functions, DML (Data Manipulation Language), DDL (Data Definition Language), DQL (Data Query Language). The protocol is unified for all request types. For a single request multiple responses of different types can be sent.
## Background and motivation
Tarantool wire protocol is called **IProto**, and is used by database connectors written in different languages and accessing the database via network. The protocol describes how to distinguish different message types and what data can be stored in each message. Tarantool response messages can be of the following kind:
* A response, which represents a single reply to a request or completes a chain of replies. There are two response types of this kind: OK and ERROR. Error response is trivial, and contains an error code and message. OK response may contain useful payload, such as result set rows or metadata.
* A response which is a part of a chain of replies - a so called PUSH-message. Multiple push messages can be sent in response to a single request, but they never indicate an end of reply stream: the end is always flagged by a response of the previous kind.
In support of this response set 2 main challenges appear:
1. How to unify responses;
2. How to support multiple messages inside a single request.
A response which contains a payload can contain either data or metadata, or both. If it is necessary to share the same response metadata among multiple PUSH messages, the metadata can be assigned a numeric identifier (PUSH ID) and referenced in the stream by this identifier.
The metadata itself can contain:
* affected row count, last autoincrement column value, flags (such metadata
is sent in response to DML statements such as INSERT/UPDATE/DELETE);
* column count, names and types (used to describe result set rows that
follow).
To understand how a single request can produce multiple responses, consider the stored procedure (do not pay attention to the syntax - it does not matter here):
```SQL
FUNCTION my_sql_func(a1, a2, a3, a4) BEGIN
SELECT my_lua_func(a1);
SELECT * FROM table1;
SELECT my_c_func(a2);
INSERT INTO table1 VALUES (1, 2, 3);
RETURN a4;
END;
```
, where `my_lua_func()` is the function, written in Lua and sending its own push-messages:
```Lua
function my_lua_func(arg)
box.session.push(arg)
return arg
end
```
and `my_c_func()` is the function, written in C and returning some raw data:
```C
int
my_c_func(box_function_ctx_t *ctx) {
box_tuple_t *tuple;
/* Fill the tuple with any data. */
return box_return_tuple(ctx, tuple);
}
```
Consider each statement:
* `SELECT FROM` can split a big result set into multiple messages;
* `SELECT my_lua_func()` produces 2 messages: one is the push-message generated in `my_lua_func` and another is the result of `SELECT` itself;
* `INSERT` creates 1 message with metadata;
* `RETURN` creates a final response message.
Of course, some of messages, or even all of them can be batched and sent as a single TCP packet.
In the next section we specify code names and messages used by the protocol.
For the protocol details - code values, all header and body keys - see Tarantool [website](https://tarantool.io/).
## Detailed design
Tarantool response consists of a body and a header. Header is used to store response code and some internal metainfo such as schema version, request id (called **sync** in Tarantool). Body is used to store result data and request-dependent metainfo.
### Header
There are 3 response codes in header:
* `IPROTO_OK` - the terminal response to a successful request;
* `IPROTO_ERROR | error code` - the terminal response to a request, which ended with an error.
* `IPROTO_PUSH` - non-final response. One request can generate multuple PUSH messages;
`IPROTO_ERROR` response is trivial, and consists just of code and message.
`IPROTO_OK` and `IPROTO_PUSH` have the same body format. But
1. `IPROTO_OK` finalizes a request;
2. `IPROTO_PUSH` can have `IPROTO_PUSH_ID` field in the header, that allows to build a chain of pushes with the same `ID`. Absense of this field means, that the push is not a part of a chain. All pushes which are part of the same chain (are identified by the same id) should share chain metadata.
### Body
The common body structure:
```
+----------------------------------------------+
| IPROTO_BODY: { |
| IPROTO_METADATA: [ |
| { |
| IPROTO_FIELD_NAME: string, |
| IPROTO_FIELD_TYPE: number, |
| IPROTO_FIELD_FLAGS: number, |
| }, |
| ... |
| ], |
| |
| IPROTO_SQL_INFO: { |
| SQL_INFO_ROW_COUNT: number, |
| SQL_INFO_LAST_ID: number, |
| ... |
| }, |
| |
| IPROTO_DATA: [ |
| tuple/scalar, |
| ... |
| ] |
| } |
+----------------------------------------------+
```
Consider, how different responses use the body, and how they can be distinguished.
_A non formatted response_ has only `IPROTO_DATA` key in a body. It is the result of Lua and C DML, DDL, DQL, stored procedures calls, push messages. Such response is never linked with the next or previous messages of the same request.
_A non formatted response with metadata_ has only `IPROTO_SQL_INFO` and it is always a result of DDL/DML executed via SQL. As well as the previous type, this response is independent from other messages in the stream.
_A formatted response_ always has `IPROTO_DATA`, and can have both `IPROTO_SQL_INFO` and `IPROTO_METADATA`. It is a result of SQL DQL (`SELECT`) or SQL DML (`INSERT`). The response can be part of a stream. The first message of the stream always contains `IPROTO_METADATA` in the body and sets `IPROTO_PUSH_ID` in the header, should there be multiple messages sharing the same metadata. All other messasges in the stream contain `IPROTO_PUSH_ID` with the same value.
On the picture the state machine of the protocol is showed:
For a call to `FUNCTION my_sql_func` the following responses are sent:
```
/* Push from my_lua_func(a1). */
+----------------------------------------------+
| HEADER: IPROTO_PUSH |
+- - - - - - - - - - - - - - - - - - - - - - - +
| BODY: { |
| IPROTO_DATA: [ a1 ] |
| } |
+----------------------------------------------+
/* Result of SELECT my_lua_func(a1). */
+----------------------------------------------+
| HEADER: IPROTO_PUSH |
+- - - - - - - - - - - - - - - - - - - - - - - +
| BODY: { |
| IPROTO_DATA: [ [ a1 ] ], |
| IPROTO_METADATA: [ |
| { /* field name, type ... */ } |
| ] |
| } |
+----------------------------------------------+
/* First push of SELECT * FROM table1. */
+----------------------------------------------+
| HEADER: IPROTO_PUSH, IPROTO_PUSH_ID = <id1> |
+- - - - - - - - - - - - - - - - - - - - - - - +
| BODY: { |
| IPROTO_DATA: [ tuple1, tuple2, ... ] |
| IPROTO_METADATA: [ |
| { /* field1 name, type ... */ }, |
| { /* field2 name, type ... */ }, |
| ... |
| ] |
| } |
+----------------------------------------------+
/* From second to last push. */
+----------------------------------------------+
| HEADER: IPROTO_PUSH, IPROTO_PUSH_ID = <id1> |
+- - - - - - - - - - - - - - - - - - - - - - - +
| BODY: { |
| IPROTO_DATA: [ tuple1, tuple2, ... ] |
| } |
+----------------------------------------------+
/* Result of SELECT my_c_func(a2). */
+----------------------------------------------+
| HEADER: IPROTO_PUSH |
+- - - - - - - - - - - - - - - - - - - - - - - +
| BODY: { |
| IPROTO_DATA: [ [ tuple ] ], |
| IPROTO_METADATA: [ |
| { /* field name, type ... */ } |
| ] |
| } |
+----------------------------------------------+
/* Result of INSERT INTO table1 VALUES (1, 2, 3). */
+----------------------------------------------+
| HEADER: IPROTO_PUSH |
+- - - - - - - - - - - - - - - - - - - - - - - +
| BODY: { |
| IPROTO_SQL_INFO: { |
| SQL_INFO_ROW_COUNT: number, |
| SQL_INFO_LAST_ID: number, |
| } |
| } |
+----------------------------------------------+
/* Result of RETURN a4 */
+----------------------------------------------+
| HEADER: IPROTO_OK |
+- - - - - - - - - - - - - - - - - - - - - - - +
| BODY: { |
| IPROTO_DATA: [ a4 ] |
| } |
+----------------------------------------------+
```
## Rationale and alternatives
Another way to link pushes together exists, replacing `IPROTO_PUSH_ID`.
Pushes can be linked via flag in a header: `IPROTO_FLAG_IS_CHAIN`, that would be stored in `IPROTO_FLAGS` header value. When a multiple messages form a chain, all of them except last one contain this flag. For example:
```
IPROTO_PUSH
|
IPROTO_PUSH, IS_CHAIN
|
+--IPROTO_PUSH, IS_CHAIN
|
+--IPROTO_PUSH, IS_CHAIN
|
+--IPROTO_PUSH
|
IPROTO_PUSH
|
...
|
IPROTO_OK/ERROR
```
It is slightly simpler than `PUSH_ID`, but
1. Does not enable to mix parts of different chains, if it will be needed sometimes;
2. The last response does not contain `IS_CHAIN`, but it is actually a part of chain. `IS_CHAIN` can not be stored in the last response, because else it will not be distinguishable from the next chain. This can be solved by renaming `IS_CHAIN` to `HAS_NEXT_CHAIN` or something, but `PUSH_ID` seems better - it has no these problems, and is more scalable.
|
