1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
|
# Bytecode
This document describes the bytecode that the VM interprets or JIT compiles.
Note that this is just one part of an input file to the VM; along with it
will also be a bunch of serialized objects, and some container. This just
describes the way the executable segment of things looks. (In a sense, this
is the low-level reification of the Actions/World distinction at the level
of the compiler).
## Endianness
All integer values are stored in little endian format.
## Floats
Floating point numbers are represented according to IEEE 754.
## Header
The header appears at the start of the MoarVM bytecode file, and indicates
what it contains.
+---------------------------------------------------------+
| "MOARVM\r\n" |
| 8-byte magic string; includes \r\n to catch mangling |
| of line endings |
+---------------------------------------------------------+
| Version |
| 32-bit unsigned integer; since we'll never reach a |
| huge number of versions, this also doubles up as a |
| check that no weird big/little endian issues keep us |
| from reading the bytecode. |
+---------------------------------------------------------+
| Offset (from start of file) of the SC dependencies |
| table |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| Number of entries in the SC dependencies table |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| Offset (from start of file) of the extension ops table |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| Number of entries in the extension ops table |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| Offset (from start of file) of the frames data segment |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| Number of frames we should end up finding in the frames |
| data segment |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| Offset (from start of file) of the callsites data |
| segment |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| Number of callsites we should end up finding in the |
| callsites data segment |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| Offset (from start of file) of the strings heap |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| Number of entries in the strings heap |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| Offset (from start of file) of the SC data segment |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| Length of the SC data segment |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| Offset (from start of file) of the bytecode segment |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| Length of the bytecode segment |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| Offset (from start of file) of the annotation segment |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| Length of the annotation segment |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| HLL Name |
| 32-bit unsigned integer index into the string heap, |
| providing the name of the HLL this compilation unit |
| was compiled from. May be the empty string. |
+---------------------------------------------------------+
| Main entry point frame index + 1; 0 if no main frame |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| Library load frame index + 1; 0 if no load frame |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| Deserialization frame index + 1; 0 if none |
| 32-bit unsigned integer |
+---------------------------------------------------------+
## Strings heap
This segment contains a bunch of string data. Each string is laid out as:
+---------------------------------------------------------+
| String length in bytes |
| 32-bit unsigned integer left shifted by 1 |
| LSB is flag where 1 means UTF-8 and 0 means latin-1 |
+---------------------------------------------------------+
| String data encoded as UTF-8/latin-1 |
| Bunch of bytes, padded at end to 32 bit boundary |
+---------------------------------------------------------+
## SC Dependencies Table
This table describes the SCs (Serialization Contexts) that the bytecode in
this file references objects from. The wval opcode specifies an index in
this table and an index in the SC itself. When the bytecode file is first
loaded, we look in the known SCs table and resolve all that we can. Then,
the deserialize code for the compilation unit is run. Whenever the SC
creation opcode is used, we search all known compilation units to see if they
have any unresolved SCs, and fill in any gaps that correspond to the newly
created SC. By the time the deserialize phase for a compilation unit is over,
we expect that all SCs have been resolved. Thus, the lifetime of an SC is
equal to the lifetimes of all the compilation units that reference it,
since their code depends on it. Note that the primary way an SC is rooted
is through a compilation unit, and that these roots are established as
soon as it is created, and before it's returned to userspace (which could
allocate more) are the way we make sure it isn't collected too early.
+---------------------------------------------------------+
| Index into the string heap of the SC unique ID |
| 32-bit unsigned integer |
+---------------------------------------------------------+
## Extension ops table
+---------------------------------------------------------+
| Index into the string heap of the extension op ID |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| Operand descriptor |
| Bunch of bytes describing a single operand each, |
| zero-padded to 8 bytes |
+---------------------------------------------------------+
The operand descriptor follows the same format as used by MVMOpInfo.
The 8 bytes limit corresponds to MVM_MAX_OPERANDS.
## Frames Data
The frames data segment contains data that describes all of the frames in
the compilation unit. It also points into the bytecode segment, which contains
the bytecode we will execute for this frame. This is stored elsewhere at least
partly for the sake of demand paging and CPU cache efficiency; once we
processed the static data, it's not very interesting at runtime, so there's no
real reason for it to stay in memory, let alone be cached by the CPU. The
actual bytecode itself, on the other hand, is (at least until JIT happens) of
interest for execution.
Each frame starts with the following data.
+---------------------------------------------------------+
| Bytecode segment offset |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| Bytecode length in bytes |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| Number of locals/registers |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| Number of lexicals |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| Compilation unit unique ID |
| 32-bit string heap index |
+---------------------------------------------------------+
| Name |
| 32-bit string heap index |
+---------------------------------------------------------+
| Outer |
| 16-bit frame index of the outer frame. For no outer, |
| this is set to the current frame index. |
+---------------------------------------------------------+
| Annotation segment offset |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| Number of annotations |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| Number of handlers |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| Frame flag bits |
| 16-bit integer |
| 1 = frame has an exit handler |
| 2 = frame is a thunk |
| Remaining values reserved |
| [NEW IN VERSION 2] |
+---------------------------------------------------------+
| Number of entries in static lexical values table |
| 16-bit integer |
| [NEW IN VERSION 4] |
+---------------------------------------------------------+
| Code object SC dependency index + 1; 0 if none |
| 32-bit unsigned integer |
| [NEW IN VERSION 4] |
+---------------------------------------------------------+
| SC object index; ignored if above is 0 |
| 32-bit unsigned integer |
| [NEW IN VERSION 4] |
+---------------------------------------------------------+
| Number of local debug name mappings |
| 32-bit unsigned integer |
| [NEW IN VERSION 6] |
+---------------------------------------------------------+
This is followed, for each local, by a number indicating what kind of
local it is. These are stored as 16-bit unsigned integers.
int8 1
int16 2
int32 3
int64 4
num32 5
num64 6
str 7
obj 8
uint8 17
uint16 18
uint32 19
uint64 20
Lexicals are similar, apart from each entry is followed by a 32-bit unsigned
index into the string heap, which gives the name of the lexical.
[Conjectural: a future MoarVM may instead do these in terms of REPRs.]
Next comes the handlers table. Each handler has an entry as follows:
+---------------------------------------------------------+
| Start of protected region. Inclusive offset from start |
| of the frame's bytecode |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| End of protected region. Exclusive offset from start of |
| the frame's bytecode |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| Handler category mask bitfield |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| Handler action (see exceptions spec for values) |
| 16-bit unsigned integer |
+---------------------------------------------------------+
| Register number containing the block to invoke, for a |
| block handler. |
| 16-bit unsigned integer |
+---------------------------------------------------------+
| Handler address to go to, or where to unwind to after |
| an invoked handler. Offset from start of the frame's |
| bytecode. |
| 32-bit unsigned integer |
+---------------------------------------------------------+
From version 4 and up, this is followed by a static lexical values
table. Each entry is as follows:
+---------------------------------------------------------+
| Lexical index |
| 16-bit unsigned integer |
+---------------------------------------------------------+
| Flag |
| 16-bit unsigned integer |
| 0 = static lexical value |
| 1 = container var (cloned per frame) |
| 2 = state var (cloned per closure) |
+---------------------------------------------------------+
| SC dependency index |
| 32-bit unsigned integer |
+---------------------------------------------------------+
| SC object index |
| 32-bit unsigned integer |
+---------------------------------------------------------+
From version 6 and up, this is followed by a debug local names table. Each
entry is as follows:
+---------------------------------------------------------+
| Local index |
| 16-bit unsigned integer |
+---------------------------------------------------------+
| String heap index for the debug name |
| 32-bit unsigned integer |
+---------------------------------------------------------+
## Callsites Data
This data blob contains all of the callsite descriptors that are used in
the compilation unit. At the point of loading the bytecode, they will
be set up, and a table pointing to them created. This means that a
callsite descriptor will always be a pointer + offset away.
Each callsite consists of a 16-bit unsigned integer indicating the number
of argument flags. This is followed by the flags, taking 8 bits each. If
the number of argument flags is odd, then an extra padding byte will be
written afterwards. Since version 3, this is then followed with one index
to the string heap (in the form of a 32-bit integer) for each argument flag
that has the `MVM_CALLSITE_ARG_NAMED` bit set.
## Bytecode segment
This consists of a sequence of instructions. Instruction codes are always
16 bits in length. The first 8 bits describe an instruction "bank", and the
following 8 bits identify the instruction within that bank. Instruction banks
0 through 127 are reserved for MoarVM core ops or future needs. Instruction
banks 128 through 255 are mappable per compilation unit, and are used for
"plug-in" ops.
Opcodes may be followed by zero or more operands. The instruction set will
have the needed operands described by the following set of descriptors.
r local variable index being read, 16 bits unsigned
w local variable index being written, 16 bits unsigned
rl lexical variable being read, 16 bits unsigned for the
index within a frame and 16 bits for how many frames out
to go to locate it
wl lexical variable being written, 16 bits unsigned for the
index within a frame and 16 bits for how many frames out
to go to locate it
i16 16-bit integer constant
i32 32-bit integer constant
i64 64-bit integer constant
n32 32-bit floating point constant
n64 64-bit floating point constant
si Strings table index, 32 bits unsigned
sci Serialization Context object table index, 16 bits unsigned
csi Callsite table index, 16 bits unsigned
ins Instruction offset from frame start (for goto), 32 bits unsigned
Note that this ensures we always keep at least 16-bit alignment for ops.
Some instructions place demands on the type of value in the register.
This is perhaps most noticable when it comes to integers of different
sizes; all computations are done on them in full-width (64-bit) form,
and loading/storing them to registers representing locals of more
constrained sizes needs explict sign extension and truncate ops.
Wherever a register is specified, the kind of value in it is also
indicated. These are typechecked *once*, either at bytecode load time
or (perhaps better) on the first execution. After that, all future
interpreter executions can just plow through the instructions without
ever having to do checks.
The set of ops is listed in src/core/oplist.
## Annotation segment
This consists of a number of records, composed of:
* 32-bit unsigned integer offset into the bytecode segment
* 32-bit unsigned integer strings heap index (filename)
* 32-bit unsigned integer (line number)
|
