1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
|
# Relocatable and Relative Pointers
When memory is shared between applications and each application maps the shared memory to a different starting address
in its own address space, raw pointers cannot be used anymore. This is due to the fact that a pointer pointing to some
object in shared memory in one address space will in general not point to the same object in another address space (or
even to the shared memory), i.e. the pointer is invalid. In this case, different pointer concepts such as relative and
relocatable pointers must be used. Depending on the shared memory setup and the relationship between pointer and pointee,
there are different use cases which we need to cover.
Both `RelativePointer<T>` and `relocatable_ptr<T>` are template pointer classes that try to retain as much of the normal pointer
syntax as possible while still being valid across address space boundaries. This means that operations like
assignment, dereferencing, comparison and so on work similar to raw pointers. They do not possess reference counting
semantics in this implementation, but extensions for this functionality are possible.
Note that the naming `RelativePointer` in the codebase is currently inconsistent with `relocatable_ptr` (which mimic STL
smart pointers). This will be changed in a refactoring of `RelativePointer`.
## Shared Memory Setup
In general, we have many shared memory segments, each starting at some address and containing a specified number of
bytes. Specifically we distinguish between management and payload segments. The management segment contains metadata
used by RouDi and the runtime to communicate, store common structures such as ports and so on. The payload segment
contains the actual data samples sent by senders in some application to receivers in other applications.
## Relocatable Pointers
When pointer and pointee are located in the same shared memory segment, the lightweight (compared to a relative pointer)
relocatable pointer can be used. That is, we have the following scenario.
Pointer `p` points to object `X` of type `T` and both are stored in shared memory segment `S`.
```txt
Shared Memory S: p X
|____________________^
App1 a1 b1 c1
App2 a2 b2 c2
```
Let `a1`, `b1`, `c1` and so on be the addresses of segment `S`, pointer `p` and object `X` in application 1 and similarly `a2`, `b2` and
`c2` in application2. If application 2 maps the memory differently they will be shifted by some common offset `d` depending
on the individual memory mapping:
```
a2 = a1 + d, b2 = b1 + d, c2 = c1 + d
```
This is why storing a raw pointer to `X` will not be sufficient, the value `c1` of `p` will not point to `X` in application 2.
However, storing the difference between the location of `p` and `X` will work since it is an invariant in both address
spaces.
Thus, instead of using a raw pointer `p`, we just use a relocatable pointer `relocatable_ptr<T>` pointing to `X`. This pointer
can then be dereferenced in both address spaces and will point to `X`.
Note that if we only use relocatable pointers in this way in complex data structures that rely on pointers (such as
lists and trees) in shared memory, we can in fact copy the whole shared memory segment somewhere else (i.e. *relocate*
it, hence the name) without compromising the data structure integrity. That is, the list elements in the copy correctly
point to their successor elements in the copied memory. This also presupposes that we do not rely on other side effects
of deep copying (i.e. nontrivial copy constructors).
### Construction and Assignment of Relocatable Pointers
Assume we have an object `X` of type `T` in shared memory. Then we can construct relocatable pointers `p` in the *same
segment* (e.g. as members of objects that reside in this segment or via emplacement new at a memory address in this
segment).
Default construct a logical `nullptr`
+ `relocatable_ptr<T> p;`
Construct relocatable pointing to `X`
+ `relocatable_ptr<T> p(&X);`
Assign relocatable to point to `X`
+ `p = &X;`
### Operations of Relocatable Pointers
Most operations for raw pointers also work for relocatable pointers. In particular, a `relocatable_ptr<T>` is compatible
with a raw pointer `T*` in many cases, such as assignment, to provide seamless integration and convenient syntax.
Comparison with raw pointers
+ `if(p != nullptr) { \\do something }`
+ `T* raw = &X; if(p == raw) { \\do something }`
Assignment to raw pointer (resolves the pointer in the current address space)
+ `T* raw = p;`
Explicitly get the raw pointer
+ `T* raw = p.get();`
Dereference
+ `T& t = *p;`
Access members
+ `p->func();`
### Performance of Relocatable Pointers
Since relocatable pointers just measure the distance to the pointee from itself it just requires a number to store this
pointer difference. This number has the same size as a pointer itself, i.e. 64 bit on 64 bit architectures. All
operations use just a few simple arithmetic instructions in addition to pointer dereferencing and hence they do incur
little overhead compared to raw pointers.
## Relative Pointers
When pointer and pointee are located in different shared memory segments, it gets more complicated and relocatable
pointers are no longer appropriate. This happens in our setup with metadata in the management segment referencing data
in the payload segment.
Pointer `rp` is stored in segment `S1` and points to object `X` of type `T` in segment `S2`.
```txt
Shared Memory S1: rp S2: X
|_______________________^
App1 a1 b1 c1 d1
App2 a2 b2 c2 d2
```
It is no longer true in general that both segments will be offset by the same difference in App2 and therefore
relocatable pointers are no longer sufficient.
Relative pointers solve this problem by incorporating the information from where they need to measure differences (i.e.
*relative* to the given address). This requires an additional registration mechanism to be used by all applications
where the start addresses and the size of all segments to be used are registered. Since these start address may differ
between applications, each segment is identified by a unique id, which can be provided upon registration by the first
application. In the figure, this means that the starting addresses of both segments (`a1, a2 and c1, c2`) would have to be
registered in both applications.
Once this registration is done, relative pointers can be constructed from raw pointers similar to relocatable pointers.
However, it should be noted that relocating a memory segment will invalidate relative pointers, i.e. relative pointers
are **NOT relocatable**. This is because the registration mechanism cannot be automatically informed about the
copy of a whole segment, such a segment would have to be registered on its own (and the original segment deregistered).
### Registration
Register the start pointer `start1` of a segment with a specific size (this happens once after the memory is mapped)
+ `auto id = registerPtr(start1, size);`
Register the start pointer of the segment `start2` in another application with a specific id (again, this happens after
the other application maps the memory)
+ `registerPtr(id, start2, size);`
Unregister specific id (also invalidates relative pointers using this id, they cannot be used safely after this
operation anymore)
+ `unregisterPtr(id);`
Unregister all ids (also invalidates all relative pointers)
+ `unregisterAll();`
### Construction and Assignment of Relative Pointers
Assume we have an object `X` of type `T` in shared memory. Then we can construct relative pointers `rp` pointing to *arbitrary
registered segments*.
Default construct a logical `nullptr`
+ `RelativePointer<T> rp;`
Construct relative pointer pointing to `X` (provided the segment containing `X` was registered)
+ `RelativePointer<T> rp(&X);`
Assign relative pointer to point to `X`
+ `rp = &X;`
Copy construction
+ `RelativePointer<T> rp2(rp);`
Copy assignment
+ `RelativePointer<T> rp2 = rp;`
### Operations of Relative Pointers
As for relocatable pointers, operations for raw pointers also work for relative pointers.
comparison with raw pointers
+ `if(rp != nullptr) { \\do something }`
+ `T* raw = &X; if(p == raw) { \\do something }`
Assignment to raw (resolves the pointer in the current address space)
+ `T* raw = rp;`
Dereference
+ `T& t = *p;`
Implicitly get the raw pointer
+ `T* raw = rp;`
Explicitly get the raw pointer
+ `T* raw = rp.get();`
Access members
+ `p->func();`
### Performance of Relative Pointers
A relative pointer measures the distance to the pointee relatively to a registered address and therefore needs to store two
numbers, one for the distance itself and the other one to identify the segment base address to measure from. For this reason,
the operations incur slightly more overhead compared to relocatable pointers.
## Atomic usage
There is a technical problem using both `relocatable_ptr` and `RelativePointer` as a type in a `std::atomic`.
This is essentially impossible as an atomic requires its type to be copyable/movable (to be loaded) but on the other hand,
this copy constructor must be trivial, i.e. performable with a (shallow) memcpy. Therefore, the types used in atomic cannot implement
custom copy/move. This is not possible for `relocatable_ptr` and `RelativePointer` as both require operations performed
during copying, which cannot be done by a simple shallow copy.
To support the use case of an atomic relocatable pointer, the template `AtomicRelocatablePointer<T>` is provided. It is a
basic version of a relocatable_ptr, which lacks some of its move functionality and other operations. However, its
existing operations behave like `relocatable_ptr` but in an atomic way (but also incur additional overhead caused by
atomic loads and stores). Therefore, this object can be used concurrently by multiple threads, in contrast to
`RelativePointer` and `relocatable_ptr`. This atomic version also utilizes lock-free atomic operations internally (when
available on the target architecture) as the data to be stored is just a 64 bit word.
Note that this cannot be done as easily for `RelativePointer` as it requires more than 64 bit to store its internal
information. A construction where atomicity is guaranteed via locking could be provided in the future.
|
