1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
|
<html>
<head>
<title>Metakit Tips and Tricks</title>
<meta name="generator" content="Frontier 5.0.1 Win95"></head>
<body bgcolor="#FFFFFF" alink="#008000" vlink="#800080" link="#0000FF" text="#000000">
<table cellspacing=0 cellpadding=0 border=0 width="100%"><tr valign=top>
<td width=100><a href="http://www.equi4.com/">
<img src=e4w.gif alt="E q u i 4" align=left width=97 height=35 hspace=0 border=0></a>
</td><td align=center>
<font size=+2>Metakit Tips and Tricks</font><br>
<i>December 1999</i>
</td><td width=100></td></tr></table>
<p>
<i>This page list some ideas and background information
related to the use of Metakit.
If you have suggestions, please send them to <a href="mailto:jcw@equi4.com">jcw@equi4.com</a> ...</i>
<br><br>
<table border=0>
<tr>
<td valign=top align=center><a href="#streaming"><b>Streaming</b></a></td>
<td width=10></td>
<td>Some details about streaming and on-demand loading</td>
</tr>
<tr>
<td valign=top align=center><a href="#faster"><b>Faster</b></a></td>
<td width=10></td>
<td>What to do and what to avoid for maximum speed</td>
</tr>
<tr>
<td valign=top align=center><a href="#smaller"><b>Smaller</b></a></td>
<td width=10></td>
<td>How to manage data in the most compact format</td>
</tr>
</table>
</center>
<dl>
<dt><br><hr size=1><br><a name=streaming></a><b>Streaming</b> -
Some details about streaming and on-demand loading<p><dd><font face=Times>
When saving data, Metakit uses different formats, depending on whether
the data was saved with c4_Storage::Commit or with c4_Storage::SaveToStream.
This has a number of consequences on how and when data is loaded back in:
<blockquote><table cellpadding=5 cellspacing=0 border=1>
<tr>
<th> Saved using </th>
<th> Loaded as </th>
<th align=left> Comments </th>
</tr>
<tr>
<td align=center>
Commit<br>
Commit<br>
SaveToStream<br>
SaveToStream</td>
<td align=center>
File<br>
Stream<br>
File<br>
Stream</td>
<td>
The normal case, loads data on-demand<br>
This combination is <b>not</b> allowed!<br>
Ok, will load all data during open<br>
Loads all data serially right away</td>
</tr>
</table></blockquote>
As you can see in this overview, random-access and streaming can be mixed
in three out of four ways. Once loaded, you can again save using either
c4_Storage::Commit or c4_Storage::SaveToStream.
<p>
When random-access and streaming are mixed on a storage object, the
following will happen:<p>
<ul><li>Files are only modified (conceptually) during a call to c4_Storage::Commit.
<li>c4_Storage::SaveToStream makes a snapshot of the current (uncommitted) state.
</ul><p>
It is therefore possible to open a file, apply a few changes, save the
changed contents to a stream, and then close the file without committing
those changes. Streaming does not interfere with the commit/rollback mechanism.
</font><br>
<dt><br><hr size=1><br><a name=faster></a><b>Faster</b> -
What to do and what to avoid for maximum speed<p><dd><font face=Times>
The data structures used in Metakit can have surprising effects on performance,
both positively and negatively.
Here are a few suggestions which should help in most cases:
<p>
<li><b>Preallocate rows</b> - if the number or rows to add is known in advance,
use c4_View::SetSize to set the view size accordingly. Fixed-size properties
will need to do fewer internal reallocations if pre-allocated.
<p>
When inserting several rows, consider inserting them all with a dummy value
first, and then using a loop to adjust each of the inserted values.
<p>
<li><b>Limit the use of c4_Row objects</b> - The construction of c4_Row objects is
relatively slow, and can often easily be avoided. Here are three ways to
accomplish the same thing:
<blockquote><table cellpadding=5 cellspacing=0 border=1>
<tr>
<th>Slow</th>
<td>
for (int i = 0; i < n; ++i)<br>
{<br>
c4_Row temp;<br>
property (temp) = ...;<br>
view[i] = temp;<br>
}
</td>
</tr>
<tr>
<th>Faster</th>
<td>
c4_Row temp;<br>
for (int i = 0; i < n; ++i)<br>
{<br>
property (temp) = ...;<br>
view[i] = temp;<br>
}
</td>
</tr>
<tr>
<th>Fastest</th>
<td>
for (int i = 0; i < n; ++i)<br>
{<br>
c4_RowRef row = view[i];<br>
property (row) = ...;<br>
}
</td>
</tr>
</table></blockquote>
The last two statements could also have been combined as "property (view[i]) = ...;"
<p>
Note that "prop1 [value1] + prop2 [value2]" also creates several temporary
c4_Row objects, and should be avoided when top performance is critical.
<p>
<li><b>Remove views before storage objects are destroyed</b> - If a view
still exists when its underlying storage object is destroyed, Metakit has to
create a full copy of the data in memory before releasing the storage object,
to avoid even more confusing side-effects.
The following example illustrates a common source of suprises in Metakit:
<blockquote><table cellpadding=5 cellspacing=0 border=1>
<tr>
<th>Very slow<br>(on close)</th>
<td>
{<br>
c4_View view;<br>
c4_Storage storage (...);<br>
view = storage.View(...);<br>
...<br>
}
</td>
</tr>
<tr>
<th>Instant</th>
<td>
{<br>
c4_Storage storage (...);<br>
c4_View view = storage.View(...);<br>
...<br>
}
</td>
</tr>
</table></blockquote>
In the first case, the view still exists as the storage object goes out of scope.
There is a simple way to detach views from their storage objects, since they
act merely as (reference-counted) pointers: just set "view = c4_View ();"
<b>before</b> the storage object is destroyed.
</font><br>
<dt><br><hr size=1><br><a name=smaller></a><b>Smaller</b> -
How to manage data in the most compact format<p><dd><font face=Times>
Metakit implements automatic "adaptive integer sizing" and will often create
very compact data files as a result. Strings and binary properties also use a
variable format, and require no "field width" setting. Some tips to
allow you to make optimal use of these features:
<p>
<li><b>Attached views are more compact</b> - You can trade off size versus
speed by selecting either views attached to a storage object (smallest), or
unattached views (fastest). Note that "storage" does not imply the need to
store data on file in this context. It is perfectly valid to use the following
code to manage temporary data in memory without ever saving it to file:
<blockquote><table cellpadding=5 cellspacing=0 border=1>
<tr>
<td>
c4_Storage storage;<br>
c4_View view = storage.GetAt("temp[prop1:I,prop2:I]");<br>
...<br>
prop1 (view[i]) = ...;<br>
...
</td>
</tr>
</table></blockquote>
In this example, the view would use "adaptive integer sizing" even though
its data is valid only for the duration of each run.
Unattached views always use 32 bits to store integers.
<p>
<li><b>Integer formats only increase in size</b> - The representation used to
store integers in attached views (and on file)
is based on the largest value (in magnitude) ever stored in
each property and view. Storing large values in one subview will not affect
the storage used by that same property in another subview.
<p>
For integers, the number of bits used to store values is determined as follows:
<blockquote><table cellpadding=5 cellspacing=0 border=1>
<tr>
<th> Value range </th>
<th> Bits used </th>
<th align=left> Comments </th>
</tr>
<tr>
<td align=center>
0<br>
0 .. 1<br>
0 .. 3<br>
0 .. 15<br>
-128 .. 127<br>
-32768 .. 32727<br>
all others</td>
<td align=center>
0<br>
1<br>
2<br>
4<br>
8<br>
16<br>
32</td>
<td>
If all rows are 0, <b>no</b> storage space is used<br>
Booleans are stored as 8 rows per byte<br>
Tiny positive codes<br>
A hex nibble<br>
One byte<br>
A short integer<br>
A long integer</td>
</tr>
</table></blockquote>
</font><br>
</dl>
<hr size=1>
<center>
<a href="index.html" target="_top">Roadmap</a>
<a href="classes.html">Class Index</a>
<a href="samples.html">Sample Index</a>
<a href="tips.html">Tips and Tricks</a>
</center>
<hr size=1>
<font color="#000099" size=-1>© 1998 Equi4 Software. All rights reserved.</font>
</body>
</html>
|
