1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
|
<html>
<head>
<LINK REL="stylesheet" HREF="styles.css" TYPE="text/css">
<title>FOX-Toolkit</title>
<!-- HTML Copyright 2001 Paul Laufer -->
</head>
<body bgcolor=#ffffff link=#990033 vlink=#4a73ad alink=#ed004f text=#000000>
<!--header-->
<table align=center border=0 cellpadding=0 cellspacing=0 width=100% >
<tr><td bgcolor=silver colspan=5 align=right height=50><img src=art/oul_grey.gif align=left valign=top width=8 height=8><img src=art/foxlogo.png valign=bottom alt="FOX Toolkit" height=50 width=500 border=0 ></td>
<td bgcolor=#557faa valign=top align=right><img src=art/our.gif width=8 height=8></td>
</tr>
<!-- end header -->
<tr>
<td bgcolor=#557faa colspan=2 valign=top align=left> </td>
<td bgcolor=#557faa colspan=3><font color=#ffffff size=+1><center>
<!-- Page Title -->
Documentation: Serialization of Data and Objects
<!-- End Page Title -->
</center></font></td>
<td bgcolor=#557faa valign=top align=right> </td>
</tr>
<tr>
<td bgcolor=#557faa colspan=2> </td>
<td bgcolor=#ffffff valign=top align=left><img src=art/iul.gif width=8 height=8></td>
<td bgcolor=#ffffff> </td>
<td bgcolor=#ffffff valign=top align=right><img src=art/iur.gif width=8 height=8></td>
<td bgcolor=#557faa width=15> </td>
</tr>
<tr>
<td width=8 bgcolor=#557faa> </td>
<td valign=top bgcolor=#557faa link=#ffffff width=150>
<!-- start navbar content -->
<a href=fox.html><font color=#ffffff>Home</font></a><br>
<a href=news.html><font color=#ffffff>News</font></a><br>
<a href=download.html><font color=#ffffff>Download</font></a><br>
<a href=goals.html><font color=#ffffff>Goals & Approach</font></a><br>
<a href=doc.html><font color=#ffffff>Documentation</font></a><br>
<a href=faq.html><font color=#ffffff>FAQ</font></a><br>
<a href=rex.html><font color=#ffffff>FXRex</font></a><br>
<a href=screenshots.html><font color=#ffffff>Screenshots</font></a><br>
<br>
<a href=adie.html><font color=#ffffff>Adie</font></a><br>
<a href=pathfinder.html><font color=#ffffff>PathFinder</font></a><br>
<a href=calc.html><font color=#ffffff>FOX Calculator</font></a><br>
<br>
<a href=projects.html><font color=#ffffff>Projects</font></a><br>
<br>
<a href='http://fxpy.sourceforge.net'><font color=#ffffff>FXPy</font></a><br>
<a href='http://fxruby.sourceforge.net'><font color=#ffffff>FXRuby</font></a><br>
<a href='http://eiffelfox.sourceforge.net'><font color=#ffffff>EiffelFox</font></a><br>
<a href='http://eevolved.com/foxhole/'><font color=#ffffff>The FOX Hole</font></a><br>
<a href='http://takahr.dhis.portside.net/cgi-bin/rwiki.cgi?cmd=view;name=FOX+FAQ'><font color=#ffffff>Japanese Docs</font></a><br>
<br>
<center>
<a href="http://www.eff.org/br"><img SRC="art/freespeach.gif" border=0></a>
<p>
<a href="http://www.slashdot.org"><img SRC="art/slingerzbutton1.gif" border=0></a>
</center>
<!-- end navbar content -->
</td>
<td bgcolor=#ffffff> </td>
<td valign=top>
<!-- start main window content -->
<center><img src='art/foxstart.png'>
<BR><B>Documentation: Serialization of Data and Objects</B>
</center>
<p>
<p>
<b>What is Serialization?</b>
<hr>
Often, your application needs to save and load data in a machine-independent,
binary format. This data may be very simple, such as an array of
numbers, or it may be a complex networks of objects arranged in some application-defined
data structure.
<p>FOX offers some tools to make implementation of such basic save and
load facilities in an application fairly straighforward: <b>Serialization</b>
and
<b>Deserialization</b>. Serialization refers to the process
of taking a network of objects and their member data, and turning it into
a linear byte stream; deserialization of course refers to the opposite.
This process is also sometimes referred to as <i>streaming</i>, <i>flattening</i>,
or more prosaically, <i>pickling</i>.
<p>The <b>FXStream</b> classes support streaming of objects and data in
a type-safe and architecture-neutral manner; this means that a) your
data will be read in the way you wrote it out, and b) streaming works as
efficient on little-endian machines as it does on big-endian ones:- there
is no byte-order preference.
<p>The FXStream class are extremely flexible, in that you may subclass
them ad libitum to implement esoteric applications ranging from compression
to encryption, BSD sockets, UNIX pipes, clipboard, drag & drop,
and what have you. Code you write to serialize your data may be reused
to perform any of these functions simply by substituting the FXStream class
upon which they operate.
<p>Once code for an object's serialization has been written, this streaming
capability can be used for a variety of purposes:
<br>
<ul>
<li>
Saving or loading from <b><i>files</i></b>, in a machine-independent manner.</li>
<li>
Saving into <b><i>memory buffers</i></b>, or loading back from memory buffers.</li>
<li>
Loading of <b><i>resources</i></b> compiled into the application using
<b><i><a href="icons.html#RESWRAPDOC">reswrap</a></i></b>.</li>
<li>
Exchanging objects and data between applications using <b><i><a href="draganddrop.html#DRAGNDROP">Drag
and Drop</a></i></b> techniques.</li>
<li>
Just <b><i>counting the bytes</i></b>, e.g. to determine buffer sizes.</li>
<li>
Transfer objects and data over the network, e.g. via <b><i>sockets</i></b>,
<b><i>pipes</i></b>,
<b><i>PVM</i></b>,
<b><i>MPI</i></b>,
etc.</li>
</ul>
<p>
<p>
<b>Philosophy in FOX Serialization</b>
<hr>
The FOX Stream classes have been designed with a number of goals in
mind:
<br>
<ul>
<li>
<b>Speed</b>. The serialization and deserialization should be very
fast. Thus, a minimal amount of computing overhead is required; also,
I/O should be minimized.</li>
<br>
<li>
<b>Flexibility</b>. At some small expense in speed, all I/O eventually
boils down to a few basic virtual I/O functions; thus, it is possible to
derive subclasses and serialize data into byte streams with different destinations
or sources:- not just files, but also memory buffers, sockets, or perhaps
shared memory segments or mapped files.</li>
<br>
<li>
<b>Type Safety</b>. In order to make sure that the number of bytes
saved exactly matches the number of bytes loaded, all stream insertion/extraction
operators are defined for all basic machine types, and these <i>types </i>are<i>
guaranteed </i>to be the<i> same size </i>on all FOX implementations.</li>
<br>
<li>
<b>Byte Swapping</b>. Since the types are known, the FOX Stream class
is able to swap bytes upon stream deserialization. The FOX Stream
does NOT swap bytes on <i>saving</i>, but only on <i>loading</i>.
This is for the following reasons:</li>
<br>
<ul>
<li>
It is faster to serialize in a machine-natural order, so that as long as
one works on machines of the same architecture, no cost is incurred with
swapping bytes at all. Loading and saving on the same type of machine
is expected to be a very, very common case.</li>
<br>
<li>
By byte swapping on the receiving end, an <i>in-situ</i> swap can be performed,
which will lead to much better caching, and eliminates the need to temporary
buffers etc.</li>
<br> </ul>
<li>
<b>Predictability</b>. With the exception of serialization of FOX
Objects, the FOX Stream class serializes exactly as many bytes as it is
given by the application. This has a number of interesting benefits:-
for example, the FOX GIF Image loading routine works based on a FOX Stream,
permitting it to read both from files as well as from memory data arrays;
this makes handling of compiled-in or embedded resources (e.g. by using
<a href="icons.html#RESWRAPDOC">reswrap</a>)
very simple indeed.</li>
<br>
<li>
<b>Future expansion</b>. An escape tag is prepended for serialized
FOX Objects. This will in the [near] future allow deserialization
of FOX Objects that are available in <i>dynamic link libraries (DLL's).
</i>Currently, FOX can only deserialize objects that have been compiled
into the application code.</li>
</ul>
<p>
<p>
<b>So How Does It Work?</b>
<hr>
From the application programmer's point of view, it works very simply:
<blockquote>
<center><table BORDER CELLSPACING=0 COLS=1 WIDTH="90%" BGCOLOR="#FFF8E1" NOSAVE >
<tr>
<td><tt>FXuint data[100],numdata;</tt>
<p><tt>// Save my stuff to a stream</tt>
<br><tt>void savemystuff(FXStream& stream){</tt>
<br><tt> stream << numdata;
// Save the number of data values</tt>
<br><tt> steam.save(data,numdata); // Save the data</tt>
<br><tt> }</tt>
<br>
<p><tt>// Save stuff to a FILE stream</tt>
<br><tt>FXFileStream stream;</tt>
<br><tt>stream.open("datafile.dat",FXStreamSave);</tt>
<br><tt>savemystuff(stream);</tt>
<br><tt>stream.close();</tt></td>
</tr>
</table></center>
</blockquote>
As you see, this is pretty simple. Note that the code fragment doing the
actual serialization does not depend on the type of FXStream being used;
I recommend simply passing in an FXStream&, so that the same code may
be used to serialize to FXFileStreams, FXMemoryStreams or other stream
classes as yet to be invented.
<br>
<p>From the stream's point of view, things are a bit more complicated.
Saving basic types (FXchar, FXshort, etc) into an FXStream is done by C++-tradional
insertion and extraction operators <b><<</b> and <b>>></b>.
Note that all operators take a <i>reference</i>, rather than a value.
If we would save a value, regular C++ type promotions might be silenty
invoked, and more bytes might be saved than expected; by taking reference
arguments, one has to first store a value into a variable of <i>known type</i>,
then call the insertion operator.
<p>For <i>arrays</i> of basic types, the FXStream class supplies a few
regular member functions called save() and load(), one for each basic type.
Note that FOX may support a type FXlong on certain machines; FXlong is
always 64 bits, or 8 bytes, if supported by the system. If 64 bit
numbers can not be supported, FXlong is NOT defined.
<p>For FOX Objects, things are a more complex. A network of objects
can be saved into a stream, and should be restored upon a load. Of
course, upon load not all objects will occupy the same address as where
they were initially stored from. Also, objects may refer to each
other; despite that, each object should be saved at most once.
<p>FOX currently implements the object save by means of a hash table to
translate object pointers into reference numbers and vice versa.
In a nutshell, here's how it works:
<p><u>To save an object-pointer to the stream:</u>
<br>
<ol>
<li>
If the pointer is NULL, save the speciall <b>null</b> tag.</li>
<br>
<li>
Consult the hash table to see if the object pointer has been saved before.
If the object has been encountered previously, its data must already have
been saved, and the <b>reference</b> tag found in the hash table is saved
to the stream.</li>
<br>
<li>
If the object has never been encountered before, generate a new reference
tag, and add the object pointer and the reference tag to the hash table.
Subsequently, a <b>class</b> tag, an <b>escape</b> code [0 for now], and
the object's <b>class name</b> is saved to the stream. Then the object's
<b>member
data</b> are saved by calling the object's overloaded <b>save()</b> member
function.</li>
</ol>
<u>To load an object-pointer from the stream:</u>
<br>
<ol>
<li>
Read the tag. If the tag was the <b>null</b> tag, the pointer was
NULL, and a NULL is returned.</li>
<br>
<li>
If the tag was the <b>reference</b> tag, the object has already been loaded,
and the hash table is consulted to return the object-pointer.</li>
<br>
<li>
If the tag was the <b>class</b> tag, the <b>escape</b> tag is read and
[for now] discarded, and subsequently the classname is read. The
<b>FXMetaClass</b>
is localized from the class name, and a new object is constructed by means
of its <b>makeInstance() </b>function<b>. </b>The a new reference
number is generated and the reference number and the object-pointer are
stored into the hash table. Then the object member data are loaded
by calling the object's overloaded <b>load() </b>member function.</li>
</ol>
<p><br>In the current implementation, only those objects whose implementation
has been compiled into the application can be [de-] serialized.
<p>Future versions of FOX will use the <b>escape</b> code information for
additional methods to localize the <b>FXMetaClass</b> objects. In
particular, the thinking is that certain object-implementations may live
in DLL's (Dynamic Link Libraries) and the escape code will help localize
the DLL and pull it in to provide the object implementation. It is
clear that this will be a very powerful mechanism, enabling for example
drag and drop of objects whose implementations are not a-priori known at
the time the application is compiled.
<p><font color="#000000">I added the escape code so as to <b>not</b> break
people's streamed object files when this capability will be introduced.</font>
<p>
<p>
<b>Future FOX uses of Serialization</b>
<hr>
Serialization is not only intended for features such as saving/restoring
from files, and drag-and-drop of objects. Future versions of FOX
will also allow FOX GUI Widgets to be serialized or deserialized; in fact,
it is with this in mind that the two-step [Construct/Create] sequence is
so religiously carried out throughout the Library. Once FOX Widgets have
been deserialized from either an external file or perhaps from a compiled-in
[reswrapped] resource, a GUI can be created in one fell swoop with a single
call to FXApp::create().
<p>A FOX GUI Builder will be a program that builds a nice-looking GUI,
and then serializes it for incorporation into an application [using reswrap].
Using the escape-code mechanism, the FOX GUI builder will be able to build
GUI's that contain Custom Controls or Widgets written by third parties.
<br>
<p>
<p>
<b>Tips and Hints for Serialization: Byte Swapping</b>
<hr>
Proper use of the serialization mechanism will allow serialized data
to be read across different machines, with different byte orders.
In the scope of ``predictability,'' FOX's stream mechanism does NOT contain
any tags or markers, nor does it contain things like byte order and such,
with the exception of course being the saving of object-pointers.
<p>It <i>does</i> however try to help:
<blockquote>
<br><font face="Courier New,Courier">FXbool FXStream::<b>isLittleEndian</b>();</font></blockquote>
<p><br>returns <i>TRUE</i> for Little Endian machines [e.g. i386 and Alpha
CPU's] and <i>FALSE</i> for Big Endian machines [e.g. 68k, SPARC CPU's].
<br>Note that <b>FXbool</b> is defined as <b>FXuchar</b>, NOT as C++ <b>bool</b>.
[I've never been able to find a statement that says how big the standard
type <b>bool</b> is, but I'm pretty sure a char is 1 byte!].
<p>Thus, the following chunk of code may be executed before saving any
actual application data:
<br>
<center><table BORDER CELLSPACING=0 COLS=1 WIDTH="90%" BGCOLOR="#FFF8E1" NOSAVE >
<tr>
<td><tt>FXbool endianness=FXStream::isLittleEndian();</tt>
<br><tt>stream << endianness;</tt>
<br><tt>....</tt>
<br><i><tt>save the data</tt></i>
<br><tt>....</tt></td>
</tr>
</table></center>
<p>Then upon loading:
<br>
<br>
<center><table BORDER CELLSPACING=0 COLS=1 WIDTH="90%" BGCOLOR="#FFF8E1" NOSAVE >
<tr>
<td><tt>FXbool endianness;</tt>
<br><tt>stream >> endianness;</tt>
<br><tt>stream.swapBytes(endianness!=FXStream::isLittleEndian());</tt>
<br><tt>....</tt>
<br><i><tt>load the data</tt></i>
<br><tt>....</tt></td>
</tr>
</table></center>
<p>In other words, the bytes are swapped <b><i>on input</i></b>, if and
only if<i> </i>the byte order of the saving application <i>differs</i>
from the loading one.
<p>
<p>
<b>Tips and Hints for Serialization: Container Object</b>
<hr>
Many applications have one so-called <b>container</b> object, which
may not itself participate in serialization for one reason or another.
For example, the FOX FXApp object is normally created by the main startup
routine of an application, and will probably never be serialized [although
its member data may be].
<p>In order to accomodate references to such an object without saving it,
the FXStream class allows you to specify a <b>container</b> object.
During serialization, when a pointer to the container object is encountered,
only a reference tag is saved to the stream; likewise, on deserialization
a reference to the container object is translated into the pointer passed
in with the FXStream constructor.
<p>
<p>
<b>Tips and Hints for Serialization: Use FX Types</b>
<hr>
FOX defines a number of typedefs for the basic types, such as FXchar,
FXshort, and so on. The idea is that the size of these types is <i>fixed</i>,
and the <i>same</i> on all implementations; there is an FXASSERT somewhere
that will trip if this is not true.
<p>Writing applications that should work on heterogeneous mixes of hardware
becomes simpler if variables you intend to serialize are defined in terms
of these basic types; for loop variables and such ephemeral things, you
may want to use the ``suggested'' system-specific types, as these may be
faster.
<p>The type <b>FXlong</b> may NOT be available on all platforms.
It represents a 64 bit integer type. You use this at your own risk
of potential portability loss.
<br>
<p>
<!-- end main window content -->
</td>
<td bgcolor=#ffffff> </td>
<td bgcolor=#557faa width=15> </td>
</tr>
<tr>
<td colspan=2 bgcolor="#557faa" align=center>
</td>
<td bgcolor=#ffffff valign=bottom align=left><img src=art/ill.gif width=8 height=8></td>
<td bgcolor=#ffffff> </td>
<td bgcolor=#ffffff valign=bottom align=right><img src=art/ilr.gif width=8 height=8></td>
<td bgcolor=#557faa width=15> </td>
</tr>
<tr>
<td valign=bottom align=left bgcolor=#557faa><img src=art/oll.gif width=8 height=8></td>
<td colspan=4 bgcolor=#557faa> </td>
<td valign=bottom align=right bgcolor=#557faa><img src=art/olr.gif width=8 height=8></td>
</tr>
</table>
<address>Copyright 1997-2002 <a href=mailto:jeroen@fox-toolkit.org>Jeroen van der Zijp</a></address>
<!-- Created: Mon Apr 10 11:20:32 CEST 2000 -->
<!-- hhmts start -->
<!-- hhmts end -->
</body>
</html>
|
