1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
|
<html>
<head>
<meta http-equiv="Content-Type"
content="text/html; charset=iso-8859-1">
<meta name="GENERATOR" content="Microsoft FrontPage Express 2.0">
<title>TokenizerFunction Concept</title>
</head>
<body bgcolor="#FFFFFF" text="#000000" link="#0000EE"
vlink="#551A8B" alink="#FF0000">
<p><img src="../../c++boost.gif" alt="C++ Boost" width="277"
height="86"> </p>
<h1 align="center">TokenizerFunction Concept</h1>
<p>A TokenizerFunction is a functor whose purpose is to parse a
given sequence until exactly 1 token has been found or the end is
reached. It then updates the token, and informs the caller of the
location in the sequence of the next element immediately after
the last element of the sequence that was parsed for the current
token. </p>
<h2>Refinement of</h2>
<p>Assignable, CopyConstructable</p>
<h2>Notation</h2>
<table border="1">
<tr>
<td valign="top"><tt>X</tt> </td>
<td valign="top">A type that is a model of
TokenizerFunction</td>
</tr>
<tr>
<td valign="top"><tt>func</tt> </td>
<td valign="top">Object of type <tt>X</tt> </td>
</tr>
<tr>
<td valign="top"><tt>tok</tt></td>
<td valign="top">Object of Token</td>
</tr>
<tr>
<td>next</td>
<td>iterator that points to the first unparsed element of
the sequence being parsed</td>
</tr>
<tr>
<td>end</td>
<td>iterator that points to the past the end of the
sequence being parsed</td>
</tr>
</table>
<h2>Definitions</h2>
<p>A token is the result of parsing a sequence.</p>
<h2>Valid expressions</h2>
<p>In addition to the expression in Assignable and
CopyConstructable the following expressions are valid</p>
<table border="1">
<tr>
<th>Name </th>
<th>Expression </th>
<th>Return type </th>
</tr>
<tr>
<td valign="top">Functor</td>
<td valign="top"><tt>func(next, end, tok)</tt></td>
<td valign="top"><tt>bool</tt></td>
</tr>
<tr>
<td valign="top">reset</td>
<td valign="top"><tt>reset()</tt></td>
<td valign="top"><tt>void</tt></td>
</tr>
</table>
<h2>Expression semantics</h2>
<p>In addition to the expression semantics in Assignable and
CopyConstructable, TokenizerFunction has the following expression
semantcs</p>
<table border="1">
<tr>
<th>Name </th>
<th>Expression </th>
<th>Precondition </th>
<th>Semantics </th>
<th>Postcondition </th>
</tr>
<tr>
<td>operator()</td>
<td><tt>func(next, end, tok)</tt></td>
<td><tt>next</tt> and <tt>end</tt> are valid iterators to
the same sequence. next is a reference the function is
free to modify. tok is constructed. </td>
<td>The return value indicates whether a new token was
found in the sequence (next,end]</td>
<td>If the return value is true, the new token is
assigned to tok. next is always updated to the position
where parsing should start on the subsequent call.</td>
</tr>
<tr>
<td>reset</td>
<td><tt>reset()</tt></td>
<td><tt>None</tt></td>
<td>Clears out all state variables that are used by the
object in parsing the current sequence.</td>
<td>A new sequence to parse can be given.</td>
</tr>
</table>
<h2>Complexity guarantees</h2>
<p>No guarantees. Models of TokenizerFunction are free to define
their own complexity</p>
<h2>Models</h2>
<p><a href="escaped_list_separator.htm">escaped_list_separator</a></p>
<p><a href="offset_separator.htm">offset_separator</a></p>
<p><a href="char_delimiters_separator.htm">char_delimiters_separator</a></p>
<p> </p>
<p> </p>
<hr>
<p> Copyright John R. Bandela 2001. Permission to copy, use,
modify, sell and distribute this document is granted provided
this copyright notice appears in all copies. This document is
provided "as is" without express or implied warranty,
and with no claim as to its suitability for any purpose.</p>
</body>
</html>
|
