1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
|
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta http-equiv="Content-Language" content="en-us">
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="GENERATOR" content="Microsoft FrontPage 6.0">
<meta name="ProgId" content="FrontPage.Editor.Document">
<title>Introduction</title>
</head>
<body bgcolor="#FFFFFF">
<p><img src="../../boost.png" alt="C++ Boost" width="277" height=
"86"><br></p>
<h1 align="center">Introduction</h1>
<p align="left">The boost Tokenizer package provides a flexible and easy to
use way to break of a string or other character sequence into a series of
tokens. Below is a simple example that will break up a phrase into
words.</p>
<div align="left">
<pre>
// simple_example_1.cpp
#include<iostream>
#include<boost/tokenizer.hpp>
#include<string>
int main(){
using namespace std;
using namespace boost;
string s = "This is, a test";
tokenizer<> tok(s);
for(tokenizer<>::iterator beg=tok.begin(); beg!=tok.end();++beg){
cout << *beg << "\n";
}
}
</pre>
</div>
<p align="left">You can choose how the string gets broken up. You do this
by specifying the TokenizerFunction. If you do not specify anything, the
default TokenizerFunction is char_delimiters_separator<char> which
defaults to breaking up a string based on space and punctuation. Here is an
example of using another TokenizerFunction called escaped_list_separator.
This TokenizerFunction parses a superset of comma separated value (csv)
lines. The format looks like this</p>
<p align="left">Field 1,"putting quotes around fields, allows commas",Field
3</p>
<p align="left">Below is an example that will break the previous line into
its 3 fields</p>
<div align="left">
<pre>
// simple_example_2.cpp
#include<iostream>
#include<boost/tokenizer.hpp>
#include<string>
int main(){
using namespace std;
using namespace boost;
string s = "Field 1,\"putting quotes around fields, allows commas\",Field 3";
tokenizer<escaped_list_separator<char> > tok(s);
for(tokenizer<escaped_list_separator<char> >::iterator beg=tok.begin(); beg!=tok.end();++beg){
cout << *beg << "\n";
}
}
</pre>
</div>
<p align="left">Finally, for some TokenizerFunctions you have to pass in
something into the constructor in order to do anything interesting. An
example is offset_separator. This class breaks a string into tokens based
on offsets for example</p>
<p align="left">12252001 when parsed using offsets of 2,2,4 becomes 12 25
2001. Below is an example to parse this.</p>
<div align="left">
<pre>
// simple_example_3.cpp
#include<iostream>
#include<boost/tokenizer.hpp>
#include<string>
int main(){
using namespace std;
using namespace boost;
string s = "12252001";
int offsets[] = {2,2,4};
offset_separator f(offsets, offsets+3);
tokenizer<offset_separator> tok(s,f);
for(tokenizer<offset_separator>::iterator beg=tok.begin(); beg!=tok.end();++beg){
cout << *beg << "\n";
}
}
</pre>
</div>
<p align="left"> </p>
<hr>
<p><a href="http://validator.w3.org/check?uri=referer"><img border="0" src=
"http://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01 Transitional"
height="31" width="88"></a></p>
<p>Revised
<!--webbot bot="Timestamp" s-type="EDITED" s-format="%d %B, %Y" startspan -->25 December, 2006<!--webbot bot="Timestamp" endspan i-checksum="38518" --></p>
<p><i>Copyright © 2001 John R. Bandela</i></p>
<p><i>Distributed under the Boost Software License, Version 1.0. (See
accompanying file <a href="../../LICENSE_1_0.txt">LICENSE_1_0.txt</a> or
copy at <a href=
"http://www.boost.org/LICENSE_1_0.txt">http://www.boost.org/LICENSE_1_0.txt</a>)</i></p>
</body>
</html>
|
