1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html><head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>SQLite TH3</title>
<style type="text/css">
body {
margin: auto;
font-family: Verdana, sans-serif;
padding: 8px 1%;
}
a { color: #044a64 }
a:visited { color: #734559 }
.logo { position:absolute; margin:3px; }
.tagline {
float:right;
text-align:right;
font-style:italic;
width:300px;
margin:12px;
margin-top:58px;
}
.menubar {
clear: both;
border-radius: 8px;
background: #044a64;
padding: 0px;
margin: 0px;
cell-spacing: 0px;
}
.toolbar {
text-align: center;
line-height: 1.6em;
margin: 0;
padding: 0px 8px;
}
.toolbar a { color: white; text-decoration: none; padding: 6px 12px; }
.toolbar a:visited { color: white; }
.toolbar a:hover { color: #044a64; background: white; }
.content { margin: 5%; }
.content dt { font-weight:bold; }
.content dd { margin-bottom: 25px; margin-left:20%; }
.content ul { padding:0px; padding-left: 15px; margin:0px; }
/* Things for "fancyformat" documents start here. */
.fancy img+p {font-style:italic}
.fancy .codeblock i { color: darkblue; }
.fancy h1,.fancy h2,.fancy h3,.fancy h4 {font-weight:normal;color:#044a64}
.fancy h2 { margin-left: 10px }
.fancy h3 { margin-left: 20px }
.fancy h4 { margin-left: 30px }
.fancy th {white-space:nowrap;text-align:left;border-bottom:solid 1px #444}
.fancy th, .fancy td {padding: 0.2em 1ex; vertical-align:top}
.fancy #toc a { color: darkblue ; text-decoration: none }
.fancy .todo { color: #AA3333 ; font-style : italic }
.fancy .todo:before { content: 'TODO:' }
.fancy p.todo { border: solid #AA3333 1px; padding: 1ex }
.fancy img { display:block; }
.fancy :link:hover, .fancy :visited:hover { background: wheat }
.fancy p,.fancy ul,.fancy ol { margin: 1em 5ex }
.fancy li p { margin: 1em 0 }
/* End of "fancyformat" specific rules. */
</style>
</head>
<body>
<div><!-- container div to satisfy validator -->
<a href="index.html">
<img class="logo" src="images/sqlite370_banner.gif" alt="SQLite Logo"
border="0"></a>
<div><!-- IE hack to prevent disappearing logo--></div>
<div class="tagline">Small. Fast. Reliable.<br>Choose any three.</div>
<table width=100% class="menubar"><tr>
<td width=100%>
<div class="toolbar">
<a href="about.html">About</a>
<a href="sitemap.html">Sitemap</a>
<a href="docs.html">Documentation</a>
<a href="download.html">Download</a>
<a href="copyright.html">License</a>
<a href="news.html">News</a>
<a href="support.html">Support</a>
</div>
<script>
gMsg = "Search SQLite Docs..."
function entersearch() {
var q = document.getElementById("q");
if( q.value == gMsg ) { q.value = "" }
q.style.color = "black"
q.style.fontStyle = "normal"
}
function leavesearch() {
var q = document.getElementById("q");
if( q.value == "" ) {
q.value = gMsg
q.style.color = "#044a64"
q.style.fontStyle = "italic"
}
}
function hideorshow(btn,obj){
var x = document.getElementById(obj);
var b = document.getElementById(btn);
if( x.style.display!='none' ){
x.style.display = 'none';
b.innerHTML='show';
}else{
x.style.display = '';
b.innerHTML='hide';
}
return false;
}
</script>
<td>
<div style="padding:0 1em 0px 0;white-space:nowrap">
<form name=f method="GET" action="http://www.sqlite.org/search">
<input id=q name=q type=text
onfocus="entersearch()" onblur="leavesearch()" style="width:24ex;padding:1px 1ex; border:solid white 1px; font-size:0.9em ; font-style:italic;color:#044a64;" value="Search SQLite Docs...">
<input type=submit value="Go" style="border:solid white 1px;background-color:#044a64;color:white;font-size:0.9em;padding:0 1ex">
</form>
</div>
</table>
<div class=startsearch></div>
<h1 align="center">TH3: SQLite Test Harness #3</h1>
<h2>1.0 Overview</h2>
<p>SQLite Test Harness #3 (hereafter "TH3") is one of
<a href="testing.html#harnesses">three test harnesses</a> used for testing SQLite.
TH3 is designed to meet the following objectives:</p>
<ul>
<li><p> TH3 is able to run on embedded platforms that lack the support
infrastructure of workstations.</p></li>
<li><p> TH3 tests SQLite in an as-deployed configuration using only
published and documented interfaces.
In other words, TH3 tests the compiled object code, not
the source code, thus verifying that no problems were introduced
by compiler bugs. "Fly what you test and test what you fly."</p></li>
<li><p> TH3 checks SQLite's response to out-of-memory errors, disk I/O
errors, and power loss during transaction commit. </p></li>
<li><p> TH3 exercises SQLite in a variety of run-time configurations
(UTF8 vs UTF16, different pages sizes, varying journal modes, etc.)
</p></li>
<li><p> TH3 achieves 100% branch test coverage (and 100% MC/DC)
over the SQLite core.
(Test coverage of the operating-system specific <a href="vfs.html">VFSes</a> and extensions
such as FTS and RTREE is less than 100%). </p></li>
</ul>
<p>TH3 was originally written for validation testing only, but has
subsequently been used for development testing and debugging
as well, and has proven very helpful in those roles. A full-coverage
test takes less than three minutes on a workstation and hence
serves as a fast regression test during day-to-day maintenance
of the SQLite code base.</p>
<h2>2.0 Operation</h2>
<p>TH3 is a test program generator. The output of TH3 is a program
written in ANSI-C and intended to be
linked against the SQLite library under test. The generated test
program is compiled and run on the target platform in order to verify
correct operation of SQLite on that platform.</p>
<p>The inputs to TH3 are test modules written in C or SQL and
small configuration
files that determine how to initialize SQLite. The
TH3 package includes over one thousand test
modules and several dozen configuration files.
New modules and configurations
can be added to customize TH3 for specialized applications.
Each time TH3 is run, it reads
a subset of the available test modules and configuration files to generate
a custom C program that performs all of the specified tests under all
configurations. A complete test of SQLite normally involves running
TH3 multiple times to generate multiple test programs covering different
aspects of SQLite's operation, then linking all test programs against
a common SQLite library and running them separately on the target platform.
SQLite will be found to work if all test programs pass.</p>
<p>There are no arbitrary limits in TH3. One could generate a
single test program that contained all test modules and configuration files.
However, such a test program might be too large to deploy on embedded
platforms. Hence, TH3 provides the ability to break the library of test
modules up into smaller, more easily digested pieces.</p>
<p>Each individual test module might contain dozens, hundreds, or thousands
of separate tests. The test modules can be written in C or as scripts of
SQL or a mixture of the two. About two-thirds of the existing test modules are
written SQL with the remainder either in pure C or a combination of C and SQL.
</p>
<p>Each test module file contains a header which describes the circumstances
under which the test is valid. For a particular configuration, only those
modules that are compatible with the configuration are run. </p>
<h2>3.0 Generating A Test Program</h2>
<p>The TH3 program generator is a TCL script named "<tt>mkth3.tcl</tt>".
To generate a test program, one has merely to run this script and supply
the names of files containing test modules and configurations on the
command line. Test modules are files that use the "<tt>.test</tt>" suffix
and configurations are files that use the "<tt>.cfg</tt>" suffix. A
typical invocation of mkth3.tcl might look something like the following:</p>
<blockquote><pre>
tclsh mkth3.tcl *.test *.cfg >testprog1.c
</pre></blockquote>
<p>The output from the mkth3.tcl script is a C program that contains
everything needed to run the tests - everything that is except for
the SQLite library itself. The generated test program contains
implementations for all of the support interfaces used by the test
modules and it contains the <tt>main()</tt> routine that drives the
tests. To convert the test program into a working executable, simply
compile it against SQLite:</p>
<blockquote><pre>
cc -o testprog1 testprog1.c sqlite3.c
</pre></blockquote>
<p>The compilation step shown immediately above is merely representative.
In a working installation, one would normally want
to specify optimization parameters and compile-time switches on the
compiler command line.</p>
<p>For testing on embedded systems, the mkth3.tcl script and the compiler
steps shown above are performed on an ordinary workstation using
a cross-compiler, then the resulting test program is
transfer onto the device to be run.</p>
<p>Once the test program is generated, it is run with no arguments to
perform the tests. Progress information as well as error diagnostics
appear on standard output. (Alternative output arrangements can be made
using a compile-time option for embedded devices that lack a standard
output channel.) The program returns zero if there are no
errors and non-zero if any problems were detected.</p>
<p>TH3 comes with additional TCL scripts (for example:
"th3make" and "fulltest.tcl")
which help to automate the process of running mkth3.tcl, compiling th3.c and
sqlite3.c, running the resulting test programs, and analyzing the results.
These other scripts are for convenience only and are not required in order
to make use of TH3.
<h2>4.0 Test Coverage</h2>
<p>Using one particular subset of the available TH3 test modules (the "cov1"
tests) SQLite obtained
<a href="testing.html#coverage">100% branch test coverage</a> and 100% <a href="testing.html#mcdc">MC/DC</a> as measured
by <a href="http://gcc.gnu.org/onlinedocs/gcc/Gcov.html">gcov</a>
on Linux x86 and x86_64 hardware. All releases of SQLite since
<a href="releaselog/3_6_17.html">version 3.6.17</a> (circa 2009-08-10) have been tested to this standard.
The SQLite developers
are committed to maintaining 100% branch coverage and MC/DC for all
future releases of SQLite.</p>
<p>The cov1 test set used to obtain 100% branch test coverage are only a
subset of the tests currently implemented using TH3. New test modules are
added on a regular basis.</p>
<h2>5.0 TH3 License</h2>
<p>SQLite itself is in the <a href="copyright.html">public domain</a> and
can be used for any purpose. But TH3 is proprietary and requires a license.
Members of the <a href="consortium.html">SQLite Consortium</a> get free and unlimited access to TH3.
Others can contact the SQLite developers for information on how to obtain
a license to access and use TH3.</p>
<p>Licensees of TH3 are given read access to the software configuration
management system used to manage TH3 and so can download the latest version
of TH3 (or any historical version) whenever they like.</p>
<p>Even though open-source users do not have direct access to TH3, all
users of SQLite benefit from TH3 indirectly since each version of SQLite is
validated running TH3 on multiple platforms (Linux, Windows, WinRT, Mac,
OpenBSD, Solaris Sparc) prior to release. So anyone using an official release
of SQLite can deploy their application with the confidence of knowing that
it has been tested using TH3. They simply cannot rerun those tests
themselves without purchasing a TH3 license.</p>
|
