1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<HTML>
<HEAD>
<TITLE>Boost Test Library generic recommendations</TITLE>
<LINK rel="stylesheet" type="text/css" href="../style/btl.css" media="screen">
<LINK rel="stylesheet" type="text/css" href="../style/btl-print.css" media="print">
<META http-equiv="Content-Language" content="en-us">
<META http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
</HEAD>
<BODY>
<DIV class="header"> <A href="../index.html">Boost.Test</A> > <A href="../usage/recomendations.html">Tutorials
and usage recommendations</A> > <SPAN class="current_article"> Introduction into testing</SPAN></DIV>
<DIV class="body"> <IMG src="../btl1.gif" width="252" height="43" alt="Boost Test logo">
<H1>Introduction into testing</H1>
<P class="first-line-indented">For almost everyone, the first introduction
to the craft of programming is a version of the simple "Hello World" program.
In C++, this first example might be written as</P>
<PRE class="code">#<SPAN class="reserv-word">include</SPAN> <<SPAN class="literal">ostream</SPAN>>
<SPAN class="cpp-type">int</SPAN> main()
{
std::cout << <SPAN class="literal">"Hello World\n"</SPAN>;
}
</PRE>
<P>This is a good introduction for several reasons. One is that the program
is short enough, and the logic of its execution simple enough that direct
inspection can show whether it is correct in all use cases known to the new
student programmer. If this were the complexity of all programming, there
would be no need to test anything before using it. In programming as a new
student experiences it, testing is pointless and adds unneeded complexity.</P>
<P class="first-line-indented">However, no actual programs are as simple as
an introductory lesson makes "Hello World" seem. Not even "Hello
World". In
all real programs, there are decisions to be made and multiple paths of execution
based on these decisions. These decisions could be based on user input, streaming
data, resource availability and dozens of other factors. The programmer strives
to control the inputs, and results of these decisions, but no one can keep
all of them clearly in mind once the size of the project exceeds just a few
hundred lines. Even "Hello World" hides complexities of this sort in
the simple seeming call to std::cout.</P>
<P class="first-line-indented">Since the individual programmer can no longer
determine the correctness of the program, there is a need for a different
approach. An obvious possibility is testing the program after construction.
Someone develops a set of test cases, where inputs are given to the program
such that the behavior and outputs of a correctly performing program are
known. The performance of the new program is compared to known standards
and the new program either passes or fails. If it fails, attempts are made
to fix it. If the test cases are carefully chosen, the specifics of the failure
give an indication of what in the program needs to be fixed.</P>
<P class="first-line-indented">This is an improvement over just not knowing
whether the program is working properly, but it isn't a big improvement.
If the whole program is tested at once, it is nearly impossible to develop
test cases that clearly indicate what the failure is. The system is too complex,
and the programmer still needs to understand almost all of the possible outcomes
to be able to develop tests. As always, when a problem is too big and complicated
a good idea is to try splitting it into smaller and simpler pieces.</P>
<P class="first-line-indented">This approach leads to a layered system of testing,
that is similar to the layered approach to original development and should
be integrated into it. When writing a program, the design is factored into
small units that are conceptually and structurally easier to grasp. A standard
rule for this is that one unit performs one job or embodies one concept.
These simple units are composed into larger and more complicated algorithms
by passing needed information into a unit and receiving the desired result
out of it. The units are integrated to perform the whole task. Testing should
reflect this structure of development.</P>
<P class="first-line-indented">The simplest layer is Unit Testing. A unit is
the smallest conceptually whole segment of the program. Examples of basic
units might be a single class or a single function. For each unit, the tester
(who may or may not be the programmer) attempts to determine what states
the unit can encounter while executing as part of the program. These states
include determining the range of appropriate inputs to the unit, determining
the range of possible inappropriate inputs, and recognizing any ways the
state of the rest of the program might affect execution in this unit.</P>
<P class="first-line-indented">With so many general statements, an example
will help clarify. Imagine the following procedural function is part of a
program, and the programmer wants to test it. For the sake of brevity, header
includes and namespace qualifiers have been suppressed. </P>
<PRE class="code"><SPAN class="cpp-type">double</SPAN> find_root( <SPAN class="cpp-type">double</SPAN> (*f)(<SPAN class="cpp-type">double</SPAN>),
<SPAN class="cpp-type">double</SPAN> low_guess,
<SPAN class="cpp-type">double</SPAN> high_guess,
std::<SPAN class="cpp-type">vector</SPAN><<SPAN class="cpp-type">double</SPAN>>& steps,
<SPAN class="cpp-type">double</SPAN> tolerance )
{
<SPAN class="cpp-type">double</SPAN> solution;
<SPAN class="cpp-type">bool </SPAN> converged = <SPAN class="literal">false</SPAN>;
<SPAN class="reserv-word">while</SPAN>(not converged)
{
<SPAN class="cpp-type">double</SPAN> temp = (low_guess + high_guess) / <SPAN class="literal">2.0</SPAN>;
steps.push_back( temp );
<SPAN class="cpp-type">double</SPAN> f_temp = f(temp);
<SPAN class="cpp-type">double</SPAN> f_low = f(low_guess);
<SPAN class="reserv-word">if</SPAN>(abs(f_temp) < tolerance)
{
solution = temp;
converged = true;
}
<SPAN class="reserv-word">else </SPAN><SPAN class="reserv-word">if</SPAN>(f_temp / abs(f_temp) == f_low / abs(f_low))
{
low_guess = temp;
converged = false;
}
<SPAN class="reserv-word">else</SPAN>
{
high_guess = temp;
converged = false;
}
}
<SPAN class="reserv-word">return</SPAN> solution;
}
</PRE>
<P class="first-line-indented">This code, although brief and simple is getting
long enough that it takes attention to find what is done and why. It is no
longer obvious at a glance what the intent of the program is, so careful
naming must be used to carry that intent.</P>
<P class="first-line-indented">Thanks to the control structures, there are
some obvious execution paths in the code. However, there are also a few less
obvious paths. For example, if the root finder takes many steps to converge
to an acceptable answer, the vector that is holding the history of steps
taken may need to reallocate for additional space. In this case, there are
many hidden steps in the single push_back command. These steps also include
the chance of failure, since that is always a possibility in a memory allocation. </P>
<P class="first-line-indented">A second example notes that the value of the
function at the low guess has not been tested, so there is the chance of
a zero division. Also, if the value of the function at the high guess is
zero, the root finder will miss that root entirely. It may even fall into
an infinite loop if no root lies between the low and high values.</P>
<P class="first-line-indented">In this unit, proper testing includes checking
the behavior in each possibility. It also includes checking the function
by giving inputs where the correct answer is known and checking the results
against that answer. Thus, the unit is tested in every execution path to
assure proper behavior.</P>
<P class="first-line-indented">Test cases are chosen to expose as many errors
as possible. A defining characteristic of a good test case is that the programmer
knows what the unit should do if it is functioning properly. Test cases should
be generated to exercise each available execution path. For the above snippet,
this includes the obvious and the not so obvious paths. Every path should
be tested, since every path is a possible outcome of program execution.</P>
<P class="first-line-indented">Thus, to write a good testing suite, the tester
must know the structure of the code. The most dependable way to accomplish
this is if the original programmer writes tests as part of creating the code.
In fact, it is advisable that the tests are produced before the code is written,
and updated whenever structure decisions are changed. This way, the tests
are written with a view toward how the unit should perform instead of reproducing
the programmer's thinking from writing the code. While black box testing
is also useful, it is important that someone who knows the design decisions
made and the rationale for those decisions test the code unit. A programmer
who can't devise good tests for a unit does not yet know the problem at hand
well enough to program dependably.</P>
<P class="first-line-indented">When a unit is completed and tested, it is ready
for integration with other units in the program. This is integration should
also be tested. At this point, the test cases focus on the interaction between
the units. Tests are designed to exercise each way the units can affect each
other.</P>
<P class="first-line-indented">This is the point in development where proper
unit testing really shines. If each unit is doing what it should be doing
and not creating unexpected side effects, any issues in testing a set of
integrated units must come from how they are passing information. Thus, the
nearly intractable problem of finding an error while many units interact
becomes the less intimidating problem of finding the breakdown in communications.</P>
<P class="first-line-indented">At each layer of increasing complexity, new
tests are run, and if the prior tests of the components are well designed
and all issues are fixed, new errors are isolated to the integration. This
process continues, in parallel with development, from the smallest units
to the completed program.</P>
<P class="first-line-indented">This shows that there is a need to be able to
check and test code snippets such as individual functions and classes independent
of the program of which they will become a part. That is, the need for a
means to provide predetermined inputs to the unit and to check the outputs
against expected results. Such a system must allow for both normal operation
and error conditions, and allow the programmer to produce a thorough description
of the results.</P>
<P class="first-line-indented">This is the goal and rationale for all unit
testing, and supporting testing of this sort is the purpose of the Boost.Test
library. As is shown below, Boost.Test provides a well-integrated set of
tools to support this testing effort throughout the programming and maintenance
cycles of software development.</P>
</DIV>
<DIV class="footer">
<DIV class="footer-body">
<P> © <A name="Copyright">Copyright</A> <A href='mailto:jphillip at capital dot edu (please unobscure)'>John
R. Phillips</A> 2006. <BR>
Distributed under the Boost Software License, Version 1.0. (See accompanying
file <A href="../../../../LICENSE_1_0.txt">LICENSE_1_0.txt</A> or copy
at <A href="http://www.boost.org/LICENSE_1_0.txt">www.boost.org/LICENSE_1_0.txt</A>)</P>
<P>Revised:
<!-- #BeginDate format:Sw1 -->28 February, 2006<!-- #EndDate -->
</P>
</DIV>
</DIV>
</BODY>
</HTML>
|
