1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
|
=head1 NAME
Test::LectroTest::Tutorial - How to use LectroTest to test your software
=head1 VERSION
version 0.5001
=head1 SYNOPSIS
LectroTest is an automated, specification-based testing system. To
use it, declare properties that specify the expected behavior of your
software. Then invoke LectroTest to test whether those properties
hold.
LectroTest does this by running repeated random trials against your
software. If LectroTest finds that a property doesn't hold, it emits
the counterexample that "broke" your software. You can then plug the
counterexample into your software to debug the problem. (It's also a
good idea to add the counterexample to your list of regression tests.)
=head1 OVERVIEW
Think of your software's behavior as a haystack that you're searching
for needles. Each error is a needle. You want to find the needles
and remove of them. LectroTest will search the haystack for you --
it's nice that way -- but first you must tell it about the shape of
the haystack and how to recognize a needle when it sees one.
=head2 The Haystack
The shape of the haystack is defined by a set of "generator bindings,"
in which variables are bound to the output of value generators:
x <- Int, c <- Char( charset=>"A-Z" )
The above can be read, "For all integers I<x> and all characters I<c>
in the range A through Z." The idea is that each unique instance
of the pair (I<x, c>) specifies a point in the haystack that
we can search for needles.
=head2 The Needle Recognizer
The "needle recognizer" is defined by a snippet of code that uses the
bound variables to inspect a given point in the haystack. It returns
a "thumbs up" (true) if the point is free of needles or a "thumbs
down" (false) if it finds a needle:
the_thing_we_are_testing($x, $c) >= 0;
The above asserts for each point in the haystack that the output
of the function C<the_thing_we_are_testing> must be non-negative.
=head2 Put them together to make a Property
The generator bindings and needle recognizer are combined to make a
property:
Property {
##[ x <- Int, c <- Char( charset=>"A-Z" ) ]##
the_thing_we_are_testing($x, $c) >= 0;
}, name => "the_thing_we_are_testing(...) is non-negative";
You'll note that we also added a meaningful name. Although not
strictly required, it's an excellent practice that makes life easier.
(You'll also note that we placed the generator bindings inside of the
magic delimiters C<##[ ]##>. This tells Perl that our bindings are
bindings and not regular Perl code.)
We can read the above property like so: "For all integers I<x> and all
characters I<c> in the range A through Z, we assert that
C<the_thing_we_are_testing> is non-negative."
=head2 Testing whether your Properties hold
After you define properties for your software, just add them to
a small Perl program that uses the Test::LectroTest module:
# MyProperties.l.t
use MyModule; # provides the_thing_we_are_testing
use Test::LectroTest;
Property {
##[ x <- Int, c <- Char( charset=>"A-Z" ) ]##
the_thing_we_are_testing($x, $c) >= 0;
}, name => "the_thing_we_are_testing(...) is non-negative";
Then you can test your properties simply by running the program:
$ perl MyProperties.l.t
If your properties check out, you'll see something like this:
1..1
ok 1 - 'the_thing_we_are_testing(...) is non-negative' (1000 attempts)
If something goes wrong, however, LectroTest will tell you where
it happened:
1..1
not ok 1 - 'the_thing_we_are_testing(...) is non-negative' \
falsified in 23 attempts
# Counterexample:
# $x = 4
# $c = "R"
What this says is that at the point (I<x>=4, I<c>="R") in the
haystack, there is a needle (i.e., your property doesn't hold).
With this information, you can examine your code to determine
the cause of the error.
=head1 LET'S DO IT!
Now that we have big-picture understanding of "LectroTesting," let's
try a few examples together.
[TODO: write the step-by-step tutorial examples. For now,
take a look at the slides from my LectroTest talk for two
such examples. The slides are available at the
L<LectroTest Home|/"LECTROTEST HOME">.]
=head1 SEE ALSO
L<Test::LectroTest> gives a quick overview of automatic,
specification-based testing with LectroTest.
L<Test::LectroTest::Property> explains in detail what
you can put inside of your property specifications.
L<Test::LectroTest::Generator> describes the many generators
and generator combinators that you can use to define the
shapes of the haystacks you encounter during your testing
adventures.
L<Test::LectroTest::TestRunner> describes the objects that check your
properties and tells you how to turn their control knobs. You'll want
to look here if you're interested in customizing the testing
procedure.
=head1 AUTHOR
Tom Moertel (tom@moertel.com)
=head1 INSPIRATION
The LectroTest project was inspired by Haskell's
QuickCheck module by Koen Claessen and John Hughes:
http://www.cs.chalmers.se/~rjmh/QuickCheck/.
=head1 COPYRIGHT and LICENSE
Copyright (c) 2004-13 by Thomas G Moertel. All rights reserved.
This program is free software; you can redistribute it and/or
modify it under the same terms as Perl itself.
=cut
|
