1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
|
crontab executes run_nightly
This addition is also helpful
export PATH=.:/home/regression/bin:$PATH
To rebuild the HEAD executable without running any tests
./run_nightly.py --noregression --notestregression
To run regression tests without rebuilding HEAD
./run_nightly.py --noupdate
The structure of running nightly regressions is:
run_nightly (bash script to provide python environment)
run_nightly.py
build_revision.py build the requested revision (default is HEAD)
run_regression.py run that revision of gs on all testfiles with all paramsets
test_diff.py analyze checksums: compare this run with recent and with baseline
get_baseline_log.py summarize recent changes to the baseline update log
These script use the environment provided by gsconf.py, which reads the file testing.cfg;
as well as using the command line.
run_nightly.py
build_revision.py
run_regression.py
The data flow:
All directories are relative to gsconf.root (usually /home/regression)
build_revision.py
input
from svn gs repository
output
gs build tree, located at
gsroot (if revsion==HEAD) -
root/gs.<revision> (for specific revisions)
gs installation tree (only if revision==HEAD)
installtree
logs/<revision>.update*,make*,config*,
run_regression
input
comparefiles (usually /home/regression/comparefiles) - run all "appropriate" files (tested by file extension) using gs
gsparamsets.py - sets of output devices for gs testing
output
dailydir/<revision>.db (usually /home/regression/regression/daily) - store db with (testfile-device,checksum)
logs/<revision>.regression
logs/<revision>.gs*
<recent date> is determined by the existence of the most recent checksum database, before today
test_diff <recent date>
input
baselinedb (usually /home/regression/regression/baseline/baseline.db)
dailydir (usually /home/regression/regression/daily)
<previous revision>.db
<revision>.db
output
text report of differences. For all testfile-device entries in this revision's checksum db,
if the checksum does not match the previous revision, report as a new difference
if the checksum does not match the baseline, report as a difference
logs/<revision>.differences
get_baseline_log <recent date>
input
baseline_log (usually /home/regression/regression/baseline/baseline.log)
output
text report of updates to the baseline since the specified date
Details of checksums
all checksum db are in "dailydir" (usually /home/regression/regression/daily)
checksums are calculated using the python md5 module.
these dbs are files of anydb format, with key=testfile-device, value=checksum
New differences result from the comparison with a recent db.
All differences result from the comparison with the baseline.db
Each day (or revision) a new checksum db is created.
Each day, the new checksum db is compared with the most recent day's checksum db.
Each day the checksum db is also compared with the baseline checksum db (baseline.db).
A checksum log can contain 15,000 or more entries (number of test files * number of devices in gsparamsets)
bugs:
checksum logs that are used for comparison are still named by date, and it should be revision only.
each day's log is also copied to a db named by revision (but these are not yet used)
There are tools to examine the checksum dbs
dump_checksum.py - normal use. Be aware: you get LOTS of output
dump_checksum_raw.py - for debugging
dump_checksum_plus.py - specialized
collate.py - gather results of several dbs and sort
Details of testfiles:
all testfiles in the gsconf.comparefiledir are tested. Smaller subsets (in other directories) can be used for testing.
The normal situation:
For each testfile, there is a set of compressed raster files (one for each device).
For each raster file there is an entry in the baseline checksum db.
To make the raster file and baseline checksum db up-to-date, use update_baseline.py (--help for command line options)
Under the covers:
gsconf.py - provide an object with information about the testing environment
gstestgs.py - run gs
gsparamsets.py - provide sets of output devices (one set for "normal" and one for "pdfwrite")
gstestutils.py - utilities
gsutil.py - utilities
gssum.py - calculate checksums and insert into a checksum db
rasterdb.py - create and get compressed raster files
These scripts do the work of running gs and adding checksums to the daily checksum db:
gscheck_all.py - run the following - creating test cases for later execution
gscheck_raster.py - normal
for each test file, for each device create a normal test case
gscheck_pdfwrite.py - pdfwrite
for each test file, for each device create a pdfwrite test case
gscheck_testfiles.py - test only the files in a specified list
check_* - scripts to test the other aspects of the code base (dirs, comments, docrefs, source)
Repair:
regen_baseline.py
regenerate all checksums from the raster files (>15,000 files - takes a long time)
regen_filelist.py
regenerate checksums specified in a file list from the raster files (>15,000 files - takes a long time)
Tests with larger scope than nightly:
run_series.py - run_nightly.py with a specified range of revisions.
|
