1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
|
gpscorrelate Test Framework
===========================
The gpscorrelate test suite consists of a simple framework plus test data
that allows basic regression tests to be run to validate the gpscorrelate
code. It is designed primarily to run tests on the "gpscorrelate" executable
but is flexible enough to be used as the test framework to test other
executables as well.
The test framework provides the facility to compare the output of the
program under test with an expected file, first massage that file to make it
easier to compare, perform tests that are expected to fail, and
perform arbitrary checks following the test. It is driven by a test
parameters file which contains instructions for running that test,
plus a data file containing the data expected after running the test.
Besides a locally-built version of gpscorrelate and some standard UNIX
text utilities found on every system, the test suite also requires the
"exiv2" binary executable.
TEST FILE FORMAT
================
Each test parameters file has a name of the form testN.param where N
is the test number. it consists of a series of variable assignments
of the form 'VARIABLE=data'. Several pre-set variables are available
for use in the data. Bourne shell quoting rules are in effect, and
the order does not matter.
These are the variables that can be set in the test parameters file:
COMMAND - The command to run to execute the test (mandatory)
PRECOMMAND - A command to run before COMMAND (which must have a 0
exit status)
POSTCOMMAND - A command to run after COMMAND (which must have a 0
exit status)
RESULTCODE - The return status expected by the command (default 0)
RESULTFILE - Path to a file containing the expected output of the program; this
will be compared to $OUTFILE (default result/testN.result)
SEDCOMMAND - A command passed to sed (with extended regexes) which modifies the
output file $OUTFILE; this can be used to remove parts of the file (like
absolute paths) that can be inconsistent between test runs
SKIP - Set to "true" if this test should be skipped (useful temporarily during
development)
TITLE - A short descriptive name for the test (mandatory)
These are the variables that are available for use in the parameters file:
$DATADIR - path to the data directory (data/)
$ERRFILE - path to the default standard error output file for this test
$LOGDIR - path to the log directory (log/)
$OUTFILE - path to the default test output file for this test
$PROGRAM - path to the gpscorrelate executable (../gpscorrelate); this should
not be quoted in the params file so that the user can specify
a helper app like valgrind or gdb along with gpscorrelate
$STAGINGDIR - path to an area containing sample files and directories
(staging/)
FILES AND DIRECTORIES
=====================
Test parameter files and any data files are found in the data/ directory.
Test parameter files end with the .param extension, while data files end
with the .result extension. Files created while running the test are
stored in the log/ directory. The contents of log/ is erased before the
start of each test run.
The directory staging/ contains a few small files that can be used as test
input. It should be treated as read only, and should not be modified.
Any test that expects to modify files must first copy them from staging
into $LOGDIR in the PRECOMMAND.
INVOCATION
==========
Start the tests with the command "./testsuite". If parameters are given,
they are to specific test case files to run,
e.g. "./testsuite data/test002.param" will run only test 2. When
running outside the build tree, the environment variable "srcdir" must
be set to the absolute path of the tests source directory. The program
will exit with a non-zero status code if at least one test failed.
If the -v option is given, more verbose information about test failures
is provided. If the -m option is given, the program is run with valgrind's
memcheck tool.
Some test cases don't do a good job of cleaning up after themselves after
failures. If you get persistent test failures, try deleting the 'log'
directory and running the tests again.
The test suite has been tested using exiv2 0.21.1 and libxml2 2.9.3. Older
versions have slightly different output that causes the test output comparison
to fail. There is a hack in testsuite that can be used to relax the comparison
and use such an older version of exiv2. When using MUSL 1.1, an issue allows
exiv2 to erronously overwrite read-only files which causes tests that check
this condition to fail.
|
