1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
|
.. _testcode.py:
testcode.py
===========
.. only:: html
testcode.py - a command-line interface to testcode.
Synopsis
--------
testcode.py [options] [action1 [action2...]]
Description
-----------
Run a set of actions on a set of tests.
Requires two configuration files, :ref:`jobconfig` and :ref:`userconfig`. See
testcode documentation for further details.
testcode.py provides a command-line interface to testcode, a simple framework
for comparing output from (principally numeric) programs to previous output to
reveal regression errors or miscompilation.
Actions
-------
''run'' is th default action.
compare
compare set of test outputs from a previous testcode run against the
benchmark outputs.
diff
diff set of test outputs from a previous testcode run against the benchmark
outputs.
make-benchmarks
create a new set of benchmarks and update the :ref:`userconfig` file with
the new benchmark id. Also runs the 'run' action unless the 'compare'
action is also given.
recheck
compare set of test outputs from a previous testcode run against
benchmark outputs and rerun any failed tests.
run
run a set of tests and compare against the benchmark outputs.
tidy
Remove files from previous testcode runs from the test directories.
Options
-------
-h, --help
show this help message and exit
-b BENCHMARK, --benchmark=BENCHMARK
Set the file ID of the benchmark files. If BENCHMARK is in the format
t:ID, then the test files with the corresponding ID are used. This
allows two sets of tests to be compared. Default: specified in the [user]
section of the :ref:`userconfig` file.
-c CATEGORY, --category=CATEGORY
Select the category/group of tests. Can be specified multiple times.
Wildcards or parent directories can be used to select multiple directories
by their path. Default: use the `_default_` category if run is an action
unless make-benchmarks is an action. All other cases use the `_all_`
category by default. The `_default_` category contains all tests unless
otherwise set in the :ref:`jobconfig` file.
-e EXECUTABLE, --executable=EXECUTABLE
Set the executable(s) to be used to run the tests. Can be a path or name
of an option in the :ref:`userconfig` file, in which case all test programs are
set to use that value, or in the format program_name=value, which affects
only the specified program. Only relevant to the run action. Default: exe
variable set for each program listed in the :ref:`userconfig` file.
-i, --insert
Insert the new benchmark into the existing list of benchmarks in userconfig
rather than overwriting it. Only relevant to the make-benchmarks action.
Default: False.
--jobconfig=JOBCONFIG
Set path to the job configuration file. Default: jobconfig.
--job-option=JOB_OPTION
Override/add setting to :ref:`jobconfig`. Takes three arguments. Format:
section_name option_name value. Default: none.
--older-than=OLDER_THAN
Set the age (in days) of files to remove. Only relevant to the tidy
action. Default: 14 days.
-p NPROCS, --processors=NPROCS
Set the number of processors to run each test on. Only relevant to the run
action. Default: run tests as serial jobs.
-q, --quiet
Print only minimal output. Default: False.
-s QUEUE_SYSTEM, --submit=QUEUE_SYSTEM
Submit tests to a queueing system of the specified type. Only PBS system
is currently implemented. Only relevant to the run action. Default: none.
-t TEST_ID, --test-id=TEST_ID
Set the file ID of the test outputs. If TEST_ID is in the format b:ID, then
the benchmark files with the corresponding ID are used. This allows two
sets of benchmarks to be compared. Default: unique filename based upon
date if running tests and most recent test_id if comparing tests.
--total-processors=TOT_NPROCS
Set the total number of processors to use to run as many tests as possible
at the same time. Relevant only to the run option. Default: run all tests
concurrently run if --submit is used; run tests sequentially otherwise.
--userconfig=USERCONFIG
Set path to the user configuration file. Default: userconfig.
--user-option=USER_OPTION
Override/add setting to :ref:`userconfig`. Takes three arguments. Format:
section_name option_name value. Default: none.
-v, --verbose
Increase verbosity of output. Can be specified up to two times.
The default behaviour is to print out the test and its status. (See the
--quiet option to suppress even this.) Specify -v or --verbose once to
show (if relevant) which data values caused warnings or failures.
Specify -v or --verbose twice to see all (external) commands run and all
data extracted from running the tests. Using the maximum verbosity level
is highly recommended for debugging.
Exit status
-----------
1 if one or more tests fail (run and compare actions only) and 0 otherwise.
License
-------
Modified BSD License. See LICENSE in the source code for more details.
Bugs
----
Contact James Spencer (j.spencer@imperial.ac.uk) regarding bug reports,
suggestions for improvements or code contributions.
|
