1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
|
SystemC regression test suite
=============================
This is the SystemC regression test suite.
Note: for Windows users, please also refer to README_windows.txt
CONTENTS
========
1) Organization
2) Running tests
3) Adding new tests
1) Organization
===============
The regression test suite is organized as follows. At the top level,
we have a 'scripts' directory, an 'include' directory and several
tests directories (currently 'systemc' and 'tlm').
The 'scripts' directory contains the regression test
script(s). Right now, this directory contains a single script,
'verify.pl'. This script is written in perl. It works with perl5,
which must be located in /usr/bin, or in your path if running on Windows.
The 'include' directory contains top-level include files share across
tests. The directory 'include/<toplevel>' is automatically added to a
tests include path for tests underneath the '<toplevel>' tests
directory.
The tests directories 'systemc' and 'tlm' contain the regression tests.
Each of these contains other subdirectories. Actual tests are located
in leaf directories.
An actual test directory contains one or more source code files
(ending in .cpp). If the directory contains one source code file,
the name of the test is the filename (without extension) of that
source code file. If the directory contains more than one source
code file, there must be a file with extension .f that contains a
list of source code files (one line per source code file, and the
next higher directory must be added, e.g., 'arith_big/main.cpp'). In
this case, the name of the test is the filename (without extension)
of the .f file.
The actual test directory may contain property files that tell the
regression test script how to deal with the actual test, e.g. a
'COMPILE' file tells the regression test script to only compile the
source code file(s), and a 'DONTRUN' file tells the regression test
script to skip the actual test directory. A 'DONTRUN' file in a non-
test directory tells the regression test script to skip the tests in
all its subdirectories.
The actual test directory may contain a 'golden' directory, which
contains one or more golden reference files (stripped). The
filename of these files is the same as the test name (see above), and
the extension starts with '.log'. Platform specific golden reference
files append '.<platform>' to the file, e.g. 'test.log.linux64'.
2) Running tests
================
Regression tests can be run by calling the 'verify.pl' script. But,
_do not_ put this script in your path, because the path to the
script is used to determine where the regression tests are
located.
The 'verify.pl' script is located in the 'scripts' directory of the
SystemC regression test suite. It requires the environment variable
'SYSTEMC_HOME' to be set to your SystemC installation.
Furthermore, you must use the 'CXX' environment variable to select
the appropriate target architecture, just like with 'configure' when
configuring SystemC. In some cases (e.g. for testing cross-compiled
libraries), the automatic architecture detection can be overridden
by specifying the SystemC architecture name via the '-arch' option:
# assume a 64-bit compiler on Windows/MinGW32
> ../scripts/verify.pl -arch mingw64
Valid settings for 'CXX' on the different platforms are:
- Linux (multiple architectures):
o c++, g++ -> linux[64] (default)
- Mac OS X (multiple architectures):
o c++, g++ -> macos[x64|arm64]
- Solaris 2.7 and Solaris 2.8:
o c++, g++ -> gccsparcOS5 (default)
o CC -> sparcOS5
- HP-UX 11.00 (untested):
o c++, g++ -> gcchpux11
o aCC -> hpux11
- Windows (Cygwin 1.3 or later, MinGW32/MSYS 1.0.x)
Note:
To run the regression tests on Windows, a basic Un*x shell
environment like Cygwin or MinGW/MSYS is required.
See also: README_Windows.txt
o cl -> msvc80 (Visual C++ 2005)
-> msvc90 (Visual C++ 2008)
-> msvc10 (Visual C++ 2010)
-> msvc11 (Visual C++ 2012)
-> msvc12 (Visual C++ 2013)
-> msvc14 (Visual C++ 2015)
o c++, g++ -> cygwin (default on Cygwin)
-> cygwin64 (64-bit Cygwin, untested)
-> mingw (default on MinGW/MSYS)
-> mingw64 (64-bit MinGW-w64 compiler)
Default, the 'verify.pl' script assumes that the tests are located
in '../tests' (wrt the location of the script itself). This can be
overruled with environment variable 'SYSTEMC_TEST'.
The 'verify.pl' script takes directories and names of tests as
arguments. For example
> ../scripts/verify.pl systemc
runs all the SystemC tests in the regression test suite, and
> ../scripts/verify.pl tlm
runs the TLM2 tests, and
> ../scripts/verify.pl systemc tlm
runs everything.
The directories should be relative to '$SYSTEMC_TEST'.
By specifying subdirectories, it is possible to run only a subset
of the tests in the regression test suite, e.g.
> ../scripts/verify.pl systemc/datatypes/fx
will run only the fixed-point regression tests. And if the name of
the test/directory is unique, one can enter instead
> ../scripts/verify.pl fx
which will run the same tests.
There are several options that can be specified with the 'verify.pl'
script. Most notibly, '-f <file>' which can be used to have a file
containing the tests you want to run, '-v' to run with verbose
output, and '-g' to run with debug version of the SystemC Library.
See '-h' for a list of options.
Note:
Currently, not all options are fully implemented, and some may
disappear in the future.
The (temporary) results of the regression tests are stored in the
current directory, i.e., the directory from where the 'verify.pl'
script is called. Directories for tests that pass will be cleared
(and if possible removed), unless the '-no-cleanup' option is
specified. If a test fails all intermediate results will be
kept. The directory structure of the (temporary) results is the same
as the regression tests under '$SYSTEMC_TEST'. Furthermore, in
the current directory, a regression test log file called
'verify.log' is created. This file contains the output of running
each of the tests, and it contains a summary of which tests failed
(with the phase in which they failed) and which tests passed.
Beware:
Always create a _temporary_ directory to run the tests in. You
should _never_ run the tests in the directory in which the test
sources are located (i.e. '$SYSTEMC_TEST').
2.1) Running tests in parallel
==============================
To speed up testing an alternative to 'verify.pl' is to generate a
Makefile that runs all the tests. The tests can then be executed
using the 'make' command with a number of parallel jobs. This can
significantly reduce the time it takes to run the regressions during
development. Note that running 'verify.pl' is the canonical way of
executing the tests and it should always be run before committing
any changes to the repository.
The Makefile can be generated by running the script 'verify.pl' with
the option '-M'. It is also possible to run only a subset of the tests
in parallel. For example: 'make systemc/communication -j8' will run
all tests in the 'systemc/communication' sub-category.
3) Adding new tests
===================
When adding tests to the SystemC regression test suite, please stick
to the following rules.
- Group tests by feature.
- Contributions of new tests should be put in a directory next to
the 'systemc' directory. This directory could be called 'incoming'.
The person(s) responsible for the SystemC regression test suite can
then decide when and how to put these tests into the suite.
- Every test goes into its own directory. This directory must be a
leaf directory, that is, it can contain a 'golden' directory for
the golden reference files, but it cannot contain directories
holding other tests.
- Source code files must end in '.cpp'.
- If your test contains more than one source code file, you have to
add a '.f' file, which lists the source code files (don't forget
to prepend the current directory name -- just one level).
- When running the test the first time, it will fail, because you
don't have a golden reference file. To create this golden
reference file, do the following.
o Create the directory 'golden' in your test directory.
o Make sure that the temporary test file called
'<your_test>.log.stripped' is correct.
o Copy the above file in the 'golden' directory in your test
directory, and name it '<your_test>.log' or
'<your_test>.log.<platform>' if you have different results on
different platforms. If you do get different results on
different platforms, you have to make sure that these
differences are harmless and cannot be avoided (e.g. printing
floating-point numbers with or without leading 0).
o Now run your test again, and verify that it works now.
# Taf!
|
