File: README.testing

package info (click to toggle)
plplot 5.10.0%2Bdfsg2-0.4
  • links: PTS, VCS
  • area: main
  • in suites: stretch
  • size: 25,792 kB
  • ctags: 13,517
  • sloc: ansic: 83,001; xml: 27,081; ada: 18,878; cpp: 15,966; tcl: 11,651; python: 7,075; f90: 7,058; ml: 6,974; java: 6,665; perl: 5,029; sh: 2,208; makefile: 210; lisp: 75; sed: 25; fortran: 7
file content (214 lines) | stat: -rw-r--r-- 9,769 bytes parent folder | download | duplicates (3)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
The definitive set of instructions is given at
http://www.miscdebris.net/plplot_wiki/index.php?title=Testing_PLplot.  The
material below is cut and pasted from there as of 2010-03-14.

Testing PLplot

We encourage those who build PLplot to test both their build-tree
version and their installed-examples-tree version and report any
problems back to either the plplot-general or plplot-devel mailing
lists.

Contents
1 Testing Prerequisites
2 Build-tree tests 
2.1 Invocation of legacy tests in the build tree
2.2 Invocation of tests in the build tree using the new testing framework
3 Tests of the PLplot installation 
3.1 Legacy tests (Unix-only) of the PLplot installation
3.2 Cross-platform tests of the PLplot installation using the new test framework

1 Testing Prerequisites The legacy test for the build tree requires
ctest (available as part of CMake) and bash on Unix systems or
win-bash on Windows. The legacy test for the installed examples tree
works only on Unix (Linux, Mac OS X, and presumably legacy Unix) and
requires GNU-bash, make, and pkg-config. 

In contrast to our legacy testing system, the new testing framework is
essentially identical for both the build tree and installed examples
tree. It just requires CMake (using any generator that is suitable for
the Unix or Windows platform that is being used for the test), and
bash on Unix or win-bash on Windows.

2 Build-tree tests 

Build-tree tests can only be performed if cmake is invoked with the
-DBUILD_TEST=ON option. Such tests are done from the top-level
directory of the build tree (the directory where you invoke cmake
command that configures the build of PLplot). The methods for invoking
these tests are given below both for the legacy and new testing
methods. These tests include executing all our 31 standard examples
for all language interfaces and non-interactive device drivers that we
currently support. This is a comprehensive test of the PLplot
build. For example, our standard examples exercise virtually all of
the PLplot API. Furthermore, this series of tests generates more than
2GB of plot files in various formats. The tests also include a
comparison of PostScript (-dev psc) results and stdout from each
language interface to PLplot with the corresponding results for C. In
general, these results are identical which is a stringent test of our
language bindings. (Note it is the user's responsibility to insure the
locales are consistent for all languages since inconsistent locales
can produce inconsistent stdout results which have nothing to do with
PLplot bindings or examples issues.)

You should search test results for obvious errors such as
segfaults. In addition, you can test for rendering errors by viewing
the file results using an appropriate viewer.

Results for -dev psc are a special case. To illustrate this, here are
typical -dev psc results for example 1.

x01a.psc, x01c.psc, x01cxx.psc, x01d.psc, x01f.psc, x01f95.psc,
x01j.psc, x01lua.psc, x01o.psc, x01ocaml.psc, x01p.psc, x01pdl.psc,
and x01t.psc.

These correspond to Ada, C, C++, D, Fortran 77, Fortran 95, Java, Lua,
Octave, OCaml, Python, Perl/PDL, and Tcl -dev psc (colour PostScript)
results for our standard example 1. The test referred to above
compares everything in this list, but x01c.psc against that file so
rendering errors only need to be looked for in x01c.psc (with your
favorite Postscript viewing application), and any of these files which
the report shows are different from x01c.psc. And similarly for the
-dev psc results for the rest of our 31 standard examples.

Here are typical plot file results from our standard example 1 for
devices other than -dev psc.

x01c.pdfcairo, x01c.ps, x01c.psttf, x01c.psttfc, x01c01.bmpqt,
x01c01.epsqt, x01c01.jpgqt, x01c01.pdfqt, x01c01.pngcairo,
x01c01.pngqt, x01c01.ppmqt, x01c01.svg, x01c01.svgcairo, x01c01.svgqt,
x01c01.tiffqt, and x01c01.xfig.

Since different devices are involved in all cases, these should be
looked at individually for rendering errors (and similarly for the
remaining 31 standard examples). Note such visual inspection is a huge
job so we certainly don't expect it of our testers very often, but
once every year or so and especially for newer examples that haven't
been checked before is worth doing. On Unix platforms a general
all-purpose viewer for all these file formats is the ImageMagick
display application.

2.1 Invocation of legacy tests in the build tree 

After running "make all" from the top-level of the build tree, then run 

ctest --verbose >& ctest.out

This creates test plot file results in the plplot-test subdirectory of
the build tree, and ctest.out should contain a table of comparisons of
Postscript results from each of our standard examples and each of our
language bindings against the corresponding C versions.

2.2 Invocation of tests in the build tree using the new testing framework 

One advantage of the new testing framework is that it has full
dependencies implemented (unlike ctest which requires "make all" to be
run first).

A second advantage of the new testing framework is that for Generators
with parallel execution ability, you can save a lot of time on
hardware with more than one cpu by using the parallel execution
options (e.g., the -j option for GNU make).

A final advantage of the new testing framework is the tests are finer
grained. To see all the tests that are possible run make help |grep
test

(On platforms with CMake generators other than make, you will have to
do something equivalent to this search to find all test targets.)

Three comprehensive test targets are test_diff_psc,
test_noninteractive and test_interactive.

test_diff_psc generates all -dev psc results and compares them with
the same report that is obtained from ctest. Note this target excludes
anything but -dev psc results.

test_noninteractive runs test_diff_psc as well as every other PLplot
example that produces a file. (Note that test_noninteractive is
somewhat more comprehensive than legacy ctest and considerably more
comprehensive than the test_diff_psc target.)

test_interactive runs all interactive devices for the standard C
examples as well as all special interactive examples. Very little
user-intervention is required to run these tests because, where
possible, the PLplot -np (no-pause) command-line option is used for
these tests.

3 Tests of the PLplot installation

After PLplot has been configured (with "cmake"), built (with "make"),
and installed (with "make install"), you can test the installation
using a legacy test system (implemented with Make, pkg-config, and
GNU-bash) or our new test framework (implemented with CMake and bash
[or win-bash]).

3.1 Legacy tests (Unix-only) of the PLplot installation 

You can test the PLplot installation on Unix systems by doing the
following commands:

cp -a $prefix/share/plplot$plplot_version/examples /tmp
cd /tmp/examples
make test_noninteractive >& make_test.out
make test_interactive

where "$prefix" is the installation prefix chosen at the configuration
stage, and $plplot_version is the PLplot version (currently
5.9.4). The effect of the above "cp" and "cd" commands is to copy the
examples subtree of the install tree to /tmp and build and test the
examples in the copied subtree to keep a clean install tree. However,
an alternative is to replace those two commands with

cd $prefix/share/plplot$plplot_version/examples

and build and test the install-tree examples right in the examples
subtree of the install tree with the above "make" commands.

Regardless of whether you build and test the examples in a copy of the
examples subtree of the install tree or directly in that subtree,
check all the *.out files for any errors.

N.B. the above legacy "make test_noninteractive" command does the same
tests for the installed PLplot version as ctest does for the
build-tree version of PLplot. Although the tests are the same, the
Makefile implementation is different from the ctest implementation in
the build tree so that in particular you can do these install tree
tests in parallel using the (GNU-)make -j option to greatly speed up
these tests on a multi-processor computer. However, since make is
required for this installed-examples implementation, it cannot be run
on Windows.

N.B. the above legacy "make test_interactive" command executes our
interactive examples. The results are similar to those of the
test_interactive target implemented with our new test framework, but
the implementation is very different (depending on a GNU-bash script
rather than standard CMake cross-platform commands).

3.2 Cross-platform tests of the PLplot installation using the new test
framework

Here is an example under Unix of how to test the PLplot installation
using the new testing framework. (Those using Windows platforms should
be able to infer the equivalent of these commands.)

mkdir /tmp/build_dir
cd /tmp/build_dir
cmake $prefix/share/plplot$plplot_version/examples 
make -j4 test_diff_psc >& make_test_diff_psc.out
make -j4 test_noninteractive >& make_test_noninteractive.out
make -j4 test_interactive >& make_test_interactive.out

Note these targets are essentially identical to the targets described
above for the new test framework for the build tree because the same
bash (or win-bash) scripts and essentially the same CMake logic is
used to set up these targets. Similarly, all other fine-grained
targets (which you can discover using the

make help |grep test

command) for the new test framework are available in this context as
well. N.B. the test_noninteractive and test_interactive targets
available here are more comprehensive than the same-named targets in
the legacy installation test case and are implemented in a quite
different (cross-platform) way with much better dependencies.