1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214
|
The definitive set of instructions is given at
http://www.miscdebris.net/plplot_wiki/index.php?title=Testing_PLplot. The
material below is cut and pasted from there as of 2010-03-14.
Testing PLplot
We encourage those who build PLplot to test both their build-tree
version and their installed-examples-tree version and report any
problems back to either the plplot-general or plplot-devel mailing
lists.
Contents
1 Testing Prerequisites
2 Build-tree tests
2.1 Invocation of legacy tests in the build tree
2.2 Invocation of tests in the build tree using the new testing framework
3 Tests of the PLplot installation
3.1 Legacy tests (Unix-only) of the PLplot installation
3.2 Cross-platform tests of the PLplot installation using the new test framework
1 Testing Prerequisites The legacy test for the build tree requires
ctest (available as part of CMake) and bash on Unix systems or
win-bash on Windows. The legacy test for the installed examples tree
works only on Unix (Linux, Mac OS X, and presumably legacy Unix) and
requires GNU-bash, make, and pkg-config.
In contrast to our legacy testing system, the new testing framework is
essentially identical for both the build tree and installed examples
tree. It just requires CMake (using any generator that is suitable for
the Unix or Windows platform that is being used for the test), and
bash on Unix or win-bash on Windows.
2 Build-tree tests
Build-tree tests can only be performed if cmake is invoked with the
-DBUILD_TEST=ON option. Such tests are done from the top-level
directory of the build tree (the directory where you invoke cmake
command that configures the build of PLplot). The methods for invoking
these tests are given below both for the legacy and new testing
methods. These tests include executing all our 31 standard examples
for all language interfaces and non-interactive device drivers that we
currently support. This is a comprehensive test of the PLplot
build. For example, our standard examples exercise virtually all of
the PLplot API. Furthermore, this series of tests generates more than
2GB of plot files in various formats. The tests also include a
comparison of PostScript (-dev psc) results and stdout from each
language interface to PLplot with the corresponding results for C. In
general, these results are identical which is a stringent test of our
language bindings. (Note it is the user's responsibility to insure the
locales are consistent for all languages since inconsistent locales
can produce inconsistent stdout results which have nothing to do with
PLplot bindings or examples issues.)
You should search test results for obvious errors such as
segfaults. In addition, you can test for rendering errors by viewing
the file results using an appropriate viewer.
Results for -dev psc are a special case. To illustrate this, here are
typical -dev psc results for example 1.
x01a.psc, x01c.psc, x01cxx.psc, x01d.psc, x01f.psc, x01f95.psc,
x01j.psc, x01lua.psc, x01o.psc, x01ocaml.psc, x01p.psc, x01pdl.psc,
and x01t.psc.
These correspond to Ada, C, C++, D, Fortran 77, Fortran 95, Java, Lua,
Octave, OCaml, Python, Perl/PDL, and Tcl -dev psc (colour PostScript)
results for our standard example 1. The test referred to above
compares everything in this list, but x01c.psc against that file so
rendering errors only need to be looked for in x01c.psc (with your
favorite Postscript viewing application), and any of these files which
the report shows are different from x01c.psc. And similarly for the
-dev psc results for the rest of our 31 standard examples.
Here are typical plot file results from our standard example 1 for
devices other than -dev psc.
x01c.pdfcairo, x01c.ps, x01c.psttf, x01c.psttfc, x01c01.bmpqt,
x01c01.epsqt, x01c01.jpgqt, x01c01.pdfqt, x01c01.pngcairo,
x01c01.pngqt, x01c01.ppmqt, x01c01.svg, x01c01.svgcairo, x01c01.svgqt,
x01c01.tiffqt, and x01c01.xfig.
Since different devices are involved in all cases, these should be
looked at individually for rendering errors (and similarly for the
remaining 31 standard examples). Note such visual inspection is a huge
job so we certainly don't expect it of our testers very often, but
once every year or so and especially for newer examples that haven't
been checked before is worth doing. On Unix platforms a general
all-purpose viewer for all these file formats is the ImageMagick
display application.
2.1 Invocation of legacy tests in the build tree
After running "make all" from the top-level of the build tree, then run
ctest --verbose >& ctest.out
This creates test plot file results in the plplot-test subdirectory of
the build tree, and ctest.out should contain a table of comparisons of
Postscript results from each of our standard examples and each of our
language bindings against the corresponding C versions.
2.2 Invocation of tests in the build tree using the new testing framework
One advantage of the new testing framework is that it has full
dependencies implemented (unlike ctest which requires "make all" to be
run first).
A second advantage of the new testing framework is that for Generators
with parallel execution ability, you can save a lot of time on
hardware with more than one cpu by using the parallel execution
options (e.g., the -j option for GNU make).
A final advantage of the new testing framework is the tests are finer
grained. To see all the tests that are possible run make help |grep
test
(On platforms with CMake generators other than make, you will have to
do something equivalent to this search to find all test targets.)
Three comprehensive test targets are test_diff_psc,
test_noninteractive and test_interactive.
test_diff_psc generates all -dev psc results and compares them with
the same report that is obtained from ctest. Note this target excludes
anything but -dev psc results.
test_noninteractive runs test_diff_psc as well as every other PLplot
example that produces a file. (Note that test_noninteractive is
somewhat more comprehensive than legacy ctest and considerably more
comprehensive than the test_diff_psc target.)
test_interactive runs all interactive devices for the standard C
examples as well as all special interactive examples. Very little
user-intervention is required to run these tests because, where
possible, the PLplot -np (no-pause) command-line option is used for
these tests.
3 Tests of the PLplot installation
After PLplot has been configured (with "cmake"), built (with "make"),
and installed (with "make install"), you can test the installation
using a legacy test system (implemented with Make, pkg-config, and
GNU-bash) or our new test framework (implemented with CMake and bash
[or win-bash]).
3.1 Legacy tests (Unix-only) of the PLplot installation
You can test the PLplot installation on Unix systems by doing the
following commands:
cp -a $prefix/share/plplot$plplot_version/examples /tmp
cd /tmp/examples
make test_noninteractive >& make_test.out
make test_interactive
where "$prefix" is the installation prefix chosen at the configuration
stage, and $plplot_version is the PLplot version (currently
5.9.4). The effect of the above "cp" and "cd" commands is to copy the
examples subtree of the install tree to /tmp and build and test the
examples in the copied subtree to keep a clean install tree. However,
an alternative is to replace those two commands with
cd $prefix/share/plplot$plplot_version/examples
and build and test the install-tree examples right in the examples
subtree of the install tree with the above "make" commands.
Regardless of whether you build and test the examples in a copy of the
examples subtree of the install tree or directly in that subtree,
check all the *.out files for any errors.
N.B. the above legacy "make test_noninteractive" command does the same
tests for the installed PLplot version as ctest does for the
build-tree version of PLplot. Although the tests are the same, the
Makefile implementation is different from the ctest implementation in
the build tree so that in particular you can do these install tree
tests in parallel using the (GNU-)make -j option to greatly speed up
these tests on a multi-processor computer. However, since make is
required for this installed-examples implementation, it cannot be run
on Windows.
N.B. the above legacy "make test_interactive" command executes our
interactive examples. The results are similar to those of the
test_interactive target implemented with our new test framework, but
the implementation is very different (depending on a GNU-bash script
rather than standard CMake cross-platform commands).
3.2 Cross-platform tests of the PLplot installation using the new test
framework
Here is an example under Unix of how to test the PLplot installation
using the new testing framework. (Those using Windows platforms should
be able to infer the equivalent of these commands.)
mkdir /tmp/build_dir
cd /tmp/build_dir
cmake $prefix/share/plplot$plplot_version/examples
make -j4 test_diff_psc >& make_test_diff_psc.out
make -j4 test_noninteractive >& make_test_noninteractive.out
make -j4 test_interactive >& make_test_interactive.out
Note these targets are essentially identical to the targets described
above for the new test framework for the build tree because the same
bash (or win-bash) scripts and essentially the same CMake logic is
used to set up these targets. Similarly, all other fine-grained
targets (which you can discover using the
make help |grep test
command) for the new test framework are available in this context as
well. N.B. the test_noninteractive and test_interactive targets
available here are more comprehensive than the same-named targets in
the legacy installation test case and are implemented in a quite
different (cross-platform) way with much better dependencies.
|