1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
|
PsimagLite v1.0
Copyright (c) 2011, UT-Battelle, LLC
All rights reserved
[by G.A., Oak Ridge National Laboratory]
[TestSuite contribution by E.P., Oak Ridge National Laboratory]
*********************************************************
DISCLAIMER
THE SOFTWARE IS SUPPLIED BY THE COPYRIGHT HOLDERS AND
CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
COPYRIGHT OWNER, CONTRIBUTORS, UNITED STATES GOVERNMENT,
OR THE UNITED STATES DEPARTMENT OF ENERGY BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
DAMAGE.
NEITHER THE UNITED STATES GOVERNMENT, NOR THE UNITED
STATES DEPARTMENT OF ENERGY, NOR THE COPYRIGHT OWNER, NOR
ANY OF THEIR EMPLOYEES, REPRESENTS THAT THE USE OF ANY
INFORMATION, DATA, APPARATUS, PRODUCT, OR PROCESS
DISCLOSED WOULD NOT INFRINGE PRIVATELY OWNED RIGHTS.
*********************************************************
The TestSuite is run as follows:
cd TestSuite
./testsuite.pl -a
to run all tests. Individual tests can be run with ./testsuite.pl -n N where N is the test number.
Results, and most importantly diffs, are written to
TestSuite/results
For example, for SPFv7 you will see, after running the TestSuite, files like:
TestSuite/results/dataN.diff
showing a diff between the "oracle" dataValidatedN.txt and the dataN.txt file computed with the current
(possibly buggy) version of the code.
And that's it, one then would check the diffs for each test to make sure diffs are zero or reasonably small.
(Stochastic tests, like those in SpinPhononFermion, need to be run with the same seed, of course.)
Because code changes are done in small stages from code version A --> code version B, if A is valid
but B is buggy, then we know the problem must be between A and B. Here the key words are "small changes"
between A and B. So, one can easily find the problem in version B. Were we to make "big changes" between
A and B then the TestSuite would be of no use, we'll just know that B is buggy but finding the problem
wouldn't be easy. That's why code development must be done in small stages at all times (except when writing
a code from scratch, of course)
---------------------------------------------------------------------------------------------
TestSuite apparatus.
Common files for the TestSuite (that is, common to all codes) are in PsimagLite/TestSuite
This follows usual practice of putting common files for multiple codes in PsimagLite.
Each code has a testsuite.pl script under its TestSuite directory (in SpinPhononFermion it's under v7/TestSuite/testsuite.pl)
which should be a very small script. Code "hooks" are under
spf/v7/TestSuite/TestSuiteSpf.pm
for SpinPhononFermion and
dmrgpp/TestSuite/TestSuiteDmrg.pm
for DMRG++, etc for other codes in the future.
The main features are:
* hashing of executables, so they don't have to be recompiled unless necessary
* powerful and extensible metalanguage for testing properties of results, not just raw data
---------------------------------------------------------------------------------------------
How tests are added.
Here I suppose we have an input file, the answers to the configure script, and the data that has been
"blessed," is considered valid, and will constitute the oracle.
The first thing is to give the test a number. Each test has a different number, let's assume it is N.
Then one puts the input file in
TestSuite/inputs/inputN.inp,
one creates a file with the answers to the configure script (one answer per line) in file:
TestSuite/inputs/modelN.spec,
then one creates the processing file that contains commands in a metalanguage (developed by E.P.) that tell the
TestSuite apparatus what to do. These instructions go in file:
TestSuite/inputs/processingN.txt
For SpinPhononFermion this will be mostly:
spf
observables
where the label spf tells the TestSuite to run SpinPhononFermion, and
the label observables, at least for now, just tells the TestSuite
to diff with the oracle. The metalanguage makes this extensible, you can see these commands with:
cat TestSuite/inputs/processingLibrary.txt
[spf]
Let $input = $inputsDir/input$testNum.inp
Let $raw = $resultsDir/stderrAndOut$testNum.txt
Execute runSpf($input,$raw)
[observables]
Let $result = $resultsDir/data$testNum
Let $oracle = $oraclesDir/dataValidated$testNum
Let $diff = $resultsDir/data$testNum.diff
Diff $result $oracle > $diff
One could process the raw data and get properties, and diff these properties, instead of the raw data. This is particularly
important in DMRG++ because the data files are so large. You can see the flexibility of this processing scheme in DMRG++'s TestSuite:
https://github.com/g1257/dmrgpp/blob/master/TestSuite/inputs/processingLibrary.txt
The valid data file needs to be copied to:
TestSuite/oracles/dataValidatedN.txt
---------------------------------------------------------------------------------------------
Limitations and Future Work:
* Each test could be run in parallel with trivial parallelization
* MPI is not supported
* smart diff could be improved
|
