1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
|
Running regression tests:
-------------------------
0) Prerequisites
Install the prerequisite helper programs on your system:
cmake, python3 (not 2), ImageMagick 6.5.9.3 or newer, diff
There are binary installer packages of these tools available for Mac,
Win, Linux, BSD, and other systems. (except maybe diff for Win)
Next, get a working qmake GUI build of the main openscad binary working.
For Windows(TM) this means get a cross-build working from within linux.
See README.md for how to do this.
Then, install MCAD under openscad/libraries.
$ cd openscad
$ git submodule update --init
A) Building test environment
NOTE: If cmake was used to build openscad itself (as opposed to the qmake build),
then the "cmake" and "make" commands are already automatically run for tests.
Only "cd tests" is needed to begin running ctest in that case.
Linux, Mac:
$ cd tests
$ cmake .
$ make
Windows(TM):
Cross-build from within linux:
64-bit:
$ ./scripts/release-common.sh mingw64 tests
$ # result is .zip file under ./mingw64/
32-bit:
$ ./scripts/release-common.sh mingw32 tests
$ # result is .zip file under ./mingw32/
B) Running tests
Linux, Mac:
$ ctest Runs tests enabled by default
$ ctest -R <regex> Runs only matching tests, e.g. ctest -R dxf
$ ctest -C <configs> Adds extended tests belonging to configs.
Valid configs:
Default - Run default tests
Heavy - Run more time consuming tests (> ~10 seconds)
Examples - test all examples
Bugs - test known bugs (tests will fail)
All - test everything
Win:
Unzip the OpenSCAD-Tests-YYYY.MM.DD file onto a Windows(TM) machine.
There will be a script called OpenSCAD-Test-Console.py in the parent folder.
Double-click it, and it will open a console, from which you can type the ctest
commands listed above.
C) Automatically upload test results (experimental)
It's possible to automatically upload tests results to an external
server. This is good for CI, as well as being able to easily report
bugs.
To enable this feature, add '-DOPENSCAD_UPLOAD_TESTS=1' to the cmake
cmd-line, e.g.: cmake -DOPENSCAD_UPLOAD_TESTS=1 .
Adding a new test:
------------------
1) create a test file at an appropriate location under testdata/
2) if the test is non-obvious, create a human readable description as comments in the test (or in another file in the same directory in case the file isn't human readable)
3) if a new test app was written, this must be added to tests/CMakeLists.txt
4) Add the tests to the test apps for which you want them to run (in tests/CMakeLists.txt)
5) rebuild the test environment
6) run the test with the environment variable TEST_GENERATE=1, e.g.:
$ TEST_GENERATE=1 ctest -R mytest
(this will generate a mytest-expected.txt file which is used for regression testing)
7) manually verify that the output is correct (tests/regression/<testapp>/mytest-expected.<suffix>)
8) run the test normally and verify that it passes:
$ ctest -R mytest
Adding a new example:
---------------------
This is almost the same as adding a new regression test:
1) Create the example under examples/
2) run the test with the environment variable TEST_GENERATE=1, e.g.:
$ TEST_GENERATE=1 ctest -C Examples -R exampleNNN
(this will generate a exampleNNN-expected.txt file which is used for regression testing)
3) manually verify that the output is correct (tests/regression/<testapp>/exampleNNN.<suffix>)
4) run the test normally and verify that it passes:
$ ctest -C Examples -R exampleNNN
Troubleshooting:
------------------------------
0. Headless unix servers
If you are attempting to run the tests on a unix-like system but only
have shell-console access, you may be able to run the tests by using a
virtual framebuffer program like Xvnc or Xvfb. For example:
$ Xvfb :5 -screen 0 800x600x24 &
$ DISPLAY=:5 ctest
or
$ xvfb-run ctest
Some versions of Xvfb may fail, however.
1. Trouble finding libraries on unix
To help CMAKE find eigen, OpenCSG, CGAL, Boost, and GLEW, you can use
environment variables, just like for the main qmake & openscad.pro. Examples:
OPENSCAD_LIBRARIES=$HOME cmake .
CGALDIR=$HOME/CGAL-3.9 BOOSTDIR=$HOME/boost-1.47.0 cmake .
Valid variables are as follows:
BOOSTDIR, CGALDIR, EIGENDIR, GLEWDIR, OPENCSGDIR, OPENSCAD_LIBRARIES
When running, this might help find your locally built libraries (assuming
you installed into $HOME)
Linux: export LD_LIBRARY_PATH=$HOME/lib:$HOME/lib64
Mac: export DYLD_LIBRARY_PATH=$HOME/lib
2. Location of logs
Logs of test runs are found in tests/build/Testing/Temporary
A pretty-printed index.html is in a subdir of tests/build/Testing/Temporary
Expected results are found in tests/regression/*
Actual results are found in tests/build/testname-output/*
3. Image-based tests takes a long time, they fail, and the log says 'return -11'
Imagemagick may have crashed while comparing the expected images to the
test-run generated (actual) images. You can try using the alternate
ImageMagick comparison method by by erasing CMakeCache, and re-running
cmake with -DCOMPARATOR=ncc. This will enable the Normalized Cross
Comparison method which is less accurate but won't usually crash.
4. Testing images fails with 'morphology not found" for ImageMagick in the log
Your version of imagemagick is old. Upgrade, or pass -DCOMPARATOR=old to
cmake. The comparison will be of lowered reliability.
5. Locale errors
"terminate called after throwing an instance of 'std::runtime_error'
what(): locale::facet::_S_create_c_locale name not valid"
Is a boost/libstdc++ bug. Fix like so before running:
$ export LC_MESSAGES=
6. I want to build without OpenGL
There is an unsupported way to do this, by defining NULLGL to Cmake:
mkdir nullglbin
cd nullglbin && cmake .. -DNULLGL=1 && make
The resulting openscad_nogui binary will fail most tests, but may be
useful for debugging and outputting 3d-formats like STL on systems without GL.
This option may break in the future and require tweaking to get working again.
7. Proprietary GL driver issues
There are sporadic reports of problems running on remote machines with
proprietary GL drivers. Try doing a web search for your exact error
message to see solutions and workarounds that others have found.
8. Windows + MSVC:
The MSVC build was last tested circa 2012. The last time it worked,
these were the necessary commands to run.
> Start the 'QT command prompt'
> cd \where\you\installed\openscad
> cd tests
> cmake . -DCMAKE_BUILD_TYPE=Release
> sed -i s/\/MD/\/MT/ CMakeCache.txt
> cmake .
> nmake -f Makefile
9. Other issues
The OpenSCAD User Manual Wiki has a section on buildling. Please check
there for possible updates and workarounds:
https://en.wikibooks.org/wiki/OpenSCAD_User_Manual
Please report build errors (after double checking the instructions) in
the github issue tracker
https://github.com/openscad/openscad/issues
10. Migration away from dedicated regression tests:
In 2013 the test programs underwent a major change. The following notes
are leftover.
"This test still needs an intermediate script that mangles away timestamps and
near-zero floating point numbers:
* cgalstlsanitytest
Some tests are yet to be converted:
* csgtexttest -- verify whether this is not redundant with dumptest
These look like tests, but are not actually in use:
* modulecachetest
* cgalcachetest
"
|
