2010-10-31 00:40:24 +04:00
Running regression tests:
-------------------------
2011-12-07 04:56:58 +04:00
Prerequisites: cmake, python, ImageMagick 6.5.9.3 or newer
2010-10-31 00:40:24 +04:00
2014-01-19 19:18:10 +04:00
First, get a working qmake GUI build of the main openscad binary and install MCAD.
See the main README.
2013-01-27 01:49:37 +04:00
2011-11-26 22:25:31 +04:00
A) Building test environment
2010-10-31 00:40:24 +04:00
2011-11-26 22:25:31 +04:00
Linux, Mac:
$ cd tests
$ cmake .
$ make
2011-12-31 20:02:55 +04:00
Windows + MSVC:
2011-10-16 05:14:35 +04:00
2013-01-27 01:49:37 +04:00
From the QT command prompt:
2011-10-16 05:14:35 +04:00
2011-12-07 04:56:58 +04:00
> cd tests
> cmake . -DCMAKE_BUILD_TYPE=Release
> sed -i s/\/MD/\/MT/ CMakeCache.txt
> cmake .
> nmake -f Makefile
2011-11-26 22:25:31 +04:00
2012-01-03 06:44:50 +04:00
Cross compiling Linux->Win32:
2011-12-31 20:02:55 +04:00
2012-01-03 06:44:50 +04:00
Please see openscad/tests/CMingw-cross-env.cmake for instructions.
2011-12-31 20:02:55 +04:00
2011-11-26 22:25:31 +04:00
B) Running tests
$ ctest Runs tests enabled by default
$ ctest -R <regex> Runs only matching tests, e.g. ctest -R dxf
$ ctest -C <configs> Adds extended tests belonging to configs.
Valid configs:
Default - Run default tests
Heavy - Run more time consuming tests (> ~10 seconds)
Examples - test all examples
2014-01-19 19:18:10 +04:00
Bugs - test known bugs (tests will fail)
2011-11-26 22:25:31 +04:00
All - test everything
2011-11-05 18:20:53 +04:00
2014-02-27 09:14:30 +04:00
C) Automatically upload test results (experimental)
It's possible to automatically upload tests results to an external
server. This is good for CI, as well as being able to easily report
bugs.
To enable this feature, add '-DOPENSCAD_UPLOAD_TESTS=1' to the cmake cmd-line, e.g.:
cmake -DOPENSCAD_UPLOAD_TESTS=1 .
2014-01-19 19:18:10 +04:00
Adding a new test:
------------------
2010-07-11 21:22:24 +04:00
1) create a test file at an appropriate location under testdata/
2012-02-07 04:42:11 +04:00
2) if the test is non-obvious, create a human readable description as comments in the test (or in another file in the same directory in case the file isn't human readable)
2011-09-04 04:58:09 +04:00
3) if a new test app was written, this must be added to tests/CMakeLists.txt
4) Add the tests to the test apps for which you want them to run (in tests/CMakeLists.txt)
5) run the test with the environment variable TEST_GENERATE=1, e.g.:
2011-04-29 20:02:54 +04:00
$ TEST_GENERATE=1 ctest -R mytest
2010-07-11 21:22:24 +04:00
(this will generate a mytest-expected.txt file which is used for regression testing)
2011-09-04 04:58:09 +04:00
6) manually verify that the output is correct (tests/regression/<testapp>/mytest-expected.<suffix>)
7) run the test normally and verify that it passes:
2010-07-11 21:22:24 +04:00
$ ctest -R mytest
2012-02-18 22:16:40 +04:00
Adding a new example:
---------------------
This is almost the same as adding a new regression test:
1) Create the example under examples/
2) run the test with the environment variable TEST_GENERATE=1, e.g.:
$ TEST_GENERATE=1 ctest -C Examples -R exampleNNN
(this will generate a exampleNNN-expected.txt file which is used for regression testing)
3) manually verify that the output is correct (tests/regression/<testapp>/exampleNNN.<suffix>)
4) run the test normally and verify that it passes:
$ ctest -C Examples -R exampleNNN
2013-06-25 15:28:30 +04:00
Migration away from dedicated regression tests:
-----------------------------------------------
2012-02-18 22:16:40 +04:00
2013-09-21 19:35:36 +04:00
This test still needs an intermediate script that mangles away timestamps and
2013-06-25 15:28:30 +04:00
near-zero floating point numbers:
2013-07-02 08:28:56 +04:00
* cgalstlsanitytest
2013-06-25 15:28:30 +04:00
Some tests are yet to be converted:
* csgtexttest -- verify whether this is not redundant with dumptest
These look like tests, but are not actually in use:
* modulecachetest
* cgalcachetest
2011-12-07 04:56:58 +04:00
Troubleshooting:
2011-09-19 02:53:33 +04:00
------------------------------
2011-12-26 22:34:58 +04:00
0. Headless unix servers
2011-12-07 07:48:19 +04:00
2011-12-26 22:34:58 +04:00
If you are attempting to run the tests on a unix-like system but only
have shell-console access, you may be able to run the tests by using a
virtual framebuffer program like Xvnc or Xvfb. For example:
$ Xvfb :5 -screen 0 800x600x24 &
2011-12-07 07:48:19 +04:00
$ DISPLAY=:5 ctest
2012-06-03 00:15:28 +04:00
or
$ xvfb-run ctest
2011-12-26 22:34:58 +04:00
Some versions of Xvfb may fail, however.
1. Trouble finding libraries on unix
2011-12-07 08:20:17 +04:00
2012-08-19 17:10:33 +04:00
To help CMAKE find eigen, OpenCSG, CGAL, Boost, and GLEW, you can use
2011-12-14 03:53:12 +04:00
environment variables, just like for the main qmake & openscad.pro. Examples:
2011-12-07 04:56:58 +04:00
2012-06-03 00:15:28 +04:00
OPENSCAD_LIBRARIES=$HOME cmake .
CGALDIR=$HOME/CGAL-3.9 BOOSTDIR=$HOME/boost-1.47.0 cmake .
2011-12-26 22:34:58 +04:00
Valid variables are as follows:
2011-12-14 03:53:12 +04:00
2012-08-19 17:10:33 +04:00
BOOSTDIR, CGALDIR, EIGENDIR, GLEWDIR, OPENCSGDIR, OPENSCAD_LIBRARIES
2011-12-07 07:48:19 +04:00
2012-06-03 00:15:28 +04:00
When running, this might help find your locally built libraries (assuming
you installed into $HOME)
2012-06-04 09:46:19 +04:00
Linux: export LD_LIBRARY_PATH=$HOME/lib:$HOME/lib64
Mac: export DYLD_LIBRARY_PATH=$HOME/lib
2012-06-03 00:15:28 +04:00
2011-12-26 22:34:58 +04:00
2. Location of logs
2011-12-07 04:56:58 +04:00
2011-09-19 02:53:33 +04:00
Logs of test runs are found in tests/build/Testing/Temporary
2011-12-26 22:34:58 +04:00
A pretty-printed index.html is in a subdir of tests/build/Testing/Temporary
2011-09-19 02:53:33 +04:00
Expected results are found in tests/regression/*
Actual results are found in tests/build/testname-output/*
2011-12-07 07:48:19 +04:00
2012-01-01 03:02:34 +04:00
3. Image-based tests takes a long time, they fail, and the log says 'return -11'
2011-12-07 08:20:17 +04:00
2011-12-26 22:34:58 +04:00
Imagemagick may have crashed while comparing the expected images to the
test-run generated (actual) images. You can try using the alternate
ImageMagick comparison method by by erasing CMakeCache, and re-running
cmake with -DCOMPARATOR=ncc. This will enable the Normalized Cross
Comparison method.
2011-12-07 08:20:17 +04:00
2012-01-01 03:02:34 +04:00
4. Testing images fails with 'morphology not found" for ImageMagick in the log
2011-12-07 08:31:59 +04:00
2011-12-14 03:53:12 +04:00
Your version of imagemagick is old. Upgrade, or pass -DCOMPARATOR=old to
cmake. The comparison will be of lowered reliability.
2011-12-07 10:47:15 +04:00
2012-06-03 00:42:36 +04:00
5. Locale errors
"terminate called after throwing an instance of 'std::runtime_error'
what(): locale::facet::_S_create_c_locale name not valid"
Is a boost/libstdc++ bug. Fix like so:
$ export LC_MESSAGES=
6. Other issues
2011-12-17 23:17:11 +04:00
2011-12-26 22:34:58 +04:00
The OpenSCAD User Manual has a section on buildling. Please check there
for updates:
2011-12-17 23:17:11 +04:00
http://en.wikibooks.org/wiki/OpenSCAD_User_Manual