2010-10-31 00:40:24 +04:00
Running regression tests:
-------------------------
2011-12-07 04:56:58 +04:00
Prerequisites: cmake, python, ImageMagick 6.5.9.3 or newer
2010-10-31 00:40:24 +04:00
2011-11-26 22:25:31 +04:00
A) Building test environment
2010-10-31 00:40:24 +04:00
2011-11-26 22:25:31 +04:00
Linux, Mac:
$ cd tests
$ cmake .
$ make
2011-12-31 20:02:55 +04:00
Windows + MSVC:
2011-10-16 05:14:35 +04:00
First, get a normal build working by following instructions at
http://en.wikibooks.org/wiki/OpenSCAD_User_Manual/Building_on_Windows
Then, from the QT command prompt:
2011-12-07 04:56:58 +04:00
> cd tests
> cmake . -DCMAKE_BUILD_TYPE=Release
> sed -i s/\/MD/\/MT/ CMakeCache.txt
> cmake .
> nmake -f Makefile
2011-11-26 22:25:31 +04:00
2012-01-03 06:44:50 +04:00
Cross compiling Linux->Win32:
2011-12-31 20:02:55 +04:00
2012-01-03 06:44:50 +04:00
Please see openscad/tests/CMingw-cross-env.cmake for instructions.
2011-12-31 20:02:55 +04:00
2011-11-26 22:25:31 +04:00
B) Running tests
$ ctest Runs tests enabled by default
$ ctest -R <regex> Runs only matching tests, e.g. ctest -R dxf
$ ctest -C <configs> Adds extended tests belonging to configs.
Valid configs:
Default - Run default tests
Heavy - Run more time consuming tests (> ~10 seconds)
Examples - test all examples
All - test everything
2011-11-05 18:20:53 +04:00
2011-12-14 03:53:12 +04:00
Adding a new regression test:
2010-10-31 00:40:24 +04:00
------------------------------
2010-07-11 21:22:24 +04:00
1) create a test file at an appropriate location under testdata/
2012-02-07 04:42:11 +04:00
2) if the test is non-obvious, create a human readable description as comments in the test (or in another file in the same directory in case the file isn't human readable)
2011-09-04 04:58:09 +04:00
3) if a new test app was written, this must be added to tests/CMakeLists.txt
4) Add the tests to the test apps for which you want them to run (in tests/CMakeLists.txt)
5) run the test with the environment variable TEST_GENERATE=1, e.g.:
2011-04-29 20:02:54 +04:00
$ TEST_GENERATE=1 ctest -R mytest
2010-07-11 21:22:24 +04:00
(this will generate a mytest-expected.txt file which is used for regression testing)
2011-09-04 04:58:09 +04:00
6) manually verify that the output is correct (tests/regression/<testapp>/mytest-expected.<suffix>)
7) run the test normally and verify that it passes:
2010-07-11 21:22:24 +04:00
$ ctest -R mytest
2012-02-18 22:16:40 +04:00
Adding a new example:
---------------------
This is almost the same as adding a new regression test:
1) Create the example under examples/
2) run the test with the environment variable TEST_GENERATE=1, e.g.:
$ TEST_GENERATE=1 ctest -C Examples -R exampleNNN
(this will generate a exampleNNN-expected.txt file which is used for regression testing)
3) manually verify that the output is correct (tests/regression/<testapp>/exampleNNN.<suffix>)
4) run the test normally and verify that it passes:
$ ctest -C Examples -R exampleNNN
2011-12-07 04:56:58 +04:00
Troubleshooting:
2011-09-19 02:53:33 +04:00
------------------------------
2011-12-26 22:34:58 +04:00
0. Headless unix servers
2011-12-07 07:48:19 +04:00
2011-12-26 22:34:58 +04:00
If you are attempting to run the tests on a unix-like system but only
have shell-console access, you may be able to run the tests by using a
virtual framebuffer program like Xvnc or Xvfb. For example:
$ Xvfb :5 -screen 0 800x600x24 &
2011-12-07 07:48:19 +04:00
$ DISPLAY=:5 ctest
2012-06-03 00:15:28 +04:00
or
$ xvfb-run ctest
2011-12-26 22:34:58 +04:00
Some versions of Xvfb may fail, however.
1. Trouble finding libraries on unix
2011-12-07 08:20:17 +04:00
To help CMAKE find eigen2, OpenCSG, CGAL, Boost, and GLEW, you can use
2011-12-14 03:53:12 +04:00
environment variables, just like for the main qmake & openscad.pro. Examples:
2011-12-07 04:56:58 +04:00
2012-06-03 00:15:28 +04:00
OPENSCAD_LIBRARIES=$HOME cmake .
CGALDIR=$HOME/CGAL-3.9 BOOSTDIR=$HOME/boost-1.47.0 cmake .
2011-12-26 22:34:58 +04:00
Valid variables are as follows:
2011-12-14 03:53:12 +04:00
2011-12-15 07:59:15 +04:00
BOOSTDIR, CGALDIR, EIGEN2DIR, GLEWDIR, OPENCSGDIR, OPENSCAD_LIBRARIES
2011-12-07 07:48:19 +04:00
2012-06-03 00:15:28 +04:00
When running, this might help find your locally built libraries (assuming
you installed into $HOME)
export LD_LIBRARY_PATH=$HOME/lib:$HOME/lib64
2011-12-26 22:34:58 +04:00
2. Location of logs
2011-12-07 04:56:58 +04:00
2011-09-19 02:53:33 +04:00
Logs of test runs are found in tests/build/Testing/Temporary
2011-12-26 22:34:58 +04:00
A pretty-printed index.html is in a subdir of tests/build/Testing/Temporary
2011-09-19 02:53:33 +04:00
Expected results are found in tests/regression/*
Actual results are found in tests/build/testname-output/*
2011-12-07 07:48:19 +04:00
2012-01-01 03:02:34 +04:00
3. Image-based tests takes a long time, they fail, and the log says 'return -11'
2011-12-07 08:20:17 +04:00
2011-12-26 22:34:58 +04:00
Imagemagick may have crashed while comparing the expected images to the
test-run generated (actual) images. You can try using the alternate
ImageMagick comparison method by by erasing CMakeCache, and re-running
cmake with -DCOMPARATOR=ncc. This will enable the Normalized Cross
Comparison method.
2011-12-07 08:20:17 +04:00
2012-01-01 03:02:34 +04:00
4. Testing images fails with 'morphology not found" for ImageMagick in the log
2011-12-07 08:31:59 +04:00
2011-12-14 03:53:12 +04:00
Your version of imagemagick is old. Upgrade, or pass -DCOMPARATOR=old to
cmake. The comparison will be of lowered reliability.
2011-12-07 10:47:15 +04:00
2012-01-01 03:02:34 +04:00
5. Other issues
2011-12-17 23:17:11 +04:00
2011-12-26 22:34:58 +04:00
The OpenSCAD User Manual has a section on buildling. Please check there
for updates:
2011-12-17 23:17:11 +04:00
http://en.wikibooks.org/wiki/OpenSCAD_User_Manual