2010-10-31 00:40:24 +04:00
Running regression tests:
-------------------------
2011-12-07 04:56:58 +04:00
Prerequisites: cmake, python, ImageMagick 6.5.9.3 or newer
2010-10-31 00:40:24 +04:00
2013-09-13 05:03:27 +04:00
First, get a working qmake GUI build of the main openscad binary. It is
used by the tests. Next, get MCAD installed by using
'git submodule update --init'
2013-01-27 01:49:37 +04:00
2011-11-26 22:25:31 +04:00
A) Building test environment
2010-10-31 00:40:24 +04:00
2011-11-26 22:25:31 +04:00
Linux, Mac:
$ cd tests
$ cmake .
$ make
2011-12-31 20:02:55 +04:00
Windows + MSVC:
2011-10-16 05:14:35 +04:00
2013-01-27 01:49:37 +04:00
From the QT command prompt:
2011-10-16 05:14:35 +04:00
2011-12-07 04:56:58 +04:00
> cd tests
> cmake . -DCMAKE_BUILD_TYPE=Release
> sed -i s/\/MD/\/MT/ CMakeCache.txt
> cmake .
> nmake -f Makefile
2011-11-26 22:25:31 +04:00
2012-01-03 06:44:50 +04:00
Cross compiling Linux->Win32:
2011-12-31 20:02:55 +04:00
2012-01-03 06:44:50 +04:00
Please see openscad/tests/CMingw-cross-env.cmake for instructions.
2011-12-31 20:02:55 +04:00
2011-11-26 22:25:31 +04:00
B) Running tests
$ ctest Runs tests enabled by default
$ ctest -R <regex> Runs only matching tests, e.g. ctest -R dxf
$ ctest -C <configs> Adds extended tests belonging to configs.
Valid configs:
Default - Run default tests
Heavy - Run more time consuming tests (> ~10 seconds)
Examples - test all examples
All - test everything
2011-11-05 18:20:53 +04:00
2011-12-14 03:53:12 +04:00
Adding a new regression test:
2010-10-31 00:40:24 +04:00
------------------------------
2010-07-11 21:22:24 +04:00
1) create a test file at an appropriate location under testdata/
2012-02-07 04:42:11 +04:00
2) if the test is non-obvious, create a human readable description as comments in the test (or in another file in the same directory in case the file isn't human readable)
2011-09-04 04:58:09 +04:00
3) if a new test app was written, this must be added to tests/CMakeLists.txt
4) Add the tests to the test apps for which you want them to run (in tests/CMakeLists.txt)
5) run the test with the environment variable TEST_GENERATE=1, e.g.:
2011-04-29 20:02:54 +04:00
$ TEST_GENERATE=1 ctest -R mytest
2010-07-11 21:22:24 +04:00
(this will generate a mytest-expected.txt file which is used for regression testing)
2011-09-04 04:58:09 +04:00
6) manually verify that the output is correct (tests/regression/<testapp>/mytest-expected.<suffix>)
7) run the test normally and verify that it passes:
2010-07-11 21:22:24 +04:00
$ ctest -R mytest
2012-02-18 22:16:40 +04:00
Adding a new example:
---------------------
This is almost the same as adding a new regression test:
1) Create the example under examples/
2) run the test with the environment variable TEST_GENERATE=1, e.g.:
$ TEST_GENERATE=1 ctest -C Examples -R exampleNNN
(this will generate a exampleNNN-expected.txt file which is used for regression testing)
3) manually verify that the output is correct (tests/regression/<testapp>/exampleNNN.<suffix>)
4) run the test normally and verify that it passes:
$ ctest -C Examples -R exampleNNN
2013-06-25 15:28:30 +04:00
Migration away from dedicated regression tests:
-----------------------------------------------
2012-02-18 22:16:40 +04:00
2013-09-21 19:35:36 +04:00
This test still needs an intermediate script that mangles away timestamps and
2013-06-25 15:28:30 +04:00
near-zero floating point numbers:
2013-07-02 08:28:56 +04:00
* cgalstlsanitytest
2013-06-25 15:28:30 +04:00
Some tests are yet to be converted:
* csgtexttest -- verify whether this is not redundant with dumptest
These look like tests, but are not actually in use:
* modulecachetest
* cgalcachetest
2011-12-07 04:56:58 +04:00
Troubleshooting:
2011-09-19 02:53:33 +04:00
------------------------------
2011-12-26 22:34:58 +04:00
0. Headless unix servers
2011-12-07 07:48:19 +04:00
2011-12-26 22:34:58 +04:00
If you are attempting to run the tests on a unix-like system but only
have shell-console access, you may be able to run the tests by using a
virtual framebuffer program like Xvnc or Xvfb. For example:
$ Xvfb :5 -screen 0 800x600x24 &
2011-12-07 07:48:19 +04:00
$ DISPLAY=:5 ctest
2012-06-03 00:15:28 +04:00
or
$ xvfb-run ctest
2011-12-26 22:34:58 +04:00
Some versions of Xvfb may fail, however.
1. Trouble finding libraries on unix
2011-12-07 08:20:17 +04:00
2012-08-19 17:10:33 +04:00
To help CMAKE find eigen, OpenCSG, CGAL, Boost, and GLEW, you can use
2011-12-14 03:53:12 +04:00
environment variables, just like for the main qmake & openscad.pro. Examples:
2011-12-07 04:56:58 +04:00
2012-06-03 00:15:28 +04:00
OPENSCAD_LIBRARIES=$HOME cmake .
CGALDIR=$HOME/CGAL-3.9 BOOSTDIR=$HOME/boost-1.47.0 cmake .
2011-12-26 22:34:58 +04:00
Valid variables are as follows:
2011-12-14 03:53:12 +04:00
2012-08-19 17:10:33 +04:00
BOOSTDIR, CGALDIR, EIGENDIR, GLEWDIR, OPENCSGDIR, OPENSCAD_LIBRARIES
2011-12-07 07:48:19 +04:00
2012-06-03 00:15:28 +04:00
When running, this might help find your locally built libraries (assuming
you installed into $HOME)
2012-06-04 09:46:19 +04:00
Linux: export LD_LIBRARY_PATH=$HOME/lib:$HOME/lib64
Mac: export DYLD_LIBRARY_PATH=$HOME/lib
2012-06-03 00:15:28 +04:00
2011-12-26 22:34:58 +04:00
2. Location of logs
2011-12-07 04:56:58 +04:00
2011-09-19 02:53:33 +04:00
Logs of test runs are found in tests/build/Testing/Temporary
2011-12-26 22:34:58 +04:00
A pretty-printed index.html is in a subdir of tests/build/Testing/Temporary
2011-09-19 02:53:33 +04:00
Expected results are found in tests/regression/*
Actual results are found in tests/build/testname-output/*
2011-12-07 07:48:19 +04:00
2012-01-01 03:02:34 +04:00
3. Image-based tests takes a long time, they fail, and the log says 'return -11'
2011-12-07 08:20:17 +04:00
2011-12-26 22:34:58 +04:00
Imagemagick may have crashed while comparing the expected images to the
test-run generated (actual) images. You can try using the alternate
ImageMagick comparison method by by erasing CMakeCache, and re-running
cmake with -DCOMPARATOR=ncc. This will enable the Normalized Cross
Comparison method.
2011-12-07 08:20:17 +04:00
2012-01-01 03:02:34 +04:00
4. Testing images fails with 'morphology not found" for ImageMagick in the log
2011-12-07 08:31:59 +04:00
2011-12-14 03:53:12 +04:00
Your version of imagemagick is old. Upgrade, or pass -DCOMPARATOR=old to
cmake. The comparison will be of lowered reliability.
2011-12-07 10:47:15 +04:00
2012-06-03 00:42:36 +04:00
5. Locale errors
"terminate called after throwing an instance of 'std::runtime_error'
what(): locale::facet::_S_create_c_locale name not valid"
Is a boost/libstdc++ bug. Fix like so:
$ export LC_MESSAGES=
6. Other issues
2011-12-17 23:17:11 +04:00
2011-12-26 22:34:58 +04:00
The OpenSCAD User Manual has a section on buildling. Please check there
for updates:
2011-12-17 23:17:11 +04:00
http://en.wikibooks.org/wiki/OpenSCAD_User_Manual