openscad/doc/testing.txt

Running regression tests:
-------------------------

Prerequisites: cmake, python, ImageMagick 6.5.9.3 or newer

A) Building test environment

Linux, Mac:
$ cd tests
$ cmake .
$ make

Windows + MSVC:

First, get a normal build working by following instructions at 
http://en.wikibooks.org/wiki/OpenSCAD_User_Manual/Building_on_Windows
Then, from the QT command prompt:

> cd tests
> cmake . -DCMAKE_BUILD_TYPE=Release
> sed -i s/\/MD/\/MT/ CMakeCache.txt
> cmake .
> nmake -f Makefile

Cross compiling Linux->Windows:

Additional prerequisities: wine, mingw-cross-env

Note: CTest harness will only run under Wine under *nix, not under Win32.

Follow the instructions for installing the mingw-env cross build system at 
http://en.wikibooks.org/wiki/OpenSCAD_User_Manual , especially the part 
about including the cross-compiler in your PATH. If you do cross-builds,
it's recommended to use out-of-source builds (mkdir buildx; cd buildx; cmake ..)
in order to keep your native *nix build separate from your cross-build. 

$ cd tests
$ mkdir build-mingw32
$ cd build-mingw32
$ cmake .. -DMINGW_CROSS_ENV_DIR=<mingw-env-dir> -DCMAKE_TOOLCHAIN_FILE=CMingw-env-cross.cmake
$ make

B) Running tests

$ ctest               Runs tests enabled by default
$ ctest -R <regex>    Runs only matching tests, e.g. ctest -R dxf
$ ctest -C <configs>  Adds extended tests belonging to configs.
                      Valid configs:
                      Default  - Run default tests
                      Heavy    - Run more time consuming tests (> ~10 seconds)
                      Examples - test all examples
                      All      - test everything

Adding a new regression test:
------------------------------

1) create a test file at an appropriate location under testdata/
2) if the test is non-obvious, create a human readable description of the test in the same directory (e.g testdata/scad/mytest.txt)
3) if a new test app was written, this must be added to tests/CMakeLists.txt
4) Add the tests to the test apps for which you want them to run (in tests/CMakeLists.txt)
5) run the test with the environment variable TEST_GENERATE=1, e.g.:
   $ TEST_GENERATE=1 ctest -R mytest
   (this will generate a mytest-expected.txt file which is used for regression testing)
6) manually verify that the output is correct (tests/regression/<testapp>/mytest-expected.<suffix>)
7) run the test normally and verify that it passes:
  $ ctest -R mytest

Troubleshooting:
------------------------------

0. Headless unix servers

If you are attempting to run the tests on a unix-like system but only
have shell-console access, you may be able to run the tests by using a 
virtual framebuffer program like Xvnc or Xvfb. For example:

$ Xvfb :5 -screen 0 800x600x24 &
$ DISPLAY=:5 ctest

Some versions of Xvfb may fail, however. 

1. Trouble finding libraries on unix

 To help CMAKE find eigen2, OpenCSG, CGAL, Boost, and GLEW, you can use 
 environment variables, just like for the main qmake & openscad.pro. Examples:

 OPENSCAD_LIBRARIES=~ cmake .
 CGALDIR=~/CGAL-3.9 BOOSTDIR=~/boost-1.47.0 cmake .

 Valid variables are as follows:

 BOOSTDIR, CGALDIR, EIGEN2DIR, GLEWDIR, OPENCSGDIR, OPENSCAD_LIBRARIES

2. Location of logs
 
Logs of test runs are found in tests/build/Testing/Temporary
A pretty-printed index.html is in a subdir of tests/build/Testing/Temporary
Expected results are found in tests/regression/*
Actual results are found in tests/build/testname-output/*

3. Image-based tests takes a long time, they fail, and the log says 'return -11'

Imagemagick may have crashed while comparing the expected images to the 
test-run generated (actual) images. You can try using the alternate 
ImageMagick comparison method by by erasing CMakeCache, and re-running 
cmake with -DCOMPARATOR=ncc. This will enable the Normalized Cross 
Comparison method.

4. Testing images fails with 'morphology not found" for ImageMagick in the log

Your version of imagemagick is old. Upgrade, or pass -DCOMPARATOR=old to 
cmake. The comparison will be of lowered reliability.

5. Other issues

The OpenSCAD User Manual has a section on buildling. Please check there 
for updates:

http://en.wikibooks.org/wiki/OpenSCAD_User_Manual