Running regression tests: ------------------------- 0) Prerequisites Install the prerequisite helper programs on your system: cmake, python2 (not 3), ImageMagick 6.5.9.3 or newer, diff There are binary installer packages of these tools available for Mac, Win, Linux, BSD, and other systems. (except maybe diff for Win) Next, get a working qmake GUI build of the main openscad binary working. For Windows(TM) this means get a cross-build working from within linux. See README.md for how to do this. Then, install MCAD under openscad/libraries. $ cd openscad $ git submodule update --init A) Building test environment Linux, Mac: $ cd tests $ cmake . $ make Windows(TM): Cross-build from within linux: 64-bit: $ source ./scripts/setenv-mingw-xbuild.sh 64 $ ./scripts/release-common.sh mingw64 tests $ # result is .zip file under ./mingw64/ 32-bit: $ source ./scripts/setenv-mingw-xbuild.sh 32 $ ./scripts/release-common.sh mingw32 tests $ # result is .zip file under ./mingw32/ B) Running tests Linux, Mac: $ ctest Runs tests enabled by default $ ctest -R Runs only matching tests, e.g. ctest -R dxf $ ctest -C Adds extended tests belonging to configs. Valid configs: Default - Run default tests Heavy - Run more time consuming tests (> ~10 seconds) Examples - test all examples Bugs - test known bugs (tests will fail) All - test everything Win: Unzip the OpenSCAD-Tests-YYYY.MM.DD file onto a Windows(TM) machine. There will be a script called OpenSCAD-Test-Console.py in the parent folder. Double-click it, and it will open a console, from which you can type the ctest commands listed above. C) Automatically upload test results (experimental) It's possible to automatically upload tests results to an external server. This is good for CI, as well as being able to easily report bugs. To enable this feature, add '-DOPENSCAD_UPLOAD_TESTS=1' to the cmake cmd-line, e.g.: cmake -DOPENSCAD_UPLOAD_TESTS=1 . Adding a new test: ------------------ 1) create a test file at an appropriate location under testdata/ 2) if the test is non-obvious, create a human readable description as comments in the test (or in another file in the same directory in case the file isn't human readable) 3) if a new test app was written, this must be added to tests/CMakeLists.txt 4) Add the tests to the test apps for which you want them to run (in tests/CMakeLists.txt) 5) run the test with the environment variable TEST_GENERATE=1, e.g.: $ TEST_GENERATE=1 ctest -R mytest (this will generate a mytest-expected.txt file which is used for regression testing) 6) manually verify that the output is correct (tests/regression//mytest-expected.) 7) run the test normally and verify that it passes: $ ctest -R mytest Adding a new example: --------------------- This is almost the same as adding a new regression test: 1) Create the example under examples/ 2) run the test with the environment variable TEST_GENERATE=1, e.g.: $ TEST_GENERATE=1 ctest -C Examples -R exampleNNN (this will generate a exampleNNN-expected.txt file which is used for regression testing) 3) manually verify that the output is correct (tests/regression//exampleNNN.) 4) run the test normally and verify that it passes: $ ctest -C Examples -R exampleNNN Troubleshooting: ------------------------------ 0. Headless unix servers If you are attempting to run the tests on a unix-like system but only have shell-console access, you may be able to run the tests by using a virtual framebuffer program like Xvnc or Xvfb. For example: $ Xvfb :5 -screen 0 800x600x24 & $ DISPLAY=:5 ctest or $ xvfb-run ctest Some versions of Xvfb may fail, however. 1. Trouble finding libraries on unix To help CMAKE find eigen, OpenCSG, CGAL, Boost, and GLEW, you can use environment variables, just like for the main qmake & openscad.pro. Examples: OPENSCAD_LIBRARIES=$HOME cmake . CGALDIR=$HOME/CGAL-3.9 BOOSTDIR=$HOME/boost-1.47.0 cmake . Valid variables are as follows: BOOSTDIR, CGALDIR, EIGENDIR, GLEWDIR, OPENCSGDIR, OPENSCAD_LIBRARIES When running, this might help find your locally built libraries (assuming you installed into $HOME) Linux: export LD_LIBRARY_PATH=$HOME/lib:$HOME/lib64 Mac: export DYLD_LIBRARY_PATH=$HOME/lib 2. Location of logs Logs of test runs are found in tests/build/Testing/Temporary A pretty-printed index.html is in a subdir of tests/build/Testing/Temporary Expected results are found in tests/regression/* Actual results are found in tests/build/testname-output/* 3. Image-based tests takes a long time, they fail, and the log says 'return -11' Imagemagick may have crashed while comparing the expected images to the test-run generated (actual) images. You can try using the alternate ImageMagick comparison method by by erasing CMakeCache, and re-running cmake with -DCOMPARATOR=ncc. This will enable the Normalized Cross Comparison method which is less accurate but won't usually crash. 4. Testing images fails with 'morphology not found" for ImageMagick in the log Your version of imagemagick is old. Upgrade, or pass -DCOMPARATOR=old to cmake. The comparison will be of lowered reliability. 5. Locale errors "terminate called after throwing an instance of 'std::runtime_error' what(): locale::facet::_S_create_c_locale name not valid" Is a boost/libstdc++ bug. Fix like so before running: $ export LC_MESSAGES= 6. I want to build without OpenGL There is an unsupported way to do this, by defining NULLGL to Cmake: mkdir nullglbin cd nullglbin && cmake .. -DNULLGL=1 && make The resulting openscad_nogui binary will fail most tests, but may be useful for debugging and outputting 3d-formats like STL on systems without GL. This option may break in the future and require tweaking to get working again. 7. Proprietary GL driver issues There are sporadic reports of problems running on remote machines with proprietary GL drivers. Try doing a web search for your exact error message to see solutions and workarounds that others have found. 8. Windows + MSVC: The MSVC build was last tested circa 2012. The last time it worked, these were the necessary commands to run. > Start the 'QT command prompt' > cd \where\you\installed\openscad > cd tests > cmake . -DCMAKE_BUILD_TYPE=Release > sed -i s/\/MD/\/MT/ CMakeCache.txt > cmake . > nmake -f Makefile 9. Other issues The OpenSCAD User Manual Wiki has a section on buildling. Please check there for possible updates and workarounds: http://en.wikibooks.org/wiki/OpenSCAD_User_Manual Please report build errors (after double checking the instructions) in the github issue tracker http://github.com/openscad/openscad/issues 10. Migration away from dedicated regression tests: In 2013 the test programs underwent a major change. The following notes are leftover. "This test still needs an intermediate script that mangles away timestamps and near-zero floating point numbers: * cgalstlsanitytest Some tests are yet to be converted: * csgtexttest -- verify whether this is not redundant with dumptest These look like tests, but are not actually in use: * modulecachetest * cgalcachetest "