roottest migration to CMake / CTest Philipp Schoppe 14.07.2014 1 / - PowerPoint PPT Presentation

roottest migration to CMake / CTest Philipp Schoppe 14.07.2014 1 / 23

Agenda Motivation CTest Part of CMake Mode of operations CTest reports roottest → CTest Usage Status CTest advantages CTest disadvantages Summary 2 / 23

Motivation 3 / 23

Motivation ◮ Ensure SW meets its requirements ◮ Risk management ◮ Quality ensurance & improvement ◮ SW design specification ◮ But: writing tests is perceived as a burden by many developers ⇒ Provide a testing framework that is easy to use and integrates well into the development process 4 / 23

CTest 5 / 23

CTest Part of CMake Used for: ◮ automate updating ◮ configuring ◮ building ◮ testing ◮ perform memory checking ◮ submit results to a CDash 6 / 23

CTest Mode of operations Simple testing ◮ CMake configures & builds a project ◮ CMake uses commands to add tests ◮ CTest executes tests / submits results to a dashboard CTest Scripting ◮ CTest runs a script to control the whole build process 7 / 23

CTest Simple Testing enable_testing() add_test(testname Exename arg1 arg2 ... ) add_test(NAME <name> [CONFIGURATIONS [Debug|Release|...]] [WORKING_DIRECTORY dir] COMMAND <command> [arg1 [arg2 ...]]) Additional Properties: ◮ TIMEOUT ◮ DEPENDS ◮ PASS_REGULAR_EXPRESSION ◮ FAIL_REGULAR_EXPRESSION ◮ WILL_FAIL ◮ LABELS ◮ . . . (ctest –help-property-list) 8 / 23

CTest Running CTest ◮ ctest -N (list all tests) ◮ ctest -j4 (run all tests in parallel) ◮ ctest -R regex (run all tests matching regex) ◮ ctest -E regex (run all tests not matching regex) ◮ ctest -V (verbose output) ◮ ctest –print-labels ◮ ctest -L regex (run all tests that contain the label regex) 9 / 23

CTest CTest Scripting Setup programs and environments with syntax similar to CMake, then define the build process: ctest_start("Nightly") ctest_update() ctest_configure() ctest_build() ctest_test() if (WITH_COVERAGE AND CTEST_COVERAGE_COMMAND) ctest_coverage() endif (WITH_COVERAGE AND CTEST_COVERAGE_COMMAND) if (WITH_MEMCHECK AND CTEST_MEMORYCHECK_COMMAND) ctest_memcheck() endif (WITH_MEMCHECK AND CTEST_MEMORYCHECK_COMMAND) ctest_submit() http://www.vtk.org/Wiki/CMake_Scripting_Of_CTest 10 / 23

CTest Reports 11 / 23

CTest Reports 12 / 23

roottest → CTest 13 / 23

roottest → CTest Usage Create $roottest/cling/printf/CMakeLists.txt: ROOTTEST_ADD_TEST(roottest-cling-printf-star MACRO runstar.C OUTREF star.ref LABELS roottest regression cling) ◮ Creates test (target) roottest-cling-printf-star ◮ Runs root.exe -q -l -b runstar.C (+ some environment settings) ◮ Writes stdout & stderr to a logfile ◮ Compares the logfile to star.ref ◮ Checks for star.ref32 / star.ref64 ◮ Associates the test with labels 14 / 23

roottest → CTest Usage Another example: ROOTTEST_SET_TESTOWNER("Philipp Schoppe") ROOTTEST_GENERATE_EXECUTABLE(loadernotapp loadernotapp.cxx LIBRARIES Hist) ROOTTEST_ADD_TEST(roottest-root-meta-loadernotapp EXEC loadernotapp OUTREF loadernotapp.ref DEPENDS ${GENERATE_EXECUTABLE_TEST}) 15 / 23

roottest → CTest Status ◮ Migrated 349 tests ◮ Many tests left for migration ◮ Problems with python, not out-of-source ◮ Some tests are still failing (investigating...) 16 / 23

CTest advantages 17 / 23

CTest advantages ◮ Clear overview of all existing tests ◮ Clear reports on individual tests (+ URL to share) ◮ Easily specify individual test settings ◮ Selection of tests by labels and regular expressions ◮ Integrated, cross-platform build/testing framework ◮ Natural integration into CMake ◮ Many projects move to CMake ⇒ Easy integration ◮ Tool to setup a software testing process ◮ Visualization for timing and pass/fail status ◮ More intuitive to read ◮ Code coverage and memory checks 18 / 23

CTest disadvantages 19 / 23

CTest disadvantages ◮ Harder to write complex setup scripts ◮ Escape sequences can be troublesome ◮ Native regex syntax ◮ Mixing stdout / stderr may cause problems ◮ Needs time / manpower to fully migrate & validate ◮ Some tests need big setup steps ◮ Tests are not written with cross-platform support in mind 20 / 23

Summary 21 / 23

Summary ◮ Migration to CTest ⇒ Migration to more clear & systematic testing ◮ May restrict freedom of users (can be a good thing?) ◮ Most of the problems occured due to compatibility to old conventions ◮ Migration to CTest in a context of paradigm shift ◮ Introduce SW testing processes / models ◮ Unit testing (googletest?) ◮ Regression testing ◮ System testing ◮ Automatic testing, fuzzing, static analysis, code coverage, . . . ◮ Ideally: have a test engineer, never test your own code, . . . 22 / 23

Thank you for your attention 23 / 23