roottest migration to CMake / CTest Philipp Schoppe 14.07.2014 1 / - - PowerPoint PPT Presentation

▶

Apr 26, 2023 504 likes •750 views

roottest migration to CMake / CTest Philipp Schoppe 14.07.2014 1 / 23 Agenda Motivation CTest Part of CMake Mode of operations CTest reports roottest CTest Usage Status CTest advantages CTest disadvantages Summary 2 / 23

SLIDE 1

roottest migration to CMake / CTest

Philipp Schoppe 14.07.2014

1 / 23

SLIDE 2

Agenda

Motivation CTest Part of CMake Mode of operations CTest reports roottest → CTest Usage Status CTest advantages CTest disadvantages Summary

2 / 23

SLIDE 3

Motivation

3 / 23

SLIDE 4

Motivation

◮ Ensure SW meets its requirements ◮ Risk management ◮ Quality ensurance & improvement ◮ SW design specification ◮ But: writing tests is perceived as a burden by many

developers ⇒ Provide a testing framework that is easy to use and integrates well into the development process

4 / 23

SLIDE 5

CTest

5 / 23

SLIDE 6

CTest

Part of CMake

Used for:

◮ automate updating ◮ configuring ◮ building ◮ testing ◮ perform memory checking ◮ submit results to a CDash

6 / 23

SLIDE 7

CTest

Mode of operations

Simple testing

◮ CMake configures & builds a project ◮ CMake uses commands to add tests ◮ CTest executes tests / submits results to a dashboard

CTest Scripting

◮ CTest runs a script to control the whole build process

7 / 23

SLIDE 8

CTest

Simple Testing enable_testing() add_test(testname Exename arg1 arg2 ... ) add_test(NAME <name> [CONFIGURATIONS [Debug|Release|...]] [WORKING_DIRECTORY dir] COMMAND <command> [arg1 [arg2 ...]])

Additional Properties:

◮ TIMEOUT ◮ DEPENDS ◮ PASS_REGULAR_EXPRESSION ◮ FAIL_REGULAR_EXPRESSION ◮ WILL_FAIL ◮ LABELS ◮ . . . (ctest –help-property-list)

8 / 23

SLIDE 9

CTest

Running CTest

◮ ctest -N (list all tests) ◮ ctest -j4 (run all tests in parallel) ◮ ctest -R regex (run all tests matching regex) ◮ ctest -E regex (run all tests not matching regex) ◮ ctest -V (verbose output) ◮ ctest –print-labels ◮ ctest -L regex (run all tests that contain the label regex)

9 / 23

SLIDE 10

CTest

CTest Scripting

Setup programs and environments with syntax similar to CMake, then define the build process:

ctest_start("Nightly") ctest_update() ctest_configure() ctest_build() ctest_test() if (WITH_COVERAGE AND CTEST_COVERAGE_COMMAND) ctest_coverage() endif (WITH_COVERAGE AND CTEST_COVERAGE_COMMAND) if (WITH_MEMCHECK AND CTEST_MEMORYCHECK_COMMAND) ctest_memcheck() endif (WITH_MEMCHECK AND CTEST_MEMORYCHECK_COMMAND) ctest_submit() http://www.vtk.org/Wiki/CMake_Scripting_Of_CTest

10 / 23

SLIDE 11

CTest

Reports

11 / 23

SLIDE 12

CTest

Reports

12 / 23

SLIDE 13

roottest → CTest

13 / 23

SLIDE 14

roottest → CTest

Usage

Create $roottest/cling/printf/CMakeLists.txt:

ROOTTEST_ADD_TEST(roottest-cling-printf-star MACRO runstar.C OUTREF star.ref LABELS roottest regression cling)

◮ Creates test (target) roottest-cling-printf-star ◮ Runs root.exe -q -l -b runstar.C (+ some

environment settings)

◮ Writes stdout & stderr to a logfile ◮ Compares the logfile to star.ref

◮ Checks for star.ref32 / star.ref64

◮ Associates the test with labels

14 / 23

SLIDE 15

roottest → CTest

Usage

Another example:

ROOTTEST_SET_TESTOWNER("Philipp Schoppe") ROOTTEST_GENERATE_EXECUTABLE(loadernotapp loadernotapp.cxx LIBRARIES Hist) ROOTTEST_ADD_TEST(roottest-root-meta-loadernotapp EXEC loadernotapp OUTREF loadernotapp.ref DEPENDS ${GENERATE_EXECUTABLE_TEST})

15 / 23

SLIDE 16

roottest → CTest

Status

◮ Migrated 349 tests ◮ Many tests left for migration ◮ Problems with python, not out-of-source ◮ Some tests are still failing (investigating...)

16 / 23

SLIDE 17

CTest advantages

17 / 23

SLIDE 18

CTest advantages

◮ Clear overview of all existing tests ◮ Clear reports on individual tests (+ URL to share) ◮ Easily specify individual test settings ◮ Selection of tests by labels and regular expressions ◮ Integrated, cross-platform build/testing framework

◮ Natural integration into CMake ◮ Many projects move to CMake ⇒ Easy integration ◮ Tool to setup a software testing process

◮ Visualization for timing and pass/fail status ◮ More intuitive to read ◮ Code coverage and memory checks

18 / 23

SLIDE 19

CTest disadvantages

19 / 23

SLIDE 20

CTest disadvantages

◮ Harder to write complex setup scripts ◮ Escape sequences can be troublesome ◮ Native regex syntax ◮ Mixing stdout / stderr may cause problems ◮ Needs time / manpower to fully migrate & validate

◮ Some tests need big setup steps ◮ Tests are not written with cross-platform support in mind 20 / 23

SLIDE 21

Summary

21 / 23

SLIDE 22

Summary

◮ Migration to CTest ⇒ Migration to more clear & systematic

testing

◮ May restrict freedom of users (can be a good thing?)

◮ Most of the problems occured due to compatibility to old

conventions

◮ Migration to CTest in a context of paradigm shift

◮ Introduce SW testing processes / models ◮ Unit testing (googletest?) ◮ Regression testing ◮ System testing ◮ Automatic testing, fuzzing, static analysis, code coverage,

. . .

◮ Ideally: have a test engineer, never test your own code, . . .

22 / 23

SLIDE 23

Thank you for your attention

23 / 23