Continuous Integration/Testing and why you should assume every - - PowerPoint PPT Presentation

Continuous Integration/Testing

and why you should assume every change breaks your code

René Milk 4th November 2015

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Continuous Integration/Testing 2 /16

Not talking about C∞ functions

Continuous

◮ completely automatic triggering (nightly, on branch updates, ...) ◮ automatic reporting ◮ simple visualization ◮ wide accessibility ◮ metrics

, , rene.milk@wwu.de

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Continuous Integration/Testing 3 /16

No quadrature rules contained in this talk either

Integration

◮ does your software stack still build after changes? ◮ compile, run, link, work as a dependency for other software? ◮ how about on system X?

, , rene.milk@wwu.de

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Continuous Integration/Testing 4 /16

and ofc no weak solutions for PDEs

Testing

◮ automated, repeatable, deterministic, reliable ◮ Unit Testing ◮ System Testing ◮ Performance Testing

, , rene.milk@wwu.de

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Continuous Integration/Testing 5 /16

assume your every change breaks everybody’s code

◮ saves time, no need to argue about possible effects of changes ◮ It’s better to err on the side of caution. You will break code from time to time. ◮ software is modular, codebases are huge, dependency graphs are complex ◮ distributed development: multiple people, branches, features/fixes, ... ◮ http://whatever.scalzi.com/2010/06/16/the-failure-state-of-clever/

, , rene.milk@wwu.de

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Continuous Integration/Testing 6 /16

Tools for CI: Travis

◮ free for FOSS ◮ integrates with GitHub repositories ◮ explicitly supports C++, python, ruby, but if your test/software runs on a standard-ish

Ubuntu 12.04 you’re good

◮ triggers on branch push, visual feedback in pull requests and badges ◮ very easy to create different configurations

, , rene.milk@wwu.de

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Continuous Integration/Testing 7 /16

Tools for CI: Travis

language: python python:

• "2.7"

script:

• DISPLAY=:99.0 py.test -k "${PYTEST_MARKER}"

install:

• python setup.py build_ext -i

notifications: email:

• n_success: change
• n_failure: change

after_success:

• coveralls

branches: except:

• gh-pages

, , rene.milk@wwu.de

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Continuous Integration/Testing 8 /16

Tools for CI: Travis

sudo: false language: cpp matrix: include:

• os: linux

compiler: gcc addons: &gcc49 apt: sources: [’ubuntu-toolchain-r-test’] packages: [’g++-4.9’, ’gcc-4.9’] env: CXX_COMPILER=g++-4.9 COMPILER=gcc-4.9

• os: linux

compiler: gcc addons: *gcc49 env: CXX_COMPILER=g++-4.9 COMPILER=gcc-4.9 DELETE="dune-istl" , , rene.milk@wwu.de

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Continuous Integration/Testing 9 /16

Tools for CI: Travis

, , rene.milk@wwu.de

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Continuous Integration/Testing 10 /16

Tools for CI: Travis

, , rene.milk@wwu.de

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Continuous Integration/Testing 11 /16

Tools for CI: Buildbot

◮ Python framework to build CI systems, not a ready made solution ◮ Master/Slave setup ◮ good for testing complex stacks on diverse architectures ◮ much more freedom than on Travis, at a much higher maintenance and setup cost

, , rene.milk@wwu.de

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Continuous Integration/Testing 12 /16

Tools for CI: Buildbot - Example

◮ setup to deploy new masters with only 5 LOC ◮ buildbot code for that to work is 2300 LOC ◮ git repository tracks individual DUNE-modules´

repositories -> dynamic build steps -> make test, result log

◮ tied into redmine ◮ email on buildstatus change, filter by user in change

, , rene.milk@wwu.de

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Continuous Integration/Testing 13 /16

Tools for CI: Buildbot - Example (master config)

from common import config, passwords, genconfig config.CURRENT_DB = passwords.dune_db_url import redmine.db.connection redmine.db.connection.setup(config.CURRENT_DB) BuildmasterConfig = genconfig.genconfig( "http://users.../dune-stuff-demos.git", project="dune-stuff", base_port=11020, use_cmake=True)

, , rene.milk@wwu.de

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Continuous Integration/Testing 14 /16

Tools for CI: Buildbot - Example (buildbot.opts)

[clang_3.5] file = config.opts/clang-3.5 cc = clang-3.5 [gcc_5.0] file = config.opts/gcc-snapshot cc = gcc-snapshot [variants] # name = semicolon seperated list of directory names # to delete prior to buildbot config no_fem = dune-fem minimal = dune-grid;dune-localfunctions;dune-fem; dune-typetree;dune-pdelab;dune-istl

, , rene.milk@wwu.de

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Continuous Integration/Testing 15 /16

Buildbot

, , rene.milk@wwu.de

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Continuous Integration/Testing 16 /16

Testing: What?

utopic

Every line of code is unit tested, with function level granularity. System tests cover all possible software configurations and inputs. Performance testing is run on a wide variety of benchmark problems, in a restricted, possibly virtualized, environment. All of that on a very diverse set of computer architectures, operating systems, support library versions. Not viable in the real world. Configuration space is just too big. Compute time, and available hardware is limited. Test cycles need to be short to not impede development.

, , rene.milk@wwu.de

Continuous Integration/Testing 17 /16

Testing: What?

realistic

Every non-trivial piece of code is unit tested, with algorithm level granularity. System tests cover the most common configuration options. Benchmark runs are recorded.

, , rene.milk@wwu.de

Continuous Integration/Testing 18 /16

Testing: What?

minimal

The code is run for a benchmark problem with known/analytic solution.

, , rene.milk@wwu.de

Continuous Integration/Testing 19 /16

Testing: How?

◮ Use a test harness: google-test, Boost.Test, py.test

Simplifies selective test running, result gathering, behavior expectation

◮ Writing a DUNE module? Use dune-testtools for simple test parameterization ◮ record algorithm results, error norms and compare in system tests ◮ Make sure exceptions get raised on invalid data input ◮ integrate into build system (ie make test)

, , rene.milk@wwu.de

Continuous Integration/Testing 20 /16

Testing: How? - Gtest Example

typedef testing::Types<YaspGrid<1>, YaspGrid<4> Grids; template <class T> struct CornerRangeTest : public ::testing::Test { CornerRangeTest() : grid_prv(0., 1., level) {} void check() { for (auto it : grid_prv.grid().leafGridView()) check_range(it, cornerRange(it->geometry())); for (auto v : DSC::valueRange(T::dimensionworld)) EXPECT_GE(v, 0); } }; TYPED_TEST_CASE(CornerRangeTest, Grids); TYPED_TEST(CornerRangeTest, Misc) { this->check(); } , , rene.milk@wwu.de

Continuous Integration/Testing 21 /16

Testing: What tests are allowed to fail

NONE

except those where the platform requirements cannot be met

, , rene.milk@wwu.de

Continuous Integration/Testing 22 /16

Testing: What else is there?

◮ Using cmake? Make ctest submit test results to cdash

, , rene.milk@wwu.de

Continuous Integration/Testing 23 /16

Thank you for your attention.

Slides available at http://work.milk.pm

, , rene.milk@wwu.de