Mutation Testing How good your tests are 2017 whoami iOS - - PowerPoint PPT Presentation

▶
mutation testing
SMART_READER_LITE
LIVE PREVIEW

Mutation Testing How good your tests are 2017 whoami iOS - - PowerPoint PPT Presentation

Feb 27, 2023 150 likes •1.01k views

Mutation Testing How good your tests are 2017 whoami iOS Developer by day compiler hacker by night https://twitter.com/1101_debian https://lowlevelbits.org https://systemundertest.org Mars Climate Orbiter Mars Climate

slide-1
SLIDE 1

Mutation Testing

How good your tests are

2017

slide-2
SLIDE 2

whoami

  • iOS Developer by day
  • compiler hacker by night
  • https://twitter.com/1101_debian
  • https://lowlevelbits.org
  • https://systemundertest.org
slide-3
SLIDE 3

Mars Climate Orbiter

slide-4
SLIDE 4

Mars Climate Orbiter

slide-5
SLIDE 5

Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan

Therac-25

‘85 ‘86 ‘87

slide-6
SLIDE 6

Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan

Injury

Therac-25

‘85 ‘86 ‘87

slide-7
SLIDE 7

Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan

Injury Bugfix

Therac-25

It’s fine ‘85 ‘86 ‘87

slide-8
SLIDE 8

More Incidents

https://github.com/stanislaw/awesome-safety-critical#incidents

slide-9
SLIDE 9

Quality of Software

slide-10
SLIDE 10

Quality of Software

  • Formal Verification
slide-11
SLIDE 11

Quality of Software

  • Formal Verification
  • Fuzz Testing
slide-12
SLIDE 12

Quality of Software

  • Formal Verification
  • Fuzz Testing
  • Unit Testing + Code Coverage
slide-13
SLIDE 13

Quality of Software

  • Formal Verification
  • Fuzz Testing
  • Unit Testing + Code Coverage
slide-14
SLIDE 14

Unit Testing

slide-15
SLIDE 15

Unit Testing

void test() { assert(sum(5, 10) > 0); } int sum(int a, int b) { return a + b; }

Failed Tests: 0 Code Coverage: 100% Passed Tests: 1

slide-16
SLIDE 16

Mutation Testing

slide-17
SLIDE 17

Mutation Testing

run_test(program, test)

slide-18
SLIDE 18

Mutation Testing

run_test(program, test) mutant = mutate(program)

slide-19
SLIDE 19

Mutation Testing

run_test(program, test) mutant = mutate(program) result = run_test(mutant, test)

slide-20
SLIDE 20

Mutation Testing

run_test(program, test) mutant = mutate(program) result = run_test(mutant, test) if (result == Failed) report_killed_mutant(mutant, test)

slide-21
SLIDE 21

Mutation Testing

run_test(program, test) mutant = mutate(program) result = run_test(mutant, test) if (result == Failed) report_killed_mutant(mutant, test) else report_survived_mutant(mutant, test)

slide-22
SLIDE 22

Mutation Testing

void test() { assert(sum(5, 10) > 0); } int sum(int a, int b) { return a + b; }

slide-23
SLIDE 23

Mutation Testing

void test() { assert(sum(5, 10) > 0); } int sum(int a, int b) { return a + b; } int sum’(int a, int b) { return a * b; }

slide-24
SLIDE 24

Mutation Testing

void test() { assert(sum(5, 10) > 0); } int sum(int a, int b) { return a + b; } int sum’(int a, int b) { return a * b; } int sum’’(int a, int b) { return a - b; }

slide-25
SLIDE 25

Mutation Testing

void test() { assert(sum(5, 10) > 0); } int sum(int a, int b) { return a + b; } int sum’(int a, int b) { return a * b; } int sum’’(int a, int b) { return a - b; }

test passed -> mutant survived

slide-26
SLIDE 26

Mutation Testing

void test() { assert(sum(5, 10) > 0); } int sum(int a, int b) { return a + b; } int sum’(int a, int b) { return a * b; } int sum’’(int a, int b) { return a - b; }

test passed -> mutant survived test failed -> mutant killed

slide-27
SLIDE 27

Mutation Testing

Killed Mutants: 1 Mutation Score = killed / total * 100% Survived Mutants: 1 Total Mutants: 2 Mutation Score: 50%

slide-28
SLIDE 28

Mutation Testing

  • First proposed by Richard Lipton in 1971
slide-29
SLIDE 29

Mutation Testing

  • First proposed by Richard Lipton in 1971
  • First implemented by Timothy Budd in

1980

slide-30
SLIDE 30

Mutation Testing

  • First proposed by Richard Lipton in 1971
  • First implemented by Timothy Budd in

1980

  • Studies say that MT was able to detect

70%-90% of real faults

slide-31
SLIDE 31

Mutation Testing

  • Generates lots of data
slide-32
SLIDE 32

Mutation Testing

  • Generates lots of data
  • Time consuming
slide-33
SLIDE 33

Mutation Testing

  • Generates lots of data
  • Time consuming
  • Languages are not mutation-testing-friendly
slide-34
SLIDE 34

Mutation Testing

  • Generates lots of data
  • Time consuming
  • Languages are not mutation-testing-friendly
  • Problem of a Human Test Oracle
slide-35
SLIDE 35

Mutation Testing

  • Generates lots of data
  • Time consuming
  • Languages are not mutation-testing-friendly
  • Problem of a Human Test Oracle
  • "Excuse me, but I write good tests"
slide-36
SLIDE 36

Mull

slide-37
SLIDE 37

Mull

  • Smart mutant selection
slide-38
SLIDE 38

Mull

  • Smart mutant selection
  • Control over data generation
slide-39
SLIDE 39

Mull

  • Smart mutant selection
  • Control over data generation
  • Runtime compilation
slide-40
SLIDE 40

Mull

  • Smart mutant selection
  • Control over data generation
  • Runtime compilation
  • Operates on LLVM IR level
slide-41
SLIDE 41

Mull

  • Smart mutant selection
  • Control over data generation
  • Runtime compilation
  • Operates on LLVM IR level
  • Language agnostic*
slide-42
SLIDE 42

Language agnostic

Frontend

  • Clang
  • Rust
  • Swift
  • …

Backend LLVM

slide-43
SLIDE 43

Language agnostic

Frontend

  • Clang
  • Rust
  • Swift
  • …

Backend LLVM

Source Code Machine Code

slide-44
SLIDE 44

Language agnostic

Frontend

  • Clang
  • Rust
  • Swift
  • …

Backend LLVM

Source Code Machine Code LLVM IR

slide-45
SLIDE 45

Frontend Backend

Source Code Machine Code LLVM IR

OS

Language agnostic

slide-46
SLIDE 46

Frontend Backend

Source Code Machine Code LLVM IR

OS Mull

Machine Code

JIT

Language agnostic

slide-47
SLIDE 47

Mutant Selection

slide-48
SLIDE 48

Mutant Selection

slide-49
SLIDE 49

Mutant Selection

slide-50
SLIDE 50

Mutant Selection

slide-51
SLIDE 51

Mutant Selection

slide-52
SLIDE 52

Mutant Selection

slide-53
SLIDE 53

Before:

391 modules 85 minutes

IRTests*: 238 tests

* Part of LLVM’s test suite

slide-54
SLIDE 54

Before:

391 modules 85 minutes

After:

124 modules 48 minutes

IRTests*: 238 tests

* Part of LLVM’s test suite

slide-55
SLIDE 55

Mutation Control

slide-56
SLIDE 56

Mutation Control

slide-57
SLIDE 57

Distance: 2 Number of mutants: ~1.5k Real execution time: ~1 hour

IRTests*: 238 tests

* Part of LLVM’s test suite

slide-58
SLIDE 58

IRTests*: 238 tests

Distance: 2 Number of mutants: ~1.5k Real execution time: ~1 hour Distance: 29 Number of mutants: ~18k Approximate execution time: ~11 days

* Part of LLVM’s test suite

slide-59
SLIDE 59

Mutation Control

slide-60
SLIDE 60

System Design

slide-61
SLIDE 61

System Design

Mull

slide-62
SLIDE 62

System Design

Core

  • Driver
  • Reporter
  • Mutation


Operators

slide-63
SLIDE 63

System Design

Core

  • Driver
  • Reporter
  • Mutation


Operators

Toolchain

  • JIT Compiler
  • Object Cache
slide-64
SLIDE 64

System Design

Test Framework

  • Test Finder
  • Test Runner

Core

  • Driver
  • Reporter
  • Mutation


Operators

Toolchain

  • JIT Compiler
  • Object Cache
slide-65
SLIDE 65

System Design

Test Framework

  • Test Finder
  • Test Runner

Google Test Core

  • Driver
  • Reporter
  • Mutation


Operators

Toolchain

  • JIT Compiler
  • Object Cache
  • Test Finder
  • Test Runner
slide-66
SLIDE 66

System Design

Test Framework

  • Test Finder
  • Test Runner

Google Test

  • Test Finder
  • Test Runner

XCTest Core

  • Driver
  • Reporter
  • Mutation


Operators

Toolchain

  • JIT Compiler
  • Object Cache
slide-67
SLIDE 67

Showcase

  • LLVM - compilers and dev tools, C++.
  • fmt - formatting library, C++.
  • Nom - parser combinators library, Rust.
  • CryptoSwift - collection of cryptographic algorithms, Swift.
slide-68
SLIDE 68

Example #1

slide-69
SLIDE 69

template<typename T> unsigned ComputeEditDistance(ArrayRef<T> FromArray, ArrayRef<T> ToArray, bool AllowReplacements = true, unsigned MaxEditDistance = 0) { typename ArrayRef<T>::size_type m = FromArray.size(); typename ArrayRef<T>::size_type n = ToArray.size(); const unsigned SmallBufferSize = 64; unsigned SmallBuffer[SmallBufferSize]; std::unique_ptr<unsigned[]> Allocated; unsigned *Row = SmallBuffer; if (n + 1 > SmallBufferSize) { Row = new unsigned[n + 1]; Allocated.reset(Row); } for (unsigned i = 1; i <= n; ++i) Row[i] = i; for (typename ArrayRef<T>::size_type y = 1; y <= m; ++y) { Row[0] = y; unsigned BestThisRow = Row[0]; unsigned Previous = y - 1; for (typename ArrayRef<T>::size_type x = 1; x <= n; ++x) { int OldRow = Row[x]; if (AllowReplacements) { Row[x] = std::min( Previous + (FromArray[y-1] == ToArray[x-1] ? 0u : 1u), std::min(Row[x-1], Row[x])+1); } else { if (FromArray[y-1] == ToArray[x-1]) Row[x] = Previous; else Row[x] = std::min(Row[x-1], Row[x]) + 1; } Previous = OldRow; BestThisRow = std::min(BestThisRow, Row[x]); } if (MaxEditDistance && BestThisRow > MaxEditDistance) return MaxEditDistance + 1; } unsigned Result = Row[n]; return Result; }

slide-70
SLIDE 70

template<typename T> unsigned ComputeEditDistance(ArrayRef<T> FromArray, ArrayRef<T> ToArray, bool AllowReplacements = true, unsigned MaxEditDistance = 0) { typename ArrayRef<T>::size_type m = FromArray.size(); typename ArrayRef<T>::size_type n = ToArray.size(); const unsigned SmallBufferSize = 64; unsigned SmallBuffer[SmallBufferSize]; std::unique_ptr<unsigned[]> Allocated; unsigned *Row = SmallBuffer; for (typename ArrayRef<T>::size_type y = 1; y <= m; ++y) { Row[0] = y; unsigned BestThisRow = Row[0]; unsigned Previous = y - 1; for (typename ArrayRef<T>::size_type x = 1; x <= n; ++x) { int OldRow = Row[x]; if (AllowReplacements) { Row[x] = std::min( Previous + (FromArray[y-1] != ToArray[x-1] ? 0u : 1u), std::min(Row[x-1], Row[x])+1); } Previous = OldRow; BestThisRow = std::min(BestThisRow, Row[x]); } } unsigned Result = Row[n]; return Result; } template<typename T> unsigned ComputeEditDistance(ArrayRef<T> FromArray, ArrayRef<T> ToArray, bool AllowReplacements = true, unsigned MaxEditDistance = 0) { typename ArrayRef<T>::size_type m = FromArray.size(); typename ArrayRef<T>::size_type n = ToArray.size(); const unsigned SmallBufferSize = 64; unsigned SmallBuffer[SmallBufferSize]; std::unique_ptr<unsigned[]> Allocated; unsigned *Row = SmallBuffer; if (n + 1 > SmallBufferSize) { Row = new unsigned[n + 1]; Allocated.reset(Row); } for (unsigned i = 1; i <= n; ++i) Row[i] = i; for (typename ArrayRef<T>::size_type y = 1; y <= m; ++y) { Row[0] = y; unsigned BestThisRow = Row[0]; unsigned Previous = y - 1; for (typename ArrayRef<T>::size_type x = 1; x <= n; ++x) { int OldRow = Row[x]; if (AllowReplacements) { Row[x] = std::min( Previous + (FromArray[y-1] == ToArray[x-1] ? 0u : 1u), std::min(Row[x-1], Row[x])+1); } else { if (FromArray[y-1] == ToArray[x-1]) Row[x] = Previous; else Row[x] = std::min(Row[x-1], Row[x]) + 1; } Previous = OldRow; BestThisRow = std::min(BestThisRow, Row[x]); } if (MaxEditDistance && BestThisRow > MaxEditDistance) return MaxEditDistance + 1; } unsigned Result = Row[n]; return Result; }

slide-71
SLIDE 71

TEST(StringRefTest, EditDistance) { StringRef Str("hello"); EXPECT_EQ(2U, Str.edit_distance("hill")); }

slide-72
SLIDE 72

TEST(StringRefTest, EditDistance) { StringRef Hello("hello"); EXPECT_EQ(2U, Hello.edit_distance("hill")); StringRef Soylent("soylent green is people");
 StringRef People("people soiled our green"); EXPECT_EQ(19U, Soylent.edit_distance(People)); EXPECT_EQ(26U, Soylent.edit_distance(People, false)); EXPECT_EQ(9U, Soylent.edit_distance(People, true, 8)); }

Fix: r300312

slide-73
SLIDE 73

Example #2

slide-74
SLIDE 74

Triple T = Triple(""); T.setObjectFormat(Triple::ELF); EXPECT_EQ(Triple::ELF, T.getObjectFormat());

slide-75
SLIDE 75

Triple T = Triple(""); // T.setObjectFormat(Triple::ELF); EXPECT_EQ(Triple::ELF, T.getObjectFormat());

slide-76
SLIDE 76

Triple T = Triple(""); T.setObjectFormat(Triple::ELF); EXPECT_EQ(Triple::ELF, T.getObjectFormat()); T.setObjectFormat(Triple::MachO); EXPECT_EQ(Triple::MachO, T.getObjectFormat());

Fix: r294104

slide-77
SLIDE 77

Example #3

slide-78
SLIDE 78

T.setArch(Triple::mips64); EXPECT_EQ(Triple::mips64el,
 T.getLittleEndianArchVariant().getArch());

slide-79
SLIDE 79

T.setArch(Triple::mips64); EXPECT_EQ(Triple::mips64el,
 T.getLittleEndianArchVariant().getArch()); T.setArch(Triple::tce); EXPECT_EQ(Triple::tcele,
 T.getLittleEndianArchVariant().getArch());

Fix: r294095, r294096

slide-80
SLIDE 80
slide-81
SLIDE 81
slide-82
SLIDE 82

Demo

slide-83
SLIDE 83

Results

  • https://lowlevelbits.org/ADTTests/
  • https://lowlevelbits.org/IRTests/
  • https://github.com/krzyzanowskim/CryptoSwift/issues/417
slide-84
SLIDE 84

Project: https://github.com/mull-project/mull Contact: alex@lowlevelbits.org Updates: https://twitter.com/1101_debian

slide-85
SLIDE 85

Project: https://github.com/mull-project/mull Contact: alex@lowlevelbits.org Updates: https://twitter.com/1101_debian

Questions?