Mutate and Test Your Tests Benoit Baudry KTH, Sweden 1 - - PowerPoint PPT Presentation

▶
mutate and test your tests
SMART_READER_LITE
LIVE PREVIEW

Mutate and Test Your Tests Benoit Baudry KTH, Sweden 1 - - PowerPoint PPT Presentation

Aug 15, 2023 754 likes •1.25k views

Mutate and Test Your Tests Benoit Baudry KTH, Sweden 1 baudry@kth.se Test Your Tests What do you expect from test cases? Cover requirements Stress the application Prevent regressions Reveal bugs 2 Benoit Baudry, KTH,

slide-1
SLIDE 1

Mutate and Test Your Tests

Benoit Baudry KTH, Sweden

1

baudry@kth.se

slide-2
SLIDE 2

Test Your Tests

  • What do you expect from test cases?
  • Cover requirements
  • Stress the application
  • Prevent regressions
  • Reveal bugs

Benoit Baudry, KTH, EclipseCon'17

2

slide-3
SLIDE 3

Test Your Tests

  • What do you expect from test cases?
  • Cover requirements
  • Stress the application
  • Prevent regressions
  • Reveal bugs

Benoit Baudry, KTH, EclipseCon'17

3

slide-4
SLIDE 4

4

@Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); }

Benoit Baudry, KTH, EclipseCon'17

slide-5
SLIDE 5

long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;}

5

@Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); }

Benoit Baudry, KTH, EclipseCon'17

slide-6
SLIDE 6

Coverage

long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;}

6

@Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); }

Benoit Baudry, KTH, EclipseCon'17

slide-7
SLIDE 7

Coverage

7

@Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));} long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;}

Benoit Baudry, KTH, EclipseCon'17

slide-8
SLIDE 8

8

@Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));} long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;}

Are these test cases good at detecting bugs?

Benoit Baudry, KTH, EclipseCon'17

slide-9
SLIDE 9

9

@Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));} long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;}

Are these test cases good at detecting bugs? Let’s mutate our code to see.

Benoit Baudry, KTH, EclipseCon'17

slide-10
SLIDE 10

Mutation analysis

  • Tests are good if they can detect bugs
  • Principle: generate bugs and test the tests

10

Benoit Baudry, KTH, EclipseCon'17

slide-11
SLIDE 11

Mutation analysis

  • Tests are good if they can detect bugs
  • Principle: generate bugs and test the tests
  • Mutation operators
  • Mutant = Program with one mutation

11

Benoit Baudry, KTH, EclipseCon'17

slide-12
SLIDE 12

Mutation analysis

Inputs : P, TS, Ops Output : score, coverage M <- generateMutants (P, OPs) forAll (m in M) run (TS,m) if (one-test-fail) then killed <- m else alive <- m score = killed / size(M)

12

Benoit Baudry, KTH, EclipseCon'17

slide-13
SLIDE 13

PIT

  • Open source
  • Actively developed and maintained
  • State of the art
  • For Java
  • Integrates with major build systems

13

Benoit Baudry, KTH, EclipseCon'17

slide-14
SLIDE 14

PIT mutation operators

  • Conditions
  • Constants
  • Return
  • Delete method calls
  • Constructor call

14

Benoit Baudry, KTH, EclipseCon'17

slide-15
SLIDE 15

long fact(int n) { if(n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;}

15

Benoit Baudry, KTH, EclipseCon'17

slide-16
SLIDE 16

long fact(int n) { if(n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;}

16

n != 0 return 1+1 <

  • !(i<=n)

result/i result+1

Benoit Baudry, KTH, EclipseCon'17

slide-17
SLIDE 17

long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;}

17

n != 0 return 1+1 <

  • !(i<=n)

result/i result+1

Benoit Baudry, KTH, EclipseCon'17

slide-18
SLIDE 18

long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;}

18

@Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); }

Benoit Baudry, KTH, EclipseCon'17

n != 0 return 1+1 <

  • !(i<=n)

result/i result+1

slide-19
SLIDE 19

long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;}

19

@Test factorialWith5Test() { assertEqual(1, fact(0));}

Benoit Baudry, KTH, EclipseCon'17

n != 0 return 1+1 <

  • !(i<=n)

result/i result+1

slide-20
SLIDE 20

long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;}

20

@Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));}

Benoit Baudry, KTH, EclipseCon'17

n != 0 return 1+1 <

  • !(i<=n)

result/i result+1

slide-21
SLIDE 21

long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;}

Mutation reveals bugs in the test suite

21

@Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));}

Benoit Baudry, KTH, EclipseCon'17

Bugs in the test suite:

  • Weak oracle
  • Missing input

n != 0 return 1+1 <

  • !(i<=n)

result/i result+1

slide-22
SLIDE 22

long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;}

22

@Test factorialWith5Test() { long obs = fact(5);

assertEqual(120, obs); }

@Test factorialWith5Test() { assertEqual(1, fact(0));} @Test factorialWith1Test() { assertEqual(1, fact(1));}

Benoit Baudry, KTH, EclipseCon'17

n != 0 return 1+1 <

  • !(i<=n)

result/i result+1

slide-23
SLIDE 23

PIT – reporting

23

Benoit Baudry, KTH, EclipseCon'17

slide-24
SLIDE 24

PIT – reporting

24

Benoit Baudry, KTH, EclipseCon'17

slide-25
SLIDE 25

Project #LOC #Mutants Time (h) Score (%) Amazon Web Services SDK 1703892 2141690 04:25:35 76.28 XWiki Rendering Engine 104727 112609 01:59:55 50.89 Apache Commons Math 180951 104786 03:22:18 83.81 JFreeChart 134353 89592 00:41:28 58.04 Apache PdfBox 137099 79763 06:20:25 58.89 Java Git 128874 78316 16:02:00 73.86 SCIFIO 55347 62768 03:12:11 45.92 Joda-Time 85911 31233 00:16:32 81.65 Apache Commons Lang 60733 30361 00:21:02 86.17 Apache Commons Collections 51632 20394 00:05:41 85.94 Urban Airship Client Library 40885 17345 00:11:31 82.26 SAT4J 26415 17067 11:11:04 68.58 ImageJ Common 23868 15592 00:29:09 54.77 jsoup 17839 14054 00:12:49 78.34 Jaxen XPath Engine 21204 12210 00:24:40 67.13 Apache Commons Codec 17267 9233 00:07:57 87.82 Apache Commons IO 24334 8809 00:12:48 84.73 Google Gson 20068 7353 00:05:34 81.76 AuthZForce PDP Core 16602 7296 01:23:50 88.18 Apache Commons CLI 7005 2560 00:01:26 88.71 JOpt Simple 9214 2271 00:01:36 93.52

Benoit Baudry, KTH, EclipseCon'17

25

slide-26
SLIDE 26

Descartes

  • Mutation operators: extreme mutation
  • Active development
  • Open source
  • Compared to PIT
  • Less mutants
  • Different type of feedback
  • Same framework

26

Benoit Baudry, KTH, EclipseCon'17

slide-27
SLIDE 27

long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;}

Descartes - example

27

Benoit Baudry, KTH, EclipseCon'17

slide-28
SLIDE 28

long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;}

Descartes - example

28

long fact(int n){ return 0;} long fact(int n){ return 1;}

Benoit Baudry, KTH, EclipseCon'17

PIT : 7 mutants Descartes : 2 mutants

slide-29
SLIDE 29

Scalability of extreme mutation

Benoit Baudry, KTH, EclipseCon'17

29

public static boolean isValidXmlChar(int ch){ return (ch == 0x9) || (ch == 0xA) || (ch == 0xD) || (ch >= 0x20 && ch <= 0xD7FF) || (ch >= 0xE000 && ch <= 0xFFFD || (ch >= 0x10000 && ch <= 0x10FFFF); }

PIT : 45 mutants Descartes : 2 mutants

slide-30
SLIDE 30

Project Mutants PIT Mutants Descartes Time PIT Time Descartes Amazon Web Services SDK 2141690 161758 04:25:35 01:27:30 XWiki Rendering Engine 112609 5534 01:59:55 00:10:50 Apache Commons Math 104786 7150 03:22:18 00:08:30 JFreeChart 89592 7210 00:41:28 00:05:26 Apache PdfBox 79763 7559 06:20:25 00:42:11 Java Git 78316 7152 16:02:00 00:56:07 SCIFIO 62768 3627 03:12:11 00:15:26 Joda-Time 31233 4525 00:16:32 00:04:13 Apache Commons Lang 30361 3872 00:21:02 00:02:18 Apache Commons Collections 20394 3558 00:05:41 00:01:48 Urban Airship Client Library 17345 3082 00:11:31 00:09:38 SAT4J 17067 2296 11:11:04 00:56:42 ImageJ Common 15592 1947 00:29:09 00:04:08 jsoup 14054 1566 00:12:49 00:02:45 Jaxen XPath Engine 12210 1252 00:24:40 00:01:34 Apache Commons Codec 9233 979 00:07:57 00:02:02 Apache Commons IO 8809 1164 00:12:48 00:02:18 Google Gson 7353 848 00:05:34 00:01:11 AuthZForce PDP Core 7296 626 01:23:50 00:08:45 Apache Commons CLI 2560 271 00:01:26 00:00:09 JOpt Simple 2271 412 00:01:36 00:00:25

Benoit Baudry, KTH, EclipseCon'17

30

slide-31
SLIDE 31

Coarser grain than PIT

Benoit Baudry, KTH, EclipseCon'17

31

long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} long fact(int n){ return 0;} long fact(int n){ return 1;}

@Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));}

slide-32
SLIDE 32

Benoit Baudry, KTH, EclipseCon'17

32

bool equals(object other) { return other instanceof AClass &&((AClass) other).aField==aField; } bool equals(object other) { return true;} bool equals(object other) { return false;}

slide-33
SLIDE 33

Benoit Baudry, KTH, EclipseCon'17

33

bool equals(object other) { return other instanceof AClass &&((AClass) other).aField==aField; } bool equals(object other) { return true;} bool equals(object other) { return false;}

test() { AClass a = new AClass(3); AClass b = new AClass(3); AClass c = new AClass(4); assertEquals(a, b); assertFalse(a == c); }

slide-34
SLIDE 34

Benoit Baudry, KTH, EclipseCon'17

34

bool equals(object other) { return other instanceof AClass &&((AClass) other).aField==aField; } bool equals(object other) { return true;} bool equals(object other) { return false;}

test() { AClass a = new AClass(3); AClass b = new AClass(3); AClass c = new AClass(4); assertEquals(a, b); assertFalse(a == c); }

slide-35
SLIDE 35

Wrong test

Benoit Baudry, KTH, EclipseCon'17

35

bool equals(object other) { return other instanceof AClass &&((AClass) other).aField==aField; } bool equals(object other) { return true;} bool equals(object other) { return false;}

test() { AClass a = new AClass(3); AClass b = new AClass(3); AClass c = new AClass(4); assertEquals(a, b); assertFalse(a == c); }

slide-36
SLIDE 36

Benoit Baudry, KTH, EclipseCon'17

36

protected final void prepareSocket(final SSLSocket socket) { }}

slide-37
SLIDE 37

Benoit Baudry, KTH, EclipseCon'17

37

@Test void typical() throws NoSuchAlgorithmException { SdkTLSSocketFactory f = new SdkTLSSocketF(SSLContext.getDefault(),null); f.prepareSocket(new TestSSLSocket() { @Override public void setEnabledProtocols(String[] protocols) { assertTrue(Arrays.equals(protocols, new String[] {"TLSv1.2", "TLSv1.1", "TLSv1" }});}}} protected final void prepareSocket(final SSLSocket socket) { }}

slide-38
SLIDE 38

Benoit Baudry, KTH, EclipseCon'17

38

@Test void typical() throws NoSuchAlgorithmException { SdkTLSSocketFactory f = new SdkTLSSocketF(SSLContext.getDefault(),null); f.prepareSocket(new TestSSLSocket() { @Override public void setEnabledProtocols(String[] protocols) { assertTrue(Arrays.equals(protocols, new String[] {"TLSv1.2", "TLSv1.1", "TLSv1" }});}}} protected final void prepareSocket(final SSLSocket socket) { }}

slide-39
SLIDE 39

Missing oracle

Benoit Baudry, KTH, EclipseCon'17

39

@Test void typical() throws NoSuchAlgorithmException { SdkTLSSocketFactory f = new SdkTLSSocketF(SSLContext.getDefault(),null); f.prepareSocket(new TestSSLSocket() { @Override public void setEnabledProtocols(String[] protocols) { assertTrue(Arrays.equals(protocols, new String[] {"TLSv1.2", "TLSv1.1", "TLSv1" }});}}} protected final void prepareSocket(final SSLSocket socket) { }}

slide-40
SLIDE 40

Benoit Baudry, KTH, EclipseCon'17

40

private boolean isLongOption(final String token){ if (...){return false;} //Covered 27 times ... if (...){return true;} //Not covered else if (...){return true;} //Covered once return false; //Covered 7 times }

slide-41
SLIDE 41

Benoit Baudry, KTH, EclipseCon'17

41

private boolean isLongOption(final String token){ return false;} private boolean isLongOption(final String token){ return true;}

slide-42
SLIDE 42

Benoit Baudry, KTH, EclipseCon'17

42

... if (condition && isLongOption(value)){ action1(value);} else{ action2(value);} ... private boolean isLongOption(final String token){ return false;} private boolean isLongOption(final String token){ return true;}

slide-43
SLIDE 43

Benoit Baudry, KTH, EclipseCon'17

43

... if (condition && isLongOption(value)){ action1(value);} else{ action2(value);} ... private boolean isLongOption(final String token){ return false;} private boolean isLongOption(final String token){ return true;}

slide-44
SLIDE 44

Benoit Baudry, KTH, EclipseCon'17

44

... if (condition && isLongOption(value)){ action1(value);} else{ action2(value);} ... private boolean isLongOption(final String token){ return false;} private boolean isLongOption(final String token){ return true;}

Testability issue

slide-45
SLIDE 45

45

15 285 4 15 54 28 64 132 23 115 14 594 369 92 9 40 158 458 116 65 259 2 69 2 14 28 19 27 55 10 17 12 97 85 31 2 14 11 61 20 69 197 274 1516 175 408 1178 610 1840 4245 459 344 585 2931 2237 2596 252 714 456 1726 425 2004 1618 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Tested Weak pseudo-tested Strong pseudo-tested Benoit Baudry, KTH, EclipseCon'17

slide-46
SLIDE 46

Descartes – future

  • Incremental Descartes in the CI
  • Github hook
  • Support multi module Maven projects
  • Enhanced diagnosis
  • Integrate into a test augmentation pipeline

Benoit Baudry, KTH, EclipseCon'17

46

slide-47
SLIDE 47

Conclusion

  • Mutation analysis
  • Automatic generation of mutants
  • Evaluate the test suite
  • Bugs in test suites
  • Oracle
  • Input space coverage
  • Testability
  • Indirectly tested code

47

Benoit Baudry, KTH, EclipseCon'17

slide-48
SLIDE 48

Feedback welcome!

  • https://github.com/STAMP-project/pitest-descartes
  • https://github.com/hcoles/pitest
  • http://stamp-project.eu/

baudry@kth.se

48

Benoit Baudry, KTH, EclipseCon'17