Do Redundant Mutants Affect the Effectiveness and Efficiency of - - PowerPoint PPT Presentation

▶
do redundant mutants affect the effectiveness and
SMART_READER_LITE
LIVE PREVIEW

Do Redundant Mutants Affect the Effectiveness and Efficiency of - - PowerPoint PPT Presentation

Aug 21, 2022 102 likes •782 views

Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis? Ren Just 1 & Gregory M. Kapfhammer 2 & Franz Schweiggert 1 1 Ulm University, Germany 2 Allegheny College, USA 7th International Workshop on Mutation

slide-1
SLIDE 1

Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

René Just1 & Gregory M. Kapfhammer2 & Franz Schweiggert1

1Ulm University, Germany 2Allegheny College, USA

7th International Workshop on Mutation Analysis Montreal, Canada April 17, 2012

slide-2
SLIDE 2

Introduction Operator Empirical Study Conclusion

Overview of the Presentation

Challenges in Mutation Analysis

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-3
SLIDE 3

Introduction Operator Empirical Study Conclusion

Overview of the Presentation

Challenges in Mutation Analysis Equivalent Mutants

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-4
SLIDE 4

Introduction Operator Empirical Study Conclusion

Overview of the Presentation

Challenges in Mutation Analysis Equivalent Mutants Cannot be killed

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-5
SLIDE 5

Introduction Operator Empirical Study Conclusion

Overview of the Presentation

Challenges in Mutation Analysis Equivalent Mutants Cannot be killed Redundant Mutants

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-6
SLIDE 6

Introduction Operator Empirical Study Conclusion

Overview of the Presentation

Challenges in Mutation Analysis Equivalent Mutants Cannot be killed Redundant Mutants Subsumed by

  • ther mutants

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-7
SLIDE 7

Introduction Operator Empirical Study Conclusion

Overview of the Presentation

Challenges in Mutation Analysis Equivalent Mutants Cannot be killed Redundant Mutants Subsumed by

  • ther mutants

Harmful to efficiency and effectiveness

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-8
SLIDE 8

Introduction Operator Empirical Study Conclusion

Overview of the Presentation

Redundant mutants

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-9
SLIDE 9

Introduction Operator Empirical Study Conclusion

Overview of the Presentation

Redundant mutants Operator for Condi- tional Expressions without redundancy

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-10
SLIDE 10

Introduction Operator Empirical Study Conclusion

Overview of the Presentation

Redundant mutants Operator for Condi- tional Expressions without redundancy Conditional Operator Replacement (COR)

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-11
SLIDE 11

Introduction Operator Empirical Study Conclusion

Overview of the Presentation

Redundant mutants Operator for Condi- tional Expressions without redundancy Conditional Operator Replacement (COR) Unary Operator Insertion (UOI)

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-12
SLIDE 12

Introduction Operator Empirical Study Conclusion

Overview of the Presentation

Redundant mutants Operator for Condi- tional Expressions without redundancy Conditional Operator Replacement (COR) Unary Operator Insertion (UOI) Empirical Study

  • n the impact of

redundant mutants

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-13
SLIDE 13

Introduction Operator Empirical Study Conclusion

Overview of the Presentation

Redundant mutants Operator for Condi- tional Expressions without redundancy Conditional Operator Replacement (COR) Unary Operator Insertion (UOI) Empirical Study

  • n the impact of

redundant mutants Efficiency

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-14
SLIDE 14

Introduction Operator Empirical Study Conclusion

Overview of the Presentation

Redundant mutants Operator for Condi- tional Expressions without redundancy Conditional Operator Replacement (COR) Unary Operator Insertion (UOI) Empirical Study

  • n the impact of

redundant mutants Efficiency Number of mutants and runtime

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-15
SLIDE 15

Introduction Operator Empirical Study Conclusion

Overview of the Presentation

Redundant mutants Operator for Condi- tional Expressions without redundancy Conditional Operator Replacement (COR) Unary Operator Insertion (UOI) Empirical Study

  • n the impact of

redundant mutants Efficiency Number of mutants and runtime Effectiveness

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-16
SLIDE 16

Introduction Operator Empirical Study Conclusion

Overview of the Presentation

Redundant mutants Operator for Condi- tional Expressions without redundancy Conditional Operator Replacement (COR) Unary Operator Insertion (UOI) Empirical Study

  • n the impact of

redundant mutants Efficiency Number of mutants and runtime Effectiveness Accuracy

  • f mutation

score

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-17
SLIDE 17

Introduction Operator Empirical Study Conclusion

Mutating Conditional Expressions

a <op> b

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-18
SLIDE 18

Introduction Operator Empirical Study Conclusion

Mutating Conditional Expressions

a <op> b && ||

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-19
SLIDE 19

Introduction Operator Empirical Study Conclusion

Mutating Conditional Expressions

a <op> b && || Conditional Operator Replacement (COR)

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-20
SLIDE 20

Introduction Operator Empirical Study Conclusion

Mutating Conditional Expressions

a <op> b && || Conditional Operator Replacement (COR) a && b a || b a == b a != b

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-21
SLIDE 21

Introduction Operator Empirical Study Conclusion

Mutating Conditional Expressions

a <op> b && || Conditional Operator Replacement (COR) a && b a || b a == b a != b lhs rhs true false

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-22
SLIDE 22

Introduction Operator Empirical Study Conclusion

Mutating Conditional Expressions

a <op> b && || Conditional Operator Replacement (COR) a && b a || b a == b a != b lhs rhs true false Unary Operator Insertion (UOI)

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-23
SLIDE 23

Introduction Operator Empirical Study Conclusion

Mutating Conditional Expressions

a <op> b && || Conditional Operator Replacement (COR) a && b a || b a == b a != b lhs rhs true false Unary Operator Insertion (UOI) !a <op> b !(a <op> b) a <op> !b

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-24
SLIDE 24

Introduction Operator Empirical Study Conclusion

Mutating Conditional Expressions

Literals Expression a b a && b 1 1 1 1 1 Literals Expression a b a b 1 1 1 1 1 1 1 Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-25
SLIDE 25

Introduction Operator Empirical Study Conclusion

Mutating Conditional Expressions

Literals Expression a b a && b false lhs rhs a == b a b a != b true !a && b !(a && b) a && !b 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Literals Expression a b a b a != b rhs lhs true a && b a == b false !a b !(a b) a !b 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-26
SLIDE 26

Introduction Operator Empirical Study Conclusion

Mutating Conditional Expressions

Literals Expression Sufficient mutations Subsumed mutations Subsumed operator UOI a b a && b false lhs rhs a == b a b a != b true !a && b !(a && b) a && !b 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Literals Expression Sufficient mutations Subsumed mutations Subsumed operator UOI a b a b a != b rhs lhs true a && b a == b false !a b !(a b) a !b 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-27
SLIDE 27

Introduction Operator Empirical Study Conclusion

Mutating Conditional Expressions

Literals Expression Sufficient mutations Subsumed mutations Subsumed operator UOI a b a && b false lhs rhs a == b a b a != b true !a && b !(a && b) a && !b 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Literals Expression Sufficient mutations Subsumed mutations Subsumed operator UOI a b a b a != b rhs lhs true a && b a == b false !a b !(a b) a !b 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

4 Mutants are sufficient

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-28
SLIDE 28

Introduction Operator Empirical Study Conclusion

Mutating Conditional Expressions

Literals Expression Sufficient mutations Subsumed mutations Subsumed operator UOI a b a && b false lhs rhs a == b a b a != b true !a && b !(a && b) a && !b 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Literals Expression Sufficient mutations Subsumed mutations Subsumed operator UOI a b a b a != b rhs lhs true a && b a == b false !a b !(a b) a !b 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

4 Mutants are sufficient UOI Operator completely subsumed

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-29
SLIDE 29

Introduction Operator Empirical Study Conclusion

Mutating Conditional Expressions

Literals Expression Sufficient mutations Subsumed mutations Subsumed operator UOI a b a && b false lhs rhs a == b a b a != b true !a && b !(a && b) a && !b 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Literals Expression Sufficient mutations Subsumed mutations Subsumed operator UOI a b a b a != b rhs lhs true a && b a == b false !a b !(a b) a !b 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

4 Mutants are sufficient UOI Operator completely subsumed A reduction of exactly 60% ?

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-30
SLIDE 30

Introduction Operator Empirical Study Conclusion

Mutating Conditional Expressions

Two common patterns for short-circuit operators

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-31
SLIDE 31

Introduction Operator Empirical Study Conclusion

Mutating Conditional Expressions

Two common patterns for short-circuit operators

public void foo(int x){ Var v; if(flag&&(v=getVar())!=null) { v.bar(x); } ... }

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-32
SLIDE 32

Introduction Operator Empirical Study Conclusion

Mutating Conditional Expressions

Two common patterns for short-circuit operators

public void foo(int x){ Var v; if(flag&&(v=getVar())!=null) { v.bar(x); } ... }

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-33
SLIDE 33

Introduction Operator Empirical Study Conclusion

Mutating Conditional Expressions

Two common patterns for short-circuit operators

public void foo(int x){ Var v; if(flag&&(v=getVar())!=null) { v.bar(x); } ... }

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-34
SLIDE 34

Introduction Operator Empirical Study Conclusion

Mutating Conditional Expressions

Two common patterns for short-circuit operators

public void foo(int x){ Var v; if(flag&&(v=getVar())!=null) { v.bar(x); } ... }

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-35
SLIDE 35

Introduction Operator Empirical Study Conclusion

Mutating Conditional Expressions

Two common patterns for short-circuit operators

public void foo(int x){ Var v; if(flag&&(v=getVar())!=null) { v.bar(x); } ... } public void foo(int x){ Var v; if(flag||(v=getVar())!=null) { v.bar(x); } ... }

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-36
SLIDE 36

Introduction Operator Empirical Study Conclusion

Mutating Conditional Expressions

Two common patterns for short-circuit operators

public void foo(int x){ Var v; if(flag&&(v=getVar())!=null) { v.bar(x); } ... } public void foo(int x){ Var v; if(flag||(v=getVar())!=null) { v.bar(x); } ... }

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-37
SLIDE 37

Introduction Operator Empirical Study Conclusion

Mutating Conditional Expressions

Two common patterns for short-circuit operators

public void foo(int x){ Var v; if(flag&&(v=getVar())!=null) { v.bar(x); } ... } public void foo(int x){ Var v; if(flag||(v=getVar())!=null) { v.bar(x); } ... }

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-38
SLIDE 38

Introduction Operator Empirical Study Conclusion

Mutating Conditional Expressions

Two common patterns for short-circuit operators

public void foo(int x){ Var v; if(flag&&(v=getVar())!=null) { v.bar(x); } ... }

Properly handled by MAJOR

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-39
SLIDE 39

Introduction Operator Empirical Study Conclusion

Mutating Conditional Expressions

Two common patterns for short-circuit operators

public void foo(int x){ Var v; if(flag&&(v=getVar())!=null) { v.bar(x); } ... } public void foo(int x){ Var v; if(flag||(v=getVar())==null) { return; } v.bar(x); }

Properly handled by MAJOR Properly handled by MAJOR

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-40
SLIDE 40

Introduction Operator Empirical Study Conclusion

Mutating Conditional Expressions

Two common patterns for short-circuit operators

public void foo(int x){ Var v; if(flag&&(v=getVar())!=null) { v.bar(x); } ... } public void foo(int x){ Var v; if(flag||(v=getVar())==null) { return; } v.bar(x); }

Properly handled by MAJOR Properly handled by MAJOR A reduction of up to 60%

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-41
SLIDE 41

Introduction Operator Empirical Study Conclusion

COR and ROR Mutation Operators

Subsumption hierarchy Conditional Operator Replacement (COR) Reduction of up to 60%

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-42
SLIDE 42

Introduction Operator Empirical Study Conclusion

COR and ROR Mutation Operators

Subsumption hierarchy Conditional Operator Replacement (COR) Reduction of up to 60% Relational Operator Replacement (ROR)

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-43
SLIDE 43

Introduction Operator Empirical Study Conclusion

COR and ROR Mutation Operators

Subsumption hierarchy Conditional Operator Replacement (COR) Reduction of up to 60% Relational Operator Replacement (ROR) Kaminski et al. AST’11

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-44
SLIDE 44

Introduction Operator Empirical Study Conclusion

COR and ROR Mutation Operators

Subsumption hierarchy Conditional Operator Replacement (COR) Reduction of up to 60% Relational Operator Replacement (ROR) Kaminski et al. AST’11 Reduction of 50%

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-45
SLIDE 45

Introduction Operator Empirical Study Conclusion

COR and ROR Mutation Operators

Subsumption hierarchy Conditional Operator Replacement (COR) Reduction of up to 60% Relational Operator Replacement (ROR) Kaminski et al. AST’11 Reduction of 50% How prevalent are COR and ROR mutants?

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-46
SLIDE 46

Introduction Operator Empirical Study Conclusion

Investigated Applications

Files LOC Tests Mutants Generated Mutants Covered commons-math 408 39,991 2,169 80,372 72,203 commons-lang 99 19,495 2,039 31,130 29,069 commons-io 100 7,908 309 9,547 4,935 numerics4j 73 3,647 218 6,835 6,547

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-47
SLIDE 47

Introduction Operator Empirical Study Conclusion

Investigated Applications

Files LOC Tests Mutants Generated Mutants Covered commons-math 408 39,991 2,169 80,372 72,203 commons-lang 99 19,495 2,039 31,130 29,069 commons-io 100 7,908 309 9,547 4,935 numerics4j 73 3,647 218 6,835 6,547

Application differ in size and complexity

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-48
SLIDE 48

Introduction Operator Empirical Study Conclusion

Investigated Applications

Files LOC Tests Mutants Generated Mutants Covered commons-math 408 39,991 2,169 80,372 72,203 commons-lang 99 19,495 2,039 31,130 29,069 commons-io 100 7,908 309 9,547 4,935 numerics4j 73 3,647 218 6,835 6,547

Application differ in size and complexity Differences in mutation coverage

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-49
SLIDE 49

Introduction Operator Empirical Study Conclusion

Investigated Applications

Files LOC Tests Mutants Generated Mutants Covered commons-math 408 39,991 2,169 80,372 72,203 commons-lang 99 19,495 2,039 31,130 29,069 commons-io 100 7,908 309 9,547 4,935 numerics4j 73 3,647 218 6,835 6,547

Application differ in size and complexity Differences in mutation coverage How prevalent are COR and ROR mutants?

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-50
SLIDE 50

Introduction Operator Empirical Study Conclusion

Ratio of COR and ROR Mutants

20,000 40,000 60,000 80,000 numerics4j commons-io commons-lang commons-math 19.5% 14.5% 22.3% 5.1% 19.3% 26.9% 35.5% 24.3% Number of generated mutants All mutants ROR mutants COR mutants

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-51
SLIDE 51

Introduction Operator Empirical Study Conclusion

Ratio of COR and ROR Mutants

20,000 40,000 60,000 80,000 numerics4j commons-io commons-lang commons-math 19.5% 14.5% 22.3% 5.1% 19.3% 26.9% 35.5% 24.3% Number of generated mutants All mutants ROR mutants COR mutants

COR and ROR generate up to 58% of all mutants

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-52
SLIDE 52

Introduction Operator Empirical Study Conclusion

Ratio of COR and ROR Mutants

20,000 40,000 60,000 80,000 numerics4j commons-io commons-lang commons-math 19.5% 14.5% 22.3% 5.1% 19.3% 26.9% 35.5% 24.3% Number of generated mutants All mutants ROR mutants COR mutants

COR and ROR generate up to 58% of all mutants Indicates notable potential for improvements

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-53
SLIDE 53

Introduction Operator Empirical Study Conclusion

Ratio of COR and ROR Mutants

20,000 40,000 60,000 80,000 numerics4j commons-io commons-lang commons-math 19.5% 14.5% 22.3% 5.1% 19.3% 26.9% 35.5% 24.3% Number of generated mutants All mutants ROR mutants COR mutants

COR and ROR generate up to 58% of all mutants Indicates notable potential for improvements How much is the overall reduction?

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-54
SLIDE 54

Introduction Operator Empirical Study Conclusion

Decrease in Number of Mutants

Generated Generated Covered Covered (original set) (reduced set) (original set) (reduced set) commons-math 80,372 66,787 (-16.9%) 72,203 59,195 (-18.0%) commons-lang 31,130 21,074 (-32.3%) 29,069 19,112 (-34.3%) commons-io 9,547 7,319 (-23.3%) 4,935 4,168 (-15.5%) numerics4j 6,835 5,437 (-20.5%) 6,547 5,149 (-21.4%)

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-55
SLIDE 55

Introduction Operator Empirical Study Conclusion

Decrease in Number of Mutants

Generated Generated Covered Covered (original set) (reduced set) (original set) (reduced set) commons-math 80,372 66,787 (-16.9%) 72,203 59,195 (-18.0%) commons-lang 31,130 21,074 (-32.3%) 29,069 19,112 (-34.3%) commons-io 9,547 7,319 (-23.3%) 4,935 4,168 (-15.5%) numerics4j 6,835 5,437 (-20.5%) 6,547 5,149 (-21.4%)

Overall reduction

  • f up to 34%

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-56
SLIDE 56

Introduction Operator Empirical Study Conclusion

Decrease in Number of Mutants

Generated Generated Covered Covered (original set) (reduced set) (original set) (reduced set) commons-math 80,372 66,787 (-16.9%) 72,203 59,195 (-18.0%) commons-lang 31,130 21,074 (-32.3%) 29,069 19,112 (-34.3%) commons-io 9,547 7,319 (-23.3%) 4,935 4,168 (-15.5%) numerics4j 6,835 5,437 (-20.5%) 6,547 5,149 (-21.4%)

Overall reduction

  • f up to 34%

Depends on mutation coverage

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-57
SLIDE 57

Introduction Operator Empirical Study Conclusion

Decrease in Number of Mutants

Generated Generated Covered Covered (original set) (reduced set) (original set) (reduced set) commons-math 80,372 66,787 (-16.9%) 72,203 59,195 (-18.0%) commons-lang 31,130 21,074 (-32.3%) 29,069 19,112 (-34.3%) commons-io 9,547 7,319 (-23.3%) 4,935 4,168 (-15.5%) numerics4j 6,835 5,437 (-20.5%) 6,547 5,149 (-21.4%)

Overall reduction

  • f up to 34%

Depends on mutation coverage How much is the saving in runtime?

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-58
SLIDE 58

Introduction Operator Empirical Study Conclusion

Runtime Improvement

Runtime Runtime (original set) (reduced set) commons-math 300.77 271.10 (-09.9%) commons-lang 28.25 18.70 (-33.8%) commons-io 6.95 4.58 (-34.1%) numerics4j 2.85 2.08 (-26.9%)

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-59
SLIDE 59

Introduction Operator Empirical Study Conclusion

Runtime Improvement

Runtime Runtime (original set) (reduced set) commons-math 300.77 271.10 (-09.9%) commons-lang 28.25 18.70 (-33.8%) commons-io 6.95 4.58 (-34.1%) numerics4j 2.85 2.08 (-26.9%)

Divergence due to differences in test suite runtime and coverage

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-60
SLIDE 60

Introduction Operator Empirical Study Conclusion

Runtime Improvement

Runtime Runtime (original set) (reduced set) commons-math 300.77 271.10 (-09.9%) commons-lang 28.25 18.70 (-33.8%) commons-io 6.95 4.58 (-34.1%) numerics4j 2.85 2.08 (-26.9%)

Divergence due to differences in test suite runtime and coverage Significant speed-up for all applications

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-61
SLIDE 61

Introduction Operator Empirical Study Conclusion

Accuracy of the Mutation Score

Mutation Score Mutation Score (original set) (reduced set) commons-math 0.77 0.73 (- 4.5%) commons-lang 0.76 0.67 (-10.7%) commons-io 0.41 0.44 ( 8.3%) numerics4j 0.69 0.65 (- 5.9%)

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-62
SLIDE 62

Introduction Operator Empirical Study Conclusion

Accuracy of the Mutation Score

Mutation Score Mutation Score (original set) (reduced set) commons-math 0.77 0.73 (- 4.5%) commons-lang 0.76 0.67 (-10.7%) commons-io 0.41 0.44 ( 8.3%) numerics4j 0.69 0.65 (- 5.9%)

Mutation Score up to 10%

  • verestimated

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-63
SLIDE 63

Introduction Operator Empirical Study Conclusion

Accuracy of the Mutation Score

Mutation Score Mutation Score (original set) (reduced set) commons-math 0.77 0.73 (- 4.5%) commons-lang 0.76 0.67 (-10.7%) commons-io 0.41 0.44 ( 8.3%) numerics4j 0.69 0.65 (- 5.9%)

Mutation Score up to 10%

  • verestimated

Low mutation coverage may lead to underestimation

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-64
SLIDE 64

Introduction Operator Empirical Study Conclusion

Accuracy of the Mutation Score

Mutation Score Mutation Score (original set) (reduced set) commons-math 0.77 0.73 (- 4.5%) commons-lang 0.76 0.67 (-10.7%) commons-io 0.41 0.44 ( 8.3%) numerics4j 0.69 0.65 (- 5.9%)

Mutation Score up to 10%

  • verestimated

Low mutation coverage may lead to underestimation Redundant mutants tend to overestimate the mutation score

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-65
SLIDE 65

Introduction Operator Empirical Study Conclusion

Conclusion and Future Work

Conclusion: Operator for conditional expressions without redundancy Decreased number of mutants and improved runtime Increased accuracy of the mutation score

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-66
SLIDE 66

Introduction Operator Empirical Study Conclusion

Conclusion and Future Work

Conclusion: Operator for conditional expressions without redundancy Decreased number of mutants and improved runtime Increased accuracy of the mutation score Future Work: Investigate redundancies in other mutation operators Analyze whether sufficient mutants tend to be equivalent Apply constraint solver to identify equivalent mutants

Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

slide-67
SLIDE 67

Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis? Thank you for your attention! Questions?

http://www.mathematik.uni-ulm.de/sai/major