MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a - - PowerPoint PPT Presentation

▶
major an efficient and extensible tool for mutation
SMART_READER_LITE
LIVE PREVIEW

MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a - - PowerPoint PPT Presentation

May 11, 2023 412 likes •750 views

MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler e Just 1 , Franz Schweiggert 1 , and Gregory M. Kapfhammer 2 Ren 1 Ulm University, Germany 2 Allegheny College, USA 26th International Conference on Automated

slide-1
SLIDE 1

MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

Ren´ e Just1, Franz Schweiggert1, and Gregory M. Kapfhammer2

1Ulm University, Germany 2Allegheny College, USA

26th International Conference on Automated Software Engineering Lawrence, Kansas, USA November 6 - 12, 2011

slide-2
SLIDE 2

Introduction MAJOR Conclusion

Overview of MAJOR

A Tool for Mutation Analysis

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-3
SLIDE 3

Introduction MAJOR Conclusion

Overview of MAJOR

A Tool for Mutation Analysis Compiler- Integrated

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-4
SLIDE 4

Introduction MAJOR Conclusion

Overview of MAJOR

A Tool for Mutation Analysis Compiler- Integrated Fast and Scalable

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-5
SLIDE 5

Introduction MAJOR Conclusion

Overview of MAJOR

A Tool for Mutation Analysis Compiler- Integrated Domain Specific Language Fast and Scalable

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-6
SLIDE 6

Introduction MAJOR Conclusion

Overview of MAJOR

A Tool for Mutation Analysis Compiler- Integrated Domain Specific Language Fast and Scalable Configurable and Extensible

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-7
SLIDE 7

Introduction MAJOR Conclusion

Overview of MAJOR

A Tool for Mutation Analysis Compiler- Integrated Domain Specific Language Mutation Coverage Information Fast and Scalable Configurable and Extensible

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-8
SLIDE 8

Introduction MAJOR Conclusion

Overview of MAJOR

A Tool for Mutation Analysis Compiler- Integrated Domain Specific Language Mutation Coverage Information Fast and Scalable Configurable and Extensible Enables Optimized Workflow

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-9
SLIDE 9

Introduction MAJOR Conclusion

Overview of Mutation Analysis

Mutation Analysis

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-10
SLIDE 10

Introduction MAJOR Conclusion

Overview of Mutation Analysis

Mutation Analysis Methodically inject small syntactical faults into the program under test

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-11
SLIDE 11

Introduction MAJOR Conclusion

Overview of Mutation Analysis

Mutation Analysis Methodically inject small syntactical faults into the program under test Unbiased and powerful method for assessing test

  • racles and input values

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-12
SLIDE 12

Introduction MAJOR Conclusion

Overview of Mutation Analysis

Mutation Analysis Methodically inject small syntactical faults into the program under test Unbiased and powerful method for assessing test

  • racles and input values

Useful method for fault seeding during the empirical study of testing techniques

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-13
SLIDE 13

Introduction MAJOR Conclusion

Overview of Mutation Analysis

public int eval(int x){ int a=3, b=1, y; y = a * x; y += b; return y; } public int max(int a, int b){ int max = a; if(b>a){ max=b; } return max; }

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-14
SLIDE 14

Introduction MAJOR Conclusion

Overview of Mutation Analysis

public int eval(int x){ int a=3, b=1, y; y = a * x; y += b; return y; } public int max(int a, int b){ int max = a; if(b>a){ max=b; } return max; }

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-15
SLIDE 15

Introduction MAJOR Conclusion

Overview of Mutation Analysis

public int eval(int x){ int a=3, b=1, y; y = a * x; y += b; return y; } public int max(int a, int b){ int max = a; if(b>a){ max=b; } return max; }

= ⇒ = ⇒ y = a - x; y = a + x; y = a / x; if(b < a) if(b != a) if(b == a)

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-16
SLIDE 16

Introduction MAJOR Conclusion

MAJOR’s Compiler

MAJOR’s Compiler

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-17
SLIDE 17

Introduction MAJOR Conclusion

MAJOR’s Compiler

MAJOR’s Compiler Enhanced Standard Java Compiler

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-18
SLIDE 18

Introduction MAJOR Conclusion

MAJOR’s Compiler

MAJOR’s Compiler Enhanced Standard Java Compiler Source Files

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-19
SLIDE 19

Introduction MAJOR Conclusion

MAJOR’s Compiler

MAJOR’s Compiler Enhanced Standard Java Compiler Source Files Common Compiler Options

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-20
SLIDE 20

Introduction MAJOR Conclusion

MAJOR’s Compiler

MAJOR’s Compiler Enhanced Standard Java Compiler Source Files Common Compiler Options Domain Specific Language

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-21
SLIDE 21

Introduction MAJOR Conclusion

MAJOR’s Compiler

MAJOR’s Compiler Enhanced Standard Java Compiler Source Files Common Compiler Options Domain Specific Language Bytecode with Embedded Mutants

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-22
SLIDE 22

Introduction MAJOR Conclusion

MAJOR’s Domain Specific Language

// variable declaration listCOR={&&, ||, ==, !=}; // Define replacement list BIN(+)<"org"> -> {-,*}; BIN(*)<"org"> -> {/,%}; // Define own operator myOp{ BIN(&&) -> listCOR; BIN(||) -> listCOR; COR; LVR; } // Enable built-in operator AOR AOR<"org">; // Enable operator myOp myOp<"java.lang.System@println">;

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-23
SLIDE 23

Introduction MAJOR Conclusion

MAJOR’s Domain Specific Language

// variable declaration listCOR={&&, ||, ==, !=}; // Define replacement list BIN(+)<"org"> -> {-,*}; BIN(*)<"org"> -> {/,%}; // Define own operator myOp{ BIN(&&) -> listCOR; BIN(||) -> listCOR; COR; LVR; } // Enable built-in operator AOR AOR<"org">; // Enable operator myOp myOp<"java.lang.System@println">;

Specify mutation

  • perators in detail

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-24
SLIDE 24

Introduction MAJOR Conclusion

MAJOR’s Domain Specific Language

// variable declaration listCOR={&&, ||, ==, !=}; // Define replacement list BIN(+)<"org"> -> {-,*}; BIN(*)<"org"> -> {/,%}; // Define own operator myOp{ BIN(&&) -> listCOR; BIN(||) -> listCOR; COR; LVR; } // Enable built-in operator AOR AOR<"org">; // Enable operator myOp myOp<"java.lang.System@println">;

Specify mutation

  • perators in detail

Define own mutation

  • perator groups

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-25
SLIDE 25

Introduction MAJOR Conclusion

MAJOR’s Domain Specific Language

// variable declaration listCOR={&&, ||, ==, !=}; // Define replacement list BIN(+)<"org"> -> {-,*}; BIN(*)<"org"> -> {/,%}; // Define own operator myOp{ BIN(&&) -> listCOR; BIN(||) -> listCOR; COR; LVR; } // Enable built-in operator AOR AOR<"org">; // Enable operator myOp myOp<"java.lang.System@println">;

Specify mutation

  • perators in detail

Define own mutation

  • perator groups

Enable operators for a specific package, class, or method

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-26
SLIDE 26

Introduction MAJOR Conclusion

Optimized Mutation Analysis Process

1

Embed and compile all mutants

2

Run test suite on instrumented program

3

Sort tests according to their runtime

4

Perform mutation analysis with reordered test suite

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-27
SLIDE 27

Introduction MAJOR Conclusion

Optimized Mutation Analysis Process

1

Embed and compile all mutants

2

Run test suite on instrumented program

3

Sort tests according to their runtime

4

Perform mutation analysis with reordered test suite

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-28
SLIDE 28

Introduction MAJOR Conclusion

Optimized Mutation Analysis Process

1

Embed and compile all mutants

2

Run test suite on instrumented program

3

Sort tests according to their runtime

4

Perform mutation analysis with reordered test suite

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-29
SLIDE 29

Introduction MAJOR Conclusion

Optimized Mutation Analysis Process

1

Embed and compile all mutants

2

Run test suite on instrumented program

3

Sort tests according to their runtime

4

Perform mutation analysis with reordered test suite

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-30
SLIDE 30

Introduction MAJOR Conclusion

Conclusion

Key Concepts and Features: Compiler-integrated solution Furnishes its own domain specific language Provides mutation coverage information

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-31
SLIDE 31

Introduction MAJOR Conclusion

Conclusion

Key Concepts and Features: Compiler-integrated solution Furnishes its own domain specific language Provides mutation coverage information Characteristics of MAJOR: Fast and scalable technique Configurable and extensible mutation tool Enables an optimized workflow for mutation analysis

Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

slide-32
SLIDE 32

Do you want to learn more details about MAJOR?

MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler

Ren´ e Just1 and Franz Schweiggert1 and Gregory M. Kapfhammer2

1Department of Applied Information Processing, Ulm University 2Department of Computer Science, Allegheny College

IMPORTANT CONTRIBUTIONS

◮ Enhances the Java 6 Standard Edition compiler ◮ Provides its own domain specific language (DSL) ◮ Easily applicable in all Java development environments ◮ Effectively reduces mutant generation time to a minimum

CONDITIONAL MUTATION

◮ Transforms the program’s abstract syntax tree (AST) ◮ Encapsulates the mutations within conditional statements ASSIGN IDENT y BINARY * a x

⇒

ASSIGN IDENT y COND-EXPR THEN BINARY

  • a

x COND (M NO ==1) ELSE COND-EXPR THEN BINARY + a x COND (M NO ==2) ELSE BINARY * a x Figure: Multiple mutated binary expression as the right hand side of an assignment statement.

SUPPORTED FEATURES

◮ Simple compiler options enable the mutation analysis ◮ Configurable mutation operators by means of a DSL ◮ Determination of mutation coverage by running the original code

MUTATION COVERAGE

public int eval(int x){ int a = 3, b = 1, y; y = (M_NO==1)? a - x: (M_NO==2)? a + x: (M_NO==3)? a % x: (M_NO==0 && COVERED(1,3))? a * x : a * x; // original if(M_NO==4){ y -= b; }else if(M_NO==0 && COVERED(4,4)){ y += b; }else{ y += b; // original } return y; } Figure: Collecting coverage information. ◮ It is impossible to kill a mutant if it is

not reached and executed

◮ Additional instrumentation

determines the covered mutations

◮ Mutation coverage is only examined

if the tests execute the original code

◮ An external driver efficiently records

the covered mutations as ranges

◮ Only those mutants covered by a

test case are executed

IMPLEMENTATION DETAILS

Figure: Integration of the conditional mutation approach into the compilation process. listCOR={&&, ||, ==, !=}; method ="java.lang.System@println"; myOp{ BIN(&&) -> listCOR; BIN(||) -> listCOR; COR; LVR; } // Define replacement list BIN(+)<"org"> -> {-,*}; BIN(*)<"org"> -> {/,%}; // Enable built-in operator AOR AOR<"org">; // Enable operator myOp myOp<method>; Figure: DSL script to define the mutation process. package major.mutation; public class Driver{ public static int MAX_NO = 100000; public static int[] COV = new int[MAX_NO]; public static int M_NO=0; public static boolean COVERED (int from, int to){ for(int i=from; i<=to; ++i){ COV[i]++; } return false; } } Figure: Simple driver class implementation.

OPTIMIZED MUTATION ANALYSIS PROCESS

Figure: Minimizing the runtime of mutation analysis by means of test prioritization and mutation coverage.

  • 1. Embed and compile all mutants into the original program
  • 2. Run tests on original program to gather runtime and coverage
  • 3. Sort tests in ascending order according to their runtime
  • 4. Perform mutation analysis while employing the reordered test

suite and the mutation coverage information

RUNTIME OF MAJOR’S COMPILER

1 2 3 4 5 6 7 8 9 10 11 12 20000 40000 60000 80000 100000 120000 140000 Compiler runtime in seconds Number of mutants apache ant jfreechart itext java pathfinder commons math commons lang numerics4j Figure: Compiler runtime to generate and compile the mutants for all of the projects. ◮ Negligible overhead for generating and compiling the mutants ◮ Applicable on commodity workstations, even for large projects

EVALUATION OF MUTATION ANALYSIS PROCESSES

20 40 60 80 100 120 140 160 180 20 40 60 80 100 Number of killed mutants Runtime in seconds

  • ptimized order (using coverage information)

random order (using coverage information)

  • riginal order (using coverage information)
  • ptimized order (without coverage information)

random order (without coverage information)

  • riginal order (without coverage information)

Figure: Runtime of the mutation analysis processes.

FUTURE WORK

◮ Implementing new mutation operators at the semantic level ◮ Extending the domain specific language to support new operators ◮ Integrating conditional mutation into the new Java 7 compiler

rene.just@uni-ulm.de

26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011)

gkapfham@allegheny.edu

See you tomorrow for a live demonstration!