Using Non-Redundant Mutation Operators and Test Suite Prioritization to Achieve Efficient and Scalable Mutation Analysis René Just 1 , 2 & Gregory M. Kapfhammer 3 & Franz Schweiggert 2 1 University of Washington, USA 2 Ulm University, Germany 3 Allegheny College, USA 23rd International Symposium on Software Reliability Engineering November 28, 2012
Introduction Reduction Characteristics Prioritization Conclusion Mutation Analysis Background Mutation analysis assesses the quality of a test suite with artificial faults (mutants)
Introduction Reduction Characteristics Prioritization Conclusion Mutation Analysis Background Mutation analysis assesses the quality of a test suite with artificial faults (mutants) Program Test suite
Introduction Reduction Characteristics Prioritization Conclusion Mutation Analysis Background Mutation analysis assesses the quality of a test suite with artificial faults (mutants) Program Test suite Generate mutants Mutants
Introduction Reduction Characteristics Prioritization Conclusion Mutation Analysis Background Mutation analysis assesses public int max(int a, int b){ int max = a; the quality of a test suite if (b>a){ with artificial faults (mutants) max=b; } return max; } Original Program Test suite Generate mutants Mutants
Introduction Reduction Characteristics Prioritization Conclusion Mutation Analysis Background Mutation analysis assesses public int max(int a, int b){ int max = a; the quality of a test suite if (b>a){ with artificial faults (mutants) max=b; } return max; } Original Program Test suite Generate mutants Mutants
Introduction Reduction Characteristics Prioritization Conclusion Mutation Analysis Background Mutation analysis assesses public int max(int a, int b){ int max = a; the quality of a test suite if (b>a){ with artificial faults (mutants) max=b; } return max; } Original Program Test suite public int max(int a, int b){ int max = a; Generate Contains a small if (b>=a){ mutants syntactic change max=b; } return max; Mutants } Mutant
Introduction Reduction Characteristics Prioritization Conclusion Mutation Analysis Background Mutation analysis assesses public int max(int a, int b){ int max = a; the quality of a test suite if (b>a){ with artificial faults (mutants) max=b; } return max; } Original Program Test suite public int max(int a, int b){ int max = a; Generate Execute Contains a small if (b>=a){ mutants mutants syntactic change max=b; } return max; Mutation Mutants } score Mutant
Introduction Reduction Characteristics Prioritization Conclusion Mutation Analysis is Expensive public int max(int a, int b){ int max = a; if (b>a){ max=b; } return max; } Original
Introduction Reduction Characteristics Prioritization Conclusion Mutation Analysis is Expensive public int max(int a, int b){ if (b < a) int max = a; if (b <= a) if (b>a){ if (b >= a) max=b; if (b != a) } if (b == a) return max; } Original
Introduction Reduction Characteristics Prioritization Conclusion Mutation Analysis is Expensive public int max(int a, int b){ if (b < a) int max = a; if (b <= a) if (b>a){ if (b >= a) max=b; if (b != a) } if (b == a) return max; } Original Many mutants can be generated for large programs
Introduction Reduction Characteristics Prioritization Conclusion Mutation Analysis is Expensive public int max(int a, int b){ if (b < a) int max = a; if (b <= a) if (b>a){ if (b >= a) max=b; if (b != a) } if (b == a) return max; } Original Large programs include Many mutants can be comprehensive test suites generated for large programs
Introduction Reduction Characteristics Prioritization Conclusion Mutation Analysis is Expensive public int max(int a, int b){ if (b < a) int max = a; if (b <= a) if (b>a){ if (b >= a) max=b; if (b != a) } if (b == a) return max; } Original Large programs include Many mutants can be comprehensive test suites generated for large programs Executing the entire test suite for all mutants in large programs is prohibitive!
Introduction Reduction Characteristics Prioritization Conclusion Overview: Efficient Mutation Analysis Execute fewer mutants fewer times
Introduction Reduction Characteristics Prioritization Conclusion Overview: Efficient Mutation Analysis Execute fewer mutants fewer times Mutant reduction Generate Execute fewer mutants fewer mutants
Introduction Reduction Characteristics Prioritization Conclusion Overview: Efficient Mutation Analysis Execute fewer mutants fewer times Test suite Mutant prioritization reduction Reordering Generate Execute Test suite and splitting fewer mutants fewer mutants characteristics
Introduction Reduction Characteristics Prioritization Conclusion Overview: Efficient Mutation Analysis Execute fewer mutants fewer times Test suite Mutant prioritization reduction Reordering Generate Execute Test suite and splitting fewer mutants fewer mutants characteristics 27% 29% Empirical evaluation of 10 open-source projects with 560,000 mutants
Introduction Reduction Characteristics Prioritization Conclusion Reduction of Mutants Execute fewer mutants fewer times Mutant reduction Generate Execute fewer mutants fewer mutants 27% Empirical evaluation of 10 open-source projects with 560,000 mutants
Introduction Reduction Characteristics Prioritization Conclusion Reduce Number of Generated Mutants Mutation operators may introduce redundancy: Redundant mutants are subsumed by other mutants a + b �→ a - b (replace binary operator) a + b �→ a + (-b) (insert unary operator) Use only non-redundant mutation operators Avoid the generation of such subsumed mutants
Introduction Reduction Characteristics Prioritization Conclusion Reduce Number of Generated Mutants Mutation operators may introduce redundancy: Redundant mutants are subsumed by other mutants a + b �→ a - b (replace binary operator) a + b �→ a + (-b) (insert unary operator) Use only non-redundant mutation operators Avoid the generation of such subsumed mutants Number of generated mutants reduced by 27%
Introduction Reduction Characteristics Prioritization Conclusion Reduce Number of Generated Mutants Mutation operators may introduce redundancy: Redundant mutants are subsumed by other mutants a + b �→ a - b (replace binary operator) a + b �→ a + (-b) (insert unary operator) Use only non-redundant mutation operators Avoid the generation of such subsumed mutants Number of generated More than 410,000 gen- erated mutants remaining mutants reduced by 27%
Introduction Reduction Characteristics Prioritization Conclusion Reduce Number of Generated Mutants Mutation operators may introduce redundancy: Redundant mutants are subsumed by other mutants a + b �→ a - b (replace binary operator) a + b �→ a + (-b) (insert unary operator) Use only non-redundant mutation operators Avoid the generation of such subsumed mutants Number of generated More than 410,000 gen- erated mutants remaining mutants reduced by 27% Executing all non-redundant mutants is still prohibitive!
Introduction Reduction Characteristics Prioritization Conclusion Reduce Number of Executed Mutants Exploit necessary conditions: Mutants not covered (reached) cannot be detected Determine covered mutants for the test suite Only execute the covered mutants
Introduction Reduction Characteristics Prioritization Conclusion Reduce Number of Executed Mutants Exploit necessary conditions: Mutants not covered (reached) cannot be detected Determine covered mutants for the test suite Only execute the covered mutants Total reduction of executed mutants of more than 50%
Introduction Reduction Characteristics Prioritization Conclusion Reduce Number of Executed Mutants Exploit necessary conditions: Mutants not covered (reached) cannot be detected Determine covered mutants for the test suite Only execute the covered mutants Mutation analysis runtime Total reduction of executed still up to 13 hours mutants of more than 50%
Introduction Reduction Characteristics Prioritization Conclusion Reduce Number of Executed Mutants Exploit necessary conditions: Mutants not covered (reached) cannot be detected Determine covered mutants for the test suite Only execute the covered mutants Mutation analysis runtime Total reduction of executed still up to 13 hours mutants of more than 50% Further optimizations beyond the reduction of mutants are necessary!
Introduction Reduction Characteristics Prioritization Conclusion Optimized Workflow for Mutation Analysis Execute fewer mutants fewer times Test suite prioritization Reordering Test suite and splitting characteristics 29% Empirical evaluation of 10 open-source projects with 560,000 mutants
Introduction Reduction Characteristics Prioritization Conclusion Motivating Example for Reordering Mutants: 1, 2, 3, 4, 5
Introduction Reduction Characteristics Prioritization Conclusion Motivating Example for Reordering Mutants: Test case t 1 : Test case t 2 : Test case t 3 : 1, 2, 3, 4, 5 5 seconds 2 seconds 1 second
Recommend
More recommend
