Introduction Regression Testing Empirical Evaluation Conclusion An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization ‡ Adam M. Smith and † Gregory M. Kapfhammer ‡ Department of Computer Science † Department of Computer Science University of Pittsburgh Allegheny College ACM Symposium on Applied Computing March 8 - 12, 2009 Featuring an image from www.campusbicycle.com An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization 1 / 20
Introduction Regression Testing Empirical Evaluation Conclusion Important Contributions JD 100 Program Reduction 80 Execution Time (ms) Analysis 60 Technique 40 20 Test Suite Prioritization 0 2OPT DGR GRD HGS Prioritization Technique Regression Testing Techniques Detailed Empirical Results Implement and empirically evaluate the efficiency and effectiveness of cost-aware greedy methods for regression test suite reduction and prioritization An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization 2 / 20
Introduction Regression Testing Empirical Evaluation Conclusion Important Contributions JD 100 Program Reduction 80 Execution Time (ms) Analysis 60 Technique 40 20 Test Suite Prioritization 0 2OPT DGR GRD HGS Prioritization Technique Regression Testing Techniques Detailed Empirical Results Implement and empirically evaluate the efficiency and effectiveness of cost-aware greedy methods for regression test suite reduction and prioritization An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization 2 / 20
Introduction Regression Testing Empirical Evaluation Conclusion Important Contributions JD 100 Program Reduction 80 Execution Time (ms) Analysis 60 Technique 40 20 Test Suite Prioritization 0 2OPT DGR GRD HGS Prioritization Technique Regression Testing Techniques Detailed Empirical Results Implement and empirically evaluate the efficiency and effectiveness of cost-aware greedy methods for regression test suite reduction and prioritization An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization 2 / 20
Introduction Regression Testing Empirical Evaluation Conclusion Important Contributions JD 100 Program Reduction 80 Execution Time (ms) Analysis 60 Technique 40 20 Test Suite Prioritization 0 2OPT DGR GRD HGS Prioritization Technique Regression Testing Techniques Detailed Empirical Results Implement and empirically evaluate the efficiency and effectiveness of cost-aware greedy methods for regression test suite reduction and prioritization An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization 2 / 20
Introduction Regression Testing Empirical Evaluation Conclusion Regression Testing and Bicycles Efficiency: Low wind resistance and time to destination An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization 3 / 20
Introduction Regression Testing Empirical Evaluation Conclusion Regression Testing and Bicycles Effectiveness: Transports all required materials and no break downs An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization 3 / 20
Introduction Regression Testing Empirical Evaluation Conclusion Regression Testing and Bicycles Cost: Frame material and components cause price to vary considerably An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization 3 / 20
Introduction Regression Testing Empirical Evaluation Conclusion Regression Testing Techniques Before After Before After Reduction Prunes the Test Suite Prioritization Reorders the Tests It is expensive to run a test suite T = � T 1 , . . . , T n � . Reduction discards some of the n tests in an attempt to decrease testing time while still preserving objectives like coverage or fault detection . An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization 4 / 20
Introduction Regression Testing Empirical Evaluation Conclusion Regression Testing Techniques Before After Before After Reduction Prunes the Test Suite Prioritization Reorders the Tests It is expensive to run a test suite T = � T 1 , . . . , T n � . Reduction discards some of the n tests in an attempt to decrease testing time while still preserving objectives like coverage or fault detection . An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization 4 / 20
Introduction Regression Testing Empirical Evaluation Conclusion Regression Testing Techniques Before After Before After Reduction Prunes the Test Suite Prioritization Reorders the Tests It is expensive to run a test suite T = � T 1 , . . . , T n � . Reduction discards some of the n tests in an attempt to decrease testing time while still preserving objectives like coverage or fault detection . An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization 4 / 20
Introduction Regression Testing Empirical Evaluation Conclusion Regression Testing Techniques Before After Before After Reduction Prunes the Test Suite Prioritization Reorders the Tests It is expensive to run a test suite T = � T 1 , . . . , T n � . Reduction discards some of the n tests in an attempt to decrease testing time while still preserving objectives like coverage or fault detection . An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization 4 / 20
Introduction Regression Testing Empirical Evaluation Conclusion Regression Testing Techniques Before After Before After Reduction Prunes the Test Suite Prioritization Reorders the Tests It is expensive to run a test suite T = � T 1 , . . . , T n � . Prioritization searches through the n ! = n × n − 1 × . . . × 1 orderings for those that maximize an objective function like coverage or fault detection . An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization 4 / 20
Introduction Regression Testing Empirical Evaluation Conclusion Finding the Overlap in Coverage R R 1 R 3 R 4 R 6 R 7 R 5 R 2 T 8 T 2 T 4 T 12 T 1 T 3 T 11 T 7 T 9 T 10 T 5 T 6 R j → T i means that requirement R j is covered by test T i Test suite reduction discards the test cases that redundantly cover the test requirements T = � T 2 , T 3 , T 6 , T 9 � covers all of the test requirements An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization 5 / 20
Introduction Regression Testing Empirical Evaluation Conclusion Finding the Overlap in Coverage R R 1 R 3 R 4 R 6 R 7 R 5 R 2 T 8 T 2 T 4 T 12 T 1 T 3 T 11 T 7 T 9 T 10 T 5 T 6 R j → T i means that requirement R j is covered by test T i Test suite reduction discards the test cases that redundantly cover the test requirements T = � T 2 , T 3 , T 6 , T 9 � covers all of the test requirements An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization 5 / 20
Introduction Regression Testing Empirical Evaluation Conclusion Finding the Overlap in Coverage R R 2 R 1 R 3 R 4 R 7 R 5 R 6 T 8 T 2 T 4 T 12 T 1 T 3 T 11 T 7 T 9 T 10 T 5 T 6 R j → T i means that requirement R j is covered by test T i Test suite reduction discards the test cases that redundantly cover the test requirements T = � T 2 , T 3 , T 6 , T 9 � covers all of the test requirements An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization 5 / 20
Introduction Regression Testing Empirical Evaluation Conclusion Greedy Approaches to Regression Testing Original Test Suite Harrold, Gupta, Soffa (HGS) T 1 T 2 T 4 T 3 Delayed Greedy (DGR) Traditional Greedy (GRD) Reduction Technique 2-Optimal Greedy (2OPT) First Output First Residual Second Output Hypothesis : Using the exe- T 1 T 4 T 2 T 3 T 2 T 3 cution time of a test case can Prioritized Test Suite improve the reduced and pri- T 1 T 4 T 2 T 3 oritized test suites Compare (i) greedy choices (cost, coverage, and ratio) and (ii) algorithms An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization 6 / 20
Introduction Regression Testing Empirical Evaluation Conclusion Greedy Approaches to Regression Testing Original Test Suite Harrold, Gupta, Soffa (HGS) T 1 T 2 T 4 T 3 Delayed Greedy (DGR) Traditional Greedy (GRD) Reduction Technique 2-Optimal Greedy (2OPT) First Output First Residual Second Output Hypothesis : Using the exe- T 1 T 4 T 2 T 3 T 2 T 3 cution time of a test case can Prioritized Test Suite improve the reduced and pri- T 1 T 4 T 2 T 3 oritized test suites Compare (i) greedy choices (cost, coverage, and ratio) and (ii) algorithms An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization 6 / 20
Introduction Regression Testing Empirical Evaluation Conclusion Greedy Approaches to Regression Testing Original Test Suite Harrold, Gupta, Soffa (HGS) T 1 T 2 T 4 T 3 Delayed Greedy (DGR) Traditional Greedy (GRD) Reduction Technique 2-Optimal Greedy (2OPT) First Output First Residual Second Output Hypothesis : Using the exe- T 1 T 4 T 2 T 3 T 2 T 3 cution time of a test case can Prioritized Test Suite improve the reduced and pri- T 1 T 4 T 2 T 3 oritized test suites Compare (i) greedy choices (cost, coverage, and ratio) and (ii) algorithms An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization 6 / 20
Recommend
More recommend
