Challenges and Solutions Regression Testing Techniques Conducting Empirical Studies Generating Synthetic Coverage Using Synthetic Coverage Information to Evaluate Test Suite Prioritizers Gregory M. Kapfhammer † Department of Computer Science Allegheny College, Pennsylvania, USA http://www.cs.allegheny.edu/~gkapfham/ Chennai Mathematical Institute, February 2008 † In Conjunction with Mary Lou Soffa, Kristen Walcott (UVa/CS) Suvarshi Bhadra, Joshua Geiger, Adam Smith, Gavilan Steinman, Yuting Zhang (Allegheny/CS) Featuring images from Embroidery and Tapestry Weaving , Grace Christie (Project Gutenberg) Using Synthetic Coverage Information to , Evaluate Test Suite Prioritizers 1 / 36
Challenges and Solutions Regression Testing Techniques Conducting Empirical Studies Generating Synthetic Coverage Presentation Outline Challenges and Solutions 1 Regression Testing Techniques 2 Conducting Empirical Studies 3 Generating Synthetic Coverage 4 Empirical Evaluation 5 Future Work 6 Conclusions 7 Using Synthetic Coverage Information to , Evaluate Test Suite Prioritizers 2 / 36
Challenges and Solutions Regression Testing Techniques Conducting Empirical Studies Generating Synthetic Coverage The Challenge of Software Testing I shall not deny that the construction of these testing programs has been a major intellectual effort: to convince oneself that one has not overlooked “a relevant state” and to convince oneself that the testing programs generate them all is no simple matter. The encouraging thing is that (as far as we know!) it could be done. Edsger W. Dijkstra , Communications of the ACM, 1968 Additional Challenge : empirically evaluating the efficiency and effectiveness of software testing techniques Using Synthetic Coverage Information to , Evaluate Test Suite Prioritizers 3 / 36
Challenges and Solutions Regression Testing Techniques Conducting Empirical Studies Generating Synthetic Coverage Important Contributions sss − covers ssm − covers ssl − covers 1.0 1.0 1.0 0.8 0.8 0.8 0.6 0.6 0.6 Overlap Score Overlap Score Overlap Score 0.4 0.4 0.4 0.2 0.2 0.2 0.0 0.0 0.0 NCO JCO TCO NCO JCO TCO NCO JCO TCO Overlap Metric Overlap Metric Overlap Metric Synthetic Coverage Generators Detailed Experimental Results A comprehensive framework that supports the empirical evaluation of regression test suite prioritizers Using Synthetic Coverage Information to , Evaluate Test Suite Prioritizers 4 / 36
Challenges and Solutions Regression Testing Techniques Conducting Empirical Studies Generating Synthetic Coverage Presentation Outline Challenges and Solutions 1 Regression Testing Techniques 2 Conducting Empirical Studies 3 Generating Synthetic Coverage 4 Empirical Evaluation 5 Future Work 6 Conclusions 7 Using Synthetic Coverage Information to , Evaluate Test Suite Prioritizers 5 / 36
Challenges and Solutions Regression Testing Techniques Conducting Empirical Studies Generating Synthetic Coverage Approaches to Software Testing Structural Testing Random Testing Input Space Regression Specification Testing Testing Testing isolates defects and establishes a confidence in the correctness of a software application Using Synthetic Coverage Information to , Evaluate Test Suite Prioritizers 6 / 36
Challenges and Solutions Regression Testing Techniques Conducting Empirical Studies Generating Synthetic Coverage What is a Test Case? Expected Output Set Up Method Test Input Output Verdict Under Test Oracle Tear Down Overview Test suite executor runs each test case independently Each test invokes a method within the program and then compares the actual and expected output values Using Synthetic Coverage Information to , Evaluate Test Suite Prioritizers 7 / 36
Challenges and Solutions Regression Testing Techniques Conducting Empirical Studies Generating Synthetic Coverage Test Coverage Monitoring Program Adequacy Criterion Residual Test Instrumented Instrumentation Requirements Program Test Suite Execution Cummulative Covered Adequacy Calculator Requirements Overview Structural adequacy criteria focus on the coverage of nodes, edges, paths, and definition-use associations Instrumentation probes track the coverage of test requirements Using Synthetic Coverage Information to , Evaluate Test Suite Prioritizers 8 / 36
Challenges and Solutions Regression Testing Techniques Conducting Empirical Studies Generating Synthetic Coverage Finding the Overlap in Coverage R R 2 R 3 R 6 R 7 R 5 R 1 R 4 T 8 T 2 T 4 T 12 T 1 T 3 T 11 T 7 T 9 T 10 T 5 T 6 R j → T i means that requirement R j is covered by test T i T = � T 2 , T 3 , T 6 , T 9 � covers all of the test requirements Include the remaining tests so that they can redundantly cover the requirements Using Synthetic Coverage Information to , Evaluate Test Suite Prioritizers 9 / 36
Challenges and Solutions Regression Testing Techniques Conducting Empirical Studies Generating Synthetic Coverage Regression Test Suite Prioritization Original Test Suite GRT Repeat Test Suite Modified Test Suite Begin Coverage Report Testing Results End Prioritization Test Suite Execution Program VSRT Repeat Overview Prioritization re-orders the tests so that they cover the requirements more effectively Researchers and practitioners need to determine whether the prioritized test suite is better than the original ordering Using Synthetic Coverage Information to , Evaluate Test Suite Prioritizers 10 / 36
Challenges and Solutions Regression Testing Techniques Conducting Empirical Studies Generating Synthetic Coverage Presentation Outline Challenges and Solutions 1 Regression Testing Techniques 2 Conducting Empirical Studies 3 Generating Synthetic Coverage 4 Empirical Evaluation 5 Future Work 6 Conclusions 7 Using Synthetic Coverage Information to , Evaluate Test Suite Prioritizers 11 / 36
Challenges and Solutions Regression Testing Techniques Conducting Empirical Studies Generating Synthetic Coverage Using Real World Applications Regression Test Suite Program Under Test It is difficult to systematically study the efficiency and effectiveness trade-offs because coverage overlap varies Using Synthetic Coverage Information to , Evaluate Test Suite Prioritizers 12 / 36
Challenges and Solutions Regression Testing Techniques Conducting Empirical Studies Generating Synthetic Coverage Coverage Effectiveness Metric Cover R ( T 1 ) Cover � n − 1 i = 1 R ( T i ) Cover R ( T ) C ( T , t ) T n Done Covered Test Reqs . . . � t ( n ) Area C ( T , t ) 0 Testing Time ( t ) T 1 Done T n − 1 Done Prioritize to increase the CE of a test suite CE = Actual Ideal ∈ [ 0 , 1 ] Using Synthetic Coverage Information to , Evaluate Test Suite Prioritizers 13 / 36
Challenges and Solutions Regression Testing Techniques Conducting Empirical Studies Generating Synthetic Coverage Characterizing a Test Suite Test Information Test Case Cost (sec) Requirements R 1 R 2 R 3 R 4 R 5 T 1 5 � � T 2 10 � � � � T 3 4 � � � Total Testing Time = 19 seconds Formulating the Metrics CE considers the execution time of each test while CE u assumes that all test cases execute for a unit cost Using Synthetic Coverage Information to , Evaluate Test Suite Prioritizers 14 / 36
Challenges and Solutions Regression Testing Techniques Conducting Empirical Studies Generating Synthetic Coverage Coverage Effectiveness Values Calculating CE and CE u Ordering CE CE u T 1 T 2 T 3 . 3789 . 4 T 1 T 3 T 2 . 5053 . 4 T 2 T 1 T 3 . 3789 . 5333 T 2 T 3 T 1 . 4316 . 6 T 3 T 1 T 2 . 5789 . 4557 T 3 T 2 T 1 . 5789 . 5333 Observations Including test case costs does impact the CE metric Depending upon the characteristics of the test suite, we may see CE = CE u , CE > CE u , or CE < CE u Using Synthetic Coverage Information to , Evaluate Test Suite Prioritizers 15 / 36
Challenges and Solutions Regression Testing Techniques Conducting Empirical Studies Generating Synthetic Coverage Comparing Prioritization Techniques Comparing Test Suite Prioritizers RAP Prioritization Technique ORP RVP GPR GPV GPC GPO 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Coverage Effectiveness Does this result generalize to other applications? Using Synthetic Coverage Information to , Evaluate Test Suite Prioritizers 16 / 36
Challenges and Solutions Regression Testing Techniques Conducting Empirical Studies Generating Synthetic Coverage Presentation Outline Challenges and Solutions 1 Regression Testing Techniques 2 Conducting Empirical Studies 3 Generating Synthetic Coverage 4 Empirical Evaluation 5 Future Work 6 Conclusions 7 Using Synthetic Coverage Information to , Evaluate Test Suite Prioritizers 17 / 36
Challenges and Solutions Regression Testing Techniques Conducting Empirical Studies Generating Synthetic Coverage Test Suites and Requirements Test Requirements Regression Test Suite R = { R 1 , . . . , R m } T = � T 1 , . . . , T n � T i ∈ T R j ∈ R covers ( i ) denotes the set of requirements that T i covers coveredby ( j ) denotes the set of test cases that cover R j Goal : automatically generate a synthetic regression test suite T that covers the requirements in R Using Synthetic Coverage Information to , Evaluate Test Suite Prioritizers 18 / 36
Recommend
More recommend
