The Theory and Practice of Software Testing: Can we Test it? Yes we - - PowerPoint PPT Presentation

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion

The Theory and Practice of Software Testing: Can we Test it? Yes we Can! Gregory M. Kapfhammer† Department of Computer Science Allegheny College, Pennsylvania, USA http://www.cs.allegheny.edu/~gkapfham/

SGT Global, February 2008

†In Conjunction with Mary Lou Soffa, Kristen Walcott (UVa/CS)

Suvarshi Bhadra, Joshua Geiger, Adam Smith, Gavilan Steinman, Yuting Zhang (Allegheny/CS) Featuring images from Embroidery and Tapestry Weaving, Grace Christie (Project Gutenberg) 1 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion

Presentation Outline

1

Software Testing Challenges

2

Structural Testing

3

Regression Testing

4

Mutation Testing

5

Future Work

6

Conclusion

2 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion

The Challenge of Software Testing

I shall not deny that the construction of these testing programs has been a major intellectual effort: to convince oneself that one has not overlooked “a relevant state” and to convince oneself that the testing programs generate them all is no simple matter. The encouraging thing is that (as far as we know!) it could be done. Edsger W. Dijkstra, Communications of the ACM, 1968 Important Question: What are your software development and testing challenges? What are your best solutions?

3 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion

Modern Software is Complex

Complex source code, database, files, and network communication Can we increase reliability by simplifying software?

4 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion

Defect Locations

Virtual

P

Machine Graphical Interface Database File System System Operating

Byte Code Input a 5 print ... exit Final Result: 45 Output

Defects may exist in the individual components or the interactions

5 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion

Approaches to Software Testing

Testing Structural Testing Specification Testing

Regression Testing

Mutation

Testing isolates defects and establishes a confidence in the correctness of a software application

6 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion

Presentation Outline

1

Software Testing Challenges

2

Structural Testing

3

Regression Testing

4

Mutation Testing

5

Future Work

6

Conclusion

7 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion

What is a Test Case?

Input Method Under Test Output Test Oracle Expected Output Verdict Tear Down Set Up

Overview

Test suite executor (JUnit) runs each test case independently Each test invokes a method within the program and then compares the actual and expected output values

8 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion

Test Coverage Monitoring

Program Instrumentation Adequacy Criterion Residual Test Requirements Cummulative Adequacy Calculator Instrumented Program Test Suite Execution Covered Requirements

Overview

Structural adequacy criteria focus on the coverage of nodes, edges, paths, and definition-use associations Instrumentation probes track the coverage of test requirements

9 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion

Calculating the Coverage of a Test

Program Instrumentation Test Suite Adequacy Criterion Instrumented Program Test Coverage Monitoring Instrumented Test Suite Coverage Results Adequacy Calculation Test Requirements Adequacy Measurements Database

Calculating Coverage Use instrumentation probes to capture and analyze a test suite’s coverage of the program Regression Testing The adequacy measurements can be used to support both test suite reduction and prioritization

10 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion

Presentation Outline

1

Software Testing Challenges

2

Structural Testing

3

Regression Testing

4

Mutation Testing

5

Future Work

6

Conclusion

11 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion

Finding the Overlap in Coverage

T2 T3 T6 T9 R R1 R2 T4 T8 R3 T12 T1 R4 T11 T5 R5 R6 T7 R7 T10

Rj → Ti means that requirement Rj is covered by test Ti T = T2, T3, T6, T9 covers all of the test requirements May include the remaining tests so that they can redundantly cover the requirements

12 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion

Reducing and Prioritizing the Tests

Begin Coverage Report End VSRT Repeat Program Reduction

• r Prioritization

Original Test Suite Modified Test Suite Test Suite Execution Testing Results GRT Repeat

Regression Testing Overview Reduction creates a smaller test suite that covers the same requirements as the original suite. Prioritization re-orders the tests so that they cover the requirements more effectively. Techniques use heuristics to solve NP-complete problems.

13 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion

Evaluating a Test Prioritization

Testing Time

. . .

Covered Test Reqs

T1 Done Tn−1 Done Tn Done Cover R(T1) Cover n−1

i=1 R(Ti)

Cover R(T) Area t(n) C(T, t)

C(T,t)

(t)

Prioritize to increase the CE of a test suite CE = Actual Ideal ∈ [0, 1]

14 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion

Characterizing a Test Suite

Test Information

Test Case Cost (sec) Requirements R1 R2 R3 R4 R5 T1 5

• T2

10

• T3

4

• Total Testing Time = 19 seconds

Formulating the Metrics CE considers the execution time of each test while CEu assumes that all test cases execute for a unit cost

15 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion

Coverage Effectiveness Values

Calculating CE and CEu

Ordering CE CEu T1 T2 T3 .3789 .4 T1 T3 T2 .5053 .4 T2 T1 T3 .3789 .5333 T2 T3 T1 .4316 .6 T3 T1 T2 .5789 .4557 T3 T2 T1 .5789 .5333

Observations

Including test case costs does impact the CE metric Depending upon the characteristics of the test suite, we may see CE = CEu, CE > CEu, or CE < CEu

16 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion

Presentation Outline

1

Software Testing Challenges

2

Structural Testing

3

Regression Testing

4

Mutation Testing

5

Future Work

6

Conclusion

17 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion

Mutation Testing Techniques

Program Mutant Generator Mutant Programs Test Suite Execution Mutation Score

Mutant Creation

A mutation testing tool (e.g., µJava or Jumble) inserts defects into the program under test Question: Why are we inserting faults into the the programs that we are testing?

Test Quality Goal: measure the quality of the test suite by determining whether or not it can differentiate between faulty and non-faulty programs

18 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion

Average Percentage of Faults Detected

Program Fault Seeding Faulty Program Test Suite Execution Test Results APFD Calculator Test Suite APFD Score Fault Table

Fault Seeding

Use known faults or a mutation testing tool (e.g., µJava or Jumble) to insert defects into the program Determine which test(s) are able to detect the seeded faults and construct a fault table

APFD Calculation A test ordering has a higher APFD score if it rapidly detects the faults

19 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion

Presentation Outline

1

Software Testing Challenges

2

Structural Testing

3

Regression Testing

4

Mutation Testing

5

Future Work

6

Conclusion

20 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion

Search-Based Test Suite Prioritization

20 40 60 80 100 120 0.50 0.55 0.60 0.65 0.70 0.75 Test Prioritization Index Coverage Effectiveness Value Coverage Effectiveness Values

Use heuristic search (HC, SANN, GA) to prioritize the test suite

21 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion

Detailed Empirical Evaluations

New Testing Techniques Real World Programs

Systematically study the efficiency and effectiveness trade-offs of different software testing techniques

22 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!

Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusion

Conclusions

Concluding Remarks

Software development and testing is fun and exciting! There are many new developments in research and practical tools – some of which are ready for use today! What are your favorite software testing tools and techniques?

Resources

Conferences: ICSE, FSE, ISSTA, ASE, ICSM, ISSRE Journals: TSE, TOSEM, IST, JSS, JSME Many articles are available online from Google Scholar http://www.cs.allegheny.edu/~gkapfham/

23 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!