Gregory M. Kapfhammer Department of Computer Science Allegheny - - PowerPoint PPT Presentation

Introduction Challenges Potential Way Forward Conclusion

Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Gregory M. Kapfhammer

Department of Computer Science Allegheny College http://www.cs.allegheny.edu/~gkapfham/ 1st International Workshop on Regression Testing Co-Located with the 4th IEEE International Conference on Software Testing, Verification and Validation Berlin, Germany, March 2011

1 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Presentation Overview

Regression Testing Community

2 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Presentation Overview

Regression Testing Community Current Trends

2 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Presentation Overview

Regression Testing Community Current Trends Increased Interest

2 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Presentation Overview

Regression Testing Community Current Trends Increased Interest

Software Testing Empirical Studies

2 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Presentation Overview

Regression Testing Community Current Trends Increased Interest

Software Testing Empirical Studies

Techniques

2 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Presentation Overview

Regression Testing Community Current Trends Increased Interest

Software Testing Empirical Studies

Techniques Challenges

2 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Presentation Overview

Regression Testing Community Current Trends Increased Interest

Software Testing Empirical Studies

Techniques Challenges Solutions

2 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Presentation Overview

Regression Testing Community Current Trends Increased Interest

Software Testing Empirical Studies

Techniques Challenges Solutions Statistics

2 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Presentation Overview

Regression Testing Community Current Trends Increased Interest

Software Testing Empirical Studies

Techniques Challenges Solutions Statistics Medicine

2 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Presentation Overview

Regression Testing Community Current Trends Increased Interest

Software Testing Empirical Studies

Techniques Challenges Solutions Statistics Medicine Social Sciences

2 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Presentation Overview

Regression Testing Community Current Trends Increased Interest

Software Testing Empirical Studies

Techniques Challenges Solutions Statistics Medicine Social Sciences Mutually Beneficial Sharing of All Artifacts Used in Experimentation

2 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Model of Regression Testing

Start 3 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Model of Regression Testing

Start Coverage Report 3 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Model of Regression Testing

Start Coverage Report Selection Reduction Prioritization 3 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Model of Regression Testing

Start Coverage Report Selection Reduction Prioritization Original Test Suite 3 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Model of Regression Testing

Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite 3 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Model of Regression Testing

Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring 3 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Model of Regression Testing

Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion 3 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Model of Regression Testing

Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Test Results 3 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Model of Regression Testing

Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Test Results End 3 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Model of Regression Testing

Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Test Results End

Use the Coverage Report During the Next Round of Regression Testing

3 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Model of Regression Testing

Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Test Results End

Use the Same Test Suite for the Next Round of Regression Testing

Standard Repeat 3 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Model of Regression Testing

Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Test Results End Standard Repeat

Make a New Test Suite for the Next Round of Regression Testing

Version Specific Repeat 3 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Model of Regression Testing

Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Test Results End Standard Repeat Version Specific Repeat

Practitioners are unwilling to use methods for “lack of empirical studies” [12]

3 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Model of Regression Testing

Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Test Results End Standard Repeat Version Specific Repeat

Practitioners are unwilling to use methods for “lack of empirical studies” [12] Testing tools do not produce outputs in the best format (e.g., per-test coverage)

3 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Model of Regression Testing

Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Test Results End Standard Repeat Version Specific Repeat

Practitioners are unwilling to use methods for “lack of empirical studies” [12] Experimental assessment could stagnate due to inaccessibility of artifacts

3 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Model of Experimental Evaluation

Start 4 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Model of Experimental Evaluation

Start Programs Test Suites 4 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Model of Experimental Evaluation

Start Programs Test Suites Conduct Experiments 4 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Model of Experimental Evaluation

Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations 4 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Model of Experimental Evaluation

Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Data Sets 4 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Model of Experimental Evaluation

Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Data Sets Visualization Statistical Analysis 4 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Model of Experimental Evaluation

Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Data Sets Visualization Statistical Analysis Graphs Diagrams Summarized Data Sets Models 4 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Model of Experimental Evaluation

Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Data Sets Visualization Statistical Analysis Graphs Diagrams Summarized Data Sets Models

Iteratively Perform Visualization and Statistical Analysis

Repeat 4 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Model of Experimental Evaluation

Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Data Sets Visualization Statistical Analysis Graphs Diagrams Summarized Data Sets Models

Iteratively Perform Visualization and Statistical Analysis

Repeat End

Conduct Experiments with Additional Programs, Test Suites, and Techniques

Repeat 4 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Model of Experimental Evaluation

Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Data Sets Visualization Statistical Analysis Graphs Diagrams Summarized Data Sets Models Repeat End

Conduct Experiments with Additional Programs, Test Suites, and Techniques

Repeat

56% of Papers Surveyed by Yoo and Harman Only Used the SIR Programs [3]

4 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Model of Experimental Evaluation

Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Data Sets Visualization Statistical Analysis Graphs Diagrams Summarized Data Sets Models Repeat End

Conduct Experiments with Additional Programs, Test Suites, and Techniques

Repeat

Few Papers Report on the Efficiency of the Regression Testing Techniques

4 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Model of Experimental Evaluation

Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Data Sets Visualization Statistical Analysis Graphs Diagrams Summarized Data Sets Models Repeat End

Conduct Experiments with Additional Programs, Test Suites, and Techniques

Repeat

Authors Do Not Release Tools That Conduct Experiments and Analyze Results

4 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Model of Experimental Evaluation

Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Data Sets Visualization Statistical Analysis Graphs Diagrams Summarized Data Sets Models Repeat End

Conduct Experiments with Additional Programs, Test Suites, and Techniques

Repeat

Without Using Data Mining Methods, Researchers May Miss Important Trends

4 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Thought Provoking Assertions

Jonathan B. Buckheit and David L. Donoho Department of Statistics Stanford University

5 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Thought Provoking Assertions

Jonathan B. Buckheit and David L. Donoho Department of Statistics Stanford University

For a field to qualify as a sci- ence, it is important first and foremost that published work be reproducible by others.

5 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Thought Provoking Assertions

Jonathan B. Buckheit and David L. Donoho Department of Statistics Stanford University

For a field to qualify as a sci- ence, it is important first and foremost that published work be reproducible by others.

Does the Regression Testing Community Want to be Scientific?

5 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Thought Provoking Assertions

Jonathan B. Buckheit and David L. Donoho Department of Statistics Stanford University

For a field to qualify as a sci- ence, it is important first and foremost that published work be reproducible by others.

Does the Regression Testing Community Want to be Scientific? What Does it Mean for Re- search to be Reproducible?

5 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Thought Provoking Assertions

Gary King Department of Govern- ment Harvard University

5 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Thought Provoking Assertions

Gary King Department of Govern- ment Harvard University

Reproducible: sufficient infor- mation exists with which to under- stand, evaluate, and build upon a prior work if a third party can replicate the results without any ad- ditional information from the author.

5 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Thought Provoking Assertions

Gary King Department of Govern- ment Harvard University

Reproducible: sufficient infor- mation exists with which to under- stand, evaluate, and build upon a prior work if a third party can replicate the results without any ad- ditional information from the author.

Can the Regression Testing Com- munity Adhere to This Standard?

5 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Reproducible Research Through Sharing

Artifacts Used in Experiments

6 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Reproducible Research Through Sharing

Artifacts Used in Experiments Test Suite

6 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Reproducible Research Through Sharing

Artifacts Used in Experiments Test Suite ...

6 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Reproducible Research Through Sharing

Artifacts Used in Experiments Test Suite ... Data Sets

6 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Reproducible Research Through Sharing

Artifacts Used in Experiments Test Suite ... Data Sets Repository

6 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Reproducible Research Through Sharing

Artifacts Used in Experiments Test Suite ... Data Sets Repository Dataverse

6 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Reproducible Research Through Sharing

Artifacts Used in Experiments Test Suite ... Data Sets Repository Dataverse Web Site

6 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Reproducible Research Through Sharing

Artifacts Used in Experiments Test Suite ... Data Sets Repository Dataverse Web Site Version Control

6 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Reproducible Research Through Sharing

Artifacts Used in Experiments Test Suite ... Data Sets Repository Dataverse Web Site Version Control Deposit the Artifacts from Experimentation in One or More Repositories

6 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Reproducible Research Through Sharing

Artifacts Used in Experiments Test Suite ... Data Sets Repository Dataverse Web Site Version Control External Researchers External Researchers Access the Repositories

6 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Reproducible Research Through Sharing

Artifacts Used in Experiments Test Suite ... Data Sets Repository Dataverse Web Site Version Control Internal Researchers Internal Researchers Access the Repositories

6 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Reproducible Research Through Sharing

Artifacts Used in Experiments Test Suite ... Data Sets Repository Dataverse Web Site Version Control Internal Researchers Internal Researchers Access the Repositories Costs of Reproducible Research?

6 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Reproducible Research Through Sharing

Artifacts Used in Experiments Test Suite ... Data Sets Repository Dataverse Web Site Version Control Internal Researchers Internal Researchers Access the Repositories Costs of Reproducible Research? Benefits of Reproducible Research?

6 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Reproducible Research Through Sharing

Artifacts Used in Experiments Test Suite ... Data Sets Repository Dataverse Web Site Version Control Internal Researchers Internal Researchers Access the Repositories Costs of Reproducible Research? Benefits of Reproducible Research? Increase the Citation of Your Paper [17]

6 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Reproducible Research Through Sharing

Artifacts Used in Experiments Test Suite ... Data Sets Repository Dataverse Web Site Version Control Internal Researchers Internal Researchers Access the Repositories Costs of Reproducible Research? Benefits of Reproducible Research? Allow Others to Replicate Your Experiments [15]

6 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Reproducible Research Through Sharing

Artifacts Used in Experiments Test Suite ... Data Sets Repository Dataverse Web Site Version Control Internal Researchers Internal Researchers Access the Repositories Costs of Reproducible Research? Benefits of Reproducible Research? Lower the Barriers to Entry for New Researchers [16]

6 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Reproducible Research Through Sharing

Artifacts Used in Experiments Test Suite ... Data Sets Repository Dataverse Web Site Version Control Internal Researchers Internal Researchers Access the Repositories Costs of Reproducible Research? Benefits of Reproducible Research? Find Your Own Deliverables More Easily [13]

6 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Practical Suggestions: Pick Tools Carefully

7 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Practical Suggestions: Pick Tools Carefully

Use the R Language for Statistical Com- putation When:

7 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Practical Suggestions: Pick Tools Carefully

Use the R Language for Statistical Com- putation When: Conducting Experiments

7 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Practical Suggestions: Pick Tools Carefully

Use the R Language for Statistical Com- putation When: Conducting Experiments Visualizing Data

7 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Practical Suggestions: Pick Tools Carefully

Use the R Language for Statistical Com- putation When: Conducting Experiments Visualizing Data Data Mining

7 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Practical Suggestions: Pick Tools Carefully

Use the R Language for Statistical Com- putation When: Conducting Experiments Visualizing Data Data Mining Simple Shell or Full-Featured IDEs

7 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Practical Suggestions: Pick Tools Carefully

Use the R Language for Statistical Com- putation When: Conducting Experiments Visualizing Data Data Mining Simple Shell or Full-Featured IDEs Advanced Data Analysis Methods

7 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Practical Suggestions: Use Tree Models

sel_method: TOU3, TOU4, TOU5 RM child_density < 0.875 0.9674

8 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Practical Suggestions: Use Tree Models

sel_method: TOU3, TOU4, TOU5 RM child_density < 0.875 0.9674 sel_method

8 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Practical Suggestions: Use Tree Models

sel_method: TOU3, TOU4, TOU5 RM child_density < 0.875 0.9674 sel_method sel_method: TOU3, TOU4, TOU5 GB mutation_rate < 0.17 cross_operator: OX1, VR

8 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Practical Suggestions: Use Tree Models

sel_method: TOU3, TOU4, TOU5 RM child_density < 0.875 0.9674 sel_method sel_method: TOU3, TOU4, TOU5 GB mutation_rate < 0.17 cross_operator: OX1, VR sel_method

8 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Practical Suggestions: Use Tree Models

sel_method: TOU3, TOU4, TOU5 RM child_density < 0.875 0.9674 sel_method sel_method: TOU3, TOU4, TOU5 GB mutation_rate < 0.17 cross_operator: OX1, VR sel_method The sel_method variable is always the most important parameter

8 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Practical Suggestions: Use Tree Models

sel_method: TOU3, TOU4, TOU5 RM child_density < 0.875 0.9674 sel_method sel_method: TOU3, TOU4, TOU5 GB mutation_rate < 0.17 cross_operator: OX1, VR sel_method Importance of sel_method holds for all case study applications

8 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Practical Suggestions: Use Tree Models

sel_method: TOU3, TOU4, TOU5 RM child_density < 0.875 0.9674 sel_method sel_method: TOU3, TOU4, TOU5 GB mutation_rate < 0.17 cross_operator: OX1, VR sel_method How does the selection method impact efficiency and effectiveness?

8 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Practical Suggestions: Use Tree Models

sel_method: TOU3, TOU4, TOU5 RM child_density < 0.875 0.9674 sel_method sel_method: TOU3, TOU4, TOU5 GB mutation_rate < 0.17 cross_operator: OX1, VR sel_method For More Details, Please See Conrad et al. [22]

8 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Practical Suggestions: Use Open Source

9 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Practical Suggestions: Use Open Source

Gelations is a Genetic Algorithm-Based Test Suite Prioritizer

9 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Practical Suggestions: Use Open Source

Visit http://gelations.googlecode.com/ for More Details

9 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Practical Suggestions: Use Open Source

Visit http://raise.googlecode.com/ for Greedy Algorithms

9 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Conclusions and Future Work

The Potential Way Forward May Seem Daunting

10 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Conclusions and Future Work

The Potential Way Forward May Seem Daunting

Once you get your courage up and believe that you can do important prob- lems, then you can.

Richard Hamming

10 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Conclusions and Future Work

The Potential Way Forward May Seem Daunting

Once you get your courage up and believe that you can do important prob- lems, then you can.

Richard Hamming Share One Item With Your Next Paper

10 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Conclusions and Future Work

Once you get your courage up and believe that you can do important prob- lems, then you can.

Richard Hamming Share One Item With Your Next Paper Leader of the Regres- sion Testing Community

Your Picture Here

Your Help is Welcomed and Appreciated!

10 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Conclusions and Future Work

Once you get your courage up and believe that you can do important prob- lems, then you can.

Richard Hamming Share One Item With Your Next Paper Leader of the Regres- sion Testing Community

Your Picture Here

Your Help is Welcomed and Appreciated! Read the Paper and Contact Me with Comments

10 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Conclusions and Future Work

Share One Item With Your Next Paper Leader of the Regres- sion Testing Community

Your Picture Here

Your Help is Welcomed and Appreciated! Read the Paper and Contact Me with Comments Future Work: Complete Case Study of Reproducible Research in Regression Testing

✓ Data Sets ✓ Tools ✓ Visualizations

10 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward

Introduction Challenges Potential Way Forward Conclusion

Conclusions and Future Work

Share One Item With Your Next Paper Leader of the Regres- sion Testing Community

Your Picture Here

Your Help is Welcomed and Appreciated! Read the Paper and Contact Me with Comments Acknowledgments:

✓ Neil Elliott ✓ Jonathan Kauffman ✓ Molly Mattis ✓ Zachary Williams

provided feedback on drafts of the paper

10 / 10 Empirically Evaluating Regression Testing Techniques: Challenges, Solutions, and a Potential Way Forward