parameter tuning for search based test data generation
play

Parameter Tuning for Search-Based Test-Data Generation Revisited - PowerPoint PPT Presentation

Aug 10, 2023 •146 likes •725 views

Parameter Tuning for Search-Based Test-Data Generation Revisited Support for Previous Results Anton Kotelyanskii Gregory M. Kapfhammer creative commons licensed ( BY-NC-ND ) ickr photo shared by sunface13 Software Testing Software

  1. Parameter Tuning for Search-Based Test-Data Generation Revisited Support for Previous Results Anton Kotelyanskii Gregory M. Kapfhammer creative commons licensed ( BY-NC-ND ) �ickr photo shared by sunface13

  2. Software Testing

  3. Software Testing Test Suites

  4. Software Testing Test Suites Automatic Generation

  5. Software Testing Test Suites Automatic Generation Confronting Challenges

  6. Software Testing Test Suites Automatic Generation Confronting Challenges Evaluation Strategies

  7. Empirical Studies

  8. Empirical Studies Challenges

  9. Empirical Studies Challenges Importance

  10. Empirical Studies Challenges Importance Replication

  11. Empirical Studies Challenges Importance Replication Rarity

  13. EvoSuite creative commons licensed ( BY-SA ) �ickr photo shared by mcclanahoochie

  14. EvoSuite Amazing test suite generator creative commons licensed ( BY-SA ) �ickr photo shared by mcclanahoochie

  15. EvoSuite Amazing test suite generator Uses a genetic algorithm creative commons licensed ( BY-SA ) �ickr photo shared by mcclanahoochie

  16. EvoSuite Amazing test suite generator Uses a genetic algorithm Input : A Java class creative commons licensed ( BY-SA ) �ickr photo shared by mcclanahoochie

  17. EvoSuite Amazing test suite generator Uses a genetic algorithm Input : A Java class Output : A JUnit test suite creative commons licensed ( BY-SA ) �ickr photo shared by mcclanahoochie

  18. EvoSuite Amazing test suite generator Uses a genetic algorithm Input : A Java class Output : A JUnit test suite http://www.evosuite.org/ creative commons licensed ( BY-SA ) �ickr photo shared by mcclanahoochie

  19. Parameter Tuning

  20. Parameter Tuning RSM : Response surface methodology

  21. Parameter Tuning RSM : Response surface methodology SPOT : Sequential parameter optimization toolbox

  22. Parameter Tuning RSM : Response surface methodology SPOT : Sequential parameter optimization toolbox Successfully applied to many diverse problems!

  23. Defaults or Tuned Values?

  25. Experiment Design creative commons licensed ( BY-NC ) �ickr photo shared by Michael Kappel

  26. Experiment Design Eight EvoSuite parameters creative commons licensed ( BY-NC ) �ickr photo shared by Michael Kappel

  27. Experiment Design Eight EvoSuite parameters Ten projects from SF100 creative commons licensed ( BY-NC ) �ickr photo shared by Michael Kappel

  28. Experiment Design Eight EvoSuite parameters Ten projects from SF100 475 Java classes for subjects creative commons licensed ( BY-NC ) �ickr photo shared by Michael Kappel

  29. Experiment Design Eight EvoSuite parameters Ten projects from SF100 475 Java classes for subjects 100 trials after parameter tuning creative commons licensed ( BY-NC ) �ickr photo shared by Michael Kappel

  30. Experiment Design Eight EvoSuite parameters Ten projects from SF100 475 Java classes for subjects 100 trials after parameter tuning Aiming to improve statement coverage creative commons licensed ( BY-NC ) �ickr photo shared by Michael Kappel

  31. Parameters Parameter Name Minimum Maximum Population Size 5 99 Chromosome Length 5 99 Rank Bias 1.01 1.99 Number of Mutations 1 10 Max Initial Test Count 1 10 Crossover Rate 0.01 0.99 Constant Pool Use Probability 0.01 0.99 Test Insertion Probability 0.01 0.99

  32. Experiments

  33. Experiments 184 days of computation time estimated

  34. Experiments 184 days of computation time estimated Cluster of 70 computers running for weeks

  35. Experiments 184 days of computation time estimated Cluster of 70 computers running for weeks Identi�ed 139 "easy" and 21 "hard" classes

  36. Experiments 184 days of computation time estimated Cluster of 70 computers running for weeks Identi�ed 139 "easy" and 21 "hard" classes Mann-Whitney U-test and

  37. Experiments 184 days of computation time estimated Cluster of 70 computers running for weeks Identi�ed 139 "easy" and 21 "hard" classes Mann-Whitney U-test and Vargha-Delaney e�ect size

  38. Results Category E�ect Size p-value Results Across Trials and Classes 0.5029 0.1045 No "Easy" and "Hard" Classes 0.5048 0.0314

  39. Results Category E�ect Size p-value Results Across Trials and Classes 0.5029 0.1045 No "Easy" and "Hard" Classes 0.5048 0.0314 Using lower-is-better inverse statement coverage

  40. Results Category E�ect Size p-value Results Across Trials and Classes 0.5029 0.1045 No "Easy" and "Hard" Classes 0.5048 0.0314 Using lower-is-better inverse statement coverage E�ect size greater than 0.5 means that tuning is worse

  41. Results Category E�ect Size p-value Results Across Trials and Classes 0.5029 0.1045 No "Easy" and "Hard" Classes 0.5048 0.0314 Using lower-is-better inverse statement coverage E�ect size greater than 0.5 means that tuning is worse Testing shows we do not always reject the null hypothesis

  42. Results Category E�ect Size p-value Results Across Trials and Classes 0.5029 0.1045 No "Easy" and "Hard" Classes 0.5048 0.0314 Using lower-is-better inverse statement coverage E�ect size greater than 0.5 means that tuning is worse Testing shows we do not always reject the null hypothesis Additional empirical results in the QSIC 2014 paper!

  44. Discussion creative commons licensed ( BY ) photo shared by Startup Stock Photos

  45. Discussion Tuning improved scores for 11 classes creative commons licensed ( BY ) photo shared by Startup Stock Photos

  46. Discussion Tuning improved scores for 11 classes Otherwise, same as or worse than defaults creative commons licensed ( BY ) photo shared by Startup Stock Photos

  47. Discussion Tuning improved scores for 11 classes Otherwise, same as or worse than defaults A "soft �oor" may exist for parameter tuning creative commons licensed ( BY ) photo shared by Startup Stock Photos

  48. Discussion Tuning improved scores for 11 classes Otherwise, same as or worse than defaults A "soft �oor" may exist for parameter tuning Additional details in the QSIC 2014 paper! creative commons licensed ( BY ) photo shared by Startup Stock Photos

  49. Practical Implications

  50. Practical Implications Fundamental Challenges

  51. Practical Implications Fundamental Challenges Tremendous Con�dence

  52. Practical Implications Fundamental Challenges Tremendous Con�dence Great Opportunities

  54. Important Contributions creative commons licensed ( BY-NC-ND ) �ickr photo shared by sunface13

  55. Important Contributions Comprehensive Experiments creative commons licensed ( BY-NC-ND ) �ickr photo shared by sunface13

  56. Important Contributions Comprehensive Experiments Conclusive Con�rmation creative commons licensed ( BY-NC-ND ) �ickr photo shared by sunface13

  57. Important Contributions Comprehensive Experiments Conclusive Con�rmation For EvoSuite, Defaults = Tuned creative commons licensed ( BY-NC-ND ) �ickr photo shared by sunface13

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Data Mining II Optimization & Parameter Tuning Heiko Paulheim Why Parameter Tuning?

Data Mining II Optimization & Parameter Tuning Heiko Paulheim Why Parameter Tuning?

Data Mining II Optimization & Parameter Tuning Heiko Paulheim Why Parameter Tuning? What we have seen so far many learning algorithms for classification, regression, ... Many of those have parameters k and distance

797 views • 52 slides
Data Mining II Optimization & Parameter Tuning Heiko Paulheim Why Parameter Tuning?

Data Mining II Optimization & Parameter Tuning Heiko Paulheim Why Parameter Tuning?

Data Mining II Optimization & Parameter Tuning Heiko Paulheim Why Parameter Tuning? What we have seen so far many learning algorithms for classification, regression, ... Many of those have parameters k and distance

586 views • 54 slides
Search Based Test Data Generation for Server-side Web Application Testing Nadia Alshahwan and

Search Based Test Data Generation for Server-side Web Application Testing Nadia Alshahwan and

Search Based Test Data Generation for Server-side Web Application Testing Nadia Alshahwan and Mark Harman CREST Centre University College London Automated web application testing using search based software Engineering (ASE 2011) Hill

588 views • 15 slides
Measurement-Based TCP Parameter Tuning in Cloud Data Centers Simon Jouet

Measurement-Based TCP Parameter Tuning in Cloud Data Centers Simon Jouet

Measurement-Based TCP Parameter Tuning in Cloud Data Centers Simon Jouet University of Glasgow [s.jouet.1@research.gla.ac.uk] Background TCP CongesHon Control For a transport endpoint embedded in a

546 views • 9 slides
On Automated Parameter Tuning, with Applications in Next-Generation Manufacturing Lars Kotthofg

On Automated Parameter Tuning, with Applications in Next-Generation Manufacturing Lars Kotthofg

On Automated Parameter Tuning, with Applications in Next-Generation Manufacturing Lars Kotthofg and Patrick Johnson Artifjcially Intelligent Manufacturing Center University of Wyoming larsko,pjohns27@uwyo.edu UCC, 02 April 2019 1 Big

567 views • 41 slides
S. Scalabrino G. Grano D. Di Nucci A. De Lucia M. Guerra H. Gall R. Oliveto OCELOT a

S. Scalabrino G. Grano D. Di Nucci A. De Lucia M. Guerra H. Gall R. Oliveto OCELOT a

S. Scalabrino G. Grano D. Di Nucci A. De Lucia M. Guerra H. Gall R. Oliveto OCELOT a Search-Based Test-Data Generation Tool for C Testing is expensive Automatic Test Generation Search-Based Test Generation TIOBE Ranking TIOBE Ranking

398 views • 28 slides
10/16/19 Parameters and Parameter Tuning Genetic Algorithms History Taxonomy

10/16/19 Parameters and Parameter Tuning Genetic Algorithms History Taxonomy

10/16/19 Parameters and Parameter Tuning Genetic Algorithms History Taxonomy Parameter Tuning vs Parameter Control EA calibration Parameter Tuning Testing Effort Parameters and Recommendations Parameter

569 views • 11 slides
Model-Based Testing: Automated Generation of Test Cases, Test Data, and Test Procedures from

Model-Based Testing: Automated Generation of Test Cases, Test Data, and Test Procedures from

Model-Based Testing: Automated Generation of Test Cases, Test Data, and Test Procedures from SysML Models Jrg Brauer and Jan Peleska Verified Systems International GmbH support@verified.de Space Tech Expo Europe 2015-11-19 Motivation

499 views • 26 slides
MODEL-BASED, MUTATION-DRIVEN TEST CASE GENERATION VIA HEURISTIC-GUIDED BRANCHING SEARCH Andreas

MODEL-BASED, MUTATION-DRIVEN TEST CASE GENERATION VIA HEURISTIC-GUIDED BRANCHING SEARCH Andreas

MODEL-BASED, MUTATION-DRIVEN TEST CASE GENERATION VIA HEURISTIC-GUIDED BRANCHING SEARCH Andreas Fellner FMCAD Student Forum Wien, October 4th 2017 TEST CASE GENERATION WITH 1 Andreas Fellner TEST CASE GENERATION WITH Abstract Model UML /

358 views • 34 slides
A UML-Based Testing Approach Using Sequence Diagrams, Statecharts, and OCL Constraints Dehla

A UML-Based Testing Approach Using Sequence Diagrams, Statecharts, and OCL Constraints Dehla

A UML-Based Testing Approach Using Sequence Diagrams, Statecharts, and OCL Constraints Dehla Sokenou TU Berlin Softwaretechnik Overview of the Test System test data test generation driver test case abstract test data generation test

444 views • 20 slides
Pruning the Search Space in Path-based Test Generation Motivation SE S ebastien Bardin

Pruning the Search Space in Path-based Test Generation Motivation SE S ebastien Bardin

Pruning the Search Space in Path-based Test Generation Motivation SE S ebastien Bardin Heuristics sebastien.bardin@cea.fr Experiments CEA-LIST, Software Security Labs Conclusion (joint work with Philippe Herrmann) S ebastien Bardin,

837 views • 44 slides
Learning to Branch Balcan, Dick, Sandholm, Vitercik Introduction Parameter tuning tedious and

Learning to Branch Balcan, Dick, Sandholm, Vitercik Introduction Parameter tuning tedious and

Learning to Branch Balcan, Dick, Sandholm, Vitercik Introduction Parameter tuning tedious and time-consuming Algorithm configuration using Machine Learning Focus on tree search algorithms Branch-and-Bound Tree Search Widely

396 views • 17 slides
Automated Test Data Generation for Coverage: Lakhotia et al. Outline Havent We Solved This

Automated Test Data Generation for Coverage: Lakhotia et al. Outline Havent We Solved This

Havent We Solved This Problem Yet? Automated Test Data Generation for Coverage: Lakhotia et al. Outline Havent We Solved This Problem Yet? Motivation Background Search Based Kiran Lakhotia 1 Phil McMinn 2 Mark Harman 1 Testing

811 views • 46 slides
Discussion Regarding Uneconomic Adjustment Policy & Parameter Tuning Market and Product

Discussion Regarding Uneconomic Adjustment Policy & Parameter Tuning Market and Product

Discussion Regarding Uneconomic Adjustment Policy & Parameter Tuning Market and Product Development Team Joint MSC/Stakeholder Meeting September 25, 2008 Uneconomic Adjustment Policy/Parameter Tuning Overview/Context Greg Cook Manager,

876 views • 66 slides
Hyperparameter tuning in caret Dr. Shirin Glander Data Scientist DataCamp Hyperparameter

Hyperparameter tuning in caret Dr. Shirin Glander Data Scientist DataCamp Hyperparameter

DataCamp Hyperparameter Tuning in R HYPERPARAMETER TUNING IN R Hyperparameter tuning in caret Dr. Shirin Glander Data Scientist DataCamp Hyperparameter Tuning in R Voter dataset from US 2016 election Split intro training and test set

779 views • 22 slides
Linkages Between Parameter Tuning and Scarcity Pricing Frank A. Wolak Chair, Market Surveillance

Linkages Between Parameter Tuning and Scarcity Pricing Frank A. Wolak Chair, Market Surveillance

Linkages Between Parameter Tuning and Scarcity Pricing Frank A. Wolak Chair, Market Surveillance Committee (MSC) Joint Stakeholder/MSC Meeting July 30, 2008 Common Features of Scarcity Pricing and Parameter Tuning Scarcity pricing occurs

731 views • 13 slides
Rare metabolic diseases: the miglustat experience Fran Platt Fran Platt Department of

Rare metabolic diseases: the miglustat experience Fran Platt Fran Platt Department of

Rare metabolic diseases: the miglustat experience Fran Platt Fran Platt Department of Pharmacology Department of Pharmacology University of Oxford University of Oxford The lysosome lysosome is an organelle involved in The is an organelle

550 views • 25 slides
Local-Scale Planning, Climate Change and Resilience 22nd Council of UArctic Meeting Stockholm,

Local-Scale Planning, Climate Change and Resilience 22nd Council of UArctic Meeting Stockholm,

Proposal for a new UArctic Thematic Network: Local-Scale Planning, Climate Change and Resilience 22nd Council of UArctic Meeting Stockholm, Sweden September 18-20, 2019 Dr. Jeff Birchall University of Alberta, Canada Global Climate Change

342 views • 16 slides
Market Operator User Group Dublin, 10 October 2018 1 Agenda The Agenda for the session is as

Market Operator User Group Dublin, 10 October 2018 1 Agenda The Agenda for the session is as

Market Operator User Group Dublin, 10 October 2018 1 Agenda The Agenda for the session is as follows: Item Presenter Welcome Liam Ryan Ex-Ante Market Liam McAllister Balancing Market Diarmaid Gillespie, Simon Tweed Settlement Sean

1.07k views • 83 slides
Small Area Models for Linking Deprivation to Local Areas in Italy ( draft ) Gennaro PUNZO

Small Area Models for Linking Deprivation to Local Areas in Italy ( draft ) Gennaro PUNZO

InGrid Summer School Reaching out to hard-to-survey groups among the poor HIVA-KU Leuven, Leuven - Belgium, 30 May -3 June 2016 Small Area Models for Linking Deprivation to Local Areas in Italy ( draft ) Gennaro PUNZO Universit y of

198 views • 18 slides
How to do successful gene expression analysis Jan Hellemans, PhD Center for Medical Genetics

How to do successful gene expression analysis Jan Hellemans, PhD Center for Medical Genetics

How to do successful gene expression analysis Jan Hellemans, PhD Center for Medical Genetics Biogazelle qPCR meeting June 25 th 2010 Sienna, Italy Introduction qPCR: reference technology for nucleic acid quantification

939 views • 47 slides
Longitudinal Survey in United Kingdom: Assessing Construct, Method and Item Bias in the General

Longitudinal Survey in United Kingdom: Assessing Construct, Method and Item Bias in the General

An Integrated Approach to Bias in a Longitudinal Survey in United Kingdom: Assessing Construct, Method and Item Bias in the General Health Questionnaire (GHQ-12). Isabel Bentez, Byron Adams & Jia He Outline contents Background: Bias

322 views • 16 slides
Banking Crises and Exports: Lessons from the Past Leonardo Iacovone, Veronika Zavacka 3rd FIW

Banking Crises and Exports: Lessons from the Past Leonardo Iacovone, Veronika Zavacka 3rd FIW

Previous literature Methodology and Data Results Robustness Conclusions Banking Crises and Exports: Lessons from the Past Leonardo Iacovone, Veronika Zavacka 3rd FIW Research Conference, Vienna December 11, 2009 Previous literature

346 views • 13 slides
How CECL Will Impact Your Credit Union & What You Can Do to Prepare For It: Part 2 Randy C

How CECL Will Impact Your Credit Union & What You Can Do to Prepare For It: Part 2 Randy C

How CECL Will Impact Your Credit Union & What You Can Do to Prepare For It: Part 2 Randy C Thompson, Ph.D. TCT Risk Solutions, LLC 1 Impact on ALLL Calculation Grade Balance Loss Factor ALLL Req. Baseline ALLL A+ $ 755,000 0.00%

812 views • 44 slides

More recommend