State of the Art and Open Issues in Software Testing State of the - PowerPoint PPT Presentation

State of the Art and Open Issues in Software Testing

State of the Art and Open Issues in Software Testing

Software Testing d e a t g t i s e v i n d n a Most used approach 18 13.5 9 4.5 0 2000 2002 2004 2006 2008 2010 2012 2014 2016 2018 Number of papers on testing at ICSE from 2000 to 2018

Software Testing d e a t g t i s e v i n d n a Most used approach

Software Testing d e a t g t i s e v i n d n a Most used approach

Testing – A Travelogue

A Travelogue – Goal Discuss most successful research in software testing since 2000 Identify most significant challenges and opportunities

A Travelogue – Approach Over 50 colleagues About 30 responses Two questions 1. What do you think are the most significant contributions to testing since 2000? 2. What do you think are the biggest open challenges and opportunities in this area?

In a Nutshell

Research Contributions Practical Contributions Automated Test Input Generation Frameworks for Test Execution Continuous Integration Dynamic Symbolic Execution Search-based Testing Random Testing Combined Techniques Testing Strategies Combinatorial Testing Model-Based Testing Challenges/Opportunities Mining/Learning from Field Data Testing Real-World Systems Oracles Probabilistic Approaches Empirical Studies & Infrastructure Testing Non-Functional Prop. Domain-Based Testing Regression Testing Leveraging Cloud and Crowd

So Many Things, So Little Time… Automated Test Input Generation Regression Testing Empirical Studies 
 & 
 Practical Contributions Leveraging Cloud and Crowd Infrastructure

So Many Things, So Little Time… Automated Test Input Generation Regression Testing Empirical Studies 
 & 
 Practical Contributions Leveraging Cloud and Crowd Infrastructure

Automated Test Input Generation { Achieve coverage goal Input Reach a given point/state Not new, but resurgence • Symbolic execution • Search-based testing Technical improvements • Random/fuzz testing Powerful machines Powerful decision procedures • Combined techniques Careful engineering

Automated Test Input Generation { Achieve coverage goal Input Reach a given point/state Not new, but resurgence • Symbolic execution • Search-based testing Technical improvements • Random/fuzz testing Powerful machines Powerful decision procedures • Combined techniques Careful engineering

Symbolic Execution SS: x=x 0 , y=y 0 SS: x=x 0 , y=y 0 , z=x 0 +y 0 SS: x=x 0 , y=y 0 , z=x 0 +y 0 SS: x=x 0 , y=y 0 foo (x, y) { Normal execution: PC: true PC: y 0 > 0 ∧ x 0 +y 0 < y 0 PC: y 0 > 0 PC: y 0 > 0 if(y > 0) { Input: x=4, y=3 z = x + y; Outcome: “OK” if(z < y) Symbolic execution: Symbolic execution: fail(); Input: x=x 0 , y=y 0 Input: x=x 0 , y=y 0 } Outcome: Outcome: print(“OK”); failure PC: y 0 > 0 ∧ 
 x 0 = -1 
 solver x 0 + y 0 < y 0 y 0 = 4

Symbolic Execution SS: x=x 0 , y=y 0 , z=cxf(x 0 , y 0 ) 
 SS: x=x 0 , y=y 0 , z=x 0 +y 0 foo (x, y) { Normal execution: PC: y 0 > 0 ∧ cxf(x 0 , y 0 ) < y 0 PC: y 0 > 0 ∧ x 0 +y 0 >10 if(x > 0) { Input: x=4, y=3 z = cxf(x, y); Outcome: “OK” if(z < y) Symbolic execution: Symbolic execution: fail(); Input: x=x 0 , y=y 0 Input: x=x 0 , y=y 0 } Outcome: Outcome: print(“OK”); failure failure ? PC: y 0 > 0 ∧ 
 PC: y 0 > 0 ∧ 
 solver x 0 + y 0 < y 0 cxf(x 0 , y 0 )<y 0

Dynamic Symbolic Execution Automated Test Input Generation SS: x=x 0 , y=y 0 , z=cxf(x 0 , y 0 ) SS: x=x 0 , y=y 0 , z=cxf(x 0 , y 0 ) SS: x=x 0 , y=y 0 SS: x=x 0 , y=y 0 SS: x=x 0 , y=y 0 , z=cxf(x 0 , y 0 ) SS: x=x 0 , y=y 0 , z=cxf(x 0 , y 0 ) 
 foo (x, y) { Normal execution: Success Stories CS: x=4, y=3, z=69 CS: x=4, y=3 CS: x=4, y=3 CS: x=4, y=3, z=69 CS: x=4, y=3, z=69 PC: x 0 > y 0 ∧ cxf(x 0 , y 0 ) < y 0 if(x > 0) { Input: x=4, y=3 Open Challenges • Academia : countless citations (e.g., over 1300 PC: x 0 > 0 ∧ cxf(x 0 , y 0 ) > y 0 PC: x 0 > 0 ∧ cxf(x 0 , y 0 ) < y 0 PC: true PC: x 0 > 0 PC: x 0 > 0 z = cxf(x, y); Outcome: “OK” • Highly structured inputs if(z < y) for DART), publications, and applications Symbolic execution: Symbolic execution: Symbolic execution: • External libraries • Tools : Crest, Klee, Pex, Sage, Symbolic JPF, … fail(); Input: x=x 0 , y=y 0 Input: x=x 0 , y=y 0 Input: x=x 0 , y=y 0 • Large complex programs • Industry : Microsoft, NASA, IBM, Fujitsu, … } Outcome: Outcome: Outcome: • Oracle problem print(“OK”); failure failure failure PC: x 0 > 0 ∧ 
 PC: x 0 > 0 ∧ 
 PC: x 0 > 0 ∧ 
 x 0 = 10 
 solver 69 < y 0 cxf(x 0 , y 0 )<y 0 cxf(4, 3) < y 0 y 0 = 80

So Many Things, So Little Time… Automated Test Input Generation Regression Testing Empirical Studies 
 & 
 Practical Contributions Leveraging Cloud and Crowd Infrastructure

Regression Testing ? Common Problem Changes require rapid modification and testing for quick release • (time to market pressures) This causes released software to have many defects • Test suite T Research Question How can we test well to gain confidence in the changes in an efficient way before release of changed software? Approach Focus on changes • Automate (as much as possible) the regression testing process • Program P Program P'

Regression Testing Process and Issues Test-suite Regression Test-suite Test suite Tval Test suite T' Prioritized maintenance prioritization test selection Test suite T' Test suite T augmentation Test-suite Obsolete test cases Redundant test cases Modified Minimized e T e s t - c a s Test-suite test suite test suite Test suite Taug o n m a n i p u l a t i minimization

Regression Testing Process and Issues Test-suite Regression Test-suite Test suite Tval Test suite T' Prioritized maintenance prioritization test selection Test suite T' Test suite T augmentation Test-suite Obsolete test cases Redundant test cases Modified Minimized e T e s t - c a s Test-suite test suite test suite Test suite Taug o n m a n i p u l a t i minimization

RTS Algorithm 1. Build JIG for P 2. Collect coverage data test cases G if() tc1 tc2 tc3 e1 e2 X e1 edges doA doB X X e2 3. Build G’ and compare 4. Select affected tests test cases G G’ tc1 tc2 tc3 if() if() if() if() e1 e1 e2 e2 X e1 edges doA doA doB doB doB doA doA doC doC doC X X e2

Regression Testing Process and Issues Test-suite Regression Test-suite Test suite Tval Test suite T' Prioritized maintenance prioritization test selection Test suite T' Test suite T augmentation Test-suite Obsolete test cases Redundant test cases Modified Minimized e T e s t - c a s Test-suite test suite test suite Test suite Taug o n m a n i p u l a t i minimization

BERT Phase I : Test case Change Generation of generator analyzer test cases for Code changes C changed code Program P Program P' Tests for C TC Test runner & Behavioral comparator Test suite T Raw behavioral Phase II : Behavioral comparison differences Phase III : Behavioral Differential behavior differences analysis and reporting Behavioral analyzer differences

BERT Test case Change generator analyzer Focus on a small 
 Code changes C code fraction ➡ thorough 
 ➡ immediate 
 feedback Program P Program P' Tests for C TC Test runner & Behavioral comparator Analyze differential 
 Test suite T behavior 
 Raw behavioral Raw behavioral differences differences ➡ no oracles Behavioral differences Behavioral analyzer differences

Regression Testing Process and Issues Test-suite Regression Test-suite Test suite Tval Test suite T' Prioritized maintenance prioritization test selection Test suite T' Test suite T augmentation Test-suite Obsolete test cases Redundant test cases Modified Minimized e T e s t - c a s Test-suite test suite test suite Test suite Taug o n m a n i p u l a t i minimization

Overview of MINTS Test-related data Coverage Cost Fault detection Test suite data data data Minimization MINTS criteria Minimized Test suite tool Criterion #1 Criterion #2 Testing team (suitably encoded) (or timeout) Minimization Solution problem Criterion #n Minimization policy Solver 1 Solver n

Regression Testing Process and Issues Test-suite Regression Test-suite Test suite Tval Test suite T' Prioritized maintenance prioritization test selection Test suite T' Test suite T augmentation Test-suite Obsolete test cases Redundant test cases Modified Minimized e T e s t - c a s Test-suite test suite test suite Test suite Taug o n m a n i p u l a t i minimization

Regression Testing Open Issues? • Greater industrial uptake • Requires better efforts to understand practitioners’ problems and needs • Industrial case studies may help • Augmentation • Test suite repair

So Many Things, So Little Time… Automated Test Input Generation Regression Testing Empirical Studies 
 & 
 Practical Contributions Leveraging Cloud and Crowd Infrastructure