Leveraging Field Data for Impact Analysis and Regression Testing Alessandro Orso joint work with Taweesup Apiwattanapong Mary Jean Harrold {orso|term|harrold}@cc.gatech.edu This work was supported in part by National Science Foundation awards CCR-9988294, CCR-0096321, CCR-0205422, SBE-0123532, and EIA-0196145 to Georgia Tech, and by the State of Georgia to Georgia Tech under the Yamacraw Mission ESEC/FSE 2003
Motivation Fundamental shift in SW development • Software virtually everywhere • Most computers interconnected • Large amount of user resources Opportunity to use field data and resources in SE • Testing and analysis limited by the use of in-house inputs and configurations • Limits can be overcome by augment these tasks with field data ESEC/FSE 2003
Overall Picture User User User Program User User User User P User User User User User User User User User Impact Analysis Field Execution Data Regression Testing ESEC/FSE 2003
Gathering Field Data Program P User A User B m 1 m 2 m 1 m 2 m 1 m 2 m 1 m 2 m 1 m 2 m 3 m 4 m 3 m 4 m 3 m 4 m 3 m 4 m 3 m 4 m 5 m 6 m 5 m 6 m 5 m 6 m 5 m 6 m 5 m 6 m 1 m 2 m 3 m 4 m 5 m 6 execution data A1 X X B1 X X X X B2 X X X A2 X X X X C1 ESEC/FSE 2003
Impact Analysis Assesses the effects of changes on a software system Predictive: help decide which changes to perform and how to implement changes Our approach • Program-sensitive impact analysis • User-sensitive impact analysis ESEC/FSE 2003
Program Sensitive Impact Analysis Input: Output: 1. Field execution data {m2,m5,m6} Impact set = m 1 m 2 m 3 m 4 m 5 m 6 Step 1 A1 X X • Identify user executions through B1 X X X X methods in C B2 X X X • Identify methods covered by A2 X X such executions X X C1 covered methods = {m1,m2,m3,m5,m6} Step 2 2. Change • Static forward slice from C C={m2, m5} forward slice = {m2,m4,m5,m6} Step 3 • Intersect covered methods and forward slice ESEC/FSE 2003
User-sensitive Impact Analysis Input: Output: 1. Field execution data 60% 1. Collective impact = m 1 m 2 m 3 m 4 m 5 m 6 2. Affected users = 100% A1 X X B1 X X X X B2 X X X Collective impact A2 X X • Percentage of executions through X X C1 at least one changed method 2. Change 3/5 = 60% C={m5, m6} Affected users • Percentage of users that executed at least once one changed method 3/3 = 100% ESEC/FSE 2003
Regression Testing Performed after P is changed to P’ to provide confidence that • Changed parts behave as intended • Unchanged parts are not adversely affected by modifications Three important issues • Tests in T to rerun on P’ ( selection ) • New tests for P’ ( augmentation ) • Order of execution of tests ( prioritization ) ESEC/FSE 2003
Regression Testing Using Field Data Input: Output: {t2, t3} 1. Field execution data 1. Tests T’ to be rerun on P’ = 2. Critical methods = m 1 m 2 m 3 m 4 m 5 m 6 CM[m2] = {m3,m5} A1 X X CM[m4] = {m1} B1 X X X X B2 X X X For each changed method m in C A2 X X • Add all tests through m to T’ 2. Change • Compute the impact set for m C={m2, m4} impact set = {m1,m2,m3,m5,m6} 3. In-house tests for P • For each t in T’ mark methods in m 1 m 2 m 3 m 4 m 5 m 6 impact set exercised by t • Remove marked methods from t1 X X X impact set X X t2 X t3 X X X ESEC/FSE 2003
Regression Testing Using Field Data Input: Output: {t2, t3} 1. Field execution data 1. Tests T’ to be rerun on P’ = 2. Critical methods = m 1 m 2 m 3 m 4 m 5 m 6 CM[m2] = {m3,m5} A1 X X CM[m4] = {m1} B1 X X X X B2 X X X For each changed method m in C A2 X X • Add all tests through m to T’ 2. Change • Compute the impact set for m C={m2, m4} C={m2, m4} impact set = {m1,m2,m4,m6} 3. In-house tests for P • For each t in T’ mark methods in m 1 m 2 m 3 m 4 m 5 m 6 impact set exercised by t • Remove marked methods from t1 X X X impact set X X t2 X t3 X X X ESEC/FSE 2003
Empirical Studies Subject • JABA: Java Architecture for Bytecode Analysis • 60 KLOC, 550 classes, 2,800 Methods Data • Field data: 1,100 executions (14 users, 12 weeks) • In-house data: 195 test cases, 63% method coverage • Changes: 20 real changes extracted from JABA’s CVS repository ESEC/FSE 2003
Study 1: Impact Analysis Research question Does field data yield different results than in-house data in terms of impact sets? Experimental setup • Computed impact sets for the 20 changes • Using field data • Using in-house data • Compared impact sets for the two datasets ESEC/FSE 2003
Study 1: Impact Analysis Number of methods Field InHouse Field - InHouse InHouse - Field 1000 800 600 400 200 0 C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12 C13 C14 C15 C16 C17 C18 C19 C20 Changes 100 636 96 Field InHouse ESEC/FSE 2003
Study 2: Regression Testing Research question Does the use of field data actually result in additional testing requirements? Experimental setup Computation of the set of critical methods for the 20 real changes ESEC/FSE 2003
Study 2: Regression Testing 400 350 #Critical Methods 300 250 200 150 100 50 0 C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12 C13 C14 C15 C16 C17 C18 C19 C20 Changes ESEC/FSE 2003
Related Work • Perpetual/Residual testing (Clarke, Osterweil, Richardson, and Young) • Expectation-Driven Event Monitoring (EDEM) (Hilbert, Redmiles, and Taylor) • Echelon (Srivastava and Thiagarajan) • Impact analysis based on whole-path profiling (Law and Rothermel) ESEC/FSE 2003
Final Remarks Conclusion • Two new techniques for impact analysis and regression testing based on field data • Empirical evaluation on a real subject with real users • Results showing that using field data considerably affect these tasks Open Issues and future work • Study on the stability of user behaviors • Collection of additional data • Clustering of field data • Capture and replay of users’ executions ESEC/FSE 2003
For more information: http://gamma.cc.gatech.edu Questions? ESEC/FSE 2003
Recommend
More recommend
