Towards Regression Testing for Database Applications Gregory M. - - PowerPoint PPT Presentation

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions

Towards Regression Testing for Database Applications

Gregory M. Kapfhammer†

Department of Computer Science Allegheny College, Pennsylvania, USA http://cs.allegheny.edu/~gkapfham/

ASTReNet and SOSoRNet, King’s College London, 2007

†In Conjunction with Mary Lou Soffa (UVa/CS), Panos Chrysanthis (Pitt/CS), Bruce Childers (Pitt/CS)

Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions

Testing Database Applications

Research Contribution A regression testing framework for traditional database

• applications. Future research includes service-oriented

applications that use Grid-enabled databases.

Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions

An Interesting Defect Report

Database Server Crashes When you run a complex query against Microsoft SQL Server 2000, the SQL Server scheduler may stop responding. Additionally, you receive an error message that resembles the following: Date Time server Error: 17883 Severity: 1, State: 0 Date Time server Process 52:0 (94c) ... An Input-Dependent Defect This problem occurs when one or more of the following conditions are true: The query contains a U N I O N clause or a U N I O N A L L clause that affects many columns. The query contains several J O I N statements. The query has a large estimated cost. BUG 473858 (SQL Server 8.0)

Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions

An Interesting Defect Report

Database Server Crashes When you run a complex query against Microsoft SQL Server 2000, the SQL Server scheduler may stop responding. Additionally, you receive an error message that resembles the following: Date Time server Error: 17883 Severity: 1, State: 0 Date Time server Process 52:0 (94c) ... An Input-Dependent Defect This problem occurs when one or more of the following conditions are true: The query contains a U N I O N clause or a U N I O N A L L clause that affects many columns. The query contains several J O I N statements. The query has a large estimated cost. BUG 473858 (SQL Server 8.0)

Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions

Real World Example

A Severe Defect

The Risks Digest, Volume 22, Issue 64, 2003

Jeppesen reports airspace boundary problems

About 350 airspace boundaries contained in Jeppesen NavData are incorrect, the FAA has warned. The error occurred at Jeppesen after a software upgrade when information was pulled from a database containing 20,000 airspace boundaries worldwide for the March NavData update, which takes effect March 20.

An Important Point Practically all use of databases occurs from within application programs [Silberschatz et al., 2006, pg. 311]

Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions

Real World Example

A Severe Defect

The Risks Digest, Volume 22, Issue 64, 2003

Jeppesen reports airspace boundary problems

About 350 airspace boundaries contained in Jeppesen NavData are incorrect, the FAA has warned. The error occurred at Jeppesen after a software upgrade when information was pulled from a database containing 20,000 airspace boundaries worldwide for the March NavData update, which takes effect March 20.

An Important Point Practically all use of databases occurs from within application programs [Silberschatz et al., 2006, pg. 311]

Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions

Program and Database Interactions

1

D

e

D P m

update select insert delete

Basic Operation Program P creates SQL statements in order to view and/or modify the state of the relational database SQL Construction Static analysis does not reveal the exact SQL command since the program constructs the full SQL statement at run-time

Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions

Database Interaction Granularity

P m i m j Dl Dk

R R2

1

E F G H A B C D I

R3

J K L

Database Interactions Program P interacts with two relational databases Dk and Dl at different levels of granularity (relation, record, attribute, ...)

Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions

Overview of the Coverage Monitoring Process

Program Instrumentation Test Suite Adequacy Criterion Instrumented Program Test Coverage Monitoring Instrumented Test Suite Coverage Results Adequacy Calculation Test Requirements Adequacy Measurements Database

Calculating Coverage Use instrumentation probes to capture and analyze a program’s interaction with the databases Regression Testing The adequacy measurements can be used to support both test suite reduction and prioritization

Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions

Database-Aware Coverage Trees

Test Case Method Invocation Database Interaction Point Database Relation Attribute Record Attribute Value

Instrumentation Probes Use static and dynamic (load-time) instrumentation techniques to insert coverage monitoring probes Coverage Trees Store the coverage results in a tree in order to support the calculation of many types of coverage (e.g., data flow or call tree)

Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions

Comparing the Coverage Trees

Tree Characteristics Tree DB? Context Probe Time Tree Space CCT × Partial Low - Moderate Low DCT × Full Low Moderate - High DI-CCT

• Partial

Moderate Moderate DI-DCT

• Full

Moderate High Table Legend Database? ∈ {×, } Context ∈ {Partial, Full} Probe Time Overhead ∈ {Low, Moderate, High} Tree Space Overhead ∈ {Low, Moderate, High}

Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions

Database-Aware Regression Testing

Begin Coverage Report End VSRT Repeat Program and Database Reduction

• r Prioritization

Original Test Suite Modified Test Suite Test Suite Execution Testing Results GRT Repeat

Regression Testing Overview Reduction aims to find a smaller test suite that covers the same requirements as the original suite. Prioritization re-orders the tests so that they cover the requirements more effectively.

Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions

Finding the Overlap in Coverage

T2 T3 T6 T9 R R1 R2 T4 T8 R3 T12 T1 R4 T11 T5 R5 R6 T7 R7 T10

Test Suite Reduction Rj → Ti means that requirement Rj is covered by test Ti T = T2, T3, T6, T9 cover all of the test requirements

Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions

Measuring Coverage Effectiveness

Testing Time

. . .

Covered Test Reqs

T1 Done Tn−1 Done Tn Done Cover R(T1) Cover n−1

i=1 R(Ti)

Cover R(T) Area t(n) C(T, t)

C(T,t)

(t) Prioritize to increase the CE of a test suite CE = Actual Ideal

Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions

Configuring the Regression Testing Framework

Type Data Flow Coverage Tree Path Configuration of the Regression Tester Technique Test Cost Requirements Reduction Prioritization Type Greedy HGS Delayed Greedy Reverse Random Overlap-Aware Not Overlap-Aware Cost Coverage Ratio Unit Actual Traditional Database-Aware Unique Dominant Type of Tree Super Path Containing Path DCT CCT

Regression tester uses several algorithms and test requirements

Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions

Characterizing the Case Study Applications

Test Suites Application # Tests Test NCSS / Total NCSS R M 13 227/548 = 50.5% F F 16 330/558 = 59.1% P I 15 203/579 = 35.1% S T 25 365/620 = 58.9% T M 27 355/748 = 47.5% G B 51 769/1455 = 52.8%

Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions

Details About the Database Interactions

Static Interaction Counts Application executeUpdate executeQuery Total R M 3 4 7 F F 3 4 7 P I 3 2 5 S T 4 3 7 T M 36 9 45 G B 11 23 34 Dynamic Interaction Counts Database interactions that occur in iterative or recursive computations are executed more frequently

Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions

Reducing the Size of the Test Suite

(Size of Reduced Test Suite, Reduction Factor)

App Rel Attr Rec Attr Value RM (13) (7, .462) (7, .462) (10, .300) (9, .308) FF (16) (7, .563) (7, .563) (11, .313) (11, .313) PI (15) (6, .600) (6, .600) (8, .700) (7, .533) ST (25) (5, .800) (5, .760) (11, .560) (10, .600) TM (27) (14, .481) (14, .481) (15, .449) (14, .481) GB (51) (33, .352) (33, .352) (33, .352) (32, .373) All (24.5) (12, .510) (12.17, .503) (14.667, .401) (13.83, .435)

Reduction factor for test suite size varies from .352 to .8

Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions

Reducing the Testing Time

Rc Av Database Interaction GB 0.2 0.4 0.6 0.8 1 Reduction Factor

• Time

Rc Av 0.78 0.78 0.94 0.94 0.06 0.06 0.81 0.81 0.79 0.79 0.36 0.39 GRO GRC GRV GRR RVR RAR

GRO reduces test execution time even though it removes few tests

Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions

Preserving Requirement Coverage

Rc Av Database Interaction GB 0.2 0.4 0.6 0.8 1 Preservation Factor

• Coverage

Rc Av 1. 1. 0.3 0.09 0.98 0.99 0.96 0.98 0.91 0.94 0.71 0.72 GRO GRC GRV GRR RVR RAR

GRO guarantees coverage preservation while the others do not

Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions

Improving Coverage Effectiveness

Rc Av Database Interaction GB 0.2 0.4 0.6 0.8 1 Coverage Effectiveness Rc Av 0.94 0.94 0.88 0.87 0.87 0.9 0.93 0.93 0.84 0.86 0.22 0.22 0.55 0.56 GPO GPC GPV GPR RVP ORP RAP

GRO is the best choice and the original ordering is poor

Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions

Future Work: Avoid Database Restarts

T0 T1 T2 T9 T13 T15 T5 T8 T12 T3 T7 T11 T14 T16 T10 T4 T6

Use prioritization to reduce testing time by avoiding database restarts

Gregory M. Kapfhammer Towards Regression Testing for Database Applications

Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions

Conclusions and Future Work

Concluding Remarks A new perspective on software testing and an efficient and effective method for database-aware regression testing Future Work Challenges associated with grid-enabled databases Conduct experiments with larger database applications Resources http://cs.allegheny.edu/~gkapfham/research/diatoms/

Gregory M. Kapfhammer Towards Regression Testing for Database Applications