Scaling Regression Testing to Large Software Systems Alessandro - - PowerPoint PPT Presentation

▶
scaling regression testing to large software systems
SMART_READER_LITE
LIVE PREVIEW

Scaling Regression Testing to Large Software Systems Alessandro - - PowerPoint PPT Presentation

Mar 27, 2024 387 likes •722 views

Scaling Regression Testing to Large Software Systems Alessandro Orso Co-authors: Nanjuan Shi, Mary Jean Harrold College of Computing Georgia Institute of Technology Supported in part by National Science Foundation (awards CCR-0306372,

slide-1
SLIDE 1

Scaling Regression Testing to Large Software Systems

Alessandro Orso

Co-authors: Nanjuan Shi, Mary Jean Harrold College of Computing Georgia Institute of Technology

Supported in part by National Science Foundation (awards CCR-0306372, CCR-0205422, CCR- 9988294, CCR-0209322, SBE-0123532, and CCR-0080900) and Boeing Commercial Airplane Group.

slide-2
SLIDE 2

FSE 04

Testing of Evolving Software

P’ Program P Test Suite T T Trerun T - Trerun T’ P’’ T’rerun T’ - T’rerun

…

Time to rerun T time Analysis Time Time to rerun Trerun Savings

slide-3
SLIDE 3

FSE 04

Previous and Related Work

  • Ideal solution: two-phase approach
  • Class-Level analysis  subset of P
  • Stmt-Level analysis on the subset  Trerun

P T (coverage info) P’ Trerun

Retest

Stmt-Level Analysis

Time to rerun T time Analysis Time Time to rerun Trerun Savings

[OOPSLA01]

slide-4
SLIDE 4

FSE 04

Previous and Related Work

  • Ideal solution: two-phase approach
  • Class-Level analysis  subset of P
  • Stmt-Level analysis on the subset  Trerun

P T (coverage info) P’ Trerun

Retest

Stmt-Level Analysis

Time to rerun T time Analysis time Time to rerun Trerun

Precise, Expensive Techniques

Low-Level Analysis

Jboss, web application server, 1 million LOC

slide-5
SLIDE 5

FSE 04

Previous and Related Work

Time to rerun T time Analysis time Time to rerun Trerun

P T (coverage info) P’ Trerun

Efficient, Imprecise Techniques

High-Level Analysis

slide-6
SLIDE 6

FSE 04

Previous and Related Work

Time to rerun T time Analysis time Time to rerun Trerun

P T (coverage info) P’ Trerun

Efficient, Imprecise Techniques

High-Level Analysis

Related Work

  • Efficient, less precise techniques
  • White and Leung [CSM92]
  • Chen, Rosenblum, and Vo [ICSE94]
  • Hsia et al. [SMRP97]
  • White and Abdullah [QW97]
  • Ren et al. [OOPSLA04]
  • Expensive, more precise techniques
  • Binkley [TSE97]
  • Rothermel and Harrold [TOSEM97]
  • Vokolos and Frankl [RQSSIS97]
  • Ball [ISSTA’98]
  • Rothermel, Harrold, and Dedhia [JSTVR00]
  • Harrold et al. [OOPSLA01]
  • Bible, Rothermel, and Rosenblum [TOSEM01]
slide-7
SLIDE 7

FSE 04

Proposed Solution Two-phase approach

  • Class-Level analysis  subset of P and P’
  • Stmt-Level analysis on the subset  Trerun

P T (coverage info) P’ Trerun

Two-Phase Technique

Class-Level Analysis Stmt-Level Analysis Subset

  • f P

Subset

  • f P’
slide-8
SLIDE 8

FSE 04

Outline

  • Background
  • Technique
  • Empirical Evaluation
  • Conclusion
slide-9
SLIDE 9

FSE 04

Motivating Example

class A { void foo() {…} } class B extends A {

}

class C extends B {} class D { void bar() {

A ref=null; switch(somevar) {

case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();

} }

class E extends D {} class F { void bar(D d) {…} }

P

slide-10
SLIDE 10

FSE 04

Motivating Example

class A { void foo() {…} } class B extends A {

}

class C extends B {} class D { void bar() {

A ref=null; switch(somevar) {

case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();

} }

class E extends D {} class F { void bar(D d) {…} }

P P’ class A {

void foo() {…} } class B extends A { void foo() {… }

}

class C extends B {} class D { void bar() {

A ref=null; switch(somevar) {

case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();

} }

class E extends D {} class F { void bar(D d) {…} }

slide-11
SLIDE 11

FSE 04

Motivating Example

class A { void foo() {…} } class B extends A {

}

class C extends B {} class D { void bar() {

A ref=null; switch(somevar) {

case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();

} }

class E extends D {} class F { void bar(D d) {…} }

P P’ class A {

void foo() {…} } class B extends A { void foo() {… }

}

class C extends B {} class D { void bar() {

A ref=null; switch(somevar) {

case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();

} }

class E extends D {} class F { void bar(D d) {…} }

slide-12
SLIDE 12

FSE 04

Motivating Example

class A { void foo() {…} } class B extends A {

}

class C extends B {} class D { void bar() {

A ref=null; switch(somevar) {

case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();

} }

class E extends D {} class F { void bar(D d) {…} }

P P’ class A {

void foo() {…} } class B extends A { void foo() {… }

}

class C extends B {} class D { void bar() {

A ref=null; switch(somevar) {

case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();

} }

class E extends D {} class F { void bar(D d) {…} }

slide-13
SLIDE 13

FSE 04

Class-Level Analysis

class B extends A {

}

P P’

class B extends A { void foo() {… }

}

slide-14
SLIDE 14

FSE 04

Class-Level Analysis

class A { void foo() {…} } class B extends A {

}

class C extends B {}

P P’ class A {

void foo() {…} } class B extends A { void foo() {… }

}

class C extends B {}

slide-15
SLIDE 15

FSE 04

Class-Level Analysis

class A { void foo() {…} } class B extends A {

}

class C extends B {} class D { void bar() {

A ref=null; switch(somevar) {

case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();

} }

P P’ class A {

void foo() {…} } class B extends A { void foo() {… }

}

class C extends B {} class D { void bar() {

A ref=null; switch(somevar) {

case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();

} }

slide-16
SLIDE 16

FSE 04

Class-Level Analysis

P / P’

A D F B E C Inheritance edge Use edge

Interclass Relation Graph (for P an P’)

class A { void foo() {…} } class B extends A { void foo() {… }

}

class C extends B {} class D { void bar() {

A ref=null; switch(somevar) {

case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();

} }

class E extends D {} class F { void bar(D d) {…} }

slide-17
SLIDE 17

FSE 04

Class-Level Analysis

P / P’

A D F B E C Inheritance edge Use edge

Interclass Relation Graph (for P an P’)

class A { void foo() {…} } class B extends A { void foo() {… }

}

class C extends B {} class D { void bar() {

A ref=null; switch(somevar) {

case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();

} }

class E extends D {} class F { void bar(D d) {…} }

slide-18
SLIDE 18

FSE 04

Class-Level Analysis

P / P’

A D F B E C Inheritance edge Use edge

Interclass Relation Graph (for P an P’)

class A { void foo() {…} } class B extends A { void foo() {… }

}

class C extends B {} class D { void bar() {

A ref=null; switch(somevar) {

case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();

} }

class E extends D {} class F { void bar(D d) {…} }

slide-19
SLIDE 19

FSE 04

Class-Level Analysis

P / P’

A D F B E C Inheritance edge Use edge

Interclass Relation Graph (for P an P’)

class A { void foo() {…} } class B extends A { void foo() {… }

}

class C extends B {} class D { void bar() {

A ref=null; switch(somevar) {

case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();

} }

class E extends D {} class F { void bar(D d) {…} }

slide-20
SLIDE 20

FSE 04

Example: Stmt-Level Analysis

ref.foo() A.foo() …

C B A

… …

A.foo() …

A C B

ref.foo()

… …

B.foo() … A.foo() B.foo()

Dangerous Edge

G’ (excerpt) G (excerpt)

class A class B {…} class C class D { void bar() {…; ref.foo(); …} } }

Subset of P’ Subset of P

class A class B {… void foo() {…} … } class C class D { void bar() {…; ref.foo(); …} } }

slide-21
SLIDE 21

FSE 04

Example: Stmt-Level Analysis

ref.foo() A.foo() …

C B A

… …

A.foo() …

A C B

ref.foo()

… …

B.foo() … A.foo() B.foo()

Dangerous Edge

G’ (excerpt) G (excerpt)

class A class B {…} class C class D { void bar() {…; ref.foo(); …} } }

Subset of P’ Subset of P

class A class B {… void foo() {…} … } class C class D { void bar() {…; ref.foo(); …} } }

Test cases to be rerun: Test cases in T that execute the call node with ref’s dynamic type being B or C

slide-22
SLIDE 22

FSE 04

Outline

  • Background
  • Technique
  • Empirical Evaluation
  • Conclusion
slide-23
SLIDE 23

FSE 04

Empirical Setup

  • Tool

DejaVOO

  • Subjects
  • Three empirical studies
  • 1. Effectiveness of Phase 1
  • 2. Precision gain of Phase 2
  • 3. Overall savings in retesting time

639 200 707 Test Cases 32 min 1,000 2,403 5 Jboss 74 min 167 824 5 Daikon 54 min 70 525 5 Jaba Retest Time KLOC Classes Versions Program

slide-24
SLIDE 24

FSE 04

RQ1: Effectiveness of Phase 1

P T (coverage info) P’ Trerun

TwoPhase

Class-Level Analysis Subset

  • f P

Subset

  • f P’

Stmt-Level Analysis

slide-25
SLIDE 25

FSE 04

RQ1: Effectiveness of Phase 1

Average: 37% Time < 14 seconds

slide-26
SLIDE 26

FSE 04

RQ3: Overall Savings in retesting Time

Time to rerun T time Analysis time Time to rerun Trerun Savings

slide-27
SLIDE 27

FSE 04

RQ3: Overall Savings in Retesting Time

RerunAll TwoPhase

slide-28
SLIDE 28

FSE 04

RQ3: Overall Savings in Retesting Time

RerunAll TwoPhase

Savings in Regression Testing Time: TwoPhase vs. RerunAll Jaba:19% Daikon:36% Jboss: 63%

slide-29
SLIDE 29

FSE 04

Outline

  • Background
  • Technique
  • Empirical Evaluation
  • Conclusion
slide-30
SLIDE 30

FSE 04

Contribution

  • Extended our existing regression-testing technique
  • Considerable increase in efficiency
  • Same precision
  • First RTS technique that, at the same time:
  • is safe
  • combines coarse and fine-grained analysis
  • handles the features of the Java language
  • Tool (DejaVOO) that implements the technique
  • Empirical results showing the effectiveness of the

technique

slide-31
SLIDE 31

FSE 04

Current and Future Work

  • Further investigate cost-effectiveness of

the technique

  • Continuous integration
  • Larger systems
  • Investigate test-suite augmentation
  • Based on changes (program differencing)
  • Based on field-data (impact analysis)
slide-32
SLIDE 32

FSE 04

Questions?