MaTRIX Maintenance-Oriented Test Requirements Identifier and - - PowerPoint PPT Presentation

▶
matrix maintenance oriented test requirements identifier
SMART_READER_LITE
LIVE PREVIEW

MaTRIX Maintenance-Oriented Test Requirements Identifier and - - PowerPoint PPT Presentation

Jan 25, 2023 317 likes •636 views

MaTRIX Maintenance-Oriented Test Requirements Identifier and Examiner Mary Jean Harrold Taweesup (Term) Apiwattanapong, Ral Santelices, Pavan Kumar Chittimalli, Alessandro Orso College of Computing, Georgia Institute of

slide-1
SLIDE 1

Mary Jean Harrold† Taweesup (Term) Apiwattanapong,† Raúl Santelices,† Pavan Kumar Chittimalli,‡ Alessandro Orso†

†College of Computing, Georgia Institute of Technology ‡Tata Research Development & Design Centre, TCS Limited

MaTRIX Maintenance-Oriented Test Requirements Identifier and Examiner

Supported by Tata Consultancy Services (TCS) Limited and by NSF

slide-2
SLIDE 2

2 TAIC PART August 2006

P’ Version

  • f P

Program P

T Regression Testing

slide-3
SLIDE 3

3 TAIC PART August 2006

P’ Version

  • f P

Program P

T T-T’ T’ T’ Regression Testing T’’ T’

slide-4
SLIDE 4

4 TAIC PART August 2006

P’ Version

  • f P

Program P

T T-T’ T’ T’ Regression Testing How well do T, T’,T,’’

  • r any test suites

exercise P’ with respect to changes? T’’ T’ How well do T, T’,T,’’

  • r any test suites

exercise P’ with respect to changes? Is there suitable guidance for creating new test cases that target the modified behavior?

slide-5
SLIDE 5

5 TAIC PART August 2006

public class E { void simple (int i) { s1 int x = i; s2 if (x > 5){ s3 x = (5/(x-5)); } s4 x = x - 1; s5 if (x == 0){ s6 print(x); } else { s7 print(10/x); } } … }

s1 s2 s3 s4 s5 s6 s7 x n F T T F

Motivating Example

c: if (x >= 5){

slide-6
SLIDE 6

6 TAIC PART August 2006

public class E { void simple (int i) { s1 int x = i; s2 if (x > 5){ s3 x = (5/(x-5)); } s4 x = x - 1; s5 if (x == 0){ s6 print(x); } else { s7 print(10/x); } } … }

s1 s2 s3 s4 s5 s6 s7 x n

c: if (x >= 5){

F T T F

Motivating Example

change

slide-7
SLIDE 7

7 TAIC PART August 2006

public class E { void simple (int i) { s1 int x = i; s2 if (x > 5){ s3 x = (5/(x-5)); } s4 x = x - 1; s5 if (x == 0){ s6 print(x); } else { s7 print(10/x); } } … }

s1 s2 s3 s4 s5 s6 s7 x n

c: if (x >= 5){

F T T F

Motivating Example

branches

slide-8
SLIDE 8

8 TAIC PART August 2006

public class E { void simple (int i) { s1 int x = i; s2 if (x > 5){ s3 x = (5/(x-5)); } s4 x = x - 1; s5 if (x == 0){ s6 print(x); } else { s7 print(10/x); } } … } du-pairs (s1,s2,x) (s1,s3,x) (s1,s4,x) (s3,s4,x) (s4,s5,x) (s4,s6,x) (s4,s7,x)

s1 s2 s3 s4 s5 s6 s7 x n

c: if (x >= 5){

F T T F

Motivating Example

du-pairs

slide-9
SLIDE 9

9 TAIC PART August 2006

public class E { void simple (int i) { s1 int x = i; s2 if (x > 5){ s3 x = (5/(x-5)); } s4 x = x - 1; s5 if (x == 0){ s6 print(x); } else { s7 print(10/x); } } … }

s1 s2 s3 s4 s5 s6 s7 x n

i=6 i=1

F T T F

Motivating Example

c: if (x >= 5){

slide-10
SLIDE 10

10 TAIC PART August 2006

public class E { void simple (int i) { s1 int x = i; s2 if (x > 5){ s3 x = (5/(x-5)); } s4 x = x - 1; s5 if (x == 0){ s6 print(x); } else { s7 print(10/x); } } … }

s1 s2 s3 s4 s5 s6 s7 x n

c: if (x >= 5){

F T T F

Motivating Example

change

i=6 i=1

slide-11
SLIDE 11

11 TAIC PART August 2006

public class E { void simple (int i) { s1 int x = i; s2 if (x > 5){ s3 x = (5/(x-5)); } s4 x = x - 1; s5 if (x == 0){ s6 print(x); } else { s7 print(10/x); } } … }

s1 s2 s3 s4 s5 s6 s7 x n

c: if (x >= 5){

F T T F

Motivating Example

branches

i=6 i=1

slide-12
SLIDE 12

12 TAIC PART August 2006

public class E { void simple (int i) { s1 int x = i; s2 if (x > 5){ s3 x = (5/(x-5)); } s4 x = x - 1; s5 if (x == 0){ s6 print(x); } else { s7 print(10/x); } } … } du-pairs (s1,s2,x) (s1,s3,x) (s1,s4,x) (s3,s4,x) (s4,s5,x) (s4,s6,x) (s4,s7,x)

s1 s2 s3 s4 s5 s6 s7 x n

c: if (x >= 5){

F T T F

Motivating Example

du-pairs

i=6 i=1 i=6

X X X X X

i=1

X X X X

slide-13
SLIDE 13

13 TAIC PART August 2006

public class E { void simple (int i) { s1 int x = i; s2 if (x > 5){ s3 x = (5/(x-5)); } s4 x = x - 1; s5 if (x == 0){ s6 print(x); } else { s7 print(10/x); } } … }

s1 s2 s3 s4 s5 s6 s7 x n

c: if (x >= 5){

F T T F

Motivating Example

i=6 i=1

Tests satisfy test requirements for criteria but don’t reveal fault in s3

slide-14
SLIDE 14

14 TAIC PART August 2006

public class E { void simple (int i) { s1 int x = i; s2 if (x > 5){ s3 x = (5/(x-5)); } s4 x = x - 1; s5 if (x == 0){ s6 print(x); } else { s7 print(10/x); } } … }

s1 s2 s3 s4 s5 s6 s7 x n

c: if (x >= 5){

F T T F

Motivating Example

i=6 i=1

Tests satisfy test requirements for criteria but don’t reveal fault in s3

Criteria require

  • Execution of the

change and entities affected by change

Criteria require

  • Execution of the

change and entities affected by change

But don’t require

  • Infection of the state

after change

  • Propagation of state

to output where it can be observed

slide-15
SLIDE 15

15 TAIC PART August 2006

s1 s2 s3 s4 s5 s6 s7 x n F T T F

i=6 i=1

Criteria require

  • Execution of the

change and entities affected by change

Criteria require

  • Execution of the

change and entities affected by change

But don’t require

  • Infection of the state

after change

  • Propagation of state

to output where it can be observed

Computation of Testing Requirements

Our technique adds these requirements to the criteria {

slide-16
SLIDE 16

16 TAIC PART August 2006

public class E { void simple (int i) { s1 int x = i; s2 if (x > 5){ s3 x = (5/(x-5)); } s4 x = x - 1; s5 if (x == 0){ s6 print(x); } else { s7 print(10/x); } } … } if (x >= 5){

SS(x) PC (i0>5) 5/(i0-5) true i0 SS’(x) PC’ ” ” (i0>5) 5/(i0-5)-1 (i0<=5) i0-1

  • r

(i0<=5)∧

…(i0!=0)

i0-1 (i0>5) ∧ ..(i0!=0) 5/(i0-5)-1 ”

”

(i0==0)

  • r

(i0>=5) 5/(i0-5)-1 (i0<5) i0-1 (i0<5) ∧

…(i0!=0)

i0-1 (i0>=5)∧ ..(i0!=0) 5/(i0-5)-1 (i0>=5) 5/(i0-5) ” ” ” ” (i0==0) true i0

  • r
  • r

Computation of Testing Requirements

PC—path condition SS—symbolic state

slide-17
SLIDE 17

17 TAIC PART August 2006

public class E { void simple (int i) { s1 int x = i; s2 if (x > 5){ s3 x = (5/(x-5)); } s4 x = x - 1; s5 if (x == 0){ s6 print(x); } else { s7 print(10/x); } } … } if (x >= 5){

SS(x) PC (i0>5) 5/(i0-5) true i0 SS’(x) PC’ ” ” (i0>5) 5/(i0-5)-1 (i0<=5) i0-1

  • r

(i0<=5)∧

…(i0!=0)

i0-1 (i0>5) ∧ ..(i0!=0) 5/(i0-5)-1 ”

”

(i0==0)

  • r

(i0>=5) 5/(i0-5)-1 (i0<5) i0-1 (i0<5) ∧

…(i0!=0)

i0-1 (i0>=5)∧ ..(i0!=0) 5/(i0-5)-1 (i0>=5) 5/(i0-5) ” ” ” ” (i0==0) true i0

  • r
  • r

Computation of Testing Requirements

PC—path condition SS—symbolic state

Conditions for propagation of infected states:

  • 1. The execution in P’ reaches si’ and the

execution in P does not reach si; or

  • 2. The execution in P’ reaches si’ and the

execution in P reaches si; however, si’ and si have different symbolic states.

slide-18
SLIDE 18

18 TAIC PART August 2006

public class E { void simple (int i) { s1 int x = i; s2 if (x > 5){ s3 x = (5/(x-5)); } s4 x = x - 1; s5 if (x == 0){ s6 print(x); } else { s7 print(10/x); } } … } if (x >= 5){

SS(x) PC (i0>5) 5/(i0-5) true i0 SS’(x) PC’ ” ” (i0>5) 5/(i0-5)-1 (i0<=5) i0-1

  • r

(i0<=5)∧

…(i0!=0)

i0-1 (i0>5) ∧ ..(i0!=0) 5/(i0-5)-1 ”

”

(i0==0)

  • r

(i0>=5) 5/(i0-5)-1 (i0<5) i0-1 (i0<5) ∧

…(i0!=0)

i0-1 (i0>=5)∧ ..(i0!=0) 5/(i0-5)-1 (i0>=5) 5/(i0-5) ” ” ” ” (i0==0) true i0

  • r
  • r

Computation of Testing Requirements

But (as we discussed yesterday)

  • symbolic execution is expensive
  • won’t scale to large programs
  • can’t be applied for entire paths
  • etc.

PC—path condition SS—symbolic state

But (as we discussed yesterday)

  • symbolic execution is expensive
  • won’t scale to large programs
  • can’t be applied for entire paths
  • etc.

Our technique has two ways to improve efficiency

slide-19
SLIDE 19

19 TAIC PART August 2006

public class E { void simple (int i) { s1 int x = i; s2 if (x > 5){ s3 x = (5/(x-5)); } s4 x = x - 1; s5 if (x == 0){ s6 print(x); } else { s7 print(10/x); } } … }

SS(x) PC SS’(x) PC’

Computation of Testing Requirements

if (x >= 5){

1. Perform partial symbolic execution (PSE) beginning immediately before the change

  • computes conditions in terms of variables

immediately before change

  • avoids symbolic execution from beginning
  • f program to change
slide-20
SLIDE 20

20 TAIC PART August 2006

public class E { void simple (int i) { s1 int x = i; s2 if (x > 5){ s3 x = (5/(x-5)); } s4 x = x - 1; s5 if (x == 0){ s6 print(x); } else { s7 print(10/x); } } … }

SS(x) PC true x0

  • SS’(x)

PC’ true x0

  • (x0>5)

5/(x0-5) (x0>=5) 5/(x0-5)

Computation of Testing Requirements

if (x >= 5){

slide-21
SLIDE 21

21 TAIC PART August 2006

public class E { void simple (int i) { s1 int x = i; s2 if (x > 5){ s3 x = (5/(x-5)); } s4 x = x - 1; s5 if (x == 0){ s6 print(x); } else { s7 print(10/x); } } … }

SS(x) PC true x0

  • SS’(x)

PC’ true x0

  • (x0>5)

5/(x0-5) (x0>=5) 5/(x0-5)

Computation of Testing Requirements

if (x >= 5){

1. Perform partial symbolic execution (PSE) beginning immediately before the change

  • computes conditions in terms of variables

immediately before change

  • avoids symbolic execution from beginning
  • f program to change

Don’t need to solve conditions—can still monitor for their satisfaction

slide-22
SLIDE 22

22 TAIC PART August 2006

public class E { void simple (int i) { s1 int x = i; s2 if (x > 5){ s3 x = (5/(x-5)); } s4 x = x - 1; s5 if (x == 0){ s6 print(x); } else { s7 print(10/x); } } … }

SS(x) PC true x0

  • SS’(x)

PC’ true x0

  • (x0>5)

5/(x0-5) (x0>=5) 5/(x0-5)

Computation of Testing Requirements

if (x >= 5){

2. Perform PSE for some specified distance (user selected) instead of to output statements

  • computes conditions on states at

intermediate points (i.e., distances)

  • bounds depth, avoids symbolic execution to
  • utputs
slide-23
SLIDE 23

23 TAIC PART August 2006

public class E { void simple (int i) { s1 int x = i; s2 if (x > 5){ s3 x = (5/(x-5)); } s4 x = x - 1; s5 if (x == 0){ s6 print(x); } else { s7 print(10/x); } } … }

SS(x) PC true x0

  • SS’(x)

PC’ true x0

  • (x0>5)

5/(x0-5) (x0>=5) 5/(x0-5)

Computation of Testing Requirements

if (x >= 5){ Distance 0—after change Distance 1—after 1 dependence Distance 2—after 2 dependences Distance 3—after 3 dependences Distance 3—after 3 dependences Distance 3—after 3 dependences And so on until output

slide-24
SLIDE 24

24 TAIC PART August 2006

public class E { void simple (int i) { s1 int x = i; s2 if (x > 5){ s3 x = (5/(x-5)); } s4 x = x - 1; s5 if (x == 0){ s6 print(x); } else { s7 print(10/x); } } … }

SS(x) PC true x0

  • SS’(x)

PC’ true x0

  • (x0>5)

5/(x0-5) (x0>=5) 5/(x0-5)

Computation of Testing Requirements

if (x >= 5){

2. Perform PSE for some specified distance (user selected) instead of to output statements

  • computes conditions on states at

intermediate points (i.e., distances)

  • bounds depth, avoids symbolic execution to
  • utputs

Greater distances improve confidence in propagation to output

slide-25
SLIDE 25

25 TAIC PART August 2006

public class E { void simple (int i) { s1 int x = i; s2 if (x > 5){ s3 x = (5/(x-5)); } s4 x = x - 1; s5 if (x == 0){ s6 print(x); } else { s7 print(10/x); } } … }

SS(x) PC true x0

  • SS’(x)

PC’ true x0

  • (x0>5)

5/(x0-5) (x0>=5) 5/(x0-5)

Computation of Testing Requirements

if (x >= 5){

Distance 1 PC’(s3) and (not PC(s3))  (x0 >= 5) and (not (x0 > 5))  (x0 >= 5) and (x0 <= 5)  (x0 == 5)

slide-26
SLIDE 26

26 TAIC PART August 2006

public class E { void simple (int i) { s1 int x = i; s2 if (x > 5){ s3 x = (5/(x-5)); } s4 x = x - 1; s5 if (x == 0){ s6 print(x); } else { s7 print(10/x); } } … }

Use of Testing Requirements

if (x >= 5){

1. Instrument program so that probe checks for condition before change (e.g., after s1) 2. Assist developer in satisfying criterion and improving confidence in testing 3. Generate test if condition can be satisfied (future work)

slide-27
SLIDE 27

27 TAIC PART August 2006

Goal:

To compare the effectiveness of our changed-based criteria with statement and all-uses coverage criteria (based on changes)

Implementation: uses differencing, Java

Pathfinder, instrumenter, data-/control-dependence analysis, etc.

Subjects: Tcas (4 versions) and Schedule (3 versions)

(each version has one fault)

Method:

  • Randomly generate 50 test suites per criterion.
  • Record the number of test suites that produce different
  • utputs.

Empirical Study: Setup

slide-28
SLIDE 28

28 TAIC PART August 2006

20 40 60 80 100 v1 v2 v3 v4 v1 v2 v3

Tcas Schedule

stmt all-uses change-based d0 d1 d2 d0 d1 d2

Percentage of test suites revealing different behaviors

  • ver 50 test suites that satisfy each criterion.

Effectiveness Study: Results

slide-29
SLIDE 29

29 TAIC PART August 2006

Conclusions

New technique

  • Identifies (creates), examines (monitors) test

requirements related to change(s)

  • Uses symbolic execution but gains efficiency
  • partial symbolic execution so avoids performing

symbolic execution from beginning of program

  • partial symbolic execution to specified distances

from change so bounds depth of symbolic execution

  • Size of symbolic execution tree related to

change instead of size of program

  • Empirical evaluation show promise of

approach

slide-30
SLIDE 30

30 TAIC PART August 2006

Current and Future Work

Current

  • Completing infrastructure
  • Performing experiments—additional subjects,

more complex changes, scalability, limitations Future

  • Expand technique to handle multiple

changes, changes involving multiple statements

  • Use conditions for automatic test-case

generation

slide-31
SLIDE 31

31 TAIC PART August 2006

Questions?