Automated Test Generation By Avoiding Redundant Tests Tao Xie - - PowerPoint PPT Presentation

▶
automated test generation by avoiding redundant tests
SMART_READER_LITE
LIVE PREVIEW

Automated Test Generation By Avoiding Redundant Tests Tao Xie - - PowerPoint PPT Presentation

Nov 03, 2023 32 likes •627 views

Automated Test Generation By Avoiding Redundant Tests Tao Xie Joint work with Darko Marinov (UIUC) and David Notkin Dept. of Computer Science & Engineering University of Washington 5 Oct. 2004, Microsoft Research 1 Motivation Tool

slide-1
SLIDE 1

1

Automated Test Generation By Avoiding Redundant Tests

  • Dept. of Computer Science & Engineering

University of Washington

5 Oct. 2004, Microsoft Research

Tao Xie

Joint work with Darko Marinov (UIUC) and David Notkin

slide-2
SLIDE 2

2

Motivation

  • Tool generated test cases
  • Many test cases
  • Important to reduce by eliminating “redundant” test cases
  • Need automation
  • Common approach
  • Identify “similar” test cases and eliminate
  • With minimal reduction of “quality” of test suite
  • Object-oriented programs
  • Test case is a sequence of method calls on an object
  • Note: Unit tests only
slide-3
SLIDE 3

3

Overview

  • Motivation
  • Rostra framework for detecting redundant tests

[ASE 04]

  • Test generation by avoiding redundant tests
  • Conclusions
slide-4
SLIDE 4

4

Example Code

public class IntStack { private int[] store; private int size; public IntStack() { … } public void push(int value) { … } public int pop() { … } public boolean isEmpty() { … } public boolean equals(Object o) { … } } [Henkel&Diwan 03]

slide-5
SLIDE 5

5

Example Generated Tests

Test 1 (T1): IntStack s1 = new IntStack(); s1.isEmpty(); s1.push(3); s1.push(2); s1.pop(); s1.push(5); Test 2 (T2): IntStack s2 = new IntStack(); s2.push(3); s2.push(5); Test 3 (T3): IntStack s3 = new IntStack(); s3.push(3); s3.push(2); s3.pop();

slide-6
SLIDE 6

6

Same inputs ⇒ Same behavior

Method Execution

  • bject state @entry
  • bject state @exit

Method arguments Method return Input = + Output = + Testing a method with the same inputs is unnecessary Assumption: deterministic method

How to represent object states?

slide-7
SLIDE 7

7

Redundant Test Cases Defined

  • Equivalent method executions
  • the same method names, signatures, and input

(equivalent object states @entry and arguments)

  • Redundant test case:
  • A test case is redundant for a test suite if the test suite

has exercised method executions equivalent to all method executions exercised by the test case

slide-8
SLIDE 8

8

Comparison with SpecExplorer

  • Code vs. models
  • SpecExplorer generates tests from models, so one may

want a different definition of redundant tests

  • One test is redundant w.r.t. another in the model but

might not be redundant for the code

  • Binary predicates vs. abstraction functions
  • s1 == s2 m(s1, s2) [boolean-returning method]
  • s1 == s2 a(s1) == a(s2) [abstraction function]
slide-9
SLIDE 9

9

Five State-Representation Techniques

  • Method-sequence representations
  • WholeSeq
  • The entire sequence
  • ModifyingSeq
  • Ignore methods that don’t modify the state
  • Concrete-state representations
  • WholeState
  • The full concrete state
  • MonitorEquals
  • Relevant parts of the concrete state
  • PairwiseEquals
  • equals() method used to compare pairs of states
slide-10
SLIDE 10

10

WholeSeq Representation

Notation: methodName(entryState, methodArgs).state [Henkel&Diwan 03] Test 1 (T1): IntStack s1 = new IntStack(); s1.isEmpty(); s1.push(3); s1.push(2); s1.pop(); s1.push(5); Test 3 (T3): IntStack s3 = new IntStack(); s3.push(3); s3.push(2); s3.pop();

Method sequences that create objects

slide-11
SLIDE 11

11

<init>( ).state

WholeSeq Representation

Notation: methodName(entryState, methodArgs).state [Henkel&Diwan 03] Test 1 (T1): IntStack s1 = new IntStack(); s1.isEmpty(); s1.push(3); s1.push(2); s1.pop(); s1.push(5); Test 3 (T3): IntStack s3 = new IntStack(); s3.push(3); s3.push(2); s3.pop();

Method sequences that create objects

slide-12
SLIDE 12

12

<init>( ).state

WholeSeq Representation

Notation: methodName(entryState, methodArgs).state [Henkel&Diwan 03] Test 1 (T1): IntStack s1 = new IntStack(); s1.isEmpty(); s1.push(3); s1.push(2); s1.pop(); s1.push(5); Test 3 (T3): IntStack s3 = new IntStack(); s3.push(3); s3.push(2); s3.pop();

Method sequences that create objects

isEmpty( ).state

slide-13
SLIDE 13

13

<init>( ).state

WholeSeq Representation

Notation: methodName(entryState, methodArgs).state [Henkel&Diwan 03] Test 1 (T1): IntStack s1 = new IntStack(); s1.isEmpty(); s1.push(3); s1.push(2); s1.pop(); s1.push(5); Test 3 (T3): IntStack s3 = new IntStack(); s3.push(3); s3.push(2); s3.pop();

Method sequences that create objects

isEmpty( ).state push( , 3).state s1.push 2

slide-14
SLIDE 14

14

<init>( ).state

WholeSeq Representation

Notation: methodName(entryState, methodArgs).state [Henkel&Diwan 03] Test 1 (T1): IntStack s1 = new IntStack(); s1.isEmpty(); s1.push(3); s1.push(2); s1.pop(); s1.push(5); Test 3 (T3): IntStack s3 = new IntStack(); s3.push(3); s3.push(2); s3.pop();

Method sequences that create objects

isEmpty( ).state push( , 3).state s1.push 2 s3.push push(<init>( ).state, 3).state 2

slide-15
SLIDE 15

15

s1.push s3.push push(<init>( ).state, 3).state push(isEmpty(<init>( ).state).state, 3).state

ModifyingSeq Representation

Test 1 (T1): IntStack s1 = new IntStack(); s1.isEmpty(); s1.push(3); s1.push(2); s1.pop(); s1.push(5); Test 3 (T3): IntStack s3 = new IntStack(); s3.push(3); s3.push(2); s3.pop();

State-modifying method sequences that create objects

2 2

slide-16
SLIDE 16

16

WholeState Representation

s1.push s2.push

store.length = 3 store[0] = 3 store[1] = 2 store[2] = 0 size = 1

Test 1 (T1): IntStack s1 = new IntStack(); s1.isEmpty(); s1.push(3); s1.push(2); s1.pop(); s1.push(5); Test 2 (T2): IntStack s2 = new IntStack(); s2.push(3); s2.push(5);

store.length = 3 store[0] = 3 store[1] = 0 store[2] = 0 size = 1 5 5

The entire concrete state reachable from the object

Comparison by isomorphism

slide-17
SLIDE 17

17

MonitorEquals Representation

s1.push s2.push store.length = 3 store[0] = 3 store[1] = 2 store[2] = 0 size = 1 Test 1 (T1): IntStack s1 = new IntStack(); s1.isEmpty(); s1.push(3); s1.push(2); s1.pop(); s1.push(5); Test 2 (T2): IntStack s2 = new IntStack(); s2.push(3); s2.push(5); store.length = 3 store[0] = 3 store[1] = 0 store[2] = 0 size = 1

5 5

The relevant part of the concrete state defined by equals (invoking

  • bj.equals(obj) and monitor field accesses)

Comparison by isomorphism

slide-18
SLIDE 18

18

PairwiseEquals Representation

s1.push s2.push Test 1 (T1): IntStack s1 = new IntStack(); s1.isEmpty(); s1.push(3); s1.push(2); s1.pop(); s1.push(5); Test 2 (T2): IntStack s2 = new IntStack(); s2.push(3); s2.push(5);

5 5

The results of equals invoked to compare pairs of states

s1.equals(s2) == true

slide-19
SLIDE 19

19

MonitorEquals vs. PairwiseEquals

  • MonitorEquals monitors field accesses during

execution of the equals() method and compares the monitored parts

  • PairwiseEquals relies only on the output of the

equals() method

  • Example of sets
slide-20
SLIDE 20

20

Detected Redundant Tests

T3, T2 MonitorEquals T3 WholeState T3, T2 PairwiseEquals T3 ModifyingSeq WholeSeq

detected redundant tests w.r.t. T1

technique

Test 1 (T1): IntStack s1 = new IntStack(); s1.isEmpty(); s1.push(3); s1.push(2); s1.pop(); s1.push(5); Test 2 (T2): IntStack s2 = new IntStack(); s2.push(3); s2.push(5); Test 3 (T3): IntStack s3 = new IntStack(); s3.push(3); s3.push(2); s3.pop();

slide-21
SLIDE 21

21

Experiment: Evaluated Test Generation Tools

  • ParaSoft Jtest 4.5
  • A commercial Java testing tool
  • Generates tests with method-call lengths up to three
  • JCrasher 0.2.7
  • An academic robustness testing tool
  • Generates tests with method-call lengths of one
slide-22
SLIDE 22

22

Questions to Be Answered

  • How much do we benefit after applying

Rostra on tests generated by Jtest and JCrasher?

  • Does redundant-test removal decrease test

suite quality?

slide-23
SLIDE 23

23

Experimental Subjects

1000 931 949 25 61 TreeMap 86 3028 398 32 38 LinkedList 47 5186 597 19 27 HeapMap 150 3743 468 14 24 FibonacciHeap 64 779 166 7 10 DisjSet 438 6205 535 17 22 BinomialHeap 56 277 246 8 13 BinSearchTree 135 519 34 7 7 BankAccount 31 470 70 8 9 ShoppingCart 14 1423 106 11 11 UBStack 6 94 44 5 5 IntStack JCrasher tests Jtest tests ncnb loc public methods methods class

slide-24
SLIDE 24

24

Assumptions About Subjects

  • Method-sequence representations assume that each

method does not modify argument state

  • MonitorEquals and PairwiseEquals representations

assume a user-defined equals()

slide-25
SLIDE 25

25

Quality of Original Test Suites

30% 53% Avg mutant killing ratio (600 mutants) 52% 77% Avg Branch cov 2 4 Avg num uncaught exceptions

JCrasher-generated tests Jtest-generated tests

slide-26
SLIDE 26

26

Redundancy among Jtest-generated Tests

  • The last three techniques detect around 90% redundant tests
  • Detected redundancy in increasing order for five techniques

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% IntStack UBStack ShoppingCart BankAccount BinSearchTree BinomialHeap DisjSet FibonacciHeap HashMap LinkedList TreeMap WholeSeq ModifyingSeq WholeState MonitorEquals PairwiseEquals

slide-27
SLIDE 27

27

Redundancy among JCrasher-generated Tests

  • The last three techniques detect over 50% on half subjects
  • JCrasher generates fewer tests and shorter tests

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 IntStack UBStack ShoppingCart BankAccount BinSearchTree BinomialHeap DisjSet FibonacciHeap HashMap LinkedList TreeMap WholeSeq ModifyingSeq WholeState MonitorEquals PairwiseEquals

slide-28
SLIDE 28

28

Quality of Minimized Test Suites

  • All five techniques on JCrasher preserve all

measurements

  • The first three techniques on Jtest preserve all

measurements.

  • Two equals techniques on Jtest decrease (with
  • nly small loss in 2 programs)
  • in branch cov %
  • in mutant killing %
slide-29
SLIDE 29

29

Comparison of Five Techniques

  • Time and space taken to find redundant tests
  • from a couple of seconds to several minutes across

subjects

  • in roughly increasing order except for pairwiseEquals

(being the least expensive)

  • The number of redundant tests found
  • in increasing order

(WholeSeq, ModifyingSeq, WholeState, MonitorEquals, PairwiseEquals)

slide-30
SLIDE 30

30

Redundancy Prediction

  • ModifySeq tends to be better than WholeSeq
  • when more state-preserving methods
  • WholeState tends to be better than ModifySeq
  • when more nonprogressive state-modifying methods
  • MonitorEquals or PairwiseEquals tend to be better

than WholeState

  • when more irrelevant object fields and these fields are

assigned different values during method sequence calls

slide-31
SLIDE 31

31

Evolution of ParaSoft Jtest

  • Version 4.5 (released in March 2002) allows

method-call lengths (1 ― 3) [studied in this work, first published in a tech report in Jan 2004]

  • Version 5.0 (released in Feb 2004) allows

method-call length of only 1

  • Recently ParaSoft notified us that

Version 6.0 (internal version, not yet released) has addressed the test redundancy issue identified by us and added back the option to generate long call sequence

slide-32
SLIDE 32

32

Overview

  • Motivation
  • Rostra framework for detecting redundant tests

[ASE 04]

  • Test generation by avoiding redundant tests
  • Conclusions
slide-33
SLIDE 33

33

Test Generation Algorithm

1. iterationNum = 1 2. put initial object states into a frontier set S 3. create an empty new frontier set S’ 4. for each state s in S { for each argument list a in A { invoke s on a (a new test) if method-exit state s’ is a new state put s’ into S’ } } 5. iterationNum++ 6. replace S with S’ 7. if (S is not empty && iterationNum <= MAXITER) goto 3 Inputs: some argument lists A for methods under test Breadth-first exploration of (object) state space

slide-34
SLIDE 34

34

Example

Inputs: push(3), push(2), push(5), pop(), isEmpty()

after new IntStack()

slide-35
SLIDE 35

35

Example

Inputs: push(3), push(2), push(5), pop(), isEmpty()

after new IntStack()

push(3) push(2) push(5) pop() isEmpty()

slide-36
SLIDE 36

36

Example

Inputs: push(3), push(2), push(5), pop(), isEmpty()

after new IntStack()

push(3) push(2) push(5) pop() isEmpty() push(3) push(2) push(5) pop() isEmpty()

…

slide-37
SLIDE 37

37

Example

Inputs: push(3), push(2), push(5), pop(), isEmpty()

after new IntStack()

push(3) push(2) push(5) pop() isEmpty() push(3) push(2) push(5) pop() isEmpty()

…

After n iterations, state space O(3^n) If the stack is a unique int stack, we need 10 different arguments to reach size of 10, state space 9864100!

slide-38
SLIDE 38

38

Symbolic Execution to Rescue

  • Execute a method on symbolic argument values
  • Inputs: push(SymbolicInt svalue), pop(), isEmpty()
  • Use code instrumentation for symbolic execution

[Khurshid et al. TACAS 03]

  • Explore paths within the method execution
  • Use reexecution to implement backtracking [not store

states inside a method execution]

  • Build a path condition for each path
  • Check satisfiability of path condition [CVC Lite, Omega]
slide-39
SLIDE 39

39

Symbolic State Representation

  • Symbolic state: < state, path condition>

(For now, assume the stack is a unique int stack)

SymbolicInt elem1 = new SymbolicInt(); SymbolicInt elem2 = new SymbolicInt(); IntStack s1 = new IntStack(); s1.push(elem1);

store.length = 3 store[0] = elem1 size = 1 State 1

slide-40
SLIDE 40

40

Symbolic State Representation

SymbolicInt elem1 = new SymbolicInt(); SymbolicInt elem2 = new SymbolicInt(); IntStack s1 = new IntStack(); s1.push(elem1); s1.push(elem2);

store.length = 3 store[0] = elem1 size = 1 State 1 store.length = 3 store[0] = elem1 store[1] = elem2 size = 2 elem1 != elem2 elem1 == elem2 store.length = 3 store[0] = elem1 size = 1 State 3 State 2

  • Symbolic state: < state, path condition>

(For now, assume the stack is a unique int stack)

slide-41
SLIDE 41

41

Equivalent Symbolic States?

  • Symbolic state: < state, path condition>
  • Equivalence: equiv state part + equiv pc

SymbolicInt elem1 = new SymbolicInt(); SymbolicInt elem2 = new SymbolicInt(); IntStack s1 = new IntStack(); s1.push(elem1); s1.push(elem2);

store.length = 3 store[0] = elem1 size = 1 State 1 store.length = 3 store[0] = elem1 store[1] = elem2 size = 2 elem1 != elem2 elem1 == elem2 store.length = 3 store[0] = elem1 size = 1 State 3 State 2

slide-42
SLIDE 42

42

Removing Irrelevant Constraints

  • Symbolic state: < state, path condition>
  • Equivalence: equiv state part + equiv pc

SymbolicInt elem1 = new SymbolicInt(); SymbolicInt elem2 = new SymbolicInt(); IntStack s1 = new IntStack(); s1.push(elem1); s1.push(elem2);

store.length = 3 store[0] = elem1 size = 1 State 1 store.length = 3 store[0] = elem1 store[1] = elem2 size = 2 elem1 != elem2 elem1 == elem2 store.length = 3 store[0] = elem1 size = 1 State 3 State 2

slide-43
SLIDE 43

43

Comparison of Path Conditions

  • Currently based on implications

S1: < s1, pc1> an old state S2: < s2, pc2> a state under consideration s1 is equiv to s2

  • if pc2 => pc1, S2 isn’t considered a new state
  • e.g. (x > deleted && deleted > y) => (x > y)
  • otherwise, S2 is a new state
  • A pure method modifies path conditions but doesn’t

produce new states

  • Merge symbolic states with equiv state parts
  • OR to connect their path conditions
slide-44
SLIDE 44

44

Heuristics

  • To make the generation faster, but have less

guarantees than testing all sequences/paths

  • symbolically explore some sequences
  • e.g. ignoring PC’s in state comparison
  • e.g. randomly choose some methods/args to invoke
  • symbolically explore some paths
  • e.g. favor paths that potentially lead to uncovered branches
  • put some parameters concrete
  • e.g. TreeMap.put(SymbolicInt key, Object value)
slide-45
SLIDE 45

45

Example – Concrete Inputs

Inputs: push(3), push(2), push(5), pop(), isEmpty()

after new IntStack()

push(3) push(2) push(5) pop() isEmpty() push(3) push(2) push(5) pop() isEmpty()

…

After n iterations, state space O(3^n) If the stack is a unique int stack, we need 10 different arguments to reach size of 10, state space 9864100! O(10)! Sym Exec

slide-46
SLIDE 46

46

Example – Symbolic Inputs

Inputs: push(SymbolicInt svalue)

after new UIntStack()

push(elem1)

If the stack is a unique int stack, we need 10 different arguments to reach size of 10, state space 9864100!

push(elem2) elem1 != elem2 push(elem3) elem1 != elem2 && elem3 != elem1 && elem3 != elem2

…

O(10)! Sym Exec

slide-47
SLIDE 47

47

From Symbolic to Concrete Inputs

Inputs: push(SymbolicInt svalue)

after new UIntStack()

push(elem1) push(elem2) elem1 != elem2 push(elem3) elem1 != elem2 && elem3 != elem1 && elem3 != elem2

…

  • Derive concrete arguments using a constraint solver [POOC]
  • Export concrete tests to executable JUnit classes

Integer push_0_1 = new Integer(-999998); Integer push_0_3 = new Integer(-999999); Integer push_0_2 = new Integer(-1000000); UIntStack u1 = new UIntStack(); u1.push(push_0_1); u1.push(push_0_2); u1.push(push_0_3); Integer push_0_1 = new Integer(-1000000); Integer push_0_2 = new Integer(-999999); UIntStack u1 = new UIntStack(); u1.push(push_0_1); u1.push(push_0_2);

slide-48
SLIDE 48

48

Preliminary Experimental Results

271.15

  • 2251
  • --after 6

8 2.41

  • 11
  • 10

symexec concrete symexec concrete 50.62 66.58 531 537 7 11.08 18.16 139 185 6 3.40 4.66 41 72 5 1.10 1.54 15 25 4 TreeMap (put, remove) 1.71

  • 10
  • 9

1.24

  • 9
  • 8

0.91

  • 8
  • --after 6

7 0.69 24.32 7 1957 6 UIntStack (push) Time (in seconds) State space (#nonequiv states) After iteration # Class

Pentium IV 2.8 GHz Sun JDK 1.4.2 with 512MB allocated mem

slide-49
SLIDE 49

49

Discussion

  • Automate code instrumentation for symbolic

execution or symbolic interpretation

  • Symbolic exec for non-primitive variables
  • SHE (Symbolic Heap Execution)
  • Build path conditions involving references
  • Test prioritization: explore how different strategies

for “on-the-fly” test generation lead to detecting faults?

slide-50
SLIDE 50

50

Related Work

  • State equivalence using observational equivalence

[Bernot et al. 91, Doong&Frankl 94, Henkel&Diwan 03]

  • Expensive because of number of sequences
  • State equivalence based on user-defined

abstraction functions [Grieskamp et al. 02]

  • Generalized symbolic execution [Khurshid et al. 03]

and Java Pathfinder test generation [Visser et al. 04]

  • Symbolic execution requiring both repOk()/precondition and

“conservative” repOk()/precondition

  • Concrete method sequence generation (slower than symbolic

Rostra)

slide-51
SLIDE 51

51

Conclusions

  • Redundant tests add cost without any benefit
  • Existing test generation tools can be potentially

improved (by incorporating Rostra framework)

  • Symbolic execution can further reduce the cost
  • The experimental results have shown
  • High redundancy among their generated tests
  • Removing them does not decrease test suite quality
  • Symbolic execution enables generation of longer

method sequences

slide-52
SLIDE 52

52

Questions?

slide-53
SLIDE 53

53

Applications

  • Assessment: compare the quality of different test

suites.

  • Selection: select a subset of automatically generated

tests to augment an existing test suite.

  • Minimization: minimize an automatically generated

test suite for correctness inspection and regression executions.

  • Generation: avoid generating and executing redundant

tests

slide-54
SLIDE 54

54

Syntactic Comparison

  • Syntactic comparison (based on of PC string)
  • Remove irrelevant constraints (e.g. elem1 == elem2)
  • Optimization during PC buildup

c1: a constraint in PC; c2: a constraint to be considered

  • if c1 => c2 , we don’t add c2 to PC (e.g. x>1 => x >=1)
  • if c2 => c1 , we remove c1 after we add c2 to PC
slide-55
SLIDE 55

55

Elapsed Real Time in Minimizing Jtest-Generated Tests (in secs)

50 100 150 200 250 300 I n t S t a c k U B S t a c k S h

  • p

p i n g C a r t B a n k A c c

  • u

n t B i n S e a r c h T r e e B i n

  • m

i a l H e a p D i s j S e t F i b

  • n

a c c i H e a p H a s h M a p L i n k e d L i s t T r e e M a p WholeSeq ModifyingSeq WholeState MonitorEquals PairwiseEquals

slide-56
SLIDE 56

56

Elapsed Real Time in Minimizing JCrasher-Generated Tests (in secs)

1 2 3 4 5 6 7 8 IntStack UBStack ShoppingCart BankAccount BinSearchTree BinomialHeap DisjSet FibonacciHeap HashMap LinkedList TreeMap WholeSeq ModifyingSeq WholeState MonitorEquals PairwiseEquals

slide-57
SLIDE 57

57

Optimization Heuristics

  • Without considering path conditions

SymbolicInt elem1 = new SymbolicInt(); SymbolicInt elem2 = new SymbolicInt(); IntStack s1 = new IntStack(); s1.push(elem1);

store.length = 3 store[0] = elem1 size = 1 State 1

slide-58
SLIDE 58

58

Optimization Heuristics

  • Without considering path conditions

store.length = 3 store[0] = elem1 store[1] = elem2 size = 2

SymbolicInt elem1 = new SymbolicInt(); SymbolicInt elem2 = new SymbolicInt(); IntStack s1 = new IntStack(); s1.push(elem1); s1.push(elem2);

elem1 != elem2 elem1 == elem2 store.length = 3 store[0] = elem1 size = 1 State 2 State 1

slide-59
SLIDE 59

59

Method-Sequence Representation with Symbolic Execution

  • Considering path conditions

SymbolicInt elem1 = new SymbolicInt(); SymbolicInt elem2 = new SymbolicInt(); IntStack s1 = new IntStack(); s1.push(elem1); s1.push(elem2);

push(<init>().state, elem1).state State 1 push( push(<init>().state, elem1).state, elem2).state elem1 != elem2 elem1 == elem2 State 3 State 2 push( push(<init>().state, elem1).state, elem2).state