Tool-Assisted Unit Test Selection Based on Operational Violations - - PowerPoint PPT Presentation

▶
tool assisted unit test selection based on operational
SMART_READER_LITE
LIVE PREVIEW

Tool-Assisted Unit Test Selection Based on Operational Violations - - PowerPoint PPT Presentation

Apr 03, 2024 472 likes •807 views

Tool-Assisted Unit Test Selection Based on Operational Violations Tao Xie David Notkin Department of Computer Science & Engineering University of Washington, Seattle, WA Oct. 8th, 2003 ASE 2003, Montreal, Canada 1 Synopsis Context:

slide-1
SLIDE 1

1

Tool-Assisted Unit Test Selection Based on Operational Violations

Tao Xie David Notkin

Department of Computer Science & Engineering University of Washington, Seattle, WA

  • Oct. 8th, 2003

ASE 2003, Montreal, Canada

slide-2
SLIDE 2

2

  • Goal:
  • From generated tests, select best candidates for manual

inspection to add oracles

Synopsis

  • Context: Automatic white-box test generation has many

benefits + Lots of tests generated for coverage and robustness

  • Problems:

– Oracles not generated for correctness checking

– Lots of tests generated impractical for inspection to add oracles

slide-3
SLIDE 3

3

Synopsis (cont.)

  • Solution: Use dynamic invariant detector to

generate properties (a.k.a operational abstractions)

  • bserved from existing test executions
  • Guide test selection for inspection
  • Guide better test generation

Benefits of specification-based testing can be obtained without the pain of writing the specifications!

slide-4
SLIDE 4

4

Outline

  • Motivation
  • Operational Violation Approach
  • Experiment
  • Related Work
  • Conclusion
slide-5
SLIDE 5

5

?

Program

Automatically generated test inputs

White-box gen

  • White-box test generation

+ Cover structural entities, e.g. statement, branch – Test oracle problem

Automatic Unit Test Generation

Jtest http://www.parasoft.com/

  • Black-box test generation

+ Guide test generation + Produce test oracles – Require a priori specs

Specs Oracles

Black-box gen

slide-6
SLIDE 6

6

Specification-Based Testing

  • Goal: generate test inputs and test oracles from

specifications

  • Tool: ParaSoft Jtest
  • Approach:
  • 1. Annotate Design by Contract (DbC) [Meyer 97]
  • Preconditions/Postconditions/Class invariants
  • 2. Generate test inputs that
  • Satisfy preconditions
  • 3. Check if test executions
  • Satisfy postconditions/invariants

Jtest

Up to range(1…3) method calls in a test

slide-7
SLIDE 7

7

Operational Abstraction Generation

[Ernst et al. 01]

  • Goal: determine properties true at runtime

(e.g. in the form of Design by Contract)

  • Tool: Daikon (dynamic invariant detector)
  • Approach
  • 1. Run test suites on a program
  • 2. Observe computed values
  • 3. Generalize

http://pag.lcs.mit.edu/daikon

slide-8
SLIDE 8

8

Automatic Unit Test Generation

  • White-box test generation

+ Cover structural entities, e.g. statement, branch – Test oracle problem

  • Black-box test generation

+ Guide test generation + Produce test oracles – Require a priori specs

Jtest

Test Selection for Inspection Operational Abstractions Based on

slide-9
SLIDE 9

9

Outline

  • Motivation
  • Operational Violation Approach
  • Experiment
  • Related Work
  • Conclusion
slide-10
SLIDE 10

10

Basic Technique

Insert as

DbC comments Annotated program

Run

Data trace

Run & Check

Violating tests Automatically generated test inputs Violated OA Selected tests

Select Detect invariants

All OA OA: Operational Abstractions The existing test suite (manual tests) Program

slide-11
SLIDE 11

11

Precondition Removal Technique

@Pre @Post @Inv The existing test suite

Run

Data trace

Detect invariants Insert as

DbC comments program

  • Overconstrained preconditions may leave

(important) legal inputs unexercised

Program Annotated @Pre

  • Solution: precondition removal technique
slide-12
SLIDE 12

12

Motivating Example [Stotts et al. 02]

public class uniqueBoundedStack { private int[] elems; private int numberOfElements; private int max; public uniqueBoundedStack() { numberOfElements = 0; max = 2; elems = new int[max]; } public int getNumberOfElements() { return numberOfElements; } …… };

A manual test suite (15 tests)

slide-13
SLIDE 13

13

Operational Violation Example

public int top(){ if (numberOfElements < 1) { System.out.println("Empty Stack"); return -1; } else { return elems[numberOfElements-1]; } }

  • Precondition Removal Technique

@pre { for (int i = 0 ; i <= this.elems.length-1; i++) $assert ((this.elems[i] >= 0)); }

@post: [($result == -1) (this.numberOfElements == 0)]

Daikon generates from manual test executions:

uniqueBoundedStack THIS = new uniqueBoundedStack (); THIS.push (-1); int RETVAL = THIS.top ();

Jtest generates a violating test input:

slide-14
SLIDE 14

14

Iterations

The existing test suite

Run

Data trace

Detect invariants Insert as DbC comments Run & Check

Violating tests Annotated program

Automatically generated test inputs

Violated OA

Select

OA Selected tests

  • Iterates until
  • No operational violations
  • User-specified max number of iteration
  • The existing tests augmented by selected tests are

run to generate operational abstractions

Program

slide-15
SLIDE 15

15

Outline

  • Motivation
  • Operational Violation Approach
  • Experiment
  • Related Work
  • Conclusion
slide-16
SLIDE 16

16

Subject Programs Studied

  • 12 programs from assignments and texts

(standard data structures)

  • Total 775 executable LOC in 127 methods
  • Accompanying manual test suites
  • ~94% branch coverage
slide-17
SLIDE 17

17

Questions to Be Answered

  • Is the number of automatically generated tests

large enough?

  • if yes, need test selection
  • Is the number of tests selected by our approach

small enough?

  • if yes, affordable inspection effort
slide-18
SLIDE 18

18

Questions to Be Answered (cont.)

  • Do the selected tests by our approach have a high

probability of exposing faults?

  • if yes, select a good subset of generated tests
  • How does our approach compare with structural

test selection approach?

  • Structural approach: select tests that exercise new

branch

slide-19
SLIDE 19

19

Measurements

  • The number of generated tests without
  • perational abstractions
  • The number of selected tests by our

approach/structural approach

  • The percentage of fault-revealing selected

tests by our approach/structural approach

  • Human inspection to determine
  • Also counting illegal inputs that exhibit abnormal

behavior, e.g. pop on empty stack leading to invalid

  • bject state
slide-20
SLIDE 20

20

Experiment Results

  • The number of generated tests without
  • perational abstraction
  • Range(24…227) Median(124)

[test containing up to 2 method calls]

  • Thousands [test containing up to 3 method calls]
  • Relatively large for inspection
  • Need test selection
slide-21
SLIDE 21

21

Experiment Results (cont.)

  • The number of selected tests
  • Our approach:
  • Range(0…25) Median(3)
  • Structural approach:
  • Range(0…5) Median(1)
  • Relatively small for inspection
  • Require affordable inspection effort
  • Our approach selects more tests than

structural approach

slide-22
SLIDE 22

22

Experiment Results (cont.)

  • The percentage of fault-revealing tests among

selected tests (median)

  • Our approach:
  • Iteration 1: 20% (Basic) 68% (Pre_Removal)
  • Iteration 2: 0% (Basic) 17% (Pre_Removal)
  • Structural approach: 0%
  • But increase confidence on the new exercised branches
  • Relatively high (our approach)
  • Select good subset of generated tests
  • Our approach complements structural approach
slide-23
SLIDE 23

23

Experiment Results (cont.)

  • Jtest’s running time on test generation and

execution dominates

  • Most programs ~5 mins
  • But 3 programs 10~20 mins

– Running Jtest several times within each iteration + Class- and method-centric + Automatic except for human inspection in the end

slide-24
SLIDE 24

24

Experiment Results (cont.)

  • Many fault-revealing tests not generated by

Jtest without operational abstractions

  • Operational abstractions guide the tool to

better generate tests

slide-25
SLIDE 25

25

Threats to Validity

  • Representative of true practice
  • Subject programs, faults, and tests
  • Instrumentation effects that bias the results
  • Faults on tools (integration scripts, Daikon,

Jtest)

slide-26
SLIDE 26

26

Outline

  • Motivation
  • Operational Violation Approach
  • Experiment
  • Related Work
  • Conclusion
slide-27
SLIDE 27

27

Related Work

  • Use of operational abstractions
  • Operational Difference [Harder et al. 03] – regression testing
  • DIDUCE [Hangal & Lam 02] – detect the sources of errors
  • Specification-based test selection [Chang & Richardson 99]
  • Structural test selection/prioritization
  • Residual/additional structural coverage techniques [Pavlopoulou

& Young 99][Rothermel et al. 01][Srivastava & Thiagarajan 02]

  • Execution profile clustering/sampling [Dicknson et al. 01]
slide-28
SLIDE 28

28

Outline

  • Motivation
  • Operational Violation Approach
  • Experiment
  • Related Work
  • Conclusion
slide-29
SLIDE 29

29

Conclusion

  • Operational Abstractions guide Test Generation and

Selection for human inspection

  • Basic technique, Precondition removal technique, Iterations
  • Experiment demonstrates its usefulness

In future work:

  • Investigate sources of variations affecting cost-

effectiveness

  • Feedback loop between specification inference and test

generation

  • Protocol specifications and algebraic specifications
slide-30
SLIDE 30

30

Questions?

slide-31
SLIDE 31

31

Iterations

Program The existing test suite Operational abstractions

generate

Automatically generated test inputs

select

Selected tests

augment collect generate

slide-32
SLIDE 32

32

Iterations

Program The existing test suite Operational abstractions

generate

Automatically generated test inputs

select

Selected tests

Add preconditions/defensive programming (illegal inputs)

generate

Fix bugs (faults exposed by legal inputs) Add oracles/augment