Test Oracles and Randomness S I T R T E V U I N L M U - - PowerPoint PPT Presentation

test oracles and randomness
SMART_READER_LITE
LIVE PREVIEW

Test Oracles and Randomness S I T R T E V U I N L M U - - PowerPoint PPT Presentation

Test Oracles and Randomness S I T R T E V U I N L M U O S C D I N E A N R D U O C D O O D C N E Ralph Guderlei and Johannes Mayer rjg@mathematik.uni-ulm.de, jmayer@mathematik.uni-ulm.de


slide-1
SLIDE 1

Test Oracles and Randomness

U N I V E R S I T Ä T U L M · S C I E N D O · D O C E N D O · C U R A N D O ·

Ralph Guderlei and Johannes Mayer

rjg@mathematik.uni-ulm.de, jmayer@mathematik.uni-ulm.de

University of Ulm, Department of Stochastics and Department of Applied Information Processing

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 1

slide-2
SLIDE 2

Introduction

  • Standard methods of software testing do not

provide information about software reliability. ⇒ random testing

  • The main problem in random testing is to

verify the actual results of the Implementation Under Test (IUT). ⇒ Oracles

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 2

slide-3
SLIDE 3

Oracles

An oracle provides methods

  • to generate expected results for test cases
  • to compare the expected results to the actual

results of the IUT.

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 3

slide-4
SLIDE 4

Oracles

An oracle provides methods

  • to generate expected results for test cases
  • to compare the expected results to the actual

results of the IUT. It consists of two parts:

  • the result generator to obtain expected results
  • the comparator to verify the actual results of

the IUT

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 3

slide-5
SLIDE 5

Standard Types of Oracles

Oracles do not apply generally, only in special cases. Standard types are

  • Perfect Oracle
  • Gold Standard Oracle
  • Parametric Oracle or Heuristic Oracle

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 4

slide-6
SLIDE 6

Perfect Oracle

  • equivalent to the IUT and completely trusted
  • accepts every input specified for the IUT
  • produces ”always” the correct result

⇒ a ”defect free” version of the IUT

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 5

slide-7
SLIDE 7

Gold Standard Oracle

Test Input Golden Implementation IUT Comparator

Test Case Input actual result golden result Pass / no Pass

Use one or more versions of an existing application system to generate expected results (e.g. a legacy system).

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 6

slide-8
SLIDE 8

Parametric Oracle

Test Input IUT Trusted Algorithm Output Converter Comparator

actual results reference parameters actual parameters Pass / no Pass

Use an algorithm to compute parameters from the actual results and compare the actual parameters to expected parameter values.

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 7

slide-9
SLIDE 9

Statistical Oracle

  • special case of a parametric oracle

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 8

slide-10
SLIDE 10

Statistical Oracle

  • special case of a parametric oracle
  • parameters are computed with statistical tools

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 8

slide-11
SLIDE 11

Statistical Oracle

  • special case of a parametric oracle
  • parameters are computed with statistical tools
  • comparison is done in a statistical way

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 8

slide-12
SLIDE 12

Statistical Oracle

  • special case of a parametric oracle
  • parameters are computed with statistical tools
  • comparison is done in a statistical way
  • generation of random input data allows a

large number of test cases

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 8

slide-13
SLIDE 13

Micropattern

Random Test Input Generator IUT Statistical Analyzer Comparator

Test Case Input Actual Results Characteristics Distributional Parameters Pass / No Pass

distributional properties of the generator and the IUT are used for the comparison

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 9

slide-14
SLIDE 14

Requirements

  • statistical characteristics of the IUT have to

be known

  • large number of input data for stable results

(> 30)

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 10

slide-15
SLIDE 15

Possible Uses

  • (Scientific) applications dealing with

randomness e.g. simulators, data analysis (e.g. in banking, image analysis)

  • Applications with complicated input data,

where reference values are difficult to obtain. As in the Example:

  • images are difficult to analyze
  • randomly generating and comparing the

mean values is simple

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 11

slide-16
SLIDE 16

Necessary Statistics

  • Xi iid random variables with mean µ and

variance σ2

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 12

slide-17
SLIDE 17

Necessary Statistics

  • Xi iid random variables with mean µ and

variance σ2

  • sample mean of n random variables:

Xn = 1

n n

  • i=1

Xi

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 12

slide-18
SLIDE 18

Necessary Statistics

  • Xi iid random variables with mean µ and

variance σ2

  • sample mean of n random variables:

Xn = 1

n n

  • i=1

Xi

  • sample variance of n random variables:

S2

n = 1 n−1 n

  • i=1

(Xi − Xn)2

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 12

slide-19
SLIDE 19

Distributional Properties

using the central limit theorem, one can assume the Xi to be (assymptotically) normally distributed. √n ¯ Xn − µ σ

d

− → N(0, 1) as n → ∞

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 13

slide-20
SLIDE 20

Simple Approach

  • expected value is known: µ0

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 14

slide-21
SLIDE 21

Simple Approach

  • expected value is known: µ0
  • compute empirical mean of the n actual

results xi: ¯ xn = 1

n

n

i=1 xi

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 14

slide-22
SLIDE 22

Simple Approach

  • expected value is known: µ0
  • compute empirical mean of the n actual

results xi: ¯ xn = 1

n

n

i=1 xi

  • pass if |¯

xn−µ0| µ0

< ε, e.g.ε = 0.1

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 14

slide-23
SLIDE 23

First Attempt

Natural choice: t-test if the mean of the actual results µ is equal to the expected result µ0.

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 15

slide-24
SLIDE 24

First Attempt

Natural choice: t-test if the mean of the actual results µ is equal to the expected result µ0. Type I error: IUT does not pass though it is correct (false alarm) Type II error: IUT does pass though it is not correct (false pass) But only the Type I error will be controlled.

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 15

slide-25
SLIDE 25

Advanced Approach

Alternative choice: use the intersection-union method to invert the test hypothesis. So the controllable probability for the Type I error becomes the probability for a false pass.

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 16

slide-26
SLIDE 26

Advanced Approach

Alternative choice: use the intersection-union method to invert the test hypothesis. So the controllable probability for the Type I error becomes the probability for a false pass. With δ > 0, one can define an interval around µ0 such that ¯ Xn / ∈ [µ0 − δ, µ0 + δ] for a given probability α.

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 16

slide-27
SLIDE 27

The Test Statistic

For a given probability α ∈ (0, 1

2),

the IUT passes if √n ¯

xn−(µ0−δ) sn

≥ tn−1, α

2

and √n ¯

xn−(µ0+δ) sn

≤ tn−1, α

2 ,

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 17

slide-28
SLIDE 28

The Test Statistic

For a given probability α ∈ (0, 1

2),

the IUT passes if √n ¯

xn−(µ0−δ) sn

≥ tn−1, α

2

and √n ¯

xn−(µ0+δ) sn

≤ tn−1, α

2 ,

where tn, α

2 denotes the (1 − α/2) - quantile of the

Student t-distribution with n − 1 degrees of freedom.

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 17

slide-29
SLIDE 29

An Example from Image Analysis

  • compute morphological

properties such as area or boundary length

  • fit stochastic models to

the given data

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 18

slide-30
SLIDE 30

Random Input - The Boolean Model

  • computationally simple

model

  • flexible
  • good fit to real data
  • the

expected mean area and mean boundary length are known in explicit form

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 19

slide-31
SLIDE 31

Micropattern Revisited

IUT Statistical Analyzer Comparator

Test Case Input Actual Results, e.g. Area Characterstics, e.g. Mean Area Theoretical Results from the Model Pass / No Pass

Generator for Boolean Model

  • Random Input Generator = Generator for the

Boolean Model

  • The IUT computes e.g. the area or the

boundary length

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 20

slide-32
SLIDE 32

Usage of the Simple Approach

Performing the test

  • for each computed characteristic (e.g. the

area)

  • for different Boolean Models

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 21

slide-33
SLIDE 33

Usage of the Simple Approach

Performing the test

  • for each computed characteristic (e.g. the

area)

  • for different Boolean Models

The simple approach was used as

  • advanced smoke test to detect severe bugs in

the program flow

  • plausibility check for the computed values

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 21

slide-34
SLIDE 34

Usage of the Advanced Approach

  • verification of the results of the simple

approach

  • choosing a small α, one can say that the IUT

produces the correct results

  • (only) with respect to the tested

characteristics

  • with probability α

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 22

slide-35
SLIDE 35

Numerical Example

  • sample of n = 500 images, error probability α = 0.0001,

here xi ˆ = boundary length

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 23

slide-36
SLIDE 36

Numerical Example

  • sample of n = 500 images, error probability α = 0.0001,

here xi ˆ = boundary length

  • ¯

xn = 0.044925685, µ0 = 0.044948, δ = 0.00044948 = 1% ∗ µ0

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 23

slide-37
SLIDE 37

Numerical Example

  • sample of n = 500 images, error probability α = 0.0001,

here xi ˆ = boundary length

  • ¯

xn = 0.044925685, µ0 = 0.044948, δ = 0.00044948 = 1% ∗ µ0

  • √n ¯

xn−(µ0−δ) sn

= 10.38 > 3.92 = t499,0.99995 and √n ¯ xn − (µ0 + δ) sn = −11.47 < −3.92 = −t499,0.99995.

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 23

slide-38
SLIDE 38

Numerical Example

  • sample of n = 500 images, error probability α = 0.0001,

here xi ˆ = boundary length

  • ¯

xn = 0.044925685, µ0 = 0.044948, δ = 0.00044948 = 1% ∗ µ0

  • √n ¯

xn−(µ0−δ) sn

= 10.38 > 3.92 = t499,0.99995 and √n ¯ xn − (µ0 + δ) sn = −11.47 < −3.92 = −t499,0.99995.

  • So the IUT passes the test.

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 23

slide-39
SLIDE 39

Conclusion

  • The approach includes information about the

reliability with respect to for the tested characteristics

  • The approach makes it possible to handle

randomness in tests

  • It is possible to test for other characteristics

than the mean

  • The approach shown above does not apply in

all cases

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 24

slide-40
SLIDE 40

Thank you for your attention.

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 25