[PPT] - Test Oracles and Randomness S I T R T E V U I N L M U PowerPoint Presentation

SLIDE 1

Test Oracles and Randomness

U N I V E R S I T Ä T U L M · S C I E N D O · D O C E N D O · C U R A N D O ·

Ralph Guderlei and Johannes Mayer

rjg@mathematik.uni-ulm.de, jmayer@mathematik.uni-ulm.de

University of Ulm, Department of Stochastics and Department of Applied Information Processing

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 1

SLIDE 2

Introduction

Standard methods of software testing do not

provide information about software reliability. ⇒ random testing

The main problem in random testing is to

verify the actual results of the Implementation Under Test (IUT). ⇒ Oracles

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 2

SLIDE 3

Oracles

An oracle provides methods

to generate expected results for test cases
to compare the expected results to the actual

results of the IUT.

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 3

SLIDE 4

Oracles

An oracle provides methods

to generate expected results for test cases
to compare the expected results to the actual

results of the IUT. It consists of two parts:

the result generator to obtain expected results
the comparator to verify the actual results of

the IUT

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 3

SLIDE 5

Standard Types of Oracles

Oracles do not apply generally, only in special cases. Standard types are

Perfect Oracle
Gold Standard Oracle
Parametric Oracle or Heuristic Oracle

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 4

SLIDE 6

Perfect Oracle

equivalent to the IUT and completely trusted
accepts every input specified for the IUT
produces ”always” the correct result

⇒ a ”defect free” version of the IUT

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 5

SLIDE 7

Gold Standard Oracle

Test Input Golden Implementation IUT Comparator

Test Case Input actual result golden result Pass / no Pass

Use one or more versions of an existing application system to generate expected results (e.g. a legacy system).

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 6

SLIDE 8

Parametric Oracle

Test Input IUT Trusted Algorithm Output Converter Comparator

actual results reference parameters actual parameters Pass / no Pass

Use an algorithm to compute parameters from the actual results and compare the actual parameters to expected parameter values.

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 7

SLIDE 9

Statistical Oracle

special case of a parametric oracle

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 8

SLIDE 10

Statistical Oracle

special case of a parametric oracle
parameters are computed with statistical tools

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 8

SLIDE 11

Statistical Oracle

special case of a parametric oracle
parameters are computed with statistical tools
comparison is done in a statistical way

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 8

SLIDE 12

Statistical Oracle

special case of a parametric oracle
parameters are computed with statistical tools
comparison is done in a statistical way
generation of random input data allows a

large number of test cases

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 8

SLIDE 13

Micropattern

Random Test Input Generator IUT Statistical Analyzer Comparator

Test Case Input Actual Results Characteristics Distributional Parameters Pass / No Pass

distributional properties of the generator and the IUT are used for the comparison

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 9

SLIDE 14

Requirements

statistical characteristics of the IUT have to

be known

large number of input data for stable results

(> 30)

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 10

SLIDE 15

Possible Uses

(Scientific) applications dealing with

randomness e.g. simulators, data analysis (e.g. in banking, image analysis)

Applications with complicated input data,

where reference values are difficult to obtain. As in the Example:

images are difficult to analyze
randomly generating and comparing the

mean values is simple

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 11

SLIDE 16

Necessary Statistics

Xi iid random variables with mean µ and

variance σ2

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 12

SLIDE 17

Necessary Statistics

Xi iid random variables with mean µ and

variance σ2

sample mean of n random variables:

Xn = 1

n n

i=1

Xi

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 12

SLIDE 18

Necessary Statistics

Xi iid random variables with mean µ and

variance σ2

sample mean of n random variables:

Xn = 1

n n

i=1

Xi

sample variance of n random variables:

S2

n = 1 n−1 n

i=1

(Xi − Xn)2

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 12

SLIDE 19

Distributional Properties

using the central limit theorem, one can assume the Xi to be (assymptotically) normally distributed. √n ¯ Xn − µ σ

d

− → N(0, 1) as n → ∞

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 13

SLIDE 20

Simple Approach

expected value is known: µ0

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 14

SLIDE 21

Simple Approach

expected value is known: µ0
compute empirical mean of the n actual

results xi: ¯ xn = 1

n

i=1 xi

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 14

SLIDE 22

Simple Approach

expected value is known: µ0
compute empirical mean of the n actual

results xi: ¯ xn = 1

n

i=1 xi

pass if |¯

xn−µ0| µ0

< ε, e.g.ε = 0.1

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 14

SLIDE 23

First Attempt

Natural choice: t-test if the mean of the actual results µ is equal to the expected result µ0.

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 15

SLIDE 24

First Attempt

Natural choice: t-test if the mean of the actual results µ is equal to the expected result µ0. Type I error: IUT does not pass though it is correct (false alarm) Type II error: IUT does pass though it is not correct (false pass) But only the Type I error will be controlled.

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 15

SLIDE 25

Advanced Approach

Alternative choice: use the intersection-union method to invert the test hypothesis. So the controllable probability for the Type I error becomes the probability for a false pass.

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 16

SLIDE 26

Advanced Approach

Alternative choice: use the intersection-union method to invert the test hypothesis. So the controllable probability for the Type I error becomes the probability for a false pass. With δ > 0, one can define an interval around µ0 such that ¯ Xn / ∈ [µ0 − δ, µ0 + δ] for a given probability α.

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 16

SLIDE 27

The Test Statistic

For a given probability α ∈ (0, 1

2),

the IUT passes if √n ¯

xn−(µ0−δ) sn

≥ tn−1, α

2

and √n ¯

xn−(µ0+δ) sn

≤ tn−1, α

2 ,

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 17

SLIDE 28

The Test Statistic

For a given probability α ∈ (0, 1

2),

the IUT passes if √n ¯

xn−(µ0−δ) sn

≥ tn−1, α

2

and √n ¯

xn−(µ0+δ) sn

≤ tn−1, α

2 ,

where tn, α

2 denotes the (1 − α/2) - quantile of the

Student t-distribution with n − 1 degrees of freedom.

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 17

SLIDE 29

An Example from Image Analysis

compute morphological

properties such as area or boundary length

fit stochastic models to

the given data

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 18

SLIDE 30

Random Input - The Boolean Model

computationally simple

model

flexible
good fit to real data
the

expected mean area and mean boundary length are known in explicit form

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 19

SLIDE 31

Micropattern Revisited

IUT Statistical Analyzer Comparator

Test Case Input Actual Results, e.g. Area Characterstics, e.g. Mean Area Theoretical Results from the Model Pass / No Pass

Generator for Boolean Model

Random Input Generator = Generator for the

Boolean Model

The IUT computes e.g. the area or the

boundary length

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 20

SLIDE 32

Usage of the Simple Approach

Performing the test

for each computed characteristic (e.g. the

area)

for different Boolean Models

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 21

SLIDE 33

Usage of the Simple Approach

Performing the test

for each computed characteristic (e.g. the

area)

for different Boolean Models

The simple approach was used as

advanced smoke test to detect severe bugs in

the program flow

plausibility check for the computed values

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 21

SLIDE 34

Usage of the Advanced Approach

verification of the results of the simple

approach

choosing a small α, one can say that the IUT

produces the correct results

(only) with respect to the tested

characteristics

with probability α

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 22

SLIDE 35

Numerical Example

sample of n = 500 images, error probability α = 0.0001,

here xi ˆ = boundary length

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 23

SLIDE 36

Numerical Example

sample of n = 500 images, error probability α = 0.0001,

here xi ˆ = boundary length

¯

xn = 0.044925685, µ0 = 0.044948, δ = 0.00044948 = 1% ∗ µ0

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 23

SLIDE 37

Numerical Example

sample of n = 500 images, error probability α = 0.0001,

here xi ˆ = boundary length

¯

xn = 0.044925685, µ0 = 0.044948, δ = 0.00044948 = 1% ∗ µ0

√n ¯

xn−(µ0−δ) sn

= 10.38 > 3.92 = t499,0.99995 and √n ¯ xn − (µ0 + δ) sn = −11.47 < −3.92 = −t499,0.99995.

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 23

SLIDE 38

Numerical Example

sample of n = 500 images, error probability α = 0.0001,

here xi ˆ = boundary length

¯

xn = 0.044925685, µ0 = 0.044948, δ = 0.00044948 = 1% ∗ µ0

√n ¯

xn−(µ0−δ) sn

= 10.38 > 3.92 = t499,0.99995 and √n ¯ xn − (µ0 + δ) sn = −11.47 < −3.92 = −t499,0.99995.

So the IUT passes the test.

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 23

SLIDE 39

Conclusion

The approach includes information about the

reliability with respect to for the tested characteristics

The approach makes it possible to handle

randomness in tests

It is possible to test for other characteristics

than the mean

The approach shown above does not apply in

all cases

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 24

SLIDE 40

Thank you for your attention.

Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 25