Test Oracles and Randomness S I T R T E V U I N L M U - PowerPoint PPT Presentation

Test Oracles and Randomness S I T R Ä T E V U I N L M U · · O S C D I N E A N R D U O C · · D O O D C N E Ralph Guderlei and Johannes Mayer rjg@mathematik.uni-ulm.de, jmayer@mathematik.uni-ulm.de University of Ulm, Department of Stochastics and Department of Applied Information Processing Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 1

Introduction • Standard methods of software testing do not provide information about software reliability. ⇒ random testing • The main problem in random testing is to verify the actual results of the Implementation Under Test (IUT). ⇒ Oracles Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 2

Oracles An oracle provides methods • to generate expected results for test cases • to compare the expected results to the actual results of the IUT. Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 3

Oracles An oracle provides methods • to generate expected results for test cases • to compare the expected results to the actual results of the IUT. It consists of two parts: • the result generator to obtain expected results • the comparator to verify the actual results of the IUT Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 3

Standard Types of Oracles Oracles do not apply generally, only in special cases. Standard types are • Perfect Oracle • Gold Standard Oracle • Parametric Oracle or Heuristic Oracle Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 4

Perfect Oracle • equivalent to the IUT and completely trusted • accepts every input specified for the IUT • produces ”always” the correct result ⇒ a ”defect free” version of the IUT Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 5

Gold Standard Oracle Golden Test Case Input golden Pass / no Pass Test Input Comparator result Implementation actual result IUT Use one or more versions of an existing application system to generate expected results (e.g. a legacy system). Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 6

Parametric Oracle actual Test Input IUT Output Converter results actual parameters Pass / no Pass reference parameters Trusted Algorithm Comparator Use an algorithm to compute parameters from the actual results and compare the actual parameters to expected parameter values. Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 7

Statistical Oracle • special case of a parametric oracle Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 8

Statistical Oracle • special case of a parametric oracle • parameters are computed with statistical tools Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 8

Statistical Oracle • special case of a parametric oracle • parameters are computed with statistical tools • comparison is done in a statistical way Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 8

Statistical Oracle • special case of a parametric oracle • parameters are computed with statistical tools • comparison is done in a statistical way • generation of random input data allows a large number of test cases Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 8

Micropattern Random Test Case Input Actual Test Input IUT Statistical Analyzer Results Generator Characteristics Distributional Parameters Pass / No Pass Comparator distributional properties of the generator and the IUT are used for the comparison Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 9

Requirements • statistical characteristics of the IUT have to be known • large number of input data for stable results ( > 30 ) Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 10

Possible Uses • (Scientific) applications dealing with randomness e.g. simulators, data analysis (e.g. in banking, image analysis) • Applications with complicated input data, where reference values are difficult to obtain. As in the Example: • images are difficult to analyze • randomly generating and comparing the mean values is simple Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 11

Necessary Statistics • X i iid random variables with mean µ and variance σ 2 Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 12

Necessary Statistics • X i iid random variables with mean µ and variance σ 2 • sample mean of n random variables: n X n = 1 � X i n i =1 Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 12

Necessary Statistics • X i iid random variables with mean µ and variance σ 2 • sample mean of n random variables: n X n = 1 � X i n i =1 • sample variance of n random variables: n 1 S 2 ( X i − X n ) 2 n = � n − 1 i =1 Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 12

Distributional Properties using the central limit theorem, one can assume the X i to be (assymptotically) normally distributed. ¯ √ n X n − µ d − → N (0 , 1) as n → ∞ σ Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 13

Simple Approach • expected value is known: µ 0 Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 14

Simple Approach • expected value is known: µ 0 • compute empirical mean of the n actual results x i : � n x n = 1 ¯ i =1 x i n Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 14

Simple Approach • expected value is known: µ 0 • compute empirical mean of the n actual results x i : � n x n = 1 ¯ i =1 x i n • pass if | ¯ x n − µ 0 | < ε , e.g. ε = 0 . 1 µ 0 Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 14

First Attempt Natural choice: t-test if the mean of the actual results µ is equal to the expected result µ 0 . Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 15

First Attempt Natural choice: t-test if the mean of the actual results µ is equal to the expected result µ 0 . Type I error: IUT does not pass though it is correct (false alarm) Type II error: IUT does pass though it is not correct (false pass) But only the Type I error will be controlled. Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 15

Advanced Approach Alternative choice: use the intersection-union method to invert the test hypothesis. So the controllable probability for the Type I error becomes the probability for a false pass. Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 16

Advanced Approach Alternative choice: use the intersection-union method to invert the test hypothesis. So the controllable probability for the Type I error becomes the probability for a false pass. With δ > 0 , one can define an interval around µ 0 such that ¯ ∈ [ µ 0 − δ, µ 0 + δ ] X n / for a given probability α . Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 16

The Test Statistic For a given probability α ∈ (0 , 1 2 ) , the IUT passes if √ n ¯ x n − ( µ 0 − δ ) ≥ t n − 1 , α s n 2 and √ n ¯ x n − ( µ 0 + δ ) ≤ t n − 1 , α 2 , s n Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 17

The Test Statistic For a given probability α ∈ (0 , 1 2 ) , the IUT passes if √ n ¯ x n − ( µ 0 − δ ) ≥ t n − 1 , α s n 2 and √ n ¯ x n − ( µ 0 + δ ) ≤ t n − 1 , α 2 , s n 2 denotes the (1 − α/ 2) - quantile of the where t n, α Student t-distribution with n − 1 degrees of freedom. Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 17

An Example from Image Analysis • compute morphological properties such as area or boundary length • fit stochastic models to the given data Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 18

Random Input - The Boolean Model • computationally simple model • flexible • good fit to real data • the expected mean area and mean boundary length are known in explicit form Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 19

Micropattern Revisited Generator Test Case Input Actual for Boolean IUT Statistical Analyzer Results, Model e.g. Area Characterstics, e.g. Mean Area Pass / No Pass Theoretical Results from the Model Comparator • Random Input Generator = Generator for the Boolean Model • The IUT computes e.g. the area or the boundary length Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 20

Usage of the Simple Approach Performing the test • for each computed characteristic (e.g. the area) • for different Boolean Models Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 21

Usage of the Simple Approach Performing the test • for each computed characteristic (e.g. the area) • for different Boolean Models The simple approach was used as • advanced smoke test to detect severe bugs in the program flow • plausibility check for the computed values Ralph Guderlei - Test Oracles and Randomness - NetObject Days 2004 – p. 21