Faults Found by Random Testing WanChi Chio Justin Molinyawe - - PowerPoint PPT Presentation

Faults Found by Random Testing

WanChi Chio Justin Molinyawe

Introduction

• Why apply random testing to object
• riented software?

○ Cost-efficient in terms of implementation

effort and execution time.

Introduction (Cont.)

• Characteristics of random testing

○ Predictability in terms of number and nature of faults. ○ Average time it takes to find a failure. ○ Distribution of detected faults.

Introduction (Cont.)

Experiments

• Predictability of Random Testing

○ Creating and using tests with AutoTest for 27 classes from Eiffel library.

• Each class was tested for 90 minutes. For

each class, testing sessions were ran 30 times with different seeds.

Unit Tests for Object- Oriented Programs

• Purpose: to verify the run of a particular

routine: ○ test routine "test_x" for routine "x"

• Hard-coded stubs must be used to replace

non-trivial routines which are not under test for a strict interpretation, but it does not usually happens in practice

Unit Tests for Object-Oriented Programs (Cont.)

• Input are created in two ways by:

○ loading previously saved objects into memory => to provide complete control

• ver the input creation

○ calling a series of constructors to create and modify data through normal means => to yield more relevant data

Unit Tests for Object-Oriented Programs (Cont.)

○ Oracle/expect output can be provided in

different levels of abstractions for validation through following specifications:

■ The exact expected output - good for

complex scenarios

■ Conditions that the output should fulfill

for any input

■ "No exception" as the expected output

• easier to be re-used for other tests

Random Tests Cases for Eiffel Library Experiment

• Testing is completely automated with test

cases created under the following criterias:

○ Unit - routines of Eiffel classes ■ Test synthesizer makes test cases where

the goal is to test each particular routine (Test Case Synthesis Algorithm)

○ Input - Objects are created through

regular constructor and routine calls.

■ To enforce diversity, the initial objects are

modified for new test cases

Random Tests Cases for Eiffel Library Experiment (Cont.)

○ Oracle ■ Contract-based by containing specifications in the form of routine pre and postconditions and class invariants

Faults and failures of Random Testing

• Usually, failures are an observed difference

between the actual and intended behavior.

• Results relied on an approximation that

maps failures to faults based on:

○ Two failures are a consequence of the same fault if and only if they manifest themselves through the same type of exception.

Test Case Synthesis Algorithm

Class C { R m(C1 P1, ... , Cn Pn); } A test case for routine m above can be executed as follows: c.m(a1, .... , an) => r where c = object of Class C r = output in Type R m = routine defined in Class C aj = argument object of Type Cj for Parameter Pj

Test Case Synthesis Algorithm (Cont.)

Using AutoTest

• Contract based testing tool.

○ Allows easy plug-in of testing strategies.

• Composed of two processes

○ Test Generator / Driver ○ Interpreter

• AutoTest runs for a duration of the set time.

Number of Faults

• Will different runs of random testing with

different seeds for the pseudo-random number generator reveal a similar number of faults?

• Will the faults be the same?

Number of Faults in Eiffel Library

Experimental Setup

• 27 classes
• Each class was tested for 30 sessions of 90

mins

• Each session had a different seed to

initialize the pseudo-random number generator.

• Total testing time was 30 * 27 * 90 = 50.6

days

Number of Faults in Eiffel Library

Experimental Results

• Is random testing predictable? Two kinds of

predictability:

○ Number of faults ○ Identity of faults and find if it is likely to detect the same faults regardless of the experiments was selected

• Normalized results give us aggregated

results.

Number of Faults

Conclusion

Results

• On average, 30% of the faults found during

experiments are found after 10 mins. After 90 mins, 8% points, on average of the

• verall number of randomly detected faults

are found.

• Relative to the number of detected faults,

random+ is highly predictable.

• Different runs of the testing process reveal

different faults.

Questions?