Software Testing Software Testing CISC 323 Winter 2006 Prof. Lamb - - PowerPoint PPT Presentation

Software Testing Software Testing

CISC 323

Winter 2006

• Prof. Lamb
• Prof. Kelly

malamb@cs.queensu.ca kelly-d@rmc.ca

2

Included in Courseware

• Reference material for the lectures

– 19 pages from “Software Testing, A Craftsman’s Approach”, by Paul C. Jorgensen

3

All Testing is Sampling

How can we sample in such a way that it is

– Efficient – Effective

• Another problem specifically for scientific

and engineering software

– Lack of accurate and detailed oracles

• Often use the TLAR method (That Looks About Right!)

Illustration of difficulties: next slides

4

5

6

7

Testing Techniques

• Purpose

– To provide reasoned ways of generating test cases – To produce test cases systematically

• Might allow some sort of test case automation

– To help in reproducibility of the tests – To address the”when are we done?” problem

• Gives some level of confidence that enough bugs have been

found

– To allow better planning – To allow the use of different techniques to address different issues – To provide data for debugging

8

Testing Techniques

• Black box testing

– Also known as functional testing – Testing without opening the software box – We’ll look at examples of testing by considering the input data

• White box testing

– Also known as structural testing – Testing by considering what the code looks like – We’ll look at examples of code coverage

9

Boundary Value Testing

(Black Box Testing)

• Focuses on the boundaries of the input space to identify

test cases

– Based on the finding that errors seem to cluster at boundaries

• Eg. code checks for <= instead of <
• Assumes

– software failures are rarely the result of the simultaneous

• ccurrence of two or more faults
• Test cases obtained by

– Holding the values of all but one input variable at their nominal values and letting that one variable assume its extreme values

• Minimum, just above minimum, nominal, just below maximum,

maximum

10

Boundary Value Testing Example

X1 X2 a b c d For a function of n variables, there are 4n+1 test cases

11

Robustness Testing

• Extension of boundary value testing

– See what happens when the extrema are exceeded

• Value slightly greater than maximum
• Value slightly less than minimum

– Forces attention on exception handling

• what happens or should happen when values fall outside

their ranges

12

Weaknesses of Boundary Value and Robustness Testing

• Assumption of independent failures
• No consideration of interactions between input values
• Meaning of the variables not taken into account
• Example

– {month, day}

• (02,31) doesn’t make semantic sense, yet this may not be

tested by this technique

13

Worst Case Testing

• Reject the single fault assumption

– Investigate what happens when more than one variable at a time is assigned an extreme value

• For a function of n variables

– Generates 5n test cases

• Limitations

– No consideration of interactions between input values – Meaning of the variables not taken into account

14

Special Value Testing

• Tester uses his/her domain knowledge to

identify risky parts of the software system

– Technique that is most dependent on the skills of the tester – No guidelines – use best engineering judgment – Can be used to “fill in” extra test cases not generated by any of the systematic techniques

15

Error in Courseware

• Software Testing A Craftsman’s Approach
• Page 62, Table 1

– What’s the goof?

16

Equivalence Class Testing

(Black Box Testing)

• Technique

– Divide the input domain into classes

• So that the requirements and/or design specifications

require exactly the same behavior from each value in the class

– Assumption

• The program is constructed so that it either succeeds or

fails for each of the values in that class

– For robustness

• For each equivalence class of valid inputs, we devise

equivalence classes of invalid inputs

17

Equivalence Class Testing

• Technique (cont’d)

– Assume the equivalence classes are disjoint – Construct tests choosing one value from each appropriate equivalence class

• Values are usually drawn at random each time the test is

run

– An equivalence class is considered covered if at least one value has been selected from it

• There is research that says this is sufficient for effective

testing

– The goal is to cover all equivalence classes

18

Equivalence Class Testing Example – Registry of Your Pet

• Fields:

– Classification:

• Dog

– age, breed, colour, M/F, neutered

• Cat

– age, colour, M/F, neutered

• Exotic:

– Mammal: species, age, restricted, endangered, origin – Other: Phylum, Class, Order, Family, Genus, Species, restricted, endangered, origin

19

Equivalence Class Testing Example – Registry of Your Pet

• Possible Valid Equivalence Classes

– Classification:

• Dog, Cat, Exotic

– Age:

• 0-200

– Breed

• Dalmatian, boxer, poodle, …

– Colour

• Black, white, brown, gray, calico,

tiger, …

– M/F – Neutered – Y/N – Exotic – Mammal, other – Mammal Species

• Monkey, ape, tiger, cheetah,

lemur, wolf, raccoon, harbour seal, …

– Restricted – Y/N – Endangered – Y/N – Origin

• Antarctica, Algeria,

Australia, Brazil, …

– Phylum – alpha (0-40) – Class – alpha (0-40) – Order – alpha (0-40) – Family – alpha (0-40) – Genus – alpha (0-40) – Species other – alpha (0-40)

• Corresponding Invalid

Equivalence Classes …?

20

Equivalence Class Testing

• Limitations

– assumes all values within an equivalence class are equally likely to cause failure – assumes the implementation of the class conforms to its specification

• If the implementation does not match the

requirement/design as specified (which it will not if a defect exists), then some values in the equivalence class will produce a different actual behavior than expected

– Those values may not be chosen during equivalence class testing

21

Equivalence Class Testing

• Controversy whether random values drawn

from the whole input space are as effective at finding defects as are random values chosen based on equivalence classes

22

Testing Techniques based on Code Coverage (White Box)

• Statement coverage

– Touch each statement once

• Branch coverage

– Execute each branch after a condition statement

• Cause each condition to be set to true, then set to false
• Path coverage

– Execute all paths through the code

• Generally infeasible in all but simplest cases
• Basis path coverage

– A compromise

23

Example of Statement, Branch, and Path Coverage

void tst(int x) { if (x>0)

pos = pos+1;

if (x%2==0)

even = even+1;

return; }

• Tests required for:

– Statement Coverage: tst(2) – Branch Coverage: tst(-1),tst(2) – Path Coverage: tst(-2),tst(-1),tst(1),tst(2)

24

Use of Statement Coverage for “When are we done?” Problem

• Weakest type of the three coverage types (statement,

branch, path) as far as effective test case generation

• Easiest to support with a tool

– Many tools available to track statement coverage

• Tools add “instrumentation” to your code to track which statements

have been executed

– Could slow your code down unacceptably

• After all your tests are run, tools report on total code coverage

– Often used to decide adequate testing

• Even for this, 100% coverage may not be feasible

– May have special conditions, dead code, error routines

• Best use is to find what parts of the code have not been tested at all

– It may be more cost effective to inspect uncovered parts than it is to cover them by testing

25

100% Coverage?

• Even 100% coverage does not guarantee that

software is fault free or reliable

• The Law of Diminishing Returns

– The more a program has been tested, the less a given test set can further contribute to the test adequacy.

26

Example Code with two Branches (part of Myers Triangle)

#include <stdio.h> #include <stdlib.h> #include <math.h> main(argc, argv) int argc; char* argv[]; { int sideA; int sideB; int sideC; double s; double Area; sideA = atoi(argv[1]); sideB = atoi(argv[2]); sideC = atoi(argv[2]); if ( (sideA == sideB) && (sideA == sideC ) ) { s = 0.5 * (sideA + sideB + sideC); Area = sqrt (s / (s - sideA) * (s - sideB) * (s - sideC) ); printf ( "area = %g\n", Area) ; } Else { puts ("not an equilateral triangle") ; return 0; }

27

Branch Coverage: 2 test cases needed

• To exercise the TRUE

branch

– use inputs

• Side A = 2
• Side B = 2
• Side C = 2

– output

• area = 1.73205
• To exercise the FALSE

branch

– use inputs

• Side A = 3
• Side B = 4
• Side C = 5

– output

• not an equilateral

triangle

28

Two Errors not Caught by Test Cases

• wrong operand to read in sideC

– should be argv[3]

• Heron’s formula for area of a triangle

– Area = [s (s - sideA) (s - sideB) (s - sideC) ] – code has division operator instead of multiplication – 2 is the only value for which the wrong equation gives the right answer

29

Flowgraphs and Path testing

• Flowgraph

– graphical representation of program’s control structure

• Path

– follows the edges of a flowgraph

• Path Selection Criteria - examples:

– every path from entry to exit (path coverage)

• impractical for most programs

– selected paths

• e.g. each loop 0, once, maxcount times

– all paths up to some length

30

McCabe Basis Path Coverage

• Select test cases that follow the control flow

paths defined for the McCabe complexity measure

• McCabe’s conjecture was that such test cases

would provide ‘adequate’ coverage

– It doesn’t

• Tools are available to support the McCabe

approach

– Claims that if more than 10 tests needed for a module, then the module is error prone

• No empirical evidence to support this claim

31

McCabe Basis Path Coverage

• Follow sample code on page 18 of

Courseware, extract from Jorgensen

• We will draw the McCabe flow graph on the

board

32

McCabe Basis Path Coverage

• Number of test cases required is

V(G) = e – n +2p Where e = number of edges on the graph n = number of nodes on the graph p = number of modules OR (number of IF statements) + 1

33

Issues in Computational Software

• From “Numerical Recipes in … - The Art of

Scientific Computing”, by William H. Press et al, Cambridge University Press, 1996

• Chapter 1, section on Error, Accuracy, and

Stability

– Extremely difficult to test for especially when you don’t know exactly what the answer should be …

34

Machine Accuracy

• Numbers are stored in some approximation

that can be packed into a fixed number of bits

• A number in integer representation is exact
• Arithmetic in integer representation is exact

as long as

– Answer is not outside the range that can be represented – Division throws away the integer remainder

35

Machine Accuracy

• Floating point numbers are represented as

s X M X B e – E Where s is a sign bit e is an exact integer exponent M is an exact positive integer mantissa B is the base of the representation (2, 8 or 16) E is a fixed integer bias for any given machine

36

Machine Accuracy

• Arithmetic for floating point numbers is not

exact

• Eg. two floating point numbers are added by

right-shifting the mantissa of the smaller one and increasing its exponent until the two

• perands have the same exponent

– Low order bits of the smaller number are lost by this shifting – If the operands differ too greatly in magnitude

• Smaller number is right-shifted into oblivion

37

Machine Accuracy

• The smallest floating point number which,

when added to 1.0, produces a floating point result different from 1.0 is called the machine accuracy, εm

• A typical computer with B=2 and a 32 bit

word length has εm around 3X10-8

• εm depends on how many bits there are in

the mantissa

38

Round Off Error

• Any arithmetic operation involving floating

point numbers introduces an error of at least εm

• Round off errors accumulate with increasing

amounts of calculation

• The error can accumulate preferentially in
• ne direction

– With N operations, the error can be N εm

39

Round Off Error

• Subtraction of two nearly equal numbers

– Correct result dependent on a few low order bits

• Example: solution of a quadratic equation

x = (-b +- (b2 – 4ac)1/2) / 2a

• Problem when ac << b2

40

Correct Solution

Define q = -1/2 [b + sgn(b)(b2 – 4ac)1/2] The roots are x1 = q/a x2 = c/q

41

Instability

• Algorithms that compute a discrete

approximation to a continuous quantity

• An algorithm can be unstable if the round off

error gets mixed into the calculation at an early stage

– Round off is successively magnified until it swamps the true answer

42

Instability Example

Golden mean given by φ = (5 1/2 – 1) / 2 ~ 0.61803398 Powers of φ n Can be computed by recurrence algorithm φ n+1 = φ n-1 - φ n But this has another undesired solution of

• 1/2 (5 1/2 + 1)

Algorithm starts giving wrong answers around n=16 on a machine with 32-bit word length

43

How to deal with this?

• Identify tricky areas and design test cases specifically

to exercise those areas

– Use engineering judgment to devise special value testing

• Code with the knowledge of machine floating point

arithmetic

• Use algorithms that are tried and tested

– Find standard algorithms in textbooks – Use well-established math libraries

• Have someone with numerical techniques knowledge

inspect your code