Testing 5 March 2020 OSU CSE 1 Importance of Testing Testing is - - PowerPoint PPT Presentation

▶
testing
SMART_READER_LITE
LIVE PREVIEW

Testing 5 March 2020 OSU CSE 1 Importance of Testing Testing is - - PowerPoint PPT Presentation

Sep 18, 2022 213 likes •705 views

Testing 5 March 2020 OSU CSE 1 Importance of Testing Testing is a ubiquitous and expensive software engineering activity It is not unusual to spend 30-40% of total project effort on testing For big and/or life-critical systems

slide-1
SLIDE 1

Testing

5 March 2020 OSU CSE 1

slide-2
SLIDE 2

Importance of Testing

  • Testing is a ubiquitous and expensive

software engineering activity

– It is not unusual to spend 30-40% of total project effort on testing – For big and/or life-critical systems (e.g., flight control), testing cost can be several times the cost of all other software engineering activities combined

5 March 2020 OSU CSE 2

slide-3
SLIDE 3

How Big is Big?

  • The method bodies we have been writing

average maybe a dozen lines of code

  • Claim: Boeing 787 Dreamliner avionics (flight

control) software has about ...

3 5 March 2020 OSU CSE

slide-4
SLIDE 4

How Big is Big?

  • The method bodies we have been writing

average maybe a dozen lines of code

  • Claim: Boeing 787 Dreamliner avionics (flight

control) software has about 6.5 million lines of code

  • Claim: Microsoft Windows 10 has about ...

4 5 March 2020 OSU CSE

slide-5
SLIDE 5

How Big is Big?

  • The method bodies we have been writing

average maybe a dozen lines of code

  • Claim: Boeing 787 Dreamliner avionics (flight

control) software has about 6.5 million lines of code

  • Claim: Microsoft Windows 10 has about 50

million lines of code

  • Claim: a modern car has about ...

5 5 March 2020 OSU CSE

slide-6
SLIDE 6

How Big is Big?

  • The method bodies we have been writing

average maybe a dozen lines of code

  • Claim: Boeing 787 Dreamliner avionics (flight

control) software has about 6.5 million lines of code

  • Claim: Microsoft Windows 10 has about 50

million lines of code

  • Claim: a modern car has about 100 million lines
  • f code (though this figure is highly dubious)

6 5 March 2020 OSU CSE

slide-7
SLIDE 7

Unit Testing: Dealing with Scale

  • Best practice is to test individual units or

components of software (one class, one method at a time)

– This is known as unit testing – Testing what happens when multiple components are put together into a larger system is known as integration testing – Testing a whole end-user system is known as system testing

5 March 2020 OSU CSE 7

slide-8
SLIDE 8

Unit Testing: Dealing with Scale

  • Best practice is to test individual units or

components of software (one class, one method at a time)

– This is known as unit testing – Testing what happens when multiple components are put together into a larger system is known as integration testing – Testing a whole end-user system is known as system testing

5 March 2020 OSU CSE 8

This is the kind of testing we will do in this course and the next.

slide-9
SLIDE 9

Unit Testing: Dealing with Scale

  • Best practice is to test individual units or

components of software (one class, one method at a time)

– This is known as unit testing – Testing what happens when multiple components are put together into a larger system is known as integration testing – Testing a whole end-user system is known as system testing

5 March 2020 OSU CSE 9

The unit being tested is known as the UUT, or unit under test.

slide-10
SLIDE 10

Testing Functional Correctness

  • What does it mean for a program unit (let’s

say a method) to be correct?

  • It does what it is supposed to do.
  • It doesn’t do what it is not supposed to do.

5 March 2020 OSU CSE 10

slide-11
SLIDE 11

“Supposed To Do”?

  • How do we know what a method is

supposed to do, and what it is not supposed to do?

– We look at its contract, which is a specification of its intended behavior

5 March 2020 OSU CSE 11

slide-12
SLIDE 12

Behaviors

5 March 2020 OSU CSE 12

Actual behaviors of the method (see body) Allowed behaviors

  • f the method

(see contract)

slide-13
SLIDE 13

Behaviors

5 March 2020 OSU CSE 13

Each point in this space is a legal input with a corresponding allowable result.

Actual behaviors of the method (see body) Allowed behaviors

  • f the method

(see contract)

slide-14
SLIDE 14

Example Method Contract

/** * Reports some factor of a number. * ... * @requires * n > 0 * @ensures * aFactor > 0 and * n mod aFactor = 0 */ private static int aFactor(int n) {...}

5 March 2020 OSU CSE 14

slide-15
SLIDE 15

Example Method Contract

/** * Reports some factor of a number. * ... * @requires * n > 0 * @ensures * aFactor > 0 and * n mod aFactor = 0 */ private static int aFactor(int n) {...}

5 March 2020 OSU CSE 15

This means: “n is divisible by aFactor”.

slide-16
SLIDE 16

Example Method Body

private static int aFactor(int n) { return 1; }

5 March 2020 OSU CSE 16

slide-17
SLIDE 17

Example Method Body

private static int aFactor(int n) { return 1; }

5 March 2020 OSU CSE 17

Is this method body correct?

slide-18
SLIDE 18

Behaviors

5 March 2020 OSU CSE 18

Actual behaviors of the method (see body) Allowed behaviors

  • f the method

(see contract)

slide-19
SLIDE 19

Behaviors

5 March 2020 OSU CSE 19

Contract for aFactor allows: n = 12 aFactor = 4

Actual behaviors of the method (see body) Allowed behaviors

  • f the method

(see contract)

(12,4)

slide-20
SLIDE 20

Behaviors

5 March 2020 OSU CSE 20

Contract for aFactor forbids: n = 12 aFactor = 5

Actual behaviors of the method (see body) Allowed behaviors

  • f the method

(see contract)

(12,4) (12,5)

slide-21
SLIDE 21

Behaviors

5 March 2020 OSU CSE 21

Contract for aFactor allows: n = 12 aFactor = 6

Actual behaviors of the method (see body) Allowed behaviors

  • f the method

(see contract)

(12,4) (12,5) (12,6)

slide-22
SLIDE 22

Behaviors

5 March 2020 OSU CSE 22

Actual behaviors of the method (see body) Allowed behaviors

  • f the method

(see contract)

Contract for aFactor allows: n = 12 aFactor = 1

(12,4) (12,5) (12,6) (12,1)

slide-23
SLIDE 23

Behaviors

5 March 2020 OSU CSE 23

Actual behaviors of the method (see body) Allowed behaviors

  • f the method

(see contract)

Body for aFactor gives: n = 12 aFactor = 1

(12,4) (12,5) (12,6) (12,1)

slide-24
SLIDE 24

Definition of Correctness

  • Body is correct if actual is a subset of allowed.

5 March 2020 OSU CSE 24

Actual behaviors of the method (see body) Allowed behaviors

  • f the method

(see contract)

slide-25
SLIDE 25

“Implements” Revisited

  • If you write class C implements I,

the Java compiler checks that for each method in I there is some method body for it in C

  • We really care about much more: that for

each method in I the method body for it in C is correct in the sense just defined

5 March 2020 OSU CSE 25

slide-26
SLIDE 26

“Implements” Revisited

  • If you write class C implements I,

the Java compiler checks that for each method in I there is some method body for it in C

  • We really care about much more: that for

each method in I the method body for it in C is correct in the sense just defined

5 March 2020 OSU CSE 26

How can you decide whether this is the case for a given method body?

slide-27
SLIDE 27

Testing

  • Testing is a technique for trying to refute

the claim that a method body is correct for the method contract

  • In other words, the goal of testing is to

show that the method body does not correctly implement the contract, i.e., that it is defective

– As a tester, you really want to think this way!

5 March 2020 OSU CSE 27

slide-28
SLIDE 28

Psychology of Testing

  • Design and coding are creative activities
  • Testing is a destructive activity

– The primary goal is to “break” the software, i.e., to show that it has defects

  • Very often the same person does both

coding and testing (not a best practice)

– You need a “split personality”: when you start testing, become paranoid and malicious – It’s surprisingly hard to do: people don’t like finding out that they made mistakes

5 March 2020 OSU CSE 28

slide-29
SLIDE 29

Testing vs. Debugging

  • Goal of testing: given some code, show

by executing it that it has a defect (i.e., there is at least one situation where the code’s actual behavior is not an allowed behavior)

  • Goal of debugging: given some source

code that has a defect, find the defect and repair it

5 March 2020 OSU CSE 29

slide-30
SLIDE 30

Incorrect (Defective) Code

  • If actual behaviors are not a subset of allowed...

5 March 2020 OSU CSE 30

Allowed behaviors

  • f the method

(see contract) Actual behaviors of the method (see body)

slide-31
SLIDE 31

Incorrect (Defective) Code

Actual behaviors of the method (see body)

  • ... and we start trying some inputs and observing

results ...

5 March 2020 OSU CSE 31

Allowed behaviors

  • f the method

(see contract)

slide-32
SLIDE 32

Incorrect (Defective) Code

  • ... one might lie outside the allowed behaviors!

5 March 2020 OSU CSE 32

Actual behaviors of the method (see body) Allowed behaviors

  • f the method

(see contract)

slide-33
SLIDE 33

Incorrect (Defective) Code

  • ... one might lie outside the allowed behaviors!

5 March 2020 OSU CSE 33

Actual behaviors of the method (see body) Allowed behaviors

  • f the method

(see contract)

If this happens, testing has succeeded (in revealing a defect in the method body).

slide-34
SLIDE 34

Test Cases

  • Each input value and corresponding

allowed/expected result is a test case

  • Test cases that do not reveal a defect in

the code do not help us refute a claim of correctness

  • Test cases like that last one should be

cherished!

5 March 2020 OSU CSE 34

slide-35
SLIDE 35

Test Plan/Test Fixture

  • A set of test cases for a given unit is called

a test plan or a test fixture for that unit

5 March 2020 OSU CSE 35

slide-36
SLIDE 36

Correct Code

  • If actual behaviors are a subset of allowed...

5 March 2020 OSU CSE 36

Actual behaviors of the method (see body) Allowed behaviors

  • f the method

(see contract)

slide-37
SLIDE 37

Correct Code

  • ... and we start trying some inputs and observing

results ...

5 March 2020 OSU CSE 37

Actual behaviors of the method (see body) Allowed behaviors

  • f the method

(see contract)

slide-38
SLIDE 38

Correct Code

  • ... then we will never find a defect.

5 March 2020 OSU CSE 38

Actual behaviors of the method (see body) Allowed behaviors

  • f the method

(see contract)

slide-39
SLIDE 39

Severe Limitation of Testing

  • “Program testing can be used to show the

presence of bugs, but never to show their absence!” — Edsger W. Dijkstra (1972)

5 March 2020 OSU CSE 39

slide-40
SLIDE 40

Designing a Test Plan

  • To make testing most likely to succeed in

revealing defects, best practices include:

– Test boundary cases: “smallest”, “largest”, “special” values based on the contract – Test routine cases – Test challenging cases, i.e., ones that, if you were writing the code (maybe you didn’t write the code being tested!), you might find difficult

  • r error-prone

5 March 2020 OSU CSE 40

slide-41
SLIDE 41

Example Method Contract #1

/** * Returns some factor of a number. * ... * @requires * n > 0 * @ensures * aFactor > 0 and * n mod aFactor = 0 */ private static int aFactor(int n) {...}

5 March 2020 OSU CSE 41

slide-42
SLIDE 42

Partial Test Plan

5 March 2020 OSU CSE 42

Inputs Results Reason

n = 1 aFactor = 1 boundary n = 2 aFactor = 1 aFactor = 2 routine challenging? (prime) n = 4 aFactor = 1 aFactor = 2 aFactor = 4 challenging? (square) n = 12 aFactor = 1 aFactor = 2 aFactor = 3 aFactor = 4 aFactor = 6 aFactor = 12 routine

slide-43
SLIDE 43

Example Method Contract #2

/** * Decrements the given NaturalNumber. * ... * @updates n * @requires * n > 0 * @ensures * n = #n – 1 */ private static void decrement(NaturalNumber n) {...}

5 March 2020 OSU CSE 43

slide-44
SLIDE 44

Partial Test Plan

5 March 2020 OSU CSE 44

Inputs Results Reason

#n = 1 n = 0 boundary #n = 2 n = 1 routine #n = 10 n = 9 challenging? (borrow) #n = 42 n = 41 routine

slide-45
SLIDE 45

Partial Test Plan

5 March 2020 OSU CSE 45

Inputs Results Reason

#n = 1 n = 0 boundary #n = 2 n = 1 routine #n = 10 n = 9 challenging? (borrow) #n = 42 n = 41 routine #n = 0

What about this “boundary” case, which is on the illegal side of the “boundary” between legal and illegal inputs?

slide-46
SLIDE 46

Partial Test Plan

5 March 2020 OSU CSE 46

Inputs Results Reason

#n = 1 n = 0 boundary #n = 2 n = 1 routine #n = 10 n = 9 challenging? (borrow) #n = 42 n = 41 routine #n = 0

This test case is worthless: it violates the requires clause, so it cannot possibly reveal a defect in the method body. Why not?

slide-47
SLIDE 47

Resources

  • Software Testing (Brian Hambling, et al., 2010)

– https://library.ohio-state.edu/record=b8532947~S7

5 March 2020 OSU CSE 47