Black Box Testing (A&O Ch. 4) (2 nd Ed. Ch. 6) Course Software - - PowerPoint PPT Presentation

Black Box Testing (A&O Ch. 4)

(2nd Ed. Ch. 6)

Course Software Testing & Verification 2019/20

Wishnu Prasetya & Gabriele Keller

Plan

• Partitioning based testing (A&O Ch 4/2nd Ed
• Ch. 6).
• Model based testing

2

Note: black box testing is an important concept. For example, system level testing is often done in a black box setup. In this lecture we will discuss two important techniques that are commonly used in such a setup. The first one, partition based testing, is discussed in length in A&O. However, the second one, model based testing (MBT), is only lightly touched in A&O. This lecture will introduce you to some basic concepts of MBT. We will borrow some key concepts from Hoare’s “Communicating of Sequential Processes” (CSP). For a more practical exposition on MBT you can look at e.g. Practical Model-Based Testing by Utting

• Lageard

(UU students can access it for free from Scienc Direct, https://www.sciencedirect.com/science/book/9780123725011)

White box testing

Testing is “white box testing” if you have knowledge of the source code of the target program to help you designing the tests, and your tests have in principle access to all the program’s variables so that they can inspect them if they are in the correct state.

• Compare this with Def 1.26 A&O (2nd Ed. See p 26).
• White box setup is common at the unit-testing level.

3

Black box testing

Testing is ”black box testing” if you do not assume full knowledge

• f the inner working of the target program. Usually this also

entails that your tests have only limited access to the program state to inspect its correctness.

• Compare with Def 1.25 in A&O (2nd Ed. See p26).
• Note that although we may have full access to the source code, at the system level

we may choose not to use this knowledge because it becomes to complicated to comprehend.

Black box testing is common at the system-testing level. We will discuss two approaches : (1) Partition-based testing, Ch 4 (2nd ed.

• Ch. 6) and (2) Model-based testing.

4

Partition-based testing

• Premise:

The input space of a program can be partitioned into “equivalence classes”, such that inputs from the same partition/equivalence-class lead to the “same kind” of

• utput/effect.
• It then makes sense to require that every partition should be

tested at least once.

• Even without source code, we can often propose a reasonable

partitioning.

5

Recall this example...

Let’s now try to come up with test cases for this function from the black box perspective.

6

Enum TriType { Isosceles, Equilateral, Scalene } TryType triangle(Float a, Float b, Float c) { ... } If a, b, c represent the sides of a triangle, this methods determines the type of the triangle.

Partitioning triangle()’s domain

7

ISOSCELES EQUIDISTANCE SCALENE NON TRIANGLE

input domain of triangle

This suggests 4 test-cases, one for every partition.

Boundary value test

Errors often lurk in the “boundaries” between partitions à test values on and around the boundaries.

8

ISOLECES EQUIDISTANCE SCALENE NON TRIANGLE triangle(1,2,3) triangle(1,2,2.999) Examples:

int incomeTax(int income, int age, int children)

Example 2

9

Fragments from its informal specification: the method calculates income tax:

• Income below 10K is not taxed.
• A person under 18y or above 70y is not taxed, as long as

the income is below 50K.

• Tax reduction applies, linear to the number of children

the person has. This method has a more complex input space than the “triangle” example.

Consider the following partitioning

• As before, we can try to come up with test cases that would cover every

partition.

• Note that just 3 test cases can cover all partitions!

à does not feel very strong. This is because the approach ignores that different parameters may ”interact”. E.g. normally when the age<18 the person will not be taxed. However, the combination of income≥50k and age<18 will trigger its own behavior, namely that the person will be taxed anyway.

10

<10k [10k..50k) ≥50k <18 [18..70] >70

income age

X

The input domain of incomeTax(..)

>0

X

#children

Combinatoric testing

• In this example the input domain is spanned by 3 characteristics,

and there are in total 8 blocks.

• A block combination over the characteristics, e.g. (<10k,<18,0)

abstractly specifies a test case.

• There are in total 3x3x2 = 18 such combinations of blocks.

11

<10k [10k..50k) ≥50k <18 [18..70] >70

income age

X

incomeTax(income,age,#children)

>0

X

#children

characteristics blocks

Combinatoric testing

• Stats: 8 blocks, 18 combinations.
• (C4.24/2nd Ed. C6.2, EACH CHOICE coverage) Each block must

be tested. |T| = (max i: 0£i<k: Bi) ; usually too weak.

• (C4.23/2nd Ed. C6.1, ALL coverage) All combinations must be

tested. |T| = (P i: 0£i<k: Bi) ; does not scale up.

12

A B C

income age

X X

#children

Let’s name the blocks for convenience: A B C A B

How about to just cover all pairs?

• Premise: most of the times, only t out of N parameters actually

influence the behavior. E.g:

– when the income is less than 10k, there will be no tax, regardless the age and #children. – When the income is ≥50k, there will be tax, regardless the age.

• If we speculate on t=2, this leads to “pair-wise” testing: to cover all

block-pairs over different characteristics.

13

A B C

income age

X X

#children

A B C A B

Pair-wise and t-wise testing

• (C4.25/2nd Ed. C6.3, pair-wise coverage). Each pair of blocks (from

different characteristics) must be tested.

• There are 9+6+6 = 21 pairs to cover. But we can cover them with just

9 test cases.

• Pair-wise coverage is stronger than EACH CHOICE, and still scalable.
• (C4.26/2nd Ed. C6.4, t-wise coverage). Generalization of pair-wise.

14

A B C

income age

X X

#children

A B C A B

Example: a test set with full pair-wise coverage

15

A B C

income age

X X

#children

A B C A B income age #children Test-case 1 A A A Test-case 2 A B B Test-case 3 A C A Test-case 4 B A B Test-case 5 B B A Test-case 6 B C B Test-case 7 C A A Test-case 8 C B B Test-case 9 C C Does’t matter

PWC, example 2

• Four characteristics, each with two blocks:
• Minimal test set that gives PWC:

16

A B A B A B

cha1 cha2 tc1 A A tc2 B B tc3 A B tc4 B A

A B

cha4 B A B B cha3 A A B B tc5 A A B A

Consider again this test set

• Despite giving full pwc these tests

miss some important cases:

– We want to fully test (B,B,x) à missing (B,B,B). – We want to fully test (C,x,x) à missing e.g. (C,A,B) and (C,C,B)

• Solution: add constraints, but...

(see next slide).

17

A B C

income age

X X

#children

A B C A B income age #children tc1 A A A tc2 A B B tc3 A C A tc4 B A B tc5 B B A tc6 B C B tc7 C A A tc8 C B B tc9 C C A <10k [10k..50k) ≥50k <18 [18..70] >70 >0

Pair-wise and t-wise testing

• Consider a program P with N characteristics [1..N]. For simplicity,

suppose each characteristic is split into B number of blocks.

• Total number of pairs = B2*V2 where V2 is the number of subsets
• f size 2 out of N characteristics. So, V2 = 𝑂

2 =

!! #!∗ !%# !

• To cover all pairs you need at least B2 number of test cases.

Expect that it can be more (as in Example 2).

• In general, finding a minimum size test set that gives full t-wise

coverage is not trivial.

• However, as pointed out before, k-wise testing ignores the

“semantic” (that some combinations should be included or excluded (because they are not sensical)). We can add constraints, though this makes the problem of calculating the minimum test set even harder.

18

Adding a bit of semantic

(C4.27/2nd Ed. C6.5, Base Choice Coverage, BCC) Decide a single base test t0. Make more tests by each time removing one block from t0, and forming combinations with all remaining blocks (of the same characteristics).

19

A B C P Q R X Y Example: t0 = (B,Q,X), generates these additional test requirements : (B,Q,Y) (A,Q,X) (C,Q,X) (B,P,X) (B,R,X)

|T| = 1 + (Si : 0£i<k : Bi - 1)

Consider again

20

A B C P Q R X Y

Base test t0 = (B,Q,X), and these additional test, giving full BCC: (B,Q,Y) (A,Q,X) (C,Q,X) (B,P,X) (B,R,X)

• What if we also need to insist on testing out all

combinations of (C,P,-) ?

• Proposal: use multiple “base tests”

MBCC

21

A B C P Q R X Y Decide the “base blocks”: (red) Choose one or more best tests. They can only use base blocks. E.g.: base test t0 = (B,Q,X) base test t1 = (C,P,X) (C4.28/2nd Ed. C6.6, Multiple Base Choices coverage). For each characteristic we decide at least one base block. Then decide a set of base tests; each only include base blocks. For each base test, generate more tests by each time removing one base block, and forming combinations with remaining non-base blocks. |T| at most M + M*(Si : 0£i<k : Bi - mi)

Example MBCC

22

A B C P Q R X Y The base blocks are marked red. The base tests: base test t0 = (B,Q,X) base test t1 = (C,P,X) We need to add these tests to get full MBCC:

• Varying t0 over non-base blocks:

(A,Q,X), (B,R,X), (B,Q,Y)

• Varying t1 over non-base blocks:

(A,P,X), (C,R,X), (C,P,Y)

• Remove duplicates (there are none above)

Example-2, MBCC

23

A B C P Q R X Y Z Red : base blocks Chosen base tests = (A,P,X), (A,Q,Y) These produce these additional test requirements:

(B,P,X) (C,P,X) (B,Q,Y) (C,Q,Y) (A,R,X) (A,R,Y) (A,P,Z) (A,Q,Z)

Some properties to note:

• base-blocks are not cross-combined except as in the base tests.
• non-base blocks are not cross-combined with each other.
• BCC and MBCC will also cover every pair of (base-block,non-base-block).

Overview of partition-based coverage

24

EACH CHOICE ALL t-Wise Multiple Base Choice Coverage Pair-Wise Base Choice Coverage

(if the base tests cover all combinations of base blocks)

Models are useful

• A model helps us in understanding the program it

models.

• It can be used as a specification that defines the

correctness of the program.

• It provides guidance on how to systematically test the

program (e.g. if the model is an FSM, we can try to cover all its prime paths, rather than just trying out random paths).

25

A simple FSM modelling some program:

b a b 𝜐 a

Model based testing (MBT)

• MBT is a popular testing technique.
• Applications:

– Testing communication protocols – Testing embedded systems – Testing GUI frontends

• Typically used in a blackbox setup.
• I will explain MBT using a simplified setup, to focus
• n illustrating MBT’s main concepts such as model

“conformance”.

26

Example: synchronously interactive systems

• Consider an interactive system that interacts with its

environment by exchanging messages. We’ll ignore the direction (to or from the system). Such an exchange is atomic and synchronous (both the system and environment first agree to do the exchange and then both will wait until the exchange is completed).

• The visible behavior of such a system can be modelled by an
• FSM. Example:

27

b a b 𝜐 a Note:

• a,b represents observable messages.
• 𝜐 represents an internal action (not

visible from the environment).

MBT, basic idea

28

3 1 2 b a b a

a model M : System S

interact

The tester (a human or algorithm) takes the role of the

• environment. He/she tests S by

interacting with it to determine if S conforms M.

Black Box MBT

• MBT is often done in a black box setup where the tester

cannot see S’s actual state. This complicates testing.

29

3 1 2 b a b a

System S

interact

model M

Trace

• A trace is a sequence of messages exchanged

by an implementation or a model.

• We can reasonably assume that traces are
• bservable.
• traces(M) = the set of all traces that are

allowed by the model M.

30

Conformance

• When do we consider an implementation S to conform

a given model M? Keep in mind that in our blackbox setup we can’t peek into I’s actual state.

• Observational-based definition (Hoare, Communicating

Sequential Processes, http://www.usingcsp.com/cspbook.pdf, p39) :

• We can observe traces.

31

S conforms M = if every possible

• bservation of the behavior of S is

described by M

Conformance

• Let ∑ be the set of possible messages that a system S can do

(so-called “alphabet” of S), and also the set of messages that a model M can do.

• Let ∑* be the set of all possible finite traces over ∑. Let’s define:
• Note: this is a quite conservative definition. A system S that

does nothing will conform any model (rationale: an S that does nothing will also not do anything dangerous). A more advanced definition takes e.g. “refusals” into account (Hoare CSP p102); this will be sketched later.

32

S conforms M, iff every trace of S is also a trace of of M.

Testing Conformance

• Given the previous def. of “S conforms M”, we can test it

with a finite-sized test suite T ⊆ ∑*. For each σ∊T that S can do, we check if σ is also allowed by M.

• Since ∑* is infinite in size, we can’t test all its traces. But we

can select T e.g. such that we cover all transitions, or all prime paths in M.

33

3 1 2 b a b a

System S

interact

model M

Limitation

• Pure trace-based conformance has its limitation though. E.g.

a system S that behaves as in the left box above would be judges as conforming the model M on the right, despite that S may refuse to do anything further after an “ab” (and the model won’t).

• To remedy this, we would need more “observation”.

34

3 1 2 b a b a

System S

interact

model M

3 1 2 4 b a b 𝜐 a

Note: extending observation with refusals

• Let V⊆ ∑. (𝜏,V) is a so-called failure-trace (CSP) of M if M can do 𝜏 (so,

𝜏∈traces(M)), and afterwhich M may refuse to do any of the action in V.

• For the above M, with ∑ = {a,b}, its failure traces:

– (𝜗,∅), (b,{b}) , (ba, {a,b}) – (abk, {a}) , with k≥0 – Also: if after the trace 𝜏 M can refuse V, it can also refuse any subset of V

• An improved definition of “conform”: S conforms M iff every failure-trace of

S is also a failure-trace of M.

35

3 1 2 b a b a

System S

interact

model M

Failure-trace-based conformance

• Under trace-based conformance: S conforms M
• S may refuse {b} after abk, M cannot refuse it. So, under

failure-trace-based conformance: S does not conform M.

36

3 1 2 b a b a

System S

interact

model M

3 1 2 4 b a b 𝜐 a

Non-determinism

• Some system can be non-deterministic, e.g. due to internal

concurrency, use of random generator, probabilistic AI.

37

3 1 2 4 b a b 𝜐 a This system can non-deterministically proceed from 3 to 4. 3 1 2 4 a a b a After doing a, this system can be in either state 1 or 3 (the tester has no control on the choice)

Non-determinism

• Non-determinism complicates testing. Consider the above

model.

– Decreasing observability: suppose S can do ab. By our trace-based definition of conformance, this conforms M, because ab∊traces(M). However, if after a the system actually moves to state 4, it would be an error to subsequently do b. Unless there is a way to obtain more information about S’ state, the possibility is indistinguishable. – Decreasing control: it might be hard to try to get S to do ab (it might keep taking the 𝜐 to go to state 4).

38

3 1 2 4 b a b 𝜐 a

System S

interact

model M

Other MBT setups

• We considered 𝑡

→

'

𝑢 where a is a synchronous action of data exchange. We ignored the direction.

• Outside our scope: different systems may require different

types of model, for example:

– You may want to distinguish between actions which are inputs for the system and actions that are outputs of the system: – Actions may be parameterized. – Actions may be asynchronous.

• Different types of models lead to a different concept of

“conformance” and therefore also a different way to test it.

39

3 1 ?b ?a

The tester can control which input action to do

• n state 0.

3 1 !b !a

The tester cannot control which output the system will do on state 0.