[PPT] - Formal Testing of Formal Testing of g Distributed Systems PowerPoint Presentation

SLIDE 1

Formal Testing of Formal Testing of g Distributed Systems Distributed Systems

R. M. Hierons
R. M. Hierons

B l U i it UK B l U i it UK Brunel University, UK Brunel University, UK rob.hierons@brunel.ac.uk rob.hierons@brunel.ac.uk http://people.brunel.ac.uk/~csstrmh http://people.brunel.ac.uk/~csstrmh

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 2

Work With Work With

Mercedes

Mercedes Merayo Merayo

Manuel Nunez

Jessica Chen

Hasan

Hasan Ural Ural

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 3

Challenges in Testing Challenges in Testing

These include:

Scale

Concurrency

Distribution

The oracle problem.

Potential solution, model

Potential solution, model-based testing: based testing:

Automate testing on the basis of a formal

Automate testing on the basis of a formal model or specification. model or specification.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 4

Model Based Testing Model Based Testing

W l b i t ti b t th W l b i t ti b t th

We only observe interactions between the

We only observe interactions between the system under test (SUT) and its environment. system under test (SUT) and its environment.

To reason about test effectiveness we

To reason about test effectiveness we assume: assume:

The behaviour of the SUT can be expressed in the

The behaviour of the SUT can be expressed in the same language as the model. same language as the model.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 5

Models for distributed and Models for distributed and networked systems networked systems

Such systems typically:

Have states and actions

Are concurrent

If we take a black

If we take a black-box view, the last issue box view, the last issue is less important is less important is less important is less important

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 6

Formal languages Formal languages

Typically use states and transitions

Typically use states and transitions b t t t t i d b ti b t t t t i d b ti between states triggered by actions. between states triggered by actions.

Many can be seen as one of:

Finite state machines

Labelled

Labelled transition systems (and input output transition systems (and input output transition systems) transition systems) transition systems) transition systems)

Former less general but the models are

Former less general but the models are easier to easier to analyse analyse.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 7

Multi Multi-port systems port systems

Physically distributed interfaces/ports.

A tester at each port.

tester tester SUT tester

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 8

Distributed testing Distributed testing

Mainly focus on the simplest approach:

The testers cannot communicate with one

The testers cannot communicate with one another another

There is no global clock

Observations are ‘local’

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 9

Motivation Motivation

Initially just testing/test generation.

Initially just testing/test generation. Th di i ill b d b th Th di i ill b d b th

The discussion will be around both

testing

testing and and

implementation/conformance relations

implementation/conformance relations.

Testing from:

input output transition systems and possibly

input output transition systems and possibly

d t i i ti fi it t t hi d t i i ti fi it t t hi

deterministic finite state machines

nondeterministic finite state machines

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 10

Testing and Testing and Testing and Testing and Observations Observations

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 11

Global Traces Global Traces

A global trace is a sequence of inputs and

utputs
utputs
utputs.
utputs.

We assume there are m ports and:

xp will denote an input at port p (from X

will denote an input at port p (from Xp)

(y

(y1,...,y ,...,ym) ∈Y, Y= Y, Y=(Y (Y1∪{-})×…×(Y (Ym∪{-}), will be }), will be an output an output

A l b l t i l t f (X A l b l t i l t f (X Y)* Y)*

A global trace is an element of (X

A global trace is an element of (X × Y)* Y)*

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 12

Consequences Consequences

Each tester observes only the interactions (

Each tester observes only the interactions (local local trace trace) at its port ) at its port trace trace) at its port ) at its port

Tester 1 SUT Tester 2 x1 x1 y1 y2

The tester at port 1 observes x

The tester at port 1 observes x1y1x1y1 and the and the tester at 2 observes y tester at 2 observes y2 only.

nly.

y1

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 13

What the testers observe What the testers observe

Given global trace z, the tester at p

Given global trace z, the tester at p b l l t b l l t ( ) ( )

bserves a local trace
bserves a local trace πp(z) .

(z) .

Tester 1 Tester 2 x1 x1 y1 y2

Networked and Distributed Networked and Distributed Systems Systems

y1

SLIDE 14

Controllability problems Controllability problems

The following test has a controllability

The following test has a controllability bl i t d d t i i i t bl i t d d t i i i t problem: introduces nondeterminism into problem: introduces nondeterminism into testing. testing.

tester SUT tester

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 15

Observability Observability problems problems

The following look the same

tester Spec tester tester SUT tester x1 x1 x1 x1 y1 y1 y2

Testers/users cannot ‘map’ output to input

1

y1 y1 y2

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 16

Equivalent global traces Equivalent global traces

Since we only observe local traces:

Global traces and ’ are indisting ishable if Global traces and ’ are indisting ishable if

Global traces z and z’ are indistinguishable if

Global traces z and z’ are indistinguishable if their projections are identical: the local traces their projections are identical: the local traces are the same. are the same.

We denote this: z

We denote this: z∼z’ z’

The following are equivalent under

The following are equivalent under ∼

x /(y /(y y )x )x /(y /(y )

x1/(y

/(y1,y ,y2)x )x1/(y /(y1,-)

x1/(y

/(y1,-)x )x1/(y /(y1, y , y2)

Both have x

Both have x1y1x1y1 at port 1 and y at port 1 and y2 at 2. at 2.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 17

Problem: Test effectiveness is not Problem: Test effectiveness is not monotonic monotonic

Example: x

Example: x1 detects a fault but x detects a fault but x1x1 does does t not. not.

tester SUT tester tester Spec tester x1 x1 x y1 y1 y2 x1 x1 y1 y1 y2

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 18

Two approaches to defining Two approaches to defining implementation relations implementation relations

We might have:

Agents at ports are entirely ‘independent’:

No external agent can receive information

No external agent can receive information regarding observations at more than one port regarding observations at more than one port

Or the local traces observed at the ports can

Or the local traces observed at the ports can be ‘brought together’ later. be ‘brought together’ later.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 19

Differences Differences

Specification

Tester 1 Tester 2 Tester 1 Tester 2

SUT

x y z Tester 1 Tester 2 x y' z' Tester 1 SUT Tester 2

Networked and Distributed Networked and Distributed Systems Systems

x y' z

SLIDE 20

Using an external network Using an external network

If we connect the testers using an external

If we connect the testers using an external t k t k ti ti network, network, sometimes sometimes we can overcome we can overcome controllability and observability problems. controllability and observability problems.

tester SUT tester tester SUT tester

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 21

But But

If a system has physically distributed

If a system has physically distributed i t f th th i l t ti l ti i t f th th i l t ti l ti interfaces then the implementation relation interfaces then the implementation relation should reflect this: should reflect this:

Even if we can connect the testers, we should

Even if we can connect the testers, we should be careful that we do not give the verdict fail be careful that we do not give the verdict fail when the when the behaviour behaviour is acceptable in use. is acceptable in use.

The users will only observe local traces

The users will only observe local traces.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 22

Past research Past research

Mainly on testing from a deterministic finite

Mainly on testing from a deterministic finite state machine (DFSM): state machine (DFSM): state machine (DFSM): state machine (DFSM):

Generating test sequences that do not suffer

Generating test sequences that do not suffer from controllability and/or observability from controllability and/or observability problems problems

Adding coordination messages (possibly

Adding coordination messages (possibly adding a minimum number). adding a minimum number).

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 23

Problems/issues Problems/issues

A DFSM can have transitions that can’t be

A DFSM can have transitions that can’t be executed without controllability problems executed without controllability problems executed without controllability problems. executed without controllability problems.

Test generation algorithms place

Test generation algorithms place conditions on the DFSM conditions on the DFSM – – they are not they are not general. general.

The methods test against the ‘traditional’

The methods test against the ‘traditional’ i l t ti l ti i l t ti l ti i i t d t i i t d t implementation relation implementation relation – aiming to do too aiming to do too much? much?

Using DFSMs is restrictive.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 24

The solution The solution

We need a good understanding of what it

We need a good understanding of what it t di ti i h t d l ith t di ti i h t d l ith means to distinguish two models with means to distinguish two models with distributed ports. distributed ports.

This gives us new implementation

This gives us new implementation relations relations relations. relations.

We want to test against these.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 25

Input Output Transition Input Output Transition Input Output Transition Input Output Transition Systems (IOTSs) Systems (IOTSs)

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 26

The models The models

These are

These are labelled labelled transition systems in transition systems in hi h di ti i h b t i t d hi h di ti i h b t i t d which we distinguish between input and which we distinguish between input and

utput.
utput.

We have states and transitions between

We have states and transitions between the states. the states.

Notation:

Normally we precede the name of an input by

Normally we precede the name of an input by ? and the name of an output by !. ? and the name of an output by !.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 27

Internal events and quiescence Internal events and quiescence

We have two special types of events:

Internal events (

Internal events (τ) are state transitions that do ) are state transitions that do not require input and do not produce output. not require input and do not produce output.

A state

A state s is quiescent if from is quiescent if from s output cannot

utput cannot

be produced without first providing input. be produced without first providing input.

If

If s is quiescent then we add a self is quiescent then we add a self-loop loop transition from transition from s with label with label δ.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 28

A simple example A simple example

A (very) simple coffee machine

?1 !tea ?2 !coffee

We have not shown the self

We have not shown the self-loops for loops for quiescence. quiescence.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 29

IOTS models IOTS models

IOTS models are more general than

IOTS models are more general than FSMs FSMs: FSMs FSMs:

They can be infinite state models

Input and output need not alternate

There can be internal (unobservable) actions.

We assume:

IOTSs are input enabled

We can observe quiescence

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 30

Implementation relations Implementation relations

There is a standard implementation

There is a standard implementation l ti (f t ti ) ll d l ti (f t ti ) ll d i relation (for testing) called relation (for testing) called ioco ioco

It requires:

If

If σ is a (suspension) trace of the specification is a (suspension) trace of the specification s and the implementation can produce output and the implementation can produce output s and the implementation can produce output and the implementation can produce output !o

after

after σ then then s must be able to produce must be able to produce

utput !
utput !o
after

after σ

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 31

Correct implementations? Correct implementations?

?1 !tea ?1 !coffee ?1 !tea ?1 ?1 ?1 !tea ?3 !choc

Networked and Distributed Networked and Distributed Systems Systems

?1 !tea ?1

SLIDE 32

Two equivalent processes Two equivalent processes

We cannot distinguish the following:

?i1 !o1 !o2 ?i1 !o2 !o1

Note: assume processes completed to

Note: assume processes completed to make them input make them input-enabled. enabled.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 33

Issue Issue

When can we ‘bring together’ local

When can we ‘bring together’ local b ti ’? b ti ’?

bservations’?
bservations’?

?i1 !o1 !o2 ?i1 !o2 !o1

In this example not after ?i

In this example not after ?i1!o !o1 or ?i

r ?i1!o

!o2

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 34

When do we make observations? When do we make observations?

For an FSM we observe the projections of

For an FSM we observe the projections of input/output sequences input/output sequences we can ‘stop’ we can ‘stop’ input/output sequences input/output sequences - we can stop we can stop after an input/output sequence. after an input/output sequence.

When can we ‘stop’ when considering

When can we ‘stop’ when considering IOTSs? Possibly: IOTSs? Possibly:

Whenever we have quiescence.

We can then ‘bring together local traces’

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 35

An implementation relation An implementation relation dioco dioco

We say that i dioco s if:

For every trace z of i that can take i to a

For every trace z of i that can take i to a quiescent state, there is some trace z’ of s quiescent state, there is some trace z’ of s such that z’ such that z’ ∼ z. z.

Thi Thi

This means:

If i has a ‘run’ z that ends in quiescence then

If i has a ‘run’ z that ends in quiescence then s has a specified behaviour that is ‘equivalent’ s has a specified behaviour that is ‘equivalent’ to z. to z.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 36

dioco dioco does not imply does not imply ioco ioco

Example:

?i1 !o1 !o2 ?i1 !o2 !o1

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 37

Result Result

If s and

If s and i are input enabled then: are input enabled then:

i ioco ioco s implies that s implies that i dioco dioco s

i ioco

ioco s implies that s implies that i dioco dioco s

Normally IOTS implementations are

Normally IOTS implementations are required to be input enabled. required to be input enabled.

So:

For input enabled specifications we have that

For input enabled specifications we have that dioco dioco is weaker than is weaker than ioco ioco.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 38

Test cases Test cases

These can be defined as processes that

These can be defined as processes that i t t ith th SUT i t t ith th SUT can interact with the SUT. can interact with the SUT.

We can have:

A global tester that interacts with every port

One local tester for each port.

In our context we cannot implement a

In our context, we cannot implement a

In our context, we cannot implement a global tester (but we can map it to a set of global tester (but we can map it to a set of local testers). local testers).

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 39

Controllability Controllability

A local tester observes only the events at

A local tester observes only the events at it t it t its port. its port.

As a result, if it has to supply an input then

As a result, if it has to supply an input then it can only know when to do this on the it can only know when to do this on the basis of its observations basis of its observations basis of its observations. basis of its observations.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 40

A controllability problem A controllability problem

The tester at port 2 does not know when to send

The tester at port 2 does not know when to send its input its input its input. its input.

tester SUT tester

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 41

The effect of The effect of nondeterminism nondeterminism

We might have pairs of allowed traces with

We might have pairs of allowed traces with fi lik th f ll i fi lik th f ll i prefixes like the following: prefixes like the following:

tester Spec tester tester Spec tester x1 x1 x1 x1 y1 y2 x1 x1 y1 y1 y2 y2 x2 x2

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 42

Choice Choice

A tester makes a choice based on its

A tester makes a choice based on its b ti b ti

bservations.
bservations.

This is the notion of ‘local choice’.

Also studied

Also studied in the context of Message in the context of Message Sequence Charts (e.g. non Sequence Charts (e.g. non-local choice local choice pathologies) pathologies) pathologies). pathologies).

Difference in problems considered and our

Difference in problems considered and our problem has additional ‘structure’ problem has additional ‘structure’

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 43

Defining controllability Defining controllability

A test case t is controllable if each tester

A test case t is controllable if each tester k ‘l l h i ’ k ‘l l h i ’ can make ‘local choices’ can make ‘local choices’

there should not be two prefixes z and z’ of traces

there should not be two prefixes z and z’ of traces that can be produced using t that look the same to that can be produced using t that look the same to a tester at port p and yet this tester should behave a tester at port p and yet this tester should behave differently after these. differently after these.

Result:

We can decide in polynomial time whether a

We can decide in polynomial time whether a test case is controllable. test case is controllable.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 44

Additional implementation Additional implementation relations? relations?

In dioco we assume traces can be brought

In dioco we assume traces can be brought t th t th d f t ti t th t th d f t ti together at the end of testing. together at the end of testing.

We have allowed the use of test case with

We have allowed the use of test case with controllability problems. controllability problems.

So there are alternative implementation

So, there are alternative implementation

So, there are alternative implementation relations. relations.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 45

An example An example

We can require that local traces are not

We can require that local traces are not brought together brought together brought together. brought together.

Makes sense if this corresponds to

Makes sense if this corresponds to expected usage. expected usage.

We require:

For every trace z of the implementation and

For every trace z of the implementation and port p there is a trace z’ of the specification port p there is a trace z’ of the specification such that such that πp(z)= (z)=πp(z’) (z’)

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 46

Can be weaker Can be weaker

Specification and implementation

?i1 !o2 !o1 ?i1 !o2 !o’1 ?i1 !o’2 !o’1

Looks ok if we cannot bring together local

Looks ok if we cannot bring together local traces. traces.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 47

Can be stronger Can be stronger

No quiescence:

!o2 !o1

Suggests: only allowing traces ending in

Suggests: only allowing traces ending in quiescence is problematic. quiescence is problematic.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 48

Additional alternatives Additional alternatives

Instead of only considering quiescent

Instead of only considering quiescent t ld t ld traces we could: traces we could:

Combine (conjoin) the previous two

Combine (conjoin) the previous two implementation relations. implementation relations.

Consider infinite traces.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 49

Using infinite traces Using infinite traces

We can compare the infinite traces of the

We can compare the infinite traces of the i l t ti ith th f th i l t ti ith th f th implementation with those of the implementation with those of the specification. specification.

This is an answer to ‘when do we bring

This is an answer to ‘when do we bring together local traces’. together local traces’.

In practice we will have to define

In practice we will have to define conservative decision procedures for conservative decision procedures for

racles.
racles.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 50

Other Types of Models Other Types of Models

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 51

The following are equivalent The following are equivalent

!o

!o1!o !o2, !o , !o2!o !o1

!o

!o1!o !o1!o !o2, !o , !o2!o !o1!o !o2

….

(!o

(!o1)1000

1000!o

!o2, !o , !o2(!o (!o1)1000

1000

….

When does this stop being reasonable?

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 52

One possible approach One possible approach

We could include time in our model.

Problem:

Local clocks need not

Local clocks need not synchronise synchronise.

We might have e.g.:

bounds in drift,

bounds in drift, i f ti b t ti t k b i f ti b t ti t k b

information about time taken by messages,

messages between testers

This is future work.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 53

Using scenarios Using scenarios

An alternative:

Allow the users and testers to effectively

Allow the users and testers to effectively synchronise synchronise at certain points. at certain points.

We can

consider

consider scenarios scenarios and; and;

add explicit

add explicit synchronisation synchronisation points in a points in a

add explicit

add explicit synchronisation synchronisation points in a points in a specification. specification.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 54

Adding probabilities Adding probabilities

Some systems have probabilistic

Some systems have probabilistic i t i t requirements. requirements.

We can add probabilities to transitions.

It is straightforward to extend

It is straightforward to extend IOTSs IOTSs to to probabilistic probabilistic IOTSs IOTSs probabilistic probabilistic IOTSs IOTSs.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 55

A Generative Approach A Generative Approach

In a state

In a state s the sum of probabilities of transitions the sum of probabilities of transitions leaving leaving s add up to 1 add up to 1 leaving leaving s add up to 1. add up to 1.

The implementation relations are similar to

The implementation relations are similar to dioco dioco – we just add requirements regarding we just add requirements regarding probabilities. probabilities. H if h i t d t t thi H if h i t d t t thi

However, if we have inputs and outputs this

However, if we have inputs and outputs this approach requires us to have probabilistic approach requires us to have probabilistic information regarding the environment. information regarding the environment.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 56

A reactive/generative approach A reactive/generative approach

Instead we can assume that:

There is no probabilistic information regarding

There is no probabilistic information regarding inputs from the environment (a reactive inputs from the environment (a reactive approach). approach).

In state

In state s, the sum of the probabilities of , the sum of the probabilities of

utputs from the SUT (including
utputs from the SUT (including δ) is 1:

) is 1:

utputs are generative.
utputs are generative.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 57

Probabilities of observations Probabilities of observations

Consider the following

?i1 !tea ?i2 !coffee ?i1 ?i2

What is the probability of observing !coffee

What is the probability of observing !coffee after ?i after ?i1?i ?i2

Networked and Distributed Networked and Distributed Systems Systems

!coffee

SLIDE 58

The problem The problem

We can have races between events at

We can have races between events at diff t t diff t t different ports. different ports.

We have no probabilistic information

We have no probabilistic information regarding the outcome of these races. regarding the outcome of these races.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 59

Possible solutions Possible solutions

Two alternatives:

Outlaw such situations (effectively say that we

Outlaw such situations (effectively say that we know nothing about the probabilities). know nothing about the probabilities).

Assume that the (unknown) environment has

Assume that the (unknown) environment has such probabilities and define corresponding such probabilities and define corresponding implementation relations. implementation relations.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 60

Finite State Machines Finite State Machines

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 61

Finite State Machines Finite State Machines

The behaviour of M in state s

The behaviour of M in state si is defined by the set is defined by the set

f input/output sequences (traces) from s
f input/output sequences (traces) from si

s2 s3 s5

a/0 a/0 a/1 b/0 b/1 b/1

f input/output sequences (traces) from s
f input/output sequences (traces) from si

s1 s4

a/0 a/1 b/0 b/1

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 62

An implementation relation for An implementation relation for distributed systems distributed systems

We say that DFSM N conforms to DFSM M if:

Every global trace of N is indistinguishable from a

Every global trace of N is indistinguishable from a global trace of M. global trace of M.

Equivalently:

For every global trace z of N there is a global trace z’

f M such that z
f M such that z ∼ z’.

z’.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 63

Conformance is weaker than Conformance is weaker than equivalence equivalence

This also shows that it is not an equivalence

This also shows that it is not an equivalence relation (second can have output y relation (second can have output y ) relation (second can have output y relation (second can have output y2). ).

s1 s2

x1/(y1,-) x2/(-, y2) x1/(y1,-) x1/(y1,-)

s1

x2/(-, y’2) x1/(y1,-)

Conforms to

Is the first an acceptable design for second?

s3

x2/(-, y’2) x2/(-, y’2)

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 64

Key components of testing Key components of testing

When testing from an FSM we want to be

When testing from an FSM we want to be bl t bl t able to: able to:

Reach states

Distinguish states (and machines)

Check output against the specification (oracle

Check output against the specification (oracle problem). problem). p ) p )

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 65

The Oracle Problem The Oracle Problem

For DFSMs this:

Can be solved in polynomial time for

Can be solved in polynomial time for controllable test sequences controllable test sequences

Otherwise is NP

Otherwise is NP-hard hard

For NFSMs:

NP

NP-hard even for controllable testing hard even for controllable testing

NP

NP hard even for controllable testing hard even for controllable testing

Polynomial if we restrict further

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 66

Reaching and distinguishing Reaching and distinguishing states states

Problem

Is there a strategy for each tester that leads to

Is there a strategy for each tester that leads to testing taking the FSM to a particular state (or testing taking the FSM to a particular state (or distinguishes two states)? distinguishes two states)?

This problem is

This problem is undecidable undecidable.

This problem is

This problem is undecidable undecidable.

Decidable for controllable testing from a

Decidable for controllable testing from a DFSM (result does not hold for NFSMs). DFSM (result does not hold for NFSMs).

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 67

Controllable testing Controllable testing

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 68

Distinguishing states Distinguishing states

If we restrict ourselves to controllable testing we

If we restrict ourselves to controllable testing we need: need: need: need:

x causes

x causes no controllability problems no controllability problems from s and s’ from s and s’

x leads to different sequences of interactions, for s

x leads to different sequences of interactions, for s and s’, at and s’, at some port some port.

We say that x

We say that x locally s locally s-distinguishes distinguishes s and s’. s and s’.

If no input sequence locally distinguishes s and

If no input sequence locally distinguishes s and ’ th ’ th l ll l ll i l t i l t s’ they are s’ they are locally s locally s-equivalent equivalent.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 69

Testing is weaker Testing is weaker

We cannot locally s

We cannot locally s-distinguish s distinguish s1 and s and s4 but x but x1x2 locally distinguishes them locally distinguishes them but x but x1x2 locally distinguishes them. locally distinguishes them.

s1 s2

x1/(y1,-) x1/(y1,-) x2/(-, y2) x2/(y1, y2) x2/(y1,-)

s4 s3

x1/(y1,-) x1/(y1,-) x2/(-, y2)

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 70

Distinguishing two states Distinguishing two states

Given port p and states s

Given port p and states s1 and s and s2 of a m

f a m-port

port FSM M with n states: FSM M with n states: FSM M with n states: FSM M with n states:

s1 and s

and s2 are locally s are locally s-distinguishable by an input distinguishable by an input sequence starting at p if and only if they are locally s sequence starting at p if and only if they are locally s- distinguished by some such input sequence of length distinguished by some such input sequence of length at most m(n at most m(n-1). 1).

This bound is ‘tight’.

The sequences can be found in low

The sequences can be found in low-order

rder

polynomial time. polynomial time.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 71

Minimality Minimality

Two possible definitions:

Def 1: A DFSM is locally s

Def 1: A DFSM is locally s-minimal if it has no minimal if it has no locally s locally s-equivalent states. equivalent states.

Def 2: A DFSM M is locally s

Def 2: A DFSM M is locally s-minimal if no minimal if no DFSM with fewer states is locally s DFSM with fewer states is locally s-equivalent equivalent to M. to M.

For initially

For initially-connected, completely connected, completely specified, single specified, single-port DFSMs, these are port DFSMs, these are the same. the same.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 72

Minimal DFSMs are not always Minimal DFSMs are not always locally s locally s-minimal minimal

We have seen that s

We have seen that s1 and s and s4 are locally s are locally s- i l t i l t equivalent equivalent

s1 s2

x1/(y1,-) x1/(y1,-) x2/(-, y2) x2/(y1, y2) x2/(y1,-)

s4 s3

x1/(y1,-) x1/(y1,-) x2/(-, y2)

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 73

Merging s Merging s-equivalent states equivalent states

A smaller acceptable design?

s1 s2

x1/(y1,-) x1/(y1,-) x1/(y1,-) x2/(-, y2) x2/(-, y2) x2/(y1,-)

s1 s2

x1/(y1,-) x1/(y1,-) x /(y ) x2/(-, y2) x2/(-, y2) x2/(-, y2) x2/(y1,-)

s3 s4 s3

x1/(y1,-) x1/(y1,-)

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 74

Minimising: smallest FSM Minimising: smallest FSM

Even smaller:

s1

x1/(y1,-) x2/(-, y2)

s1 s2

x1/(y1,-) x1/(y1,-) x2/(-, y2) x2/(-, y2) x2/(y1,-)

s4 s3

x1/(y1,-) x1/(y1,-) x2/(-, y2)

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 75

Consequences Consequences

We had two alternative definitions.

Def 1: A DFSM is locally s

Def 1: A DFSM is locally s-minimal if it has no minimal if it has no locally s locally s-equivalent states. equivalent states.

Def 2: A DFSM M is locally s

Def 2: A DFSM M is locally s-minimal if no minimal if no DFSM with fewer states is locally s DFSM with fewer states is locally s-equivalent equivalent to M. to M.

For multi

For multi-port DFSMs these differ. port DFSMs these differ.

Def 2 is ‘better’?

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 76

Canonical FSMs Canonical FSMs

Given DFSM M, we can find:

Maximal M

Maximal Mmax

max that is locally s

that is locally s-equivalent to M equivalent to M

Minimal M

Minimal Mmin

min that is locally s

that is locally s-equivalent to M equivalent to M

W fi d th ffi i tl W fi d th ffi i tl

We can find them efficiently.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 77

Results Results

DFSM N is locally s

DFSM N is locally s-equivalent to DFSM M equivalent to DFSM M if and only if N is a reduction of M if and only if N is a reduction of Mmax

max.

The set of DFSMs that are s

The set of DFSMs that are s-equivalent to equivalent to a DFSM M forms a bounded lattice a DFSM M forms a bounded lattice a DFSM M forms a bounded lattice. a DFSM M forms a bounded lattice.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 78

Refinement and testing Refinement and testing

We now know that:

FSM M FSM Mmax s-equivalence reduction Implementation N

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 79

Summary: controllable testing Summary: controllable testing

Benefits of restricting to controllable test

Benefits of restricting to controllable test f DFSM f DFSM sequences for DFSMs sequences for DFSMs

Oracle problem can be solved in polynomial

Oracle problem can be solved in polynomial time time

Have unique ‘min’ and ‘max’ machines

Can test against ‘max’ model for reduction

Can test against ‘max’ model for reduction g using traditional methods using traditional methods

Could develop from ‘max’ model?

However: limits testing

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 80

Future work Future work

Generating test cases to satisfy a test criterion

Generating test cases to satisfy a test criterion.

Generating complete test suites.

Minimising an FSM.

Testing using coordination messages but the ‘new’

Testing using coordination messages but the ‘new’ implementation relations implementation relations

Timed models.

Timed models. Enriching models with data stochastic time Enriching models with data stochastic time

Enriching models with data, stochastic time, ...

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 81

Papers ( Papers (FSMs FSMs)

B. Sarikara and G. Von Bochmann, Synthesis and Specification Issues
B. Sarikara and G. Von Bochmann, Synthesis and Specification Issues

in Protocol Testing, in Protocol Testing, IEEE Transactions on Communications IEEE Transactions on Communications, 3 , 32 4, pp. 4, pp. 389 389-395: 1984. 395: 1984. R Dssouli and G von Bochmann R Dssouli and G von Bochmann Error detection with multiple Error detection with multiple

R. Dssouli and G. von Bochmann.
R. Dssouli and G. von Bochmann. Error detection with multiple

Error detection with multiple

bservers,
bservers, Protocol Specification, Testing and Verification V

Protocol Specification, Testing and Verification V, pp. 483 , pp. 483- 494: 1985. 494: 1985.

R. Dssouli and G. von Bochmann,.
R. Dssouli and G. von Bochmann,. Conformance testing with multiple

Conformance testing with multiple

bservers,
bservers, Protocol Specification, Testing and Verification VI

Protocol Specification, Testing and Verification VI, pp. 217 , pp. 217- 229: 1986. 229: 1986.

J. Chen, R. M. Hierons, and H. Ural. Overcoming observability problems
J. Chen, R. M. Hierons, and H. Ural. Overcoming observability problems

in distributed test architectures in distributed test architectures, Information Processing Letters , Information Processing Letters, 98 98, pp. , pp. 177 177-182: 2006. 182: 2006.

R. M. Hierons and H. Ural. The effect of the distributed test architecture
R. M. Hierons and H. Ural. The effect of the distributed test architecture
n the power of testing,
n the power of testing, The Computer Journal

The Computer Journal, 51 51 4, pp. 497 4, pp. 497-510: 510: 2008. 2008.

R. M. Hierons: Canonical Finite State Machines for Distributed Systems,
R. M. Hierons: Canonical Finite State Machines for Distributed Systems,

Theoretical Computer Science Theoretical Computer Science, 411 411 2, pp. 566 2, pp. 566-580: 2010. 580: 2010.

R.M. Hierons: Reaching and Distinguishing States of Distributed

R.M. Hierons: Reaching and Distinguishing States of Distributed Systems, Systems, SIAM Journal of Computing SIAM Journal of Computing (to appear) (to appear)

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 82

Papers ( Papers (IOTSs IOTSs)

R. M. Hierons, M. G.
R. M. Hierons, M. G. Merayo

Merayo, and M. , and M. Nunuez

Nunuez. Implementation relations

. Implementation relations for the distributed test architecture, for the distributed test architecture, 20th FIP International Conference 20th FIP International Conference

n Testing Communicating Systems (
n Testing Communicating Systems (TestCom

TestCom 2008) 2008), LNCS 5074, pp. , LNCS 5074, pp. g g y ( g g y ( ) pp pp 200 200-215: 2008. 215: 2008.

R. M. Hierons, M. G.
R. M. Hierons, M. G. Merayo

Merayo, and M. Nunez. Controllable test cases for , and M. Nunez. Controllable test cases for the distributed test architecture, the distributed test architecture, 6th International Symposium on 6th International Symposium on Automated Technology for Verification and Analysis (ATVA 2008) Automated Technology for Verification and Analysis (ATVA 2008), LNCS volume 5311, pp. 201 LNCS volume 5311, pp. 201-215: 2008. 215: 2008.

R. M. Hierons and M.
R. M. Hierons and M. Núñez

Núñez: Scenarios : Scenarios-based Testing of Systems with based Testing of Systems with distributed Ports, distributed Ports, The 10th International Conference on Quality Software The 10th International Conference on Quality Software (QSIC 2010) (QSIC 2010), 2010. , 2010.

R. M. Hierons and M.
R. M. Hierons and M. Núñez

Núñez: Testing probabilistic distributed systems, : Testing probabilistic distributed systems, 30th IFIP Formal Techniques for Networked and Distributed Systems 30th IFIP Formal Techniques for Networked and Distributed Systems 30th IFIP Formal Techniques for Networked and Distributed Systems 30th IFIP Formal Techniques for Networked and Distributed Systems (FORTE 2010) (FORTE 2010), LNCS, 2010. , LNCS, 2010.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 83

Conclusions Conclusions

If a system has distributed interfaces/ports

If a system has distributed interfaces/ports then we have different implementation then we have different implementation then we have different implementation then we have different implementation relations. relations.

This can affect testing but also

This can affect testing but also development. development.

We get new notions of e.g. a design being

We get new notions of e.g. a design being i i l i i l minimal. minimal.

The effect is even greater for

The effect is even greater for nondeterministic models/systems. nondeterministic models/systems.

Networked and Distributed Networked and Distributed Systems Systems

SLIDE 84

Questions? Questions?

Networked and Distributed Networked and Distributed Systems Systems