CSE3009: (Software Architecture and Design) Yann-Gal - - PowerPoint PPT Presentation

CSE3009: 소프트웨어 구조및설계

(Software Architecture and Design)

Yann-Gaël Guéhéneuc

Département de génie informatique et de génie logiciel

Testing Method Sequences Slides by Giuliano Antoniol

Example I

Coin box of a vending machine in C++

– The coin box has a simple functionality and the code to control the physical device is omitted

2/71

– It accepts only quarters and allows vending when two quarters are received – It keeps track of the total of quarters received (totalQrts) , the current quarters received (curQrts), and whether vending is enabled (allowVend)

Example I

Coin box of a vending machine in C++

– Methods: adding a quarter, returning current quarters, resetting the coin box to its initial state, and vending

3/71

and vending

Example I

class CCoinBox { unsigned totalQtrs; unsigned curQtrs; unsigned allowVend; public: Ccoinbox() {Reset();} void AddQtrs(); void ReturnQtrs() { void CCoinBox::AddQtr() { curQtrs = curQtrs + 1; if (curQtrs > 1) allowVend = 1; } void CCoinBox::Vend() { if (isAllowedVend()) { totalQtrs = totalQtrs + 2;

4/71

void ReturnQtrs() { curQtrs = 0; } unsigned isAllowedVend() { return allowVend; } void Reset() { totalQtrs = 0; allowVend = 0; curQtrs = 0; } void Vend(); }; totalQtrs = totalQtrs + 2; curQtrs = curQtrs - 2; if (curQtrs < 2) allowVend = 0; } }

UML Statechart Transitions

State transitions caused by events, enabled

by guard conditions

Engine Off

start [transmission= neutral]

Engine On

5/71

have the following general form (all elements are optional)

event(arguments) [condition] ^target.sendEvent(arguments) /method(arguments)

event name guard executed method trigger

}

event dispatch Engine Off Engine On

UML Statechart Transitions

Reset()

Functioning

AddQtr ReturnQtr() AddQtr() Vend() Vend()

CCoinBox()

Vend() [self.curQrts = 3] Vend() [self.curQrts = 2] allowVend = 0 curQrts = 0 allowVend = 0 curQrts = 1 Vend() 6/71

• Statechart of the code, not fully specified
• One corrupt state missing: allowVend=1 curQrts=1
• Transitions missing on states

– allowVend=1 curQrts=0 : AddQrts() – allowVend=0 curQrts=0 : ReturnQrt() – allowVend=1 curQrts:[2,M] : AddQrt() ReturnQtr() Vend() [self.curQrts >= 4] ReturnQtr() allowVend = 1 curQrts = 0 allowVend = 1 curQrts:[2,M]

Example I

The state diagram indicates that the

scenario

– AddQtrs()

7/71

– AddQtrs() – ReturnQtrs() – Vend()

is possible!

⇒ Free drinks! ☺

Example I

This is a state-dependent fault

– Executing some methods is impossible in certain states where they are legal

8/71

– Some methods sequences that are theoretically not possible can be executed

Example I

Method structural testing would conclude

that, e.g., each edge has been executed

The omission may remain unforeseen even

9/71

with functional testing

Problem: a failure results from a certain sequence of methods Solution: identify minimal set of sequences of methods

Testing Method Sequences

Motivations Data Slices

– MaDUM

10/71

– MaDUM – MaDUM for Derived Classes

Method Per- / Post-conditions Conclusion

Testing Method Sequences

Motivations Data Slices

– MaDUM

11/71

– MaDUM – MaDUM for Derived Classes

Method Per- / Post-conditions Conclusion

Motivations

Using inheritance context coverage, control-

and data-flow techniques can be used to test methods

12/71

What about classes?

Motivations

Testing a class aims at finding the sequence

• f methods for which a class will be in a

state that is contradictory to its invariant or to

13/71

its methods outputs

Motivations

Testing classes for all possible sequences of

methods is not usually possible

– 10 methods ⇒ 10! sequences

14/71

– If each test requires 1s., we need 1,000 hours

Motivations

The resources required to test a class

increase exponentially with the number of its methods

15/71

It is necessary to devise a way to reduce the

number of sequences and still provide sufficient confidence

Motivations

Deriving method sequences

– Data slices – Statecharts

16/71

– Methods pre- / post-conditions – Formal specification, e.g., algebraic specification

Testing Method Sequences

Motivations Data Slices

– MaDUM

17/71

– MaDUM – MaDUM for Derived Classes

Method Per- / Post-conditions Conclusion

Data Slices

Goal: identify a minimal number of method

sequences to test

18/71

The state of an object at a single point in

time is equivalent to the aggregated state of each of its data members at that point

Data Slices

The correctness of a class depends on

– Whether the instance variables are correctly representing the intended state of an object

19/71

– Whether the methods are correctly manipulating the representation of the object

Data Slices

A class is a composition of slices

– A slice is a quantum of a class with only a single instance variable and a set of methods, such that each methods manipulate the values

20/71

that each methods manipulate the values associated with this variable

Data Slices

Bashir and Goel’s class testing strategy is to

test one slice at a time

– For each slice, test possible sequences of the

21/71

methods belonging to that slice

• Equivalent to testing a specific instance variable, i.e.,

the class partial correctness

– Repeat for all slices to demonstrate class correctness

Data Slices

A class K encapsulates a set of variables

and provides a set of methods

– K = <D(K), M(K)>

22/71

– D(K) = {di | di @ K} – M(K) = {mi | mi @ K} (where @ denotes the relationship between a class and its elements, i.e., “is in”, “belongs to”)

Data Slices

A data slice of K

– Slicedi(K) = <di, Mdi(K)>

• di ∈ D(K)

23/71

• Mdi(K) = {mi | mi @@ di}

(where @@ denotes the usage relationship) (and → denotes a modification)

Data Slices

Potential problems with data slices

– If the code is faulty and a methods that should access a variable does not

24/71

• The Sequences of methods that interact through that

variable would not be tested and the fault may remain undetected

– How to identify legal sequences of methods?

• There may be a large, possibly infinite, number of

legal sequences because a method may be used more than once in a sequence !

Testing Method Sequences

Motivations Data Slices

– MaDUM

25/71

– MaDUM – MaDUM for Derived Classes

Method Per- / Post-conditions Conclusion

Example II

class SampleStatistic { friend TestSS; // test driver! protected: int n; double x; double x2; double minValue, maxValue; public: sampleStatistic(); virtual ~SampleStatistic(); virtual void reset();

26/71

Formerly part of the GNU C++ library

virtual void reset(); virtual void operator +=(double); int samples(); double mean(); double stdDev(); double var(); double min(); double max(); double confidence(int p_percentage); double confidence(double p_value); void error(cont char * msg); };

Example II

SampleStatistic::mean() { if (n > 0) double SampleStatistics::stdDev() { if (n <= 0 || this->var() <= 0) return (0); else return (double) sqrt(var()); } double SampleStatistics::var() { if (n > 1) return ((x2 - ((x * x) / 2)) / n - 1); }

27/71

if (n > 0) return (x / n); else return (0.0); } SampleStatistic::operator+=(double value) { n += 1; x += value; x2 += (value * value); if (minValue > value) minValue = value; if (maxValue < value) maxValue = value; }

Generate MaDUM

MaDUM: Minimal Data member Usage

Matrix, n × m matrix

– n is the number of instance variables

28/71

– m is the number of methods

MaDUM reports the usage of instance

variables by methods

– Different usages: reads, reports, transforms

Generate MaDUM

MaDUM accounts for indirect use of instance

variable, through intermediate methods

29/71

Use the MaDUM to devise a testing strategy,

i.e., a testing order

Method Categories

Categories: constructors, reporters, transformers,

and others

Mdi(K) = {Rdi, C, Tdi, Odi}

– R = {r | r is a reporter method for

30/71

– Rdi = {rdi | rdi is a reporter method for variable di} – C = {ci | ci is a constructor of K} – Tdi = {mdi | mdi ∉ Rdi ∧ mdi ∉ C ∧ mdi → di} – Odi = {odi | odi @@ di ∧ odi ∉ Rdi ∧ odi ∉ C ∧ odi ∉ Tdi}

Method Categories

Categories: constructors, reporters, transformers,

and others

– Account for indirect use of data members, through intermediate methods

31/71

intermediate methods – Others

• printError()
• destructors
• …

Example II

MaDUM accounts for indirect uses

– stdDev() does not directly access x and x2, but calls var(), which does

32/71

– All SampleStatistic() does is call reset()

Test Procedure

The method categories is used to decide the

testing procedure

– Test reporters

33/71

– Test constructors – Test transformers – Test others

We would like to automate that procedure as

much as possible

Test Reporters

(Add to all classes get() and set()

methods for all their instance variables)

34/71

Systematically set and get values of instance

variables and compare the input of the former with the output of the latter

– Bashir and Goel took a different approach

Test Constructors

Constructors initialise instance variables, we

must test that

– All variables are correctly initialised

35/71

– All variables are initialised in the correct order

Run the constructor and append reporter

method(s) for each variable

Verify if correct initial state (state invariant)

– Only one simple constructor in SampleStatistic

Test Transformers

For each slice di

– Instantiate object with constructor (tested) – Create sequences, e.g., all legal permutations of

36/71

methods in Tdi

• The maximum number of sequences is |C|×|Tdi|!
• For example, |Tdi| = 7, with one constructor, there

are 5,040 possible permutations!

• Is this sufficient to test interactions among methods ?
• All paths where di is manipulated must be executed

– Append the reporter methods (tested)

Test Others

Other methods

– May not use the data members – Do not change the state of the object

37/71

– Neither report the state of any data member nor transforms the state in anyway

Yet, they must be tested as other methods Only their functionality as stand-alone

methods must be checked

Any standard test technique can be used

Example II

Slice allowVend

TallowVend(CCoinBox) = {Reset, AddQrt, ReturnQrts, Vend}

This action should not be there!

38/71

Slice curQrts

TcurQrts(CCoinBox) = {Reset, AddQrt, ReturnQrts, Vend}

As a result of the fault, possibly important

sequences are not tested

– When the code is correct, both slices show the same set of methods

Discussions

Slicing may not be helpful for many classes

(see CCoinBox)

Number of methods sequences still large

39/71

Many sequences may be impossible (illegal)

Discussions

Automation?

– Oracle, impossible sequences

Are we missing many faults by testing slices

40/71

Are we missing many faults by testing slices

independently?

– If faults lead to incorrectly defined slices

Implicitly aimed at classes with no state-

dependent behavior?

– Execute Transformers several times to reach certain states and reveal faults

Testing Method Sequences

Motivations Data Slices

– MaDUM

41/71

– MaDUM – MaDUM for Derived Classes

Method Per- / Post-conditions Conclusion

MaDUM for Derived Classes

When two classes are related through

inheritance, a class is derived from another

The derived class may add functionality or

42/71

modify the ones provided by the base class

– It inherits variables and methods of its parent

Extreme options for testing a derived class

– Flatten the derived class and retest all slices of the base class in the new context – Only test the new/redefined slices

Bashir and Goel’s Strategy

Assuming the base class has been

adequately tested, what must be tested in the derived class?

43/71

Extend MaDUM of the base class to

generate a MaDUMderived, from MaDUM

– Add a row for each newly defined or re-defined variable of the derived class – Add a column for each newly defined or re- defined method of the derived class

Example III

class SampleHistogram : public SampleStatistic { protected: short howmanybuckets; int *bucketCount; double *bucketLimit; public: SampleHistogram(double low, double hi, double bucketWidth = -1.0);

44/71

SampleHistogram(double low, double hi, double bucketWidth = -1.0); ~SampleHistogram(); virtual void reset(); // Redefined virtual void operator+=(double); // Redefined int similarSamples(double); int buckets(); double bucketThreshold(int i); int inBucket(int i); void printBuckets(ostream&); }

Filling MaDUMDerived

If a newly defined method mderived calls an

inherited method mbase of the base class, then the column of the two methods are

45/71

merged and the result is stored in the column of mderived

– C(mderived) = C(mderived) ∪ C(mbase)

Filling MaDUMDerived

Even though the base class has no a priori

knowledge about the data members defined in its derived classes, it may still act on them

46/71

through dynamic binding and polymorphism

For example, if a method m1base calls

another method m2base and the method m2 is redefined in one of the derived classes

– C(m1base) = C(m1base) ∪ C(m2derived)

Example III

The MaDUM of SampleHistogram has

eight rows, three of them for “local” instance variable, and twenty columns, eight for

47/71

“local” methods (two of them re-defined)

Both reset() and +=() are redefined or

• verridden in SampleHistogram and

invoke their counterparts

– They indirectly access all the variables of SampleStatistics

Example III

48/71

Test Procedure for Derived Class

“Local” variables

– Similar to base class testing

Retest inherited variables (i.e., their slices):

49/71

Retest inherited variables (i.e., their slices):

– Check upper right quadrant of MaDUMderived – Ascertain which inherited attributes mandate re- testing, use MaDUMderived for automation – Once inherited variables are identified, the test procedure is similar to slice-based testing in the base class, but using inherited and new/redefined methods

Test Procedure for Derived Class

An inherited variable must be re-tested if the

number of entries in its MaDUM row has increased between MaDUMbase and

50/71

MaDUMderived

For SampleHistogram, the set of inherited

variables that must be re-tested includes all inherited variables

– {n, x, x2, minValue, maxValue}

Testing Method Sequences

Motivations Data Slices

– MaDUM

51/71

– MaDUM – MaDUM for Derived Classes

Method Per- / Post-conditions Conclusion

Method Pre- and Post-conditions

Method pre-condition

– A predicate that must be true before an method is invoked

52/71

– Specifies constraints that a caller must fulfill before calling a method

Method post-condition

– A predicate that must be true after a method is invoked – Specifies constraints that the receiver must ensure after the invocation of the method

Method Pre- and Post-conditions

Possible notations

– Natural language – Object Constraint Language (OCL)

53/71

• In the context of UML

– Extensions to programming languages

• Eiffel
• JDK v1.4

Example IV

Class Queue (unbounded queue) Attribute: Number of elements in the queue, count Init(q:Queue) pre: Queue q does not exist post: Queue q exists and is empty Empty(q:Queue) pre: Queue q exists post: Returns 1 if q is empty (count=0), 0

54/71

post: Returns 1 if q is empty (count=0), 0

• therwise (count>0)

Eque(q:Queue, e:Element) pre: Queue q exists post: Element e is added to the tail of queue q, and q is not empty (count=old(count)+1) Dque(q:Queue, e:Element) pre: Queue q exists and is not empty (count>0) post: Element e is removed from q (count=old(count)-1) Top(q:Queue, e:Element) pre: Queue q exists and is not empty (count>0) post: The first element is returned (e)

Approach

Get pre- and post-conditions

– E.g., from UML documents – Or specify them

55/71

Derive method sequence constraints from

pre- and post-conditions

– They indicate which method sequences (pairs) are allowed or not and under which conditions – Automation?

Approach

Choose a criterion

– We will define 7 criteria

Derive method sequences satisfying a

56/71

Derive method sequences satisfying a

criterion from the method sequence constraints

– Automation?

Sequencing Constraints

Pre- and post-conditions imply method

sequencing constraints for pairs of methods

– Assuming m1 and m2 are two methods of a class, a sequencing constraints between and

57/71

class, a sequencing constraints between m1 and m2 is defined as a triplet (m1, m2, C) – Such a triplet indicates that m2 can be executed after m1 under condition C

Sequencing Constraints

– C is a Boolean expression / literal (True, False)

• C = True ⇒ m2 can always be executed after m1

– m1 post-condition implies m2 pre-condition

• C = False ⇒ m2 can never be executed after m1

58/71

• C = False ⇒ m2 can never be executed after m1

– m1 post-condition implies the negation of m2 pre-condition

• C = BoolExp ⇒ m2 can be executed after m1 under

some conditions

– BoolExp lists the conditions under which the sequence is possible (disjunctive normal form): C = C1 ∨ C2 ∨ …

Criteria (2/7)

Always Valid Coverage (T)

– Each always-valid constraint must be covered at least once, i.e., each (m1, m2, T)

59/71

Never Valid Coverage (F)

– Each never-valid constraint must be covered at least once, i.e., each (m1, m2, F)

Criteria (4/7)

Always/Possibly True Coverage (T/pT)

– Each always-valid constraint and each possibly-true constraint must be covered at least once, i.e., each (m1, m2, T) and each (m1, m2, C) using one of the disjunction

60/71

m2, T) and each (m1, m2, C) using one of the disjunction in C (C1, C2, or…)

Always/Possibly True Coverage+ (T/pT+)

– Each always-valid constraint and each possibly-true constraint (using each disjunction) must be covered at least once, i.e., each (m1, m2, T) and each (m1, m2, Ci) using each of the disjunction in C (C1, C2, and…)

Criteria (7/7)

Never Valid/Possibly False (F/pF)

– Each never-valid constraint and, using one of the disjunction, false constraint must be covered at least

• nce, i.e., each (m1, m2, F) and each (m1, m2, not(C))

61/71

• nce, i.e., each (m1, m2, F) and each (m1, m2, not(C))

Never Valid/Possibly False Plus (F/pF+)

– Each never-Valid constraint and each possibly false constraint must be covered at least once, not(C) = C’1 ∨ C’2 ∨ …, i.e., each (m1, m2, F) and each (m1, m2, not(C1)), (m1, m2, not(C2)), …

Always/Possibly True Plus/Never/Possibly False

Plus (T/pT+/F/pF+)

Subsumption

T/pT+/F/pF+ T/pT+ F/pF+

62/71

T/pT T F/pF F

Example IV

Sequencing constraints

• C1. (#,Init,T)
• C6. (#,Eque,F)
• C11. (#,Dque,F)
• C16. (#,Top,F)
• C2. (Init,Eque,T)
• C7. (Eque,Dque,T)
• C12. (Dque,Eque,T)
• C17. (Top,Dque,T)
• C3. (Init,Dque,F)
• C8. (Eque,Top,T)
• C13. (Dque,Init,F)
• C18. (Top,Eque,T)
• C4. (Init,Init,F)
• C9. (Eque,Eque,T)
• C14. (Dque,Dque,C)
• C19. (Top,Init,F)

C . (Init,Top,F) C . (Eque,Init,F) C . (Dque,Top,C) C . (Top,Top,T)

63/71

• C5. (Init,Top,F)
• C10. (Eque,Init,F)
• C15. (Dque,Top,C)
• C20. (Top,Top,T)

Where: C = count>0

Criterion T requires the use of

• C1. (#,Init,T)
• C2. (Init,Eque,T)
• C7. (Eque,Dque,T)
• C12. (Dque,Eque,T)
• C17. (Top,Dque,T)
• C8. (Eque,Top,T)
• C18. (Top,Eque,T)
• C9. (Eque,Eque,T)
• C20. (Top,Top,T)

Example IV (Criterion T)

• C1. (#,Init,T)

C7.(Eque,Dque,T) C12.(Dque,Eque,T)

• C2. (Init,Eque,T)

C9.(Eque,Eque,T) C8.(Eque,Top,T) C20.(Top,Top,T)

64/71

Partial tree based on sequencing constraints

C8.(Eque,Top,T) C18.(Top,Eque,T) C20.(Top,Top,T) C17.(Top,Dque,T) C12.(Dque,Eque,T) C7.(Eque,Dque,T) C12.(Dque,Eque,T) C8.(Eque,Top,T) C18.(Top,Eque,T) C8.(Eque,Top,T) C17.(Top,Dque,T) C18.(Top,Eque,T) C7.(Eque,Dque,T) C12.(Dque,Eque,T) C8.(Eque,Top,T) C17.(Top,Dque,T) C18.(Top,Eque,T) C17.(Top,Dque,T) C12.(Dque,Eque,T) C8.(Eque,Top,T)

Discussion

Automation?

– From pre- and post-conditions to sequencing constraints

65/71

– From sequencing constraints to “complete” sequences

Discussion

Several sequences can be adequate for a

particular criterion

– Which branch in the tree do we choose?

66/71

– Are they equivalent in terms of fault detection?

• The sets of statements executed in methods may be

different

– We may need to cover some pT constraints to cover certain T ones

Discussion

Selecting another criterion than T

– Criterion F

• Another representation than the tree we used before

may be necessary

67/71

may be necessary

– Criterion pT+

• The execution of sequence {m1, m2}, constrained by

(m1, m2, C), may require specific calls before the call to m1 (because of pre- and post-conditions of m1)

Empirical comparison of the different criteria

Empirical Study

From Daniels and Tai

– Three C++ programs – Mutation tool Proteum for C

68/71

– 71 mutation operators – Most mutants were killed with T criteria and all of them with T/pT criteria Criteria are not exactly the ones defined here for T/pT+ and F/pF+

Results Table

69/71

Testing Method Sequences

Motivations Data Slices

– MaDUM

70/71

– MaDUM – MaDUM for Derived Classes

Method Per- / Post-conditions Conclusion

Conclusion

Testing method in “isolation”

– Hierarchical Incremental Testing

71/71

Testing classes

– Sequences of methods – Data slices

• MaDUM

– Pre- / post-conditions