Markov Chain Usage Models and Concurrency Stacy Prowell - - PowerPoint PPT Presentation

Quick Overview Testing Complex Systems Summary

Markov Chain Usage Models and Concurrency

Stacy Prowell

sprowell@cs.utk.edu

January 2005

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary

Online

Software Quality Research Laboratory http://www.cs.utk.edu/sqrl/ Markov Chain Usage Models http://www.cs.utk.edu/~sprowell/markov.html

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary Testing as a Statistical Experiment Simple Markov Chain Models What are they good for? Tools

Outline

1

Quick Overview Testing as a Statistical Experiment Simple Markov Chain Models What are they good for? Tools

2

Testing Complex Systems Concurrency Combining Tests TGL Synchronization

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary Testing as a Statistical Experiment Simple Markov Chain Models What are they good for? Tools

Objective

Usage Based Testing The objective should not be simply “to find bugs,” because bugs are not equal. A better objective is give sufficient confidence that software release will not be harmful. Bugs to find are the ones which (1) occur most frequently in field use and (2) are most serious by some measure. To meet (1), bias testing toward expected use. To meet (2), bias testing toward critical functionality in the context of expected use.

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary Testing as a Statistical Experiment Simple Markov Chain Models What are they good for? Tools

Testing is Sampling

For most software, there is an infinite number of tests (all sequences of inputs: S∗). Only a finite sample of these can be run. How should the sample be selected? What should we infer from the sample? This is a statistical experiment.

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary Testing as a Statistical Experiment Simple Markov Chain Models What are they good for? Tools

Testing is Sampling

For most software, there is an infinite number of tests (all sequences of inputs: S∗). Only a finite sample of these can be run. How should the sample be selected? What should we infer from the sample? This is a statistical experiment.

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary Testing as a Statistical Experiment Simple Markov Chain Models What are they good for? Tools

Testing is Sampling

For most software, there is an infinite number of tests (all sequences of inputs: S∗). Only a finite sample of these can be run. How should the sample be selected? What should we infer from the sample? This is a statistical experiment.

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary Testing as a Statistical Experiment Simple Markov Chain Models What are they good for? Tools

Testing is Sampling

For most software, there is an infinite number of tests (all sequences of inputs: S∗). Only a finite sample of these can be run. How should the sample be selected? What should we infer from the sample? This is a statistical experiment.

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary Testing as a Statistical Experiment Simple Markov Chain Models What are they good for? Tools

Testing is Sampling

For most software, there is an infinite number of tests (all sequences of inputs: S∗). Only a finite sample of these can be run. How should the sample be selected? What should we infer from the sample? This is a statistical experiment.

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary Testing as a Statistical Experiment Simple Markov Chain Models What are they good for? Tools

Statistics

“In a metrics mentality, the goal is to find quantities that can be calculated easily. This metrics mentality leads to the calculation of meaningless quantities. The user thinks he has useful information, but is mistaken. A statistical mentality defines parameters, collects data, and applies statistical procedures using the data to estimate parameters. The results are meaningful, useful estimates.” - John Healy

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary Testing as a Statistical Experiment Simple Markov Chain Models What are they good for? Tools

Markov Chains

Test Cases Test cases are sequences of events. Possible next events depend on history. The event selection is stochastic. Finite Markov Chains Familiar model; states and arcs. Many tools support creating digraphs and statecharts. Nice graphical representation. Well-studied; nearly any result is computable. “All models are wrong, but some are useful.” - George Box

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary Testing as a Statistical Experiment Simple Markov Chain Models What are they good for? Tools

Markov Chains

Test Cases Test cases are sequences of events. Possible next events depend on history. The event selection is stochastic. Finite Markov Chains Familiar model; states and arcs. Many tools support creating digraphs and statecharts. Nice graphical representation. Well-studied; nearly any result is computable. “All models are wrong, but some are useful.” - George Box

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary Testing as a Statistical Experiment Simple Markov Chain Models What are they good for? Tools

Simple Markov Chain Models

Exit Error Tone hang up (1.0) Connected hang up (0.5) Off Hook disconnect (0.5) hang up (0.25) dial bad (0.25) Ring Tone dial good (0.25) Busy Tone dial busy (0.25) Ringing lift receiver (0.5) On Hook disconnect (0.5) lift receiver (0.5) incoming call (0.5) hang up (0.5) connect (0.5) hang up (1.0)

Arcs are labeled with usage events. Nodes are “states

• f use” which

provide correct sequencing. Probabilities are based on known

• r anticipated use.

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary Testing as a Statistical Experiment Simple Markov Chain Models What are they good for? Tools

What Can You Do With Them?

Generate Tests

Random walk; generate tests in order by probability. Weighted generation; generate tests in order by weight (for example, most probable tests). Coverage set; use postman algorithm to generate minimum coverage set for baseline test. All can be automated.

Compute Statistics

Static analysis of model; based on assumptions, what does use look like? Analysis of test results; what is expected reliability of delivered system and when is testing adequate?

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary Testing as a Statistical Experiment Simple Markov Chain Models What are they good for? Tools

What Can You Do With Them?

Generate Tests

Random walk; generate tests in order by probability. Weighted generation; generate tests in order by weight (for example, most probable tests). Coverage set; use postman algorithm to generate minimum coverage set for baseline test. All can be automated.

Compute Statistics

Static analysis of model; based on assumptions, what does use look like? Analysis of test results; what is expected reliability of delivered system and when is testing adequate?

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary Testing as a Statistical Experiment Simple Markov Chain Models What are they good for? Tools

Tools

Third Generation J Usage Model Builder Library (JUMBL) developed in Java at UT starting in 1999. Piloted many new ideas (weighted generation, new notations, new computations and reliability models, faster algorithms). Several usability and performance enhancements driven by industrial application. Standard interchange formats (XML) for models (MML / EMML), test records (TCML), test generation (TGL). Version 4 currently available.

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary Concurrency Combining Tests TGL Synchronization

Outline

1

Quick Overview Testing as a Statistical Experiment Simple Markov Chain Models What are they good for? Tools

2

Testing Complex Systems Concurrency Combining Tests TGL Synchronization

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary Concurrency Combining Tests TGL Synchronization

Concurrency

Network components: multiple streams of inputs. Invent strong abstractions to address this: all, some, none. Example All phones on hook. Some of the phones go off hook. Now some phones are on hook and some are off hook. Hard to translate into executable test cases.

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary Concurrency Combining Tests TGL Synchronization

Concurrency

Network components: multiple streams of inputs. Invent strong abstractions to address this: all, some, none. Example All phones on hook. Some of the phones go off hook. Now some phones are on hook and some are off hook. Hard to translate into executable test cases.

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary Concurrency Combining Tests TGL Synchronization

Concurrency

Concurrent dialogs: lots of modeless dialogs. Augment usage models with variables and predicates. Example File open, region selected, paste buffer empty, snap on... Difficult to get it right (EMML / Extended FSM).

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary Concurrency Combining Tests TGL Synchronization

Concurrency

Concurrent dialogs: lots of modeless dialogs. Augment usage models with variables and predicates. Example File open, region selected, paste buffer empty, snap on... Difficult to get it right (EMML / Extended FSM).

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary Concurrency Combining Tests TGL Synchronization

Concurrency

Consider combinations of models using operators. Lots of choices: CSP , CCS, etc. Concurrent regular expressions [Garg and Ragunath 1992] is a simple approach. Three operators: interleave, synchronous composition, and alpha closure. Only alpha closure is not regular.

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary Concurrency Combining Tests TGL Synchronization

Complex Test Cases

An Approach Use multiple models, or multiple instances of a model Combine using concurrency operators Still a Markov chain State explosion; hard to analyze and interpret analysis

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary Concurrency Combining Tests TGL Synchronization

State Explosion

Suppose we want to test a simple phone switch connected to 100 phones. Each phone can be in one of 7 states (ignoring the sink). The simple interleave model has 7100 ≈ 1084 states. We can model this system directly, but how do we analyze the resulting system? What do the analytical results even mean?

Exit Error Tone hang up (1.0) Connected hang up (0.5) Off Hook disconnect (0.5) hang up (0.25) dial bad (0.25) Ring Tone dial good (0.25) Busy Tone dial busy (0.25) Ringing lift receiver (0.5) On Hook disconnect (0.5) lift receiver (0.5) incoming call (0.5) hang up (0.5) connect (0.5) hang up (1.0)

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary Concurrency Combining Tests TGL Synchronization

Combining Tests

A Different Approach Instead of combining usage models, combine test cases generated from the models. Generate one hundred (or one thousand) test cases from the phone model, and then randomly interleave events to obtain a new test case. Each component test case is a trajectory of the new test case. Trajectory 1: A B B C B A Trajectory 2: X X Z Y Y One possible interleaved test case: A B X B X Z Y C B A Y

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary Concurrency Combining Tests TGL Synchronization

Interleave

The degree of multiplexing is controlled via channels. Fill the channels with test cases. Randomly select a channel and remove the next test event from the channel and place it in the new test case. Continue until all channels are exhausted. Preserves the order of events in the test cases.

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary Concurrency Combining Tests TGL Synchronization

Test Notation: TGL

Test sources: models, test records random, coverage, weighted, manual, ... Test operators: interleave, compose, ... General operators: repeat, randomize, ...

<TestGenerator name="phones"> <Repeat num="[100-200]"> <Tangle channels="[50-100]"> <GenTest model="phone" num="[100-1000]"/> </Tangle> </Repeat> </TestGenerator>

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary Concurrency Combining Tests TGL Synchronization

Shared Events

When one phone dials another phone, the phones have a shared event. The two trajectories must synchronize on this event. Use the synchronous composition from concurrent regular expressions, OR use submodels to enforce synchronization. Break use into everything up to the call. Then use the “make a call” and “receive a call” models, interleaved. The use a model for everything after the call. Analyze via simulation and by exploiting structure.

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary Concurrency Combining Tests TGL Synchronization

Shared Events

<TestGenerator name="phones"> <Tangle channels="100"> <Repeat num="[1000-2000]"> <Random> <GenTest model="noconnect" num="2"/> <Tangle> <GenTest model="dialout"/> <GenTest model="dialin"/> </Tangle> </Random> </Repeat> </Tangle> </TestGenerator> # Step: 1, Trajectory: 35 [On Hook]."lift receiver" # Step: 2, Trajectory: 87 [On Hook]."lift receiver" # Step: 3, Trajectory: 37 [On Hook]."lift receiver" ... # Step: 54, Trajectory: 19 [Off Hook]."dial 64" ... # Step: 61, Trajectory: 64 [On Hook]."dialed by 19" ... # Step: 384, Trajectory: 56 [Connected]."hang up" # Step: 385, Trajectory: 9 [Off Hook]."hang up"

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary Concurrency Combining Tests TGL Synchronization

Application

This approach has been applied to: Modeling of network components to determine availability (Nortel).

Each protocol (OSPF , ...) on a network switch was described using sequence-based specification. Usage models were generated and combined by interleaving their events. Allowed generating complex network traffic.

Testing medical imaging software (CTI-PET Systems).

Model each modeless dialog’s actions, and interleave the events. Use WinRunner to execute tests. Allowed hundreds of tests to be automatically generated and executed without human intervention.

SQRL: Prowell MBT: Testing Complex Systems

Quick Overview Testing Complex Systems Summary

Summary

Existing modeling tools are preserved: model construction and analysis, and test case generation. Has been successfully applied in testing networking systems and GUIs. TGL can be extended with additional operators, as needed. Overall analytical results can be obtained by simulation, or by exploitation of structure. Current JUMBL http://www.cs.utk.edu/sqrl/esp/jumbl.html implements TGL without synchronization.

SQRL: Prowell MBT: Testing Complex Systems