Quick Overview Testing Complex Systems Summary Markov Chain Usage Models and Concurrency Stacy Prowell sprowell@cs.utk.edu January 2005 SQRL: Prowell MBT: Testing Complex Systems
Quick Overview Testing Complex Systems Summary Online Software Quality Research Laboratory http://www.cs.utk.edu/sqrl/ Markov Chain Usage Models http://www.cs.utk.edu/~sprowell/markov.html SQRL: Prowell MBT: Testing Complex Systems
Testing as a Statistical Experiment Quick Overview Simple Markov Chain Models Testing Complex Systems What are they good for? Summary Tools Outline 1 Quick Overview Testing as a Statistical Experiment Simple Markov Chain Models What are they good for? Tools 2 Testing Complex Systems Concurrency Combining Tests TGL Synchronization SQRL: Prowell MBT: Testing Complex Systems
Testing as a Statistical Experiment Quick Overview Simple Markov Chain Models Testing Complex Systems What are they good for? Summary Tools Objective Usage Based Testing The objective should not be simply “to find bugs,” because bugs are not equal. A better objective is give sufficient confidence that software release will not be harmful . Bugs to find are the ones which (1) occur most frequently in field use and (2) are most serious by some measure. To meet (1), bias testing toward expected use. To meet (2), bias testing toward critical functionality in the context of expected use . SQRL: Prowell MBT: Testing Complex Systems
Testing as a Statistical Experiment Quick Overview Simple Markov Chain Models Testing Complex Systems What are they good for? Summary Tools Testing is Sampling For most software, there is an infinite number of tests (all sequences of inputs: S ∗ ). Only a finite sample of these can be run. How should the sample be selected ? What should we infer from the sample? This is a statistical experiment . SQRL: Prowell MBT: Testing Complex Systems
Testing as a Statistical Experiment Quick Overview Simple Markov Chain Models Testing Complex Systems What are they good for? Summary Tools Testing is Sampling For most software, there is an infinite number of tests (all sequences of inputs: S ∗ ). Only a finite sample of these can be run. How should the sample be selected ? What should we infer from the sample? This is a statistical experiment . SQRL: Prowell MBT: Testing Complex Systems
Testing as a Statistical Experiment Quick Overview Simple Markov Chain Models Testing Complex Systems What are they good for? Summary Tools Testing is Sampling For most software, there is an infinite number of tests (all sequences of inputs: S ∗ ). Only a finite sample of these can be run. How should the sample be selected ? What should we infer from the sample? This is a statistical experiment . SQRL: Prowell MBT: Testing Complex Systems
Testing as a Statistical Experiment Quick Overview Simple Markov Chain Models Testing Complex Systems What are they good for? Summary Tools Testing is Sampling For most software, there is an infinite number of tests (all sequences of inputs: S ∗ ). Only a finite sample of these can be run. How should the sample be selected ? What should we infer from the sample? This is a statistical experiment . SQRL: Prowell MBT: Testing Complex Systems
Testing as a Statistical Experiment Quick Overview Simple Markov Chain Models Testing Complex Systems What are they good for? Summary Tools Testing is Sampling For most software, there is an infinite number of tests (all sequences of inputs: S ∗ ). Only a finite sample of these can be run. How should the sample be selected ? What should we infer from the sample? This is a statistical experiment . SQRL: Prowell MBT: Testing Complex Systems
Testing as a Statistical Experiment Quick Overview Simple Markov Chain Models Testing Complex Systems What are they good for? Summary Tools Statistics “In a metrics mentality , the goal is to find quantities that can be calculated easily. This metrics mentality leads to the calculation of meaningless quantities. The user thinks he has useful information, but is mistaken. A statistical mentality defines parameters, collects data, and applies statistical procedures using the data to estimate parameters. The results are meaningful, useful estimates.” - John Healy SQRL: Prowell MBT: Testing Complex Systems
Testing as a Statistical Experiment Quick Overview Simple Markov Chain Models Testing Complex Systems What are they good for? Summary Tools Markov Chains Test Cases Test cases are sequences of events. Possible next events depend on history . The event selection is stochastic . Finite Markov Chains Familiar model; states and arcs. Many tools support creating digraphs and statecharts. Nice graphical representation. Well-studied; nearly any result is computable . “All models are wrong, but some are useful .” - George Box SQRL: Prowell MBT: Testing Complex Systems
Testing as a Statistical Experiment Quick Overview Simple Markov Chain Models Testing Complex Systems What are they good for? Summary Tools Markov Chains Test Cases Test cases are sequences of events. Possible next events depend on history . The event selection is stochastic . Finite Markov Chains Familiar model; states and arcs. Many tools support creating digraphs and statecharts. Nice graphical representation. Well-studied; nearly any result is computable . “All models are wrong, but some are useful .” - George Box SQRL: Prowell MBT: Testing Complex Systems
Testing as a Statistical Experiment Quick Overview Simple Markov Chain Models Testing Complex Systems What are they good for? Summary Tools Simple Markov Chain Models Ringing incoming call disconnect (0.5) (0.5) On Hook Arcs are labeled with usage events. lift receiver (0.5) Nodes are “states lift receiver Off Hook (0.5) of use” which dial good dial busy dial bad provide correct (0.25) (0.25) (0.25) disconnect sequencing. Ring Tone (0.5) Probabilities are hang up connect Busy Tone Error Tone (0.25) (0.5) based on known hang up hang up hang up Connected (0.5) (1.0) (1.0) or anticipated use. hang up (0.5) Exit SQRL: Prowell MBT: Testing Complex Systems
Testing as a Statistical Experiment Quick Overview Simple Markov Chain Models Testing Complex Systems What are they good for? Summary Tools What Can You Do With Them? Generate Tests Random walk; generate tests in order by probability. Weighted generation; generate tests in order by weight (for example, most probable tests). Coverage set; use postman algorithm to generate minimum coverage set for baseline test. All can be automated . Compute Statistics Static analysis of model; based on assumptions, what does use look like? Analysis of test results; what is expected reliability of delivered system and when is testing adequate? SQRL: Prowell MBT: Testing Complex Systems
Testing as a Statistical Experiment Quick Overview Simple Markov Chain Models Testing Complex Systems What are they good for? Summary Tools What Can You Do With Them? Generate Tests Random walk; generate tests in order by probability. Weighted generation; generate tests in order by weight (for example, most probable tests). Coverage set; use postman algorithm to generate minimum coverage set for baseline test. All can be automated . Compute Statistics Static analysis of model; based on assumptions, what does use look like? Analysis of test results; what is expected reliability of delivered system and when is testing adequate? SQRL: Prowell MBT: Testing Complex Systems
Testing as a Statistical Experiment Quick Overview Simple Markov Chain Models Testing Complex Systems What are they good for? Summary Tools Tools Third Generation J Usage Model Builder Library ( JUMBL ) developed in Java at UT starting in 1999. Piloted many new ideas (weighted generation, new notations, new computations and reliability models, faster algorithms). Several usability and performance enhancements driven by industrial application. Standard interchange formats ( XML ) for models ( MML / EMML ), test records ( TCML ), test generation ( TGL ). Version 4 currently available. SQRL: Prowell MBT: Testing Complex Systems
Concurrency Quick Overview Combining Tests Testing Complex Systems TGL Summary Synchronization Outline 1 Quick Overview Testing as a Statistical Experiment Simple Markov Chain Models What are they good for? Tools 2 Testing Complex Systems Concurrency Combining Tests TGL Synchronization SQRL: Prowell MBT: Testing Complex Systems
Concurrency Quick Overview Combining Tests Testing Complex Systems TGL Summary Synchronization Concurrency Network components: multiple streams of inputs. Invent strong abstractions to address this: all, some, none. Example All phones on hook. Some of the phones go off hook. Now some phones are on hook and some are off hook. Hard to translate into executable test cases. SQRL: Prowell MBT: Testing Complex Systems
Concurrency Quick Overview Combining Tests Testing Complex Systems TGL Summary Synchronization Concurrency Network components: multiple streams of inputs. Invent strong abstractions to address this: all, some, none. Example All phones on hook. Some of the phones go off hook. Now some phones are on hook and some are off hook. Hard to translate into executable test cases. SQRL: Prowell MBT: Testing Complex Systems
Recommend
More recommend
