Advanced Coverage Criteria Software Engineering Gordon Fraser • Saarland University Remember the Roots example? Having a million computers doing a million tests per second would be sufficient to test the Roots example four times during the lifetime of the sun. Clearly, exhaustive testing is not feasible in practice. class Roots { // Solve ax 2 + bx + c = 0 public roots(double a, double b, double c) { … } // Result: values for x double root_one, root_two; } Because we cannot do testing exhaustively, we can only sample test cases. Therefore, we can never be sure that our program is free of bugs. Because showing the absence of bugs is impossible, the Dijkstra’s law aim of testing is to show that there are bugs. Testing is successful if we find bugs (even though this is sometimes indicated with a red Testing can show the light in a GUI, suggesting presence but not the otherwise). absence of errors.
A coverage criterion describes a finite subset of test cases out of Coverage Criteria the vast/infinite number of possible tests we should execute. Possible test case We can measure coverage on any artifact produced during Coverage of... software development, e.g., structural coverage of source code, coverage of input space, coverage of complex inputs - e.g., grammar based, coverage of specification, coverage of test models, coverage of requirements, coverage of GUI elements, ... A coverage criterion can be seen as a finite set of test Coverage criteria requirements that a test suite should fulfill. There is usually more than one way to cover a test requirement, so a coverage Coverage Program criterion is not a unique Criterion under Test description of a test suite. Test Test Test = Rules Requirements Requirement Requirement “Execute statement 1”
Coverage criteria serve two main purposes: To measure Using coverage criteria adequacy of existing test suites, and to guide generation of new test cases. Even though coverage is often used to measure the quality of an 1. Adequacy: Have I got enough tests? existing test suite, coverage is not a good measurement for 2. Guidance: Where should I test more? this. Generally, coverage is only 3. Automation: Generate test that satisfies a good at telling you which parts test requirement haven’t been covered. To make use of coverage in practice we need to measure it. Measuring Code Coverage This is done by instrumenting the source code with an instrument for every single test requirement, described by the Coverage Program coverage criterion. When test Test Cases Criterion under Test cases are run on the instrumented program the instrumentation keeps track of what has been executed and Test Test what hasn’t, and so at the end Test Test Instrumented Test Covered Requirements Requirement Requirements Requirement of the execution we can analyze Program Requirement Requirement this information to point to uncovered areas and quantify the coverage. In general, instrumentation Instrumentation adds program code that does not change functional behavior but collects information. This • Instrument: Additional code that does not instrumentation might, change functional behavior but collects however, change other aspects information of the program, such as timing, public int min(A, B) { public int min(A, B) { interleaving, etc. int m = A; int m = A; if(A>B) { if(A>B) { m = B; Mark: “if body reached” } m = B; return m; } } return m;
Here is an example of how to Instrumentation measure statement coverage: Before executing a statement we simply trace that the public int min(A, B) { statement has been executed, statement[0]++; for example by adding a static int m = A; method call, incrementing a statement[1]++; public int min(A, B) { counter, etc. if(A>B) { int m = A; statement[2]++; if(A>B) { m = B; m = B; } } statement[3]++; return m; return m; } } Coverage is usually quantified as the percentage of test Coverage Value requirements satisfied. But what does that mean? # Covered test requirements Coverage value = # Total test requirements Coverage is usually quantified as the percentage of test Coverage Value requirements satisfied. But what does that mean? # Covered test requirements Coverage value = # Total test requirements I’ve got 100% statement coverage on my program. How many bugs are left?
...at the end of the day, we still don’t know what we’ve achieved by satisfying a coverage criterion. Weyuker’s Hyopthesis The adequacy of a coverage criterion can only be intuitively defined. The use of coverage has some dangerous aspects, that might Coverage is dangerous even reduce the quality of testing. If success is only • Developers write test only to satisfy quantified in a coverage metric, developers will get very e ffj cient coverage and writing test cases that • 100% coverage can detect no faults: satisfy the coverage goals, but Coverage measures what is executed , not what is checked not at finding bugs. Also, it is • Coverage metrics tell you what code is not possible to cover the entire program without detecting a tested, but cannot accurately tell you what single bug - testing is more code is tested: than just input generation (see • Low coverage means code is not well tested Mutation testing lecture). • But high coverage does not mean code is well tested Despite it’s downsides coverage has some useful sides: It is very Coverage is useful e ffj cient at telling you which parts of a program you haven’t tested at all. Intuitively, testing • It always tells you where you haven’t tested everything a little bit should be • Testing everything a bit is better than not better than testing some aspect thoroughly and neglecting the testing most of the program - unless you rest - unless you already know know where the faults are where the bugs are (which you • Coverage != coverage don’t in general). Stricter criterion → more tests • More tests = more chances of hitting bugs
Coming back to the family of Theoretical Criteria subsumes structural coverage criteria, we can Structural see that several of these criteria Path testing focus on the logical expressions in Criteria the source code. This is worth Boundary Compound interior testing condition testing having a closer look at, since this is a recurring problem in software testing. These criteria apply to MCDC testing all logical expressions (not just source code) Practical Criteria Branch and LCSAJ testing condition testing Branch testing Loop boundary Basic Statement testing testing condition testing Because the criteria we are going to consider now are Logic Coverage independent of source code we will adopt a slightly di fg erent nomenclature, to add some Predicate confusion. A logical expression is a predicate, and the predicate if(((a>b) || C) && p(x)) consists of clauses, conjoined o.m(); Clauses by boolean operators (and, or, ...). A clause contains no else boolean operators. o.n(); Predicates and clauses occur everywhere, not only in source code. For example, test models Educati ucation Individ Ind ividual ual often consist mainly of logical expressions. T T F F F F F F Education account Current purchase > – – F F T T – – Threshold 1 Current purchase > – – – – F F T T Threshold 2 Special price < F T F T – – – – scheduled price Special price < – – – – F T – – Tier 1 Special price < – – – – – – F T Tier 2 Edu Special No Special Tier 1 Special Tier 2 Special Out discount price discount price discount price discount Price Clauses Predicate
UML state charts, as another example, have predicates in terms of OCL expressions. Predicate Clauses Why is testing logical expressions important? That ʼ s because they are Translating from English difficult to get right. Translating from natural language to predicates is a process that is error prone, and • “If you leave before 6:30 AM, take Braddock to 495, if you natural language requirements are leave after 7:00 AM, take Prosperity to 50, then 50 to 495” often incomplete. • time < 6:30 → path = Braddock ∨ • time > 7:00 → path = Prosperity Incomplete! Introduction to Software Testing, Ammann and Offut Predicate Coverage (PC) • For each predicate: • Have at least one test where it evaluates to true • Have at least one test where it evaluates to false • Also known as: • Branch coverage • Decision coverage • Basic criterion
Predicate coverage is the most basic logical coverage criterion, Predicate Coverage and there are usually many di fg erent ways to satisfy it. if(((a>b) || C) && p(x)) o.m(); else o.n(); Predicate coverage is the most basic logical coverage criterion, Predicate Coverage and there are usually many di fg erent ways to satisfy it. a>b C p(x) ((a>b)||C)&&p 1 t t t t 2 t t f f 3 t f t t 4 t f f f 5 f t t t 6 f t f f 7 f f t f 8 f f f f Predicate coverage is the most basic logical coverage criterion, Predicate Coverage and there are usually many di fg erent ways to satisfy it. a>b C p(x) ((a>b)||C)&&p 1 t t t t 2 t t f f 3 t f t t 4 t f f f Clause does not 5 f t t t change value 6 f t f f 7 f f t f 8 f f f f
Recommend
More recommend
