Learning objectives Understand rationale for structural testing How - PowerPoint PPT Presentation

Learning objectives • Understand rationale for structural testing – How structural (code-based or glass-box) testing complements functional (black-box) testing Structural Testing • Recognize and distinguish basic terms – Adequacy, coverage • Recognize and distinguish characteristics of common structural criteria • Understand practical uses and limitations of structural testing (c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 1 (c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 2 “ Structural ” testing Why structural (code-based) testing? • One way of answering the question “ What is • Judging test suite thoroughness based on the missing in our test suite? ” structure of the program itself – If part of a program is not executed by any test case – Also known as “ white-box ” , “ glass-box ” , or “ code- in the suite, faults in that part cannot be exposed based ” testing – But what ’ s a “ part ” ? – To distinguish from functional (requirements-based, • Typically, a control flow element or combination: “ black-box ” testing) • Statements (or CFG nodes), Branches (or CFG edges) – “ Structural ” testing is still testing product functionality • Fragments and combinations: Conditions, paths against its specification. Only the measure of thoroughness • Complements functional testing: Another way has changed. to recognize cases that are treated differently – Recall fundamental rationale: Prefer test cases that are treated differently over cases treated the same (c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 3 (c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 4

Structural testing complements No guarantees functional testing • Executing all control flow elements does not • Control flow testing includes cases that may not guarantee finding all faults be identified from specifications alone – Execution of a faulty statement may not always – Typical case: implementation of a single item of the result in a failure specification by multiple parts of the program • The state may not be corrupted when the statement is – Example: hash table collision (invisible in interface executed with some data values spec) • Corrupt state may not propagate through execution to • Test suites that satisfy control flow adequacy eventually lead to failure criteria could fail in revealing faults that can be • What is the value of structural coverage? caught with functional criteria – Increases confidence in thoroughness of testing – Typical case: missing path faults • Removes some obvious inadequacies (c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 5 (c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 6 Structural testing in practice Statement testing • Create functional test suite first, then measure • Adequacy criterion: each statement (or node in structural coverage to identify see what is missing the CFG) must be executed at least once • Interpret unexecuted elements – may be due to natural differences between specification and • Coverage: implementation # executed statements – or may reveal flaws of the software or its development process • inadequacy of specifications that do not include cases present in # statements the implementation • coding practice that radically diverges from the specification • Rationale: a fault in a statement can only be • inadequate functional test suites revealed by executing the faulty statement • Attractive because automated – coverage measurements are convenient progress indicators – sometimes used as a criterion of completion • use with caution: does not ensure effective test suites (c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 7 (c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 8

Example int cgi_decode (char *encoded , char *decoded ) Statements or blocks? A {char *eptr = encoded ; char *dptr = decoded ; int ok = 0; T 0 = B • Nodes in a control flow graph often represent { “” , “ test ” , while (*eptr) { False True “ test+case%1Dadequacy ” } basic blocks of multiple statements char c; C 17/18 = 94% Stmt Cov . c = *eptr; if (c == '+') { – Some standards refer to basic block coverage or False T 1 = True D E node coverage *dptr = ' '; elseif (c == '%') { { “ adequate+test } %0Dexecution%7U ” } – Difference in granularity, not in concept False True 18/18 = 100% Stmt Cov . G else F int digit _high = Hex_Values[*(++eptr)]; • No essential difference *dptr = *eptr; int digit _low = Hex_Values[*(++eptr)]; } if (digit_high == -1 || digit_low == -1) { T 2 = False True – 100% node coverage <-> 100% statement coverage { “ %3D ” , “ %A ” , “ a+b ” , H I else { ok = 1; “ test ” } *dptr = 16 * digit_high + } • but levels will differ below 100% digit_low; 18/18 = 100% Stmt Cov . } – A test case that improves one will improve the other • though not by the same amount, in general M ++dptr; L *dptr = '\0'; ++eptr; return ok ; } } (c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 9 (c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 10 “ All statements ” can miss some cases Coverage is not size int cgi_decode (char *encoded , char *decoded ) • Complete statement coverage may not imply {char *eptr = encoded ; A • Coverage does not depend on the number of char *dptr = decoded ; executing all branches in int ok = 0; test cases a program while (*eptr) { B • Example: True False – T 0 , T 1 : T 1 > coverage T 0 T 1 < cardinality T 0 char c; C – Suppose block F were c = *eptr; if (c == '+') { – T 1 , T 2 : T 2 = coverage T 1 T 2 > cardinality T 1 missing False True – Statement adequacy elseif (c == '%') { D *dptr = ' '; E } • Minimizing test suite size is seldom the goal would not require false False True branch from D to L else { F int digit_high = Hex_Values [*(++eptr)]; G – small test cases make failure diagnosis easier T 3 = *dptr = *eptr; int digit_low = Hex_Values [*(++eptr)]; } if (digit_high == -1 || digit_low == -1) { { “” , “ +%0D+%4J ” } True – a failing test case in T 2 gives more information for False 100% Stmt Cov . ok = 1; else { H I fault localization than a failing test case in T 1 No false branch from D } *dptr = 16 * digit_high + digit_low; } L ++dptr; *dptr = '\0'; M ++eptr; return ok ; } } (c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 11 (c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 12

Branch testing Statements vs branches • Adequacy criterion: each branch (edge in the • Traversing all edges of a graph causes all nodes CFG) must be executed at least once to be visited • Coverage: – So test suites that satisfy the branch adequacy # executed branches criterion for a program P also satisfy the statement adequacy criterion for the same program # branches • The converse is not true (see T 3 ) – A statement-adequate (or node-adequate) test suite T 3 = { “” , “ +%0D+%4J ” } may not be branch-adequate (edge-adequate) 100% Stmt Cov. 88% Branch Cov. (7/8 branches) T 2 = { “ %3D ” , “ %A ” , “ a+b ” , “ test ” } 100% Stmt Cov. 100% Branch Cov. (8/8 branches) (c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 13 (c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 14 “ All branches ” can still miss conditions Condition testing • Sample fault: missing operator (negation) • Branch coverage exposes faults in how a computation has been decomposed into cases digit_high == 1 || digit_low == -1 – intuitively attractive: check the programmer ’ s case • Branch adequacy criterion can be satisfied by analysis varying only digit_low – but only roughly: groups cases with the same – The faulty sub-expression might never determine the outcome result • Condition coverage considers case analysis in – We might never really test the faulty condition, even more detail though we tested both outcomes of the branch – also individual conditions in a compound Boolean expression • e.g., both parts of digit_high == 1 || digit_low == -1 (c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 15 (c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 16