SLIDE 1
Test Code Coverage
White box testing
SLIDE 2 Statement Coverage – example 1
n Make sure every statement in the program is executed at
least once
n Example
n if a < b then c = a+b ; d = a*b
/* 1 */
else c = a*b ; d = a+b /* 2 */
if c < d then x = a+c ; y = b+d /* 3 */
else x = a*c ; y = b*d /* 4 */
n Statement coverage – can do with 2 tests
n Execute 1 & 3 with a < b & a+b < a*b
n a = 2 ; b = 5
n Execute 2 & 4 with a >= b & a*b >= a+b
n a = 5 ; b = 2
1 3 4 2 T T F F
Test–2
SLIDE 3 Statement Coverage
n Loops – only 1 test required
n Execute body at least once n If there are choices in the body, need to execute body with
enough tests to execute the statements in the body, analogous to example 1.
Body T F
Test–3
SLIDE 4
Statement Coverage How
n How do you know you have statement coverage?
TCC–4
SLIDE 5
Statement Coverage How
n How do you know you have statement coverage? n Instrument your program with an array of flags initialized to
false
n Set a unique flag to true in each block of statements n Run your test n If all flags are true then you have achieved statement
coverage
TCC–5
SLIDE 6 Problems with statement coverage
n A statement must be executed with different values for the
relevant variables to be fully tested
n Loop bodies may need to be iterated many times to reveal
issues
n Not all statements are equally important n Only the true branch of an if statement may be executed but
coverage may be 90% for the statement if the false branch is
TCC–6
SLIDE 7 Next level of improvement
n Path coverage
n Alias 1: Segment coverage n Alias 2: DD coverage
n Decision to decision coverage
TCC–7
SLIDE 8 Path Coverage – example 1
n Every path in the program is executed at least once n Example
n if a < b then c = a+b ; d = a*b
/* 1 */
else c = a*b ; d = a+b /* 2 */
if c < d then x = a+c ; y = b+d /* 3 */
else x = a*c ; y = b*d /* 4 */
n Path coverage – 4 tests
n Execute 1 & 3 with a < b & a+b < a*b
a = 2 ; b = 5
n Execute 2 & 4 with a >= b & a*b >= a+b a = 5 ; b = 2
Add for path
n Execute 1 & 4 with a < b & a+b >= a*b a = 0 ; b = 1 n Execute 2 & 3 with a >= b & a*b < a+b a = 1 ; b = 0
1 3 4 2 T T F F
TCC–8
SLIDE 9 Path Coverage – example 1
n Loops – 3 tests required
n Execute body zero times, once in the path, many in the path
n Once is not enough as frequently first time through is a
special case
n Path coverage usually requires exponential increase in tests
as the number of choices and loops increases
n Due to multiplication
n two loops in sequence – 9 tests n three loops in sequence – 27 tests n ten if....then...else in sequence – 1024 tests
TCC–9
SLIDE 10 Path Coverage – example 2 – the problem
n Convert an integer represented as a decimal string to a real
number.
n ASCII string "123.456" ==> 123.456 in binary
n The EBNF for the input
Input ::= +[ Spaces ] [ + , - ] [ IntegerPart ] [ '.' [ DecimalPart ] ]; IntegerPart ::= +( DecimalDigit ); DecimalPart ::= +( DecimalDigit ); DecimalDigit ::= ( '0' , '1', '2' ,'3' ,'4' ,'5' ,'6' ,'7' ,'8' , '9');
TCC–10
SLIDE 11
Path Coverage – example 2 – the algorithm
1 Skip any leading spaces 2 Determine what the sign of the number is 3 Get the integer part of the number by scanning either to the end of the number or a decimal point 4 Continue building the integer representation of the input as if there was no decimal point, meanwhile counting the number of decimal digits 5 Compute the real number representation from the sign, integer representation and count of decimal digits
TCC–11
SLIDE 12 Path Coverage – example 2 – test cases
n 2 tests are sufficient for statement coverage
n positive and negative real numbers.
n 162 tests estimated for all paths.
n 3 cases first loop – step 1 skip lead spaces n 3 cases first if statement – step 2 determine sign n 3 cases second loop – step 3 get integer part n 2 cases second if statement – step 3 check decimal point n 3 cases third loop – step 4 get decimal part
n Not all cases are possible -- for example if there is no
'.' (second if statement), then the third loop cannot not be executed one or many times, only zero times.
TCC–12
SLIDE 13
Path Coverage How
n How to you know you have path coverage?
TCC–13
SLIDE 14
Path Coverage How
n How to you know you have path coverage? n As for statement coverage have an array of flags with a flag in
each block of statements
n Compare the pattern of true flags with the expected statement
blocks in each path
n Continue until every path pattern has been matched
TCC–14
SLIDE 15 Other kinds of coverage
n Branch coverage
n Every edge from a statement node is executed at least once n At least one true and one false evaluation for each predicate
TCC–15
SLIDE 16 Other kinds of coverage
n Multi-condition coverage
n All true-false combinations of simple conditions in
compound predicates are considered at least once
n Guarantees statement, branch and predicate coverage n Does not guarantee path coverage
TCC–16
SLIDE 17 Other kinds of coverage
n Dataflow coverage
n Focuses on the points at which variables receive values
and the points at which these values are used
n And more …
n
EECS 4313 – Software Engineering Testing
TCC–17
SLIDE 18
When is testing done?
TCC–18
SLIDE 19
When is testing done?
n Short answer: Never!
TCC–19
SLIDE 20 When is testing done?
n Short answer: Never! n A bit longer answer:
n When all features of the system have been tested with all
possible inputs that could make a difference
TCC–20
SLIDE 21 When is testing done?
n Short answer: Never! n A bit longer answer:
n When all features of the system have been tested with all
possible inputs that could make a difference
n In practice this is hard to determine
TCC–21
SLIDE 22 When is testing done?
n Short answer: Never! n A bit longer answer:
n When all features of the system have been tested with all
possible inputs that could make a difference
n In practice this is hard to determine n Metrics such as code coverage can be used to give an idea of
how sufficient the testing is
TCC–22
SLIDE 23 Statement Code Metric
n Observe the system as it is running n Keep track of how many of the statements in the code were
executed at least once
n Divide by the total number of statements in the system n A comprehensive test suite is important n Typically, it is hard to get high coverage
n Anything above 70% is pretty good for a large system
TCC–23
SLIDE 24 Software engineering guideline
n Low code coverage indicates that more testing must be done n High code coverage gives little information about the quality
n Let’s see a demo (EclEmma).
TCC–24
SLIDE 25 Homework
n Short term (days)
n Install EclEmma and determine your statement test
coverage
n Demonstrate your ability to do statement test coverage in
Lab 9
n Long term (project)
n Improve your testing to raise coverage close to100%
n Your final submission must discuss test code coverage in
your Testing document
TCC–25
SLIDE 26 EclEmma
n Install EclEmma by doing the following in Eclipse
n Select Help à Install New Software n In the install dialog enter the following in the Work with
field, and press Return
n http://update.eclemma.org/
n Check the latest EclEmma version and press Next n Follow the steps in the installation wizard
n Get the User Guide at the following location
n http://eclemma.org/installation.html
