Syntax-based test coverage (part 2) - grammar-based testing - - - PowerPoint PPT Presentation

Verificação e Validação de Software Departamento de Informática Faculdade de Ciências da Universidade de Lisboa

Eduardo Marques, Vasco Thudichum Vasconcelos

Syntax-based test coverage (part 2)

• grammar-based testing -

Basic grammar concepts Syntax-based Coverage Criteria Specification-based Grammars Input Space Grammars

Grammar-based testing

Grammar-based testing

Inputs for the SUT are many times (or can be seen as) defined by a grammar. Testing the SUT with valid inputs: exercise productions of the grammar according to some criterion. Testing the SUT with invalid inputs: consider grammar-based mutations are also considered to test the SUT with invalid inputs.

Examples

XML program data (in particular XML data with associated XML schemas) Programs written in a programming language (to test compilers) ! Assorted data formats and instances of them (JSON, ASN.1, …) …

2

expr ::= id | num | expr op expr id ::= letter | letter id num ::= digit | digit num

• p ::= + | - | * | /

letter ::= a | … | z digit ::= 0 | 1 | 2 | … | 9

Example grammar

root symbol terminal symbols non-terminal symbols production rule 6 non-terminal symbols 40 (4+26+10) terminal symbols 47 production rules (3+2+2+4+26+10)

3

A simple grammar for a toy language of arithmetic expressions in BNF notation

Derivations and syntax tree

Sample derivations:

a expr => id => letter => a 43 expr => num => digit num => 4 num => 4 digit => 43 ab+12 expr => expr op expr => expr + expr => ... => ab+12

4

expr expr expr

• p

+ id num letter id a b letter 1 2 digit num digit

Syntax tree for

ab+12 expr ::= id | num | expr op expr id ::= letter | letter id num ::= digit | digit num

• p ::= + | - | * | /

letter ::= a | … | z digit ::= 0 | 1 | 2 | … | 9

Coverage criteria

Terminal Symbol Coverage (TSC) TR contains each terminal in the grammar. One test case per terminal. Toy language example: 40 test requirements Production Coverage (PDC) TR contains each production rule in the grammar. One test case per production (hence PDC subsumes TSC). Toy language example: 47 test requirements Derivation Coverage (DC) TR contains every possible string that can be derived from the grammar. One test case per derivation required. Not practical - TR usually infinite as in our toy language. When applicable, DC subsumes PDC.

5

Exercise 1

Suppose we wish to test a parser or interpreter for the example toy language.

• 1. Define a test set (i.e., a set of grammar derivations) that satisfies

TSC but not PDC.

• 2. Complete the previous test set to satisfy PDC.

6

expr ::= id | num | expr op expr id ::= letter | letter id num ::= digit | digit num

• p ::= + | - | * | /

letter ::= a | … | z digit ::= 0 | 1 | 2 | … | 9

Exercise 1 (solution)

• 1. We can consider the following tests for TSC:

a b … z (26 tests) 0 1 … 9 (10 tests) a+b a-b a*b a/b (4 tests) PDC is not satisfied since no test case covers productions id ::= letter id or num ::= digit num

• 2. To satisfy PDC we can, for instance, additionally consider:

ab 12

7

expr ::= id | num | expr op expr id ::= letter | letter id num ::= digit | digit num

• p ::= + | - | * | /

letter ::= a | … | z digit ::= 0 | 1 | 2 | … | 9

Grammar-based mutation

Mutations over a grammar can be considered with the purpose of generating invalid derivations. Invalid derivations must be recognized as errors by the software under test. Example mutation operators:

Terminal and nonterminal deletion: remove a terminal or nonterminal symbol from a production. Terminal and non-terminal duplication: duplicate a terminal or nonterminal symbol in a production. Terminal replacement: replace a terminal by another terminal. Non-terminal replacement

8

Grammar mutations - example

9

expr ::= id | num | expr op expr | ε | op expr | expr expr | expr op id ::= letter | letter id num ::= digit | digit num

• p ::= + | - | * | /

letter ::= a | … | z digit ::= 0 | 1 | 2 | … | 9

Example mutant derivations: 1) ε , 2) +a, 3) aa, 4) 12+ Exercise 2: Consider nonterminal duplication over expr. Describe the mutated grammar and provide one mutant derivation per each mutated production rule.

empty string mutant productions resulting from nonterminal deletion for expr productions

Nonterminal duplication

expr ::= id | num | expr op expr id ::= letter | letter id num ::= digit | digit num

• p ::= + | - | * | /

letter ::= a | … | z digit ::= 0 | 1 | 2 | … | 9

Exercise 2: Consider nonterminal duplication over expr. Describe the mutated grammar and provide one mutant derivation per each mutated production rule.

Exercise 2 (solution)

11

expr ::= id | num | expr op expr | id id | num num | expr expr op expr | expr op op expr | expr op expr expr id ::= letter | letter id num ::= digit | digit num

• p ::= + | - | * | /

letter ::= a | … | z digit ::= 0 | 1 | 2 | … | 9 mutant productions resulting from nonterminal duplication

Example strings from mutated rules (not all of them are invalid inputs): 1) aa (valid), 2) 1213 (valid), 3) a +1a+b, 4) a++10, 5) a+1a The “id id” and “num num” rules always produce valid inputs. Some derivations of “expr expr op expr” and “expr

• p expr expr” can be valid (e.g. ab+c, a+bc).

Derivations of “expr op op expr” are always invalid.

Specification-based Mutation

• Address specifications as finite state machines
• A finite state machine is essentially a graph G, as

defined in the Graph Coverage lecture, with a set

• f states (nodes), a set of initial states (initial

nodes), and a transition relation (the set of edges)

• Finite state descriptions can capture system

behavior at a very high level – suitable for communicating with the end user. FSM are useful for system testing.

• There are powerful analysis tools for FST: Model
• Checkers. Produce counterexamples for

properties that do not hold in the FSM

FSM _ Example

• States described by the

truth values of two variables, x and y

A&O, Figure 5.4

FSM Mutation Operators

• Constant replacement:

Replace the “next” value of a variable by a different constant

• E.g., In state FT, turn into T

the next value of y

A&O, Figure 5.5

Killing a mutant

• We need a sequence of

states that is allowed by the transition relation of the

• riginal state machine, but

not by the mutated state

• machine. Such a sequence is

precisely a test case that kills the mutant

• E.g., FF → TT → FT → TF
• Finding a test to kill a mutant
• f a FSM can be automated

using a model checker

A&O, Figure 5.5

Model Checking

• A model checker takes two

inputs: A FSM and a statement of some property, expressed in a temporal logic.

• In this case we are

interested in: It is always the case that if x=false and y=true then eventually y=false

• ☐(¬x⋀y → ♢¬y)

A&O, Figure 5.5

Counterexample

• The model checker

produces the following computation (a sequence

• f states):
• 1. FF
• 2. TT
• 3. FT
• 4. Goto 2
• If the model checker does

not produce a counterexample, we know that the mutant is equivalent

A&O, Figure 5.5

☐(¬x⋀y → ♢¬y)

Exercise 3

• Consider the mutant “wantP = false; 4”. Write a

temporal logic formula that kills the mutant

Input Space Grammars

• One common use of grammars is to define the

syntax of the inputs to a program, method, or software component

• We will look at
• regular expressions
• XML

Regular expressions

• Valid inputs:
• deposit 739 $12.35 deposit 644

$12.35 debit 739 $19.22

• Invalid inputs:
• deposit 644 $.35
• deposit 23 $12.35
• withdraw 739 $12.35

(deposit Account Amount | debit Account Amount)∗ where Account is Digit Digit Digit Digit is 0|1|2|3|4|5|6|7|8|9 Amount is $ Digit+ . Digit Digit

Mutation for regular expressions

• 1. Subexpression replacement


• 2. Subexpression deletion


• 3. Subexpression duplication

Digit is 0|1|2|3|z|5|6|7|8|9 Account is Digit Digit Amount Digit is 0|1|2|3|5|6|7|8|9 Account is Digit Digit Digit is 0|1|2|3|55|6|7|8|9 Account is Digit Digit Digit Digit

Exercise 4

(deposit Account Amount | debit Account Amount)∗ where Account is Digit Digit Digit Digit is 0|1|2|3|4|5|6|7|8|9 Amount is $ Digit+ . Digit Digit Digit is 0|1|2|3|z|5|6|7|8|9 Account is Digit Digit Amount Digit is 0|1|2|3|5|6|7|8|9 Account is Digit Digit Digit is 0|1|2|3|55|6|7|8|9 Account is Digit Digit Digit Digit

For each of the six mutants find a string that is in the original RE but not in the mutated

• ne

XML

• Commonly used in

Web applications (JSON also gaining popularity)

• XML documents

(or messages) must be well formed: opening tags matched by corresponding closing tags

A&O, Figure 5.8

XML Schemas

• XML documents

can be constrained by grammar definitions, written in XML Schemas

• In example:

unbounded number of books; 6 pieces of info per book; ISBN

• ptional; price is

decimal w/ 2 digits after the ‘,’ and min value 0; …

A&O, Figure 5.9

XML Mutation

• Idea: apply mutation to XML schemas to produce

invalid messages

• Some programs use XML parsers that validate the

messages against the grammar. If they do, it is likely that the software will usually behave correctly on invalid messages, but testers still need to verify this

• It is fairly common for programs to use XML

messages without having an explicit schema

• definition. In this case, it is very helpful for the test

engineer to develop the schema as a first step to developing tests

XML Mutation _ Exercise 5

• Given the above lines in the books schema

construct mutants that modify the values of constraining facets

• Mutate ISBN number of occurrences
• Mutate price in its various constraining facets

<xs:element name="book" maxOccurs="unbounded"> … <xs:element name="ISBN" type="xs:isbnType" minOccurs=“0"/> <xs:element name="price" type=“xs:decimal" fractionDigits=“2” minInclusive=“0”/> … </xs:element>

Exercise 5 (Cont’d)

• mutate the

constraining facets of dates

• mutate facets
• f ISBN

<xs:element name="book" maxOccurs="unbounded"> … <xs:element name="ISBN" type="xs:isbnType" minOccurs=“0"/> <xs:element name="year" type=“yearType"/> … </xs:element> <xs:simpleType name=“yearType"> <xs:restriction base="xs:int"> <xs:totalDigits value=“4"/> </xs:restriction> </xs:simpleType>
 <xs:simpleType name="isbnType"> <xs:restriction base=“xs:string"> <xs:pattern value="[0-9]{10}"/> </xs:restriction> </xs:simpleType>