Software Testing for Critical Systems Julien Fayolle and - - PowerPoint PPT Presentation

Software Testing for Critical Systems

Julien Fayolle and Sandrine-Dominique Gouraud1

1G´

enie logiciel, LRI, Universit´ e d’Orsay. {fayolle, gouraud}@lri.fr Web pages: www.lri.fr/∼fayolle and www.lri.fr/∼gouraud

University of York, CS seminar.

Fayolle & Gouraud Software Testing for Critical Systems 1

Overview of Testing

How does one test a software? ➪ Testing all entries is not possible. ➪ We don’t prove the software right. ➘ Create a test set. Functional testing: specifications, black box Structural testing: structure and code, glass box Statistical testing: draw at random from the input domain ➘ Run the program on the test set. ➘ Compare the outputs to the expected outputs (oracle).

Fayolle & Gouraud Software Testing for Critical Systems 2

Statistical Testing

Software for critical systems (planes’ motors, nuclear plants’ safety, missiles, . . . ); Intensive statistical testing; Provide quantitative figures on the test set; Provide adequacy figures on the test methods; Drawback: coverage of rare cases. factorial(integer n) if(n!=0) fact:=1; for i from n to 2 fact:=fact*i; else fact:=0;

Fayolle & Gouraud Software Testing for Critical Systems 3

Statistical Structural Testing

First answer: [Th´ evenod-Fosse & Waeselynck’91, LAAS] ➠ Combining statistical and structural testing. ➠ Compute a distribution on the input domain and draw tests inputs accordingly. ➠ How representative is the test set?

Fayolle & Gouraud Software Testing for Critical Systems 4

Test quality

1 2 3 4 7

a b c e f g h j k d

6 5

Test quality qn = 1 − (1 − q)n, where n is the number of tests and q the minimal probability to reach one element.

Fayolle & Gouraud Software Testing for Critical Systems 5

AuGuSTe

Drawback: the input distribution has to be computed by hand.

AuGuSTe.

➔ Automated Generation of Statistical Tests, [Gouraud et alii ’04] First programmed for C code. Coded in Objective Caml; uses GATEL, MuPAD, Eclipse, Prolog, C, and Java. Statistical structural testing. Drawing uniformly at random paths of a given length in the control graph. AuGuSTe web site: http://www.lri.fr/∼fayolle/Auguste.html

Fayolle & Gouraud Software Testing for Critical Systems 6

Structural coverage criteria

Given a control flow graph All paths All i-paths Branch coverage: all edges Block coverage: all instructions Predicate coverages Test quality for statistical structural tests qn minimal probability to reach one element with a test set of size n. qn = 1 − (1 − q)n.

Fayolle & Gouraud Software Testing for Critical Systems 7

Random generation of paths

Goal: optimize the test quality i.e. maximize the smallest probability to reach an element. Combinatorial description of the control graph Uniform drawing of path (criteria consideration)

7 1 2 3 6 5 d f a b c h e j k g 4

Atoms are edges G = ac(egj + fG1) + bdG1, and G1 = hj + k.

Fayolle & Gouraud Software Testing for Critical Systems 8

How does AuGuSTe work?

Inputs Program Coverage criteria Number of tests (test quality) Maximal length of the paths (elementary path)

• 1. Analysis: building the combinatorial structure of the control

graph

• 2. Drawing paths uniformly at random in the graph
• 3. Constraint resolution (GATEL)

Output Test set.

Fayolle & Gouraud Software Testing for Critical Systems 9

Testing methods comparison using mutants

Creating mutants : seed faults in the original program. A large repertoire of mutation operators: changing ≥ in ≤, > or <; changing 0 in 2; performance criteria: mutation score (kill count). stability of the fault detection ability ➳ average. Softwares for creating mutants Mothra [Offutt, DeMillo et alii’88]; SESAME [Crouzet, Th´ evenod-Fosse & Waeselynck’98]; µJava [Offutt, Ma & Kwon’03].

Fayolle & Gouraud Software Testing for Critical Systems 10

Experimental results

Function from industrial software (77 lines) 605 mutants (for some mutations). Test quality .9999 5 runs of test sets of size 850. min avg max Uniform testing 0.8950 NA 0.9150 Structural statistical testing 0.9898 0.9901 0.9915 AuGuSTe 0.9854 0.9854 0.9854

Table: Experimental mutation scores over 5 runs

Fayolle & Gouraud Software Testing for Critical Systems 11

GATEL

GATEL ➔ Test Suites Generation from Lustre Descriptions [Marre et alii, CEA and LRI] Uses Lustre descriptions. Lustre code or Lustre spec. (test) Uses constraint solving techniques, non-deterministic heuristics. Structural or functional software testing. GATEL web site :

http://www-list.cea.fr/labos/gb/LSL/test/gatel/index.html

Fayolle & Gouraud Software Testing for Critical Systems 12

Lustre code

node Alarm(stream0, s up, s down: real) returns (alarm s: bool); var upstream, downstream: bool let upstream = stream0 > s up; downstream = stream0 < s down ; alarm s = Automaton(upstream, upstream, downstream); tel; node Automaton(init, upstream, downstream: bool) returns (state: bool); let state = init -> if upstream and not pre(state) then true else if downstream and pre(state) then false else pre(state); let;

Fayolle & Gouraud Software Testing for Critical Systems 13

Lustre

Declarative language for specification and/or programming. Reactive synchronous systems. Data-flow. The specification allows the automatic generation of code.

Fayolle & Gouraud Software Testing for Critical Systems 14

Perspectives

✭ Extend the idea of uniform drawing to the synchronous data-flow languages. ✭ Application to Lustre ✭ Enrich GATEL with random generation of combinatorial structures. ✭ Provide precise bounds on the fault detection ability of AuGuSTe-like methods for imperative and declarative languages.

Fayolle & Gouraud Software Testing for Critical Systems 15