Generating Structured Test Data with Specific Properties using - PowerPoint PPT Presentation

Generating Structured Test Data with Specific Properties using Metaheuristic and Nested Monte-Carlo Search Simon Poulding Blekinge Institute of Technology Robert Feldt Chalmers University

Context software testing: generating test data for software for which inputs are highly structured Problem how to enable the test engineer to generate test data that is both valid and has desirable properties? Solution (1) allow the test engineer to specify the construction of valid test inputs using generators written in a general purpose language (2) apply metaheuristic and Monte-Carlo tree search algorithms to optimise the generation process for the desirable properties

GödelTest + Metaheuristic Search + Nested Monte-Carlo Search Application: Robustness Testing

generator @generator AZStrings begin start() = String( mult (letter)) letter() = choose(Bool) ? 'A' : 'B' end Julia implementation of GödelTest: https://github.com/simonpoulding/DataGenerators.jl

generator @generator AZStrings begin start() = String( mult (letter)) letter() = choose(Bool) ? 'A' : 'B' end choice model choice point choice point range: [0, ∞ ) range: [false,true] distribution: geometric ( p 1 ) distribution: Bernoulli ( p 2 ) Julia implementation of GödelTest: https://github.com/simonpoulding/DataGenerators.jl

generator @generator ABStrings begin start() = String( mult (letter)) letter() = choose(Bool)? 'A' : 'B' end choice point range: [0, ∞ ) distribution: geometric ( p 1 =0.5 ) “AB” probability “A” “” “BBA” 0 1 2 3 4 5

generator @generator ABStrings begin start() = String( mult (letter)) letter() = choose(Bool)? 'A' : 'B' end choice point range: [0, ∞ ) distribution: geometric ( p 1 =0.1 ) “BBABAB” probability “BBAAABAA” “ABBA” “AABAAB” 0 1 2 3 4 5

“AAABAA” ( p 1 , p 2 ) = (0.1, 0.2) “ABAAAAAA” “AAA” “BAAAAB” “BAB” “B” “” ( p 1 , p 2 ) = (0.5, 0.5) “ABBA”

GödelTest + Metaheuristic Search + Nested Monte-Carlo Search Application: Robustness Testing

“AAABAA” samples “ABAAABAA” generator “AAA” “BAAAAB” GödelSequences [3,0,4,1,2] property metrics parameters metaheuristic choice model search ( p 1 , p 2 ) = (0.1, 0.2)

target 1: size = 100 AND height = 36 height size target 2: size = 100 AND height = 6 Robert Feldt and Simon Poulding, Finding Test Data with Specific Properties via Metaheuristic Search, Proc. International Symposium on Software Reliability Engineering (ISSRE 2013)

80 60 Tree height 40 20 100 200 Tree size Boltzmann Sampler

80 80 60 60 Tree height Tree height 40 40 20 20 100 200 100 200 Tree size Tree size GödelTest using Differential Evolution

GödelTest + Metaheuristic Search + Nested Monte-Carlo Search Application: Robustness Testing

current game state child states simulation

current game state child states simulation

current game state child states simulation

current game state child states simulation

current game state child states simulation

current game state child states simulation 0.7

current game state child states 0.4 simulation 0.7

current game state child states 0.4 simulation 0.7 0.9

[ ] [0] [1] [2] [0,0] [1,0] [2,1] simulation [0,0,3] [2,1,2] 0.4 [0,0,3,0] [2,1,2,1] 0.7 0.9

size target: size = 100 Simon Poulding and Robert Feldt, Generating structured test data with specific properties using nested Monte-Carlo search, Proc. Genetic and Evolutionary Computation Conference (GECCO), 1279-1286, 2014

4000 consructions per target 3000 number of tree structure 2000 1000 0 Boltzmann NMCS 1 NMCS 2 NMCS 4 algorithm

150 time per target structure 125 100 (ms) 75 50 25 0 Boltzmann NMCS 1 NMCS 2 NMCS 4 algorithm

1.00 median coordinate 0.75 0.50 0.25 0.00 Boltzmann NMCS 1 NMCS 2 NMCS 4 NMCS 8 NMCS 16 algorithm

height size target: size = 101 AND height = 8

8 8 height height 0 0 1 101 1 101 size size ‘random’ sampler NMCS

GödelTest + Metaheuristic Search + Nested Monte-Carlo Search Application: Robustness Testing

<math> <mfrac> <mrow> <mn> 1 </mn> <mo> + </mo> <msqrt> JEuclid <mn> 5 </mn> </msqrt> </mrow> <mn> 2 </mn> </mfrac> </math> Simon Poulding and Robert Feldt Generating Controllably Invalid and Atypical Inputs for Robustness Testing, Proc. International Workshop on Testing Extra-Functional Properties (ITEQS 2017)

A 2 modifying choice model A 1 I 2 I 1 T mutating typical test set choice model

normalised information content in errors/warnings 0.005 0.015 0.01 0 T A 1 T A 1 A 2 A 2 I 1 I 1 I 2 I 2

Context software testing: generating test data for software for which inputs are highly structured Problem how to enable the test engineer to generate test data that is both valid and has desirable properties? Solution (1) allow the test engineer to specify the construction of valid test inputs using generators written in a general purpose language (2) apply metaheuristic and Monte-Carlo tree search algorithms to optimise the generation process for the desirable properties