Generating Structured Test Data with Specific Properties using - - PowerPoint PPT Presentation

▶
generating structured test data with specific properties
SMART_READER_LITE
LIVE PREVIEW

Generating Structured Test Data with Specific Properties using - - PowerPoint PPT Presentation

Dec 16, 2023 238 likes •599 views

Generating Structured Test Data with Specific Properties using Metaheuristic and Nested Monte-Carlo Search Simon Poulding Blekinge Institute of Technology Robert Feldt Chalmers University Context software testing: generating test data for

slide-1
SLIDE 1

Generating Structured Test Data with Specific Properties using Metaheuristic and Nested Monte-Carlo Search

Simon Poulding

Blekinge Institute of Technology

Robert Feldt

Chalmers University

slide-2
SLIDE 2

Problem how to enable the test engineer to generate test data that is both valid and has desirable properties? Solution (1) allow the test engineer to specify the construction of valid test inputs using generators written in a general purpose language (2) apply metaheuristic and Monte-Carlo tree search algorithms to optimise the generation process for the desirable properties Context software testing: generating test data for software for which inputs are highly structured

slide-3
SLIDE 3

+ Metaheuristic Search GödelTest + Nested Monte-Carlo Search Application: Robustness Testing

slide-4
SLIDE 4

generator

@generator AZStrings begin start() = String(mult(letter)) letter() = choose(Bool)? 'A' : 'B' end

Julia implementation of GödelTest: https://github.com/simonpoulding/DataGenerators.jl

slide-5
SLIDE 5

generator

@generator AZStrings begin start() = String(mult(letter)) letter() = choose(Bool)? 'A' : 'B' end range: [0,∞) distribution: geometric (p1)

choice point

range: [false,true] distribution: Bernoulli (p2)

choice point choice model

Julia implementation of GödelTest: https://github.com/simonpoulding/DataGenerators.jl

slide-6
SLIDE 6

generator

@generator ABStrings begin start() = String(mult(letter)) letter() = choose(Bool)? 'A' : 'B' end range: [0,∞) distribution: geometric (p1=0.5)

choice point

probability 1 2 3 4 5

“AB” “A” “” “BBA”

slide-7
SLIDE 7

generator

@generator ABStrings begin start() = String(mult(letter)) letter() = choose(Bool)? 'A' : 'B' end

choice point

probability 1 2 3 4 5

range: [0,∞) distribution: geometric (p1=0.1) “BBABAB” “BBAAABAA” “ABBA” “AABAAB”

slide-8
SLIDE 8

“AAABAA” “ABAAAAAA” “AAA” “BAAAAB” “BAB” “B” “” “ABBA” (p1, p2) = (0.1, 0.2) (p1, p2) = (0.5, 0.5)

slide-9
SLIDE 9

+ Metaheuristic Search GödelTest + Nested Monte-Carlo Search Application: Robustness Testing

slide-10
SLIDE 10

property metrics

generator choice model metaheuristic search

parameters samples GödelSequences [3,0,4,1,2] (p1, p2) = (0.1, 0.2)

“AAABAA” “ABAAABAA” “AAA” “BAAAAB”

slide-11
SLIDE 11

target 1: size = 100 AND height = 36 target 2: size = 100 AND height = 6 size height

Robert Feldt and Simon Poulding, Finding Test Data with Specific Properties via Metaheuristic Search,

  • Proc. International Symposium on Software Reliability Engineering (ISSRE 2013)
slide-12
SLIDE 12

20 40 60 80 100 200

Tree size Tree height

Boltzmann Sampler

slide-13
SLIDE 13

20 40 60 80 100 200

Tree size Tree height

20 40 60 80 100 200

Tree size Tree height

GödelTest using Differential Evolution

slide-14
SLIDE 14

+ Metaheuristic Search GödelTest + Nested Monte-Carlo Search Application: Robustness Testing

slide-15
SLIDE 15

current game state child states simulation

slide-16
SLIDE 16

current game state child states simulation

slide-17
SLIDE 17

current game state child states simulation

slide-18
SLIDE 18

current game state child states simulation

slide-19
SLIDE 19

current game state child states simulation

slide-20
SLIDE 20

current game state child states 0.7 simulation

slide-21
SLIDE 21

current game state child states 0.7 0.4 simulation

slide-22
SLIDE 22

current game state child states 0.7 0.4 0.9 simulation

slide-23
SLIDE 23

[ ] [0] [1] [2] [1,0] [0,0] [2,1] [0,0,3] [2,1,2] [0,0,3,0] [2,1,2,1] 0.7 0.4 0.9 simulation

slide-24
SLIDE 24

target: size = 100 size

Simon Poulding and Robert Feldt, Generating structured test data with specific properties using nested Monte-Carlo search,

  • Proc. Genetic and Evolutionary Computation Conference (GECCO), 1279-1286, 2014
slide-25
SLIDE 25

number of tree consructions per target structure 1000 2000 3000 4000 algorithm Boltzmann NMCS 1 NMCS 2 NMCS 4

slide-26
SLIDE 26

time per target structure (ms) 25 50 75 100 125 150 algorithm Boltzmann NMCS 1 NMCS 2 NMCS 4

slide-27
SLIDE 27

0.00 0.25 0.50 0.75 1.00 Boltzmann NMCS 1 NMCS 2 NMCS 4 NMCS 8 NMCS 16

algorithm median coordinate

slide-28
SLIDE 28

target: size = 101 AND height = 8 size height

slide-29
SLIDE 29

8 1 101

size height

8 1 101

size height

‘random’ sampler NMCS

slide-30
SLIDE 30

+ Metaheuristic Search GödelTest + Nested Monte-Carlo Search Application: Robustness Testing

slide-31
SLIDE 31

JEuclid

<math> <mfrac> <mrow> <mn> 1 </mn> <mo> + </mo> <msqrt> <mn> 5 </mn> </msqrt> </mrow> <mn> 2 </mn> </mfrac> </math>

Simon Poulding and Robert Feldt Generating Controllably Invalid and Atypical Inputs for Robustness Testing,

  • Proc. International Workshop on Testing Extra-Functional Properties (ITEQS 2017)
slide-32
SLIDE 32

T A2 A1 I1 I2 modifying choice model mutating choice model typical test set

slide-33
SLIDE 33

0.005 0.01 0.015 A1 I1 I2 A2 T T A1 A2 I1 I2 normalised information content in errors/warnings

slide-34
SLIDE 34

Problem how to enable the test engineer to generate test data that is both valid and has desirable properties? Solution (1) allow the test engineer to specify the construction of valid test inputs using generators written in a general purpose language (2) apply metaheuristic and Monte-Carlo tree search algorithms to optimise the generation process for the desirable properties Context software testing: generating test data for software for which inputs are highly structured