Time to Learn Learning Timed Automata from Tests Bernhard K. - - PowerPoint PPT Presentation

▶
time to learn learning timed automata from tests
SMART_READER_LITE
LIVE PREVIEW

Time to Learn Learning Timed Automata from Tests Bernhard K. - - PowerPoint PPT Presentation

Apr 24, 2023 546 likes •1.11k views

Institute of Software Technology tugraz Time to Learn Learning Timed Automata from Tests Bernhard K. Aichernig Kim Guldstrand Larsen Martin Tappler Florian Lorber Institute of Software Technology, Graz University of Technology, Austria

slide-1
SLIDE 1

tugraz

Institute of Software Technology

Time to Learn – Learning Timed Automata from Tests

Martin Tappler Bernhard K. Aichernig Kim Guldstrand Larsen Florian Lorber

Institute of Software Technology, Graz University of Technology, Austria Department of Computer Science, Aalborg University

August 28th, 2019

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 1 / 15

slide-2
SLIDE 2

tugraz

Institute of Software Technology

Learning a Car Alarm System

Car Alarm System

  • pen

close lock unlock armed sound flash

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 2 / 15

slide-3
SLIDE 3

tugraz

Institute of Software Technology

Learning a Car Alarm System

Car Alarm System

  • pen

close lock unlock armed sound flash

lock?,4.1 close?,15.4 armed!,17.4 lock?,3.2 close?,13.7 unlock?,14.1

  • pen?,23.1

close?,2.9 lock?,16.2 armed!,18.2

  • pen?,29.5

flash!,29.5 close?,0.7 lock?,5.4 armed!,7.4 unlock?,21.5

  • ff!,21.5

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 2 / 15

slide-4
SLIDE 4

tugraz

Institute of Software Technology

Learning a Car Alarm System

Car Alarm System

  • pen

close lock unlock armed sound flash

lock?,4.1 close?,15.4 armed!,17.4 lock?,3.2 close?,13.7 unlock?,14.1

  • pen?,23.1

close?,2.9 lock?,16.2 armed!,18.2

  • pen?,29.5

flash!,29.5 close?,0.7 lock?,5.4 armed!,7.4 unlock?,21.5

  • ff!,21.5

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 2 / 15

slide-5
SLIDE 5

tugraz

Institute of Software Technology

Learning a Car Alarm System

Car Alarm System

  • pen

close lock unlock armed sound flash

lock?,4.1 close?,15.4 armed!,17.4 lock?,3.2 close?,13.7 unlock?,14.1

  • pen?,23.1

close?,2.9 lock?,16.2 armed!,18.2

  • pen?,29.5

flash!,29.5 close?,0.7 lock?,5.4 armed!,7.4 unlock?,21.5

  • ff!,21.5

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 2 / 15

slide-6
SLIDE 6

tugraz

Institute of Software Technology

Learning a Car Alarm System

  • pen

close lock unlock armed sound flash

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 2 / 15

slide-7
SLIDE 7

tugraz

Institute of Software Technology

Motivation – Learning-Based Verification

System Under Learning Test Driver Test Cases

  • utSUT(Ping) =
  • utSUT(Con) =
  • utSUT(Con · Ping) =

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 3 / 15

slide-8
SLIDE 8

tugraz

Institute of Software Technology

Motivation – Learning-Based Verification

System Under Learning Test Driver execute tests Test Cases

  • utSUT(Ping) = ConC
  • utSUT(Con) = ConAck
  • utSUT(Con · Ping) = ConAck · Pong

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 3 / 15

slide-9
SLIDE 9

tugraz

Institute of Software Technology

Motivation – Learning-Based Verification

System Under Learning Test Driver execute tests Test Cases Learner

  • utSUT(Ping) = ConC
  • utSUT(Con) = ConAck
  • utSUT(Con · Ping) = ConAck · Pong

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 3 / 15

slide-10
SLIDE 10

tugraz

Institute of Software Technology

Motivation – Learning-Based Verification

System Under Learning Test Driver execute tests Test Cases Learner Model q0 start q1 Ping/ConnectionClosed Connect/ConnAck Ping/Pong Connect/ConnectionClosed learn consistent

  • utM(Ping) = ConC
  • utSUT(Ping) = ConC
  • utM(Con) = ConAck
  • utSUT(Con) = ConAck
  • utM(Con · Ping) = ConAck · Pong
  • utSUT(Con · Ping) = ConAck · Pong

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 3 / 15

slide-11
SLIDE 11

tugraz

Institute of Software Technology

Motivation – Learning-Based Verification

System Under Learning Test Driver execute tests Test Cases Learner Model q0 start q1 Ping/ConnectionClosed Connect/ConnAck Ping/Pong Connect/ConnectionClosed learn consistent

  • utM(Ping) = ConC
  • utSUT(Ping) = ConC
  • utM(Con) = ConAck
  • utSUT(Con) = ConAck
  • utM(Con · Ping) = ConAck · Pong
  • utSUT(Con · Ping) = ConAck · Pong

Verify Verification ◮ Model checking [Fiterau-Brostean et al., 2016], comparison of models [Aarts et al., 2012, Tappler et al., 2017] ◮ Issue: “we had to eliminate timing based behavior as well as re-transmissions” [Fiterau-Brostean et al., 2016]

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 3 / 15

slide-12
SLIDE 12

tugraz

Institute of Software Technology

Timed Automata

◮ Finite automata . . . ◮ with inputs and outputs ◮ extended with real-valued clocks

◮ used in guards ◮ reset upon transitions

◮ constraints limiting sojourn time ◮ Assumptions for

testing [Hessel et al., 2003]:

◮ output urgent:

  • utputs fire as soon as possible

◮ input enabled:

inputs must be accepted

◮ deterministic

q0 start q1 q2 q3 q4 A Lamp Touch Sensor

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 4 / 15

slide-13
SLIDE 13

tugraz

Institute of Software Technology

Timed Automata

◮ Finite automata . . . ◮ with inputs and outputs ◮ extended with real-valued clocks

◮ used in guards ◮ reset upon transitions

◮ constraints limiting sojourn time ◮ Assumptions for

testing [Hessel et al., 2003]:

◮ output urgent:

  • utputs fire as soon as possible

◮ input enabled:

inputs must be accepted

◮ deterministic

q0 start q1 q2 q3 q4 press? release? release? touch! starthold! release? endhold! A Lamp Touch Sensor

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 4 / 15

slide-14
SLIDE 14

tugraz

Institute of Software Technology

Timed Automata

◮ Finite automata . . . ◮ with inputs and outputs ◮ extended with real-valued clocks

◮ used in guards ◮ reset upon transitions

◮ constraints limiting sojourn time ◮ Assumptions for

testing [Hessel et al., 2003]:

◮ output urgent:

  • utputs fire as soon as possible

◮ input enabled:

inputs must be accepted

◮ deterministic

q0 start q1 q2 q3 q4 press? {c} release? c ≥ 5 release? c < 5 touch! starthold! c ≥ 10 release? endhold! A Lamp Touch Sensor

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 4 / 15

slide-15
SLIDE 15

tugraz

Institute of Software Technology

Timed Automata

◮ Finite automata . . . ◮ with inputs and outputs ◮ extended with real-valued clocks

◮ used in guards ◮ reset upon transitions

◮ constraints limiting sojourn time ◮ Assumptions for

testing [Hessel et al., 2003]:

◮ output urgent:

  • utputs fire as soon as possible

◮ input enabled:

inputs must be accepted

◮ deterministic

q0 start q1 q2 q3 q4 press? {c} release? c ≥ 5 release? c < 5 touch! starthold! c ≥ 10 release? endhold! A Lamp Touch Sensor

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 4 / 15

slide-16
SLIDE 16

tugraz

Institute of Software Technology

Timed Automata

◮ Finite automata . . . ◮ with inputs and outputs ◮ extended with real-valued clocks

◮ used in guards ◮ reset upon transitions

◮ constraints limiting sojourn time ◮ Assumptions for

testing [Hessel et al., 2003]:

◮ output urgent:

  • utputs fire as soon as possible

◮ input enabled:

inputs must be accepted

◮ deterministic

q0 start q1 q2 q3 q4 press? {c} release? c ≥ 5 release? c < 5 touch! starthold! c ≥ 10 release? endhold! A Lamp Touch Sensor

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 4 / 15

slide-17
SLIDE 17

tugraz

Institute of Software Technology

Timed Automata

◮ Finite automata . . . ◮ with inputs and outputs ◮ extended with real-valued clocks

◮ used in guards ◮ reset upon transitions

◮ constraints limiting sojourn time ◮ Assumptions for

testing [Hessel et al., 2003]:

◮ output urgent:

  • utputs fire as soon as possible

◮ input enabled:

inputs must be accepted

◮ deterministic

q0 start q1 q2 q3 q4 press? {c} release? c ≥ 5 release? c < 5 touch! starthold! c ≥ 10 release? endhold! A Lamp Touch Sensor release?,1.1 press?,2.3 release?,8.9 touch!,8.9

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 4 / 15

slide-18
SLIDE 18

tugraz

Institute of Software Technology

Automata Learning for Timed Systems

◮ Motivation: Model-based analysis for black-box timed systems ◮ Existing approaches:

◮ Passive learning of real-time

automata [Verwer et al., 2010, Verwer et al., 2012]: → does not distinguish inputs and outputs

◮ Active learning of event-recording

automata [Grinchtein et al., 2010, Grinchtein et al., 2006] → high runtime complexity

◮ Both: restrictions on clock resets

◮ Promising results of genetic programming in program synthesis

(e.g. mutual exclusion algorithms) [Katz and Peled, 2017]

◮ Apply genetic programing for timed automata ◮ Focus: generate models for testing

◮ input-enabled, arbitrary clock resets Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 5 / 15

slide-19
SLIDE 19

tugraz

Institute of Software Technology

Automata Learning for Timed Systems

◮ Motivation: Model-based analysis for black-box timed systems ◮ Existing approaches:

◮ Passive learning of real-time

automata [Verwer et al., 2010, Verwer et al., 2012]: → does not distinguish inputs and outputs

◮ Active learning of event-recording

automata [Grinchtein et al., 2010, Grinchtein et al., 2006] → high runtime complexity

◮ Both: restrictions on clock resets

◮ Promising results of genetic programming in program synthesis

(e.g. mutual exclusion algorithms) [Katz and Peled, 2017]

◮ Apply genetic programing for timed automata ◮ Focus: generate models for testing

◮ input-enabled, arbitrary clock resets Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 5 / 15

slide-20
SLIDE 20

tugraz

Institute of Software Technology

Automata Learning for Timed Systems

◮ Motivation: Model-based analysis for black-box timed systems ◮ Existing approaches:

◮ Passive learning of real-time

automata [Verwer et al., 2010, Verwer et al., 2012]: → does not distinguish inputs and outputs

◮ Active learning of event-recording

automata [Grinchtein et al., 2010, Grinchtein et al., 2006] → high runtime complexity

◮ Both: restrictions on clock resets

◮ Promising results of genetic programming in program synthesis

(e.g. mutual exclusion algorithms) [Katz and Peled, 2017]

◮ Apply genetic programing for timed automata ◮ Focus: generate models for testing

◮ input-enabled, arbitrary clock resets Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 5 / 15

slide-21
SLIDE 21

tugraz

Institute of Software Technology

Automata Learning for Timed Systems

◮ Motivation: Model-based analysis for black-box timed systems ◮ Existing approaches:

◮ Passive learning of real-time

automata [Verwer et al., 2010, Verwer et al., 2012]: → does not distinguish inputs and outputs

◮ Active learning of event-recording

automata [Grinchtein et al., 2010, Grinchtein et al., 2006] → high runtime complexity

◮ Both: restrictions on clock resets

◮ Promising results of genetic programming in program synthesis

(e.g. mutual exclusion algorithms) [Katz and Peled, 2017]

◮ Apply genetic programing for timed automata ◮ Focus: generate models for testing

◮ input-enabled, arbitrary clock resets Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 5 / 15

slide-22
SLIDE 22

tugraz

Institute of Software Technology

Automata Learning for Timed Systems

◮ Motivation: Model-based analysis for black-box timed systems ◮ Existing approaches:

◮ Passive learning of real-time

automata [Verwer et al., 2010, Verwer et al., 2012]: → does not distinguish inputs and outputs

◮ Active learning of event-recording

automata [Grinchtein et al., 2010, Grinchtein et al., 2006] → high runtime complexity

◮ Both: restrictions on clock resets

◮ Promising results of genetic programming in program synthesis

(e.g. mutual exclusion algorithms) [Katz and Peled, 2017]

◮ Apply genetic programing for timed automata ◮ Focus: generate models for testing

◮ input-enabled, arbitrary clock resets Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 5 / 15

slide-23
SLIDE 23

tugraz

Institute of Software Technology

Automata Learning for Timed Systems

◮ Motivation: Model-based analysis for black-box timed systems ◮ Existing approaches:

◮ Passive learning of real-time

automata [Verwer et al., 2010, Verwer et al., 2012]: → does not distinguish inputs and outputs

◮ Active learning of event-recording

automata [Grinchtein et al., 2010, Grinchtein et al., 2006] → high runtime complexity

◮ Both: restrictions on clock resets

◮ Promising results of genetic programming in program synthesis

(e.g. mutual exclusion algorithms) [Katz and Peled, 2017]

◮ Apply genetic programing for timed automata ◮ Focus: generate models for testing

◮ input-enabled, arbitrary clock resets Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 5 / 15

slide-24
SLIDE 24

tugraz

Institute of Software Technology

Genetic Programming

Evaluate Fitness Random Population Stop? Create New Population (Mutation & Crossover) Output Fittest timed automata yes no mutated timed automata

1

Create small random timed automata

2

Test system to get timed traces

3

Fitness: simulate automata

◮ # accepted traces ◮ # outputs of accepted traces ◮ determinism ◮ penalty for model size 4

New population: mutate & crossover

5

Stop if all traces accepted or

  • max. # rounds

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 6 / 15

slide-25
SLIDE 25

tugraz

Institute of Software Technology

Genetic Programming of TA – Basic

Test Evaluate Fitness Random Population Timed System Stop? Create New Population (Mutation & Crossover) Output Fittest timed traces timed automata yes no mutated timed automata

1

Create small random timed automata

2

Test system to get timed traces

3

Fitness: simulate automata

◮ # accepted traces ◮ # outputs of accepted traces ◮ determinism ◮ penalty for model size 4

New population: mutate & crossover

5

Stop if all traces accepted or

  • max. # rounds

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 6 / 15

slide-26
SLIDE 26

tugraz

Institute of Software Technology

Genetic Programming of TA – Basic

Test Evaluate Fitness Random Population Timed System Stop? Create New Population (Mutation & Crossover) Output Fittest timed traces timed automata yes no mutated timed automata

1

Create small random timed automata

2

Test system to get timed traces

3

Fitness: simulate automata

◮ # accepted traces ◮ # outputs of accepted traces ◮ determinism ◮ penalty for model size 4

New population: mutate & crossover

5

Stop if all traces accepted or

  • max. # rounds

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 6 / 15

slide-27
SLIDE 27

tugraz

Institute of Software Technology

Genetic Programming of TA – Basic

Test Evaluate Fitness Random Population Timed System Stop? Create New Population (Mutation & Crossover) Output Fittest timed traces timed automata yes no mutated timed automata

1

Create small random timed automata

2

Test system to get timed traces

3

Fitness: simulate automata

◮ # accepted traces ◮ # outputs of accepted traces ◮ determinism ◮ penalty for model size 4

New population: mutate & crossover

5

Stop if all traces accepted or

  • max. # rounds

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 6 / 15

slide-28
SLIDE 28

tugraz

Institute of Software Technology

Genetic Programming of TA – Basic

Test Evaluate Fitness Random Population Timed System Stop? Create New Population (Mutation & Crossover) Output Fittest timed traces timed automata yes no mutated timed automata

1

Create small random timed automata

2

Test system to get timed traces

3

Fitness: simulate automata

◮ # accepted traces ◮ # outputs of accepted traces ◮ determinism ◮ penalty for model size 4

New population: mutate & crossover

5

Stop if all traces accepted or

  • max. # rounds

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 6 / 15

slide-29
SLIDE 29

tugraz

Institute of Software Technology

Genetic Programming of TA – Basic

Test Evaluate Fitness Random Population Timed System Stop? Create New Population (Mutation & Crossover) Output Fittest timed traces timed automata yes no mutated timed automata

1

Create small random timed automata

2

Test system to get timed traces

3

Fitness: simulate automata

◮ # accepted traces ◮ # outputs of accepted traces ◮ determinism ◮ penalty for model size 4

New population: mutate & crossover

5

Stop if all traces accepted or

  • max. # rounds

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 6 / 15

slide-30
SLIDE 30

tugraz

Institute of Software Technology

Creating a New Population – Detailed

◮ Probabilistic choice between mutation and crossover ◮ Fitness-based selection of parents from population ◮ Repeat npop times

Choose Operation Mutate Crossover Fitness-Based Selection Apply do npop times Population New Population pcross 1 − pcross

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 7 / 15

slide-31
SLIDE 31

tugraz

Institute of Software Technology

Creating a New Population – Detailed

◮ Probabilistic choice between mutation and crossover ◮ Fitness-based selection of parents from population ◮ Repeat npop times

Choose Operation Mutate Crossover Fitness-Based Selection Apply do npop times Population New Population pcross 1 − pcross

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 7 / 15

slide-32
SLIDE 32

tugraz

Institute of Software Technology

Genetic Programming of TA – Optimized

Test Evaluate Evaluate Random Global Population Random Local Population Timed System Stop? Create New Population Create New Population Migrate Output Fittest timed traces timed automata timed automata yes no ◮ Two populations:

1

Global search: all timed traces

2

Local search: timed traces not accepted by global population

◮ From local search to global

search:

◮ Migration ◮ Crossover Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 8 / 15

slide-33
SLIDE 33

tugraz

Institute of Software Technology

Mutation & Crossover (1)

◮ Mutation operators for changing all aspects of timed automata ◮ Chosen at random ◮ An operator inspired by passive automata learning: merge location

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 9 / 15

slide-34
SLIDE 34

tugraz

Institute of Software Technology

Mutation & Crossover (1)

◮ Mutation operators for changing all aspects of timed automata ◮ Chosen at random ◮ An operator inspired by passive automata learning: merge location

Merge Location Example

q0 start q1 q2 q3 ?press {c} ?release c ≥ 5 ?release c < 5 ?release !touch merge q2 and q3 q0 start q1 q2 ?press {c} ?release c ≥ 5 ?release c < 5 !touch ?release

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 9 / 15

slide-35
SLIDE 35

tugraz

Institute of Software Technology

Mutation & Crossover (2)

◮ An operator inspired by active automata learning: split location

Split Location Example

q0 start q2 ?press {c} ?release c < 5 ?release c ≥ 5 !touch split q0 q0 start q1 q2 ?press {c} ?release c ≥ 5 ?release c < 5 ?release c < 5 ?release c ≥ 5 !touch ◮ Crossover: randomised product

◮ explore parents and synchronise on labels ◮ random combination of parents’ edges Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 10 / 15

slide-36
SLIDE 36

tugraz

Institute of Software Technology

Mutation & Crossover (2)

◮ An operator inspired by active automata learning: split location

Split Location Example

q0 start q2 ?press {c} ?release c < 5 ?release c ≥ 5 !touch split q0 q0 start q1 q2 ?press {c} ?release c ≥ 5 ?release c < 5 ?release c < 5 ?release c ≥ 5 !touch ◮ Crossover: randomised product

◮ explore parents and synchronise on labels ◮ random combination of parents’ edges Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 10 / 15

slide-37
SLIDE 37

tugraz

Institute of Software Technology

Challenge: Parameter Configuration

◮ Lots of Parameters

◮ # clocks, clock-bound range ◮ weights for fitness computation ◮ # tests, population size, # generations, test length ◮ crossover probability

→ We have guidelines

◮ Some are fixed

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 11 / 15

slide-38
SLIDE 38

tugraz

Institute of Software Technology

Challenge: Parameter Configuration

◮ Lots of Parameters

◮ # clocks, clock-bound range ◮ weights for fitness computation ◮ # tests, population size, # generations, test length ◮ crossover probability

→ We have guidelines

◮ Some are fixed

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 11 / 15

slide-39
SLIDE 39

tugraz

Institute of Software Technology

Challenge: Parameter Configuration

◮ Lots of Parameters

◮ # clocks, clock-bound range ◮ weights for fitness computation ◮ # tests, population size, # generations, test length ◮ crossover probability

→ We have guidelines

◮ Some are fixed

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 11 / 15

slide-40
SLIDE 40

tugraz

Institute of Software Technology

Experiments – a Learned Model (1)

◮ Automatic generation

  • f human-readable

models

◮ Experiments with:

◮ 40 random TA ◮ 4 TA from the

literature

◮ up to 26 locations

and 1 clock

◮ up to 10 locations

and 2 clocks

◮ Evaluation

1

learn from training data

2

simulate on test data

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 12 / 15

slide-41
SLIDE 41

tugraz

Institute of Software Technology

Experiments – a Learned Model (1)

◮ Automatic generation

  • f human-readable

models

◮ Experiments with:

◮ 40 random TA ◮ 4 TA from the

literature

◮ up to 26 locations

and 1 clock

◮ up to 10 locations

and 2 clocks

◮ Evaluation

1

learn from training data

2

simulate on test data

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 12 / 15

slide-42
SLIDE 42

tugraz

Institute of Software Technology

Experiments – a Learned Model (1)

◮ Automatic generation

  • f human-readable

models

◮ Experiments with:

◮ 40 random TA ◮ 4 TA from the

literature

◮ up to 26 locations

and 1 clock

◮ up to 10 locations

and 2 clocks

◮ Evaluation

1

learn from training data

2

simulate on test data

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 12 / 15

slide-43
SLIDE 43

tugraz

Institute of Software Technology

Experiments – a Learned Model (2)

◮ Results:

◮ successfully learned

all 44 models

◮ consistent with given

training data

◮ high accuracy on

test data

◮ Runtime:

◮ several minutes and

up to 20 hours

◮ not yet parallelized

◮ GUI demo: link in paper

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 13 / 15

slide-44
SLIDE 44

tugraz

Institute of Software Technology

Experiments – a Learned Model (2)

◮ Results:

◮ successfully learned

all 44 models

◮ consistent with given

training data

◮ high accuracy on

test data

◮ Runtime:

◮ several minutes and

up to 20 hours

◮ not yet parallelized

◮ GUI demo: link in paper

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 13 / 15

slide-45
SLIDE 45

tugraz

Institute of Software Technology

Experiments – a Learned Model (2)

◮ Results:

◮ successfully learned

all 44 models

◮ consistent with given

training data

◮ high accuracy on

test data

◮ Runtime:

◮ several minutes and

up to 20 hours

◮ not yet parallelized

◮ GUI demo: link in paper

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 13 / 15

slide-46
SLIDE 46

tugraz

Institute of Software Technology

Experiments – Evolution of Fitness

100 200 300 400 500 600 0% 50% 100% generations accepted training data Fitness of Car Alarm System Models maximum median minimum ◮ Early generations accept only initial inputs ◮ Further behaviour continuously added

→ Random generation infeasible

◮ Final generations decrease model size

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 14 / 15

slide-47
SLIDE 47

tugraz

Institute of Software Technology

Experiments – Evolution of Fitness

100 200 300 400 500 600 0% 50% 100% generations accepted training data Fitness of Car Alarm System Models maximum median minimum ◮ Early generations accept only initial inputs ◮ Further behaviour continuously added

→ Random generation infeasible

◮ Final generations decrease model size

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 14 / 15

slide-48
SLIDE 48

tugraz

Institute of Software Technology

Experiments – Evolution of Fitness

100 200 300 400 500 600 0% 50% 100% generations accepted training data Fitness of Car Alarm System Models maximum median minimum ◮ Early generations accept only initial inputs ◮ Further behaviour continuously added

→ Random generation infeasible

◮ Final generations decrease model size

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 14 / 15

slide-49
SLIDE 49

tugraz

Institute of Software Technology

Concluding Remarks

Summary

◮ Genetic Programming for timed automata including mutation,

crossover, subpopulations, and fine-grained fitness computation

◮ Evaluated on 44 timed automata used as black boxes

◮ up to 26 locations ◮ up to two clocks with arbitrary resets

◮ Implemented in a tool

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 15 / 15

slide-50
SLIDE 50

tugraz

Institute of Software Technology

Concluding Remarks

Summary

◮ Genetic Programming for timed automata including mutation,

crossover, subpopulations, and fine-grained fitness computation

◮ Evaluated on 44 timed automata used as black boxes

◮ up to 26 locations ◮ up to two clocks with arbitrary resets

◮ Implemented in a tool

Conclusion

◮ Successfully learned medium-sized models from tests ◮ Future work:

◮ active learning ◮ relaxing assumptions ◮ synthesis via model-checking-based fitness computation Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 15 / 15

slide-51
SLIDE 51

tugraz

Institute of Software Technology

Concluding Remarks

Summary

◮ Genetic Programming for timed automata including mutation,

crossover, subpopulations, and fine-grained fitness computation

◮ Evaluated on 44 timed automata used as black boxes

◮ up to 26 locations ◮ up to two clocks with arbitrary resets

◮ Implemented in a tool

Conclusion

◮ Successfully learned medium-sized models from tests ◮ Future work:

◮ active learning ◮ relaxing assumptions ◮ synthesis via model-checking-based fitness computation

Thank you!

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 15 / 15

slide-52
SLIDE 52

tugraz

Institute of Software Technology

Aarts, F., Kuppens, H., Tretmans, J., Vaandrager, F. W., and Verwer, S. (2012). Learning and testing the bounded retransmission protocol. In Heinz, J., de la Higuera, C., and Oates, T., editors, Proceedings

  • f the Eleventh International Conference on Grammatical Inference,

ICGI 2012, University of Maryland, College Park, USA, September 5-8, 2012, volume 21 of JMLR Proceedings, pages 4–18. JMLR.org. Fiterau-Brostean, P., Janssen, R., and Vaandrager, F. W. (2016). Combining model learning and model checking to analyze TCP implementations. In Chaudhuri, S. and Farzan, A., editors, Computer Aided Verification - 28th International Conference, CAV 2016, Toronto, ON, Canada, July 17-23, 2016, Proceedings, Part II, volume 9780

  • f Lecture Notes in Computer Science, pages 454–471. Springer.

Grinchtein, O., Jonsson, B., and Leucker, M. (2010). Learning of event-recording automata.

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 0 / 15

slide-53
SLIDE 53

tugraz

Institute of Software Technology

  • Theor. Comput. Sci., 411(47):4029–4054.

Grinchtein, O., Jonsson, B., and Pettersson, P. (2006). Inference of event-recording automata using timed decision trees. In Baier, C. and Hermanns, H., editors, CONCUR 2006 - Concurrency Theory, 17th International Conference, CONCUR 2006, Bonn, Germany, August 27-30, 2006, Proceedings, volume 4137 of Lecture Notes in Computer Science, pages 435–449. Springer. Hessel, A., Larsen, K. G., Nielsen, B., Pettersson, P., and Skou, A. (2003). Time-optimal real-time test case generation using UPPAAL. In FATES 2003, volume 2931 of LNCS, pages 114–130. Springer. Katz, G. and Peled, D. (2017). Synthesizing, correcting and improving code, using model checking-based genetic programming. STTT, 19(4):449–464.

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 0 / 15

slide-54
SLIDE 54

tugraz

Institute of Software Technology

Tappler, M., Aichernig, B. K., and Bloem, R. (2017). Model-based testing IoT communication via active automata learning. In 2017 IEEE International Conference on Software Testing, Verification and Validation, ICST 2017, Tokyo, Japan, March 13-17, 2017, pages 276–287. IEEE Computer Society. Verwer, S., de Weerdt, M., and Witteveen, C. (2010). A likelihood-ratio test for identifying probabilistic deterministic real-time automata from positive data. In Sempere, J. M. and García, P., editors, Grammatical Inference: Theoretical Results and Applications, 10th International Colloquium, ICGI 2010, Valencia, Spain, September 13-16, 2010. Proceedings, volume 6339 of Lecture Notes in Computer Science, pages 203–216. Springer. Verwer, S., de Weerdt, M., and Witteveen, C. (2012). Efficiently identifying deterministic real-time automata from labeled data.

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 0 / 15

slide-55
SLIDE 55

tugraz

Institute of Software Technology

Machine Learning, 86(3):295–333.

Martin Tappler FORMATS 2019 Time to Learn – Learning Timed Automata from Tests 0 / 15