Active Automata Learning: From DFA to Interface Programs and Beyond - - PowerPoint PPT Presentation

▶
active automata learning
SMART_READER_LITE
LIVE PREVIEW

Active Automata Learning: From DFA to Interface Programs and Beyond - - PowerPoint PPT Presentation

Feb 25, 2023 246 likes •967 views

Active Automata Learning: From DFA to Interface Programs and Beyond or From Languages to Program Executions or (more technically) The Power of Counterexample Analysis Bernhard Steffen, Falk Howar, Malte Isberner TU Dortmund /CMU B. Steffen

slide-1
SLIDE 1
  • B. Steffen

Summer School CPS 2014 1

Bernhard Steffen, Falk Howar, Malte Isberner TU Dortmund /CMU

Active Automata Learning:

From DFA to Interface Programs and Beyond

  • r

From Languages to Program Executions

  • r (more technically)

The Power of Counterexample Analysis

slide-2
SLIDE 2
  • B. Steffen

Summer School CPS 2014 2

learner connector try to use

X

inform about new service and device interrogate interrogate learn look for known models some service CONNECT environment

Connect Scenario

slide-3
SLIDE 3
  • B. Steffen

Summer School CPS 2014 3

Data-Dependent Control

Value-independent Data Dependencies

slide-4
SLIDE 4
  • B. Steffen

Summer School CPS 2014 4

Data is crucial for modeling

  • Interface specifications
  • relate data in input to data in subsequent output
  • Communication protocols
  • sequence numbers, identifiers, ..

(External) Mapper-Based Data Treatment Explicit Data Modelling

How to Extend w. Data?

slide-5
SLIDE 5
  • B. Steffen

Summer School CPS 2014 5

  • Background
  • Manual Treatment of Data
  • Automated Alphabet Abstraction Refinement
  • Modelling Data Explicitly
  • Conclusions

Outline

5

slide-6
SLIDE 6
  • B. Steffen

Summer School CPS 2014 6

Computer/Telephony Integrated Systems

ISDN Network Switch Model-Generator Application-PCs Application- Server

LAN

slide-7
SLIDE 7
  • B. Steffen

Summer School CPS 2014 7

The Concrete Scenario

Rational Robot

Hipermon Hipermon Hipermon Hipermon

CSTA II/III HTTP HTTP Test Coordinator

PCM Application Server PCM Application PCs

slide-8
SLIDE 8
  • B. Steffen

Summer School CPS 2014 8

Rational Robot

Hipermon

Hipermon Hipermon Hipermon

CSTA II/III

HTTP HTTP

Test Coordinator

PCM Application Server PCM Application PCs

^ ^ ^

Means of Observation

(small) learned models imposed

major test suite optimizations

slide-9
SLIDE 9
  • B. Steffen

Summer School CPS 2014 9

l Extrapolation

Hypothesis Building beyond known facts

l Regular

Extrapolation-Universe: Extended Finite Automata

l Moderated

The Extrapolation Process requires targeted interaction

Moderated, Regular Extrapolation

Neither Correct nor Complete !

slide-10
SLIDE 10
  • B. Steffen

Summer School CPS 2014 10

Abstract representation of the protocol-level behaviour. Abstraction typically concerns

replace ” symbolic names

  • details l i no time stamps etc.

Models in our Scenario

{ invokeID = 58391,

  • peration-value = 21 (cSTAEventReport),

{eventSpecificInfo. ... .hookswitch {deviceId.dialingNumber = “500” hookswitchOnHook= TRUE, ... timestamp = “20001010095551” } }}}

{obsEvent deviceId = A1 switchOnHook,

... }}

slide-11
SLIDE 11
  • B. Steffen

Summer School CPS 2014 11

Models comprise state changes as well as UPN- and CSTA-Observations.

Sketch of the Model Structure

Sys_Info Sys_Info

  • bs_CSTA
  • bs_CSTA

upnOffHook

  • bs_CSTA
  • bs_CSTA

{

{deviceId = A1 hookswitchOnHook, ... }}

device A1 display(line 1, ...) LEDs: (1,on) (2,off) ... ...

slide-12
SLIDE 12
  • B. Steffen

Summer School CPS 2014 12

Unknown System

Distinguishing Futures

OT

Lower Hypothesis Automaton Closeness & Consistency Validation Reaching Words Transitions

Active Automata Learning

slide-13
SLIDE 13
  • B. Steffen

Summer School CPS 2014 13

  1

OT

b a 1 Not closed!

Unknown System Abstract States Transition Relation

Membership Queries

slide-14
SLIDE 14
  • B. Steffen

Summer School CPS 2014 14

  1

OT

b a 1 ba bb a,b b a Closed & Consistent

Unknown System

Closure & Consistency

slide-15
SLIDE 15
  • B. Steffen

Summer School CPS 2014 15

  1

OT

b a 1 ba bb a,b b a

Counterexample: ab  L

a 1 ab 1

Unknown System

Equivalence Queries

slide-16
SLIDE 16
  • B. Steffen

Summer School CPS 2014 16

  1

OT

b ba bb a,b b a

Counterexample: ab  L

a 1 ab 1 aa aba abb 1

Unknown System

Counter Example-Based Extension

slide-17
SLIDE 17
  • B. Steffen

Summer School CPS 2014 17

  1

OT

b ba bb a 1 ab 1 aa aba abb 1

Unknown System

Not consistent: row () = row (a), but row (a)  row (aa) New Column: a

Closure & Consistency

slide-18
SLIDE 18
  • B. Steffen

Summer School CPS 2014 18

 a  1 1

OT

b ba bb a 1 aa ab 1 aba abb 1 Closed & Consistent

Unknown System

Next Iteration

slide-19
SLIDE 19
  • B. Steffen

Summer School CPS 2014 19

 a  1 1

OT

b ba bb a 1 aa ab 1 a a a,b b b aba abb 1

Unknown System

Finished!

Next Iteration

slide-20
SLIDE 20
  • B. Steffen

Summer School CPS 2014 20

Active automata learning: L*

MQ-Oracle EQ-Oracle Σ={a,b} aba  L? no ? no, bb  L!

a a a a b b b b a a,b b

slide-21
SLIDE 21
  • B. Steffen

Summer School CPS 2014 21

Summary of L* algorithm

L* infers Finite State Machine from queries:

  • 1. Pose membership queries until “saturation”
  • 2. Construct Hypothesis from obtained information
  • 3. Pose equivalence query
  • 4. if no look at counterexample and goto 1
  • 5. else return Hypothesis end
  • Has been used to learn large automata (≥100 kstates)
  • Adapted for Mealy Machines [Niese et al. 2003]
  • and for Interface Automata [Aarts et al. 2010]
  • Efficient Tool: LearnLib [TUDortmund]
slide-22
SLIDE 22
  • B. Steffen

Summer School CPS 2014 22

Summary of L* algorithm

L* infers Finite State Machine from queries:

  • 1. Pose membership queries until “saturation”
  • 2. Construct Hypothesis from obtained information
  • 3. Pose equivalence query
  • 4. if no look at counterexample and goto 1
  • 5. else return Hypothesis end
  • Has been used to learn large automata (≥100 kstates)
  • Adapted for Mealy Machines [Niese et al. 2003]
  • and for Interface Automata [Aarts et al. 2010]
  • Efficient Tool: LearnLib [TUDortmund]
slide-23
SLIDE 23
  • B. Steffen

Summer School CPS 2014 23

a b bb ε a 1 1 b 1 1 bb aa 1 1 ab 1 1 ba … … … bbb

  • ne essential suffix

All prefixes of counterexample …

Analysis of Counterexamples I

slide-24
SLIDE 24
  • B. Steffen

Summer School CPS 2014 24

a b bb ε a 1 1 b 1 1 bb aa 1 1 ab 1 1 ba … … … bbb

  • ne essential suffix

All prefixes of counterexample …

Essential suffix

Analysis of Counterexamples I

slide-25
SLIDE 25
  • B. Steffen

Summer School CPS 2014 25

Effect: Reduced Observation Table

Rivest and Shapire: Analyze counterexample separately (not in the table) Only add one ‚essential‘ suffix (i.e., witness), as column label to the table Consequence:

  • Guaranteed Consistency!
  • Improved worst case complexity

BUT: Hypothesis Automata are no longer guaranteed to be minimal! (cf. Pnueli / Mahler‘s criticism)

slide-26
SLIDE 26
  • B. Steffen

Summer School CPS 2014 26

  • Background
  • Manual Treatment of Data
  • Automated Alphabet Abstraction Refinement
  • Modelling Data Explicitly
  • Conclusions

Outline

26

slide-27
SLIDE 27
  • B. Steffen

Summer School CPS 2014 27

Simple Stack

finite capacity

slide-28
SLIDE 28
  • B. Steffen

Summer School CPS 2014 28

Mappers

slide-29
SLIDE 29
  • B. Steffen

Summer School CPS 2014 29

Learning the stack as a language

push, pop stack.push(1) stack.pop() true, false, null, 1

 L,  L

slide-30
SLIDE 30
  • B. Steffen

Summer School CPS 2014 30

Introducing outputs: Mealy machines

push, pop stack.push(1) stack.pop() true, false, null, 1 OK, NOK , null, 1

slide-31
SLIDE 31
  • B. Steffen

Summer School CPS 2014 31

Introducing outputs: Mealy machines

push1, push2, pop stack.push(1) Stack.push(2) stack.pop() OK, NOK , null, 1, 2 true, false, null, 1, 2

slide-32
SLIDE 32
  • B. Steffen

Summer School CPS 2014 32

  • Background
  • Manual Treatment of Data
  • Automated Alphabet Abstraction Refinement
  • Modelling Data Explicitly
  • Conclusions

Outline

32

slide-33
SLIDE 33
  • B. Steffen

Summer School CPS 2014 33

LearnLib

Test-driver

<presence type=… /> <iq type= “result“ /> Available OK

Static alphabet abstraction

Learning setup in Practice

Automated Alphabet Abstraction Refinement

slide-34
SLIDE 34
  • B. Steffen

Summer School CPS 2014 34

LearnLib

Test-driver

<presence type=… /> <iq type= “result“ /> Available OK

Static alphabet abstraction

LearnLib

Test-driver

CEGAR teacher

<presence type=… /> <iq type= “result“ /> Available(type=avail…) OK Available  Available(type=avail…)

Non-det. during EQ Test

Available‘  Available(type=unavail…)

Learning relative to a given representation system

Automated Alphabet Abstraction Refinement

slide-35
SLIDE 35
  • B. Steffen

Summer School CPS 2014 35

The Mod-k Stack

finite set of outputs, e.g.: odd / even

push, push’, pop stack.push(51); stack.push(2012); stack.pop() true, false, null, 51, 2012 OK, NOK , null, odd, even

slide-36
SLIDE 36
  • B. Steffen

Summer School CPS 2014 36

The Mod-k Stack

finite set of outputs, e.g.: odd / even

push, push’, pop stack.push(51); stack.push(2012); stack.pop() true, false, null, 51, 2012

push push pop / odd push push’ pop / even

OK, NOK , null, odd, even

slide-37
SLIDE 37
  • B. Steffen

Summer School CPS 2014 37

Counter Examples and Witnesses

Bern hard Steff

c1 c2 c3 c4 c5 c6

γ(α(c1)) γ(α(c2)) γ(α(c3)) γ(α(c4)) γ(α(c5)) γ(α(c6))

slide-38
SLIDE 38
  • B. Steffen

Summer School CPS 2014 38

Counter Examples and Witnesses

Bern hard Steff

c5 c6 c4

γ(α(c1)) γ(α(c2)) γ(α(c3)) γ(α(c4))

c5 c6

γ(α(c1)) γ(α(c2)) γ(α(c3)) γ(α(c4)) γ(α(c5)) γ(α(c6))

c1 c2 c3 c4 c5 c6

slide-39
SLIDE 39
  • B. Steffen

Summer School CPS 2014 39

c5 c6 c4

γ(α(c1)) γ(α(c2)) γ(α(c3)) γ(α(c4))

c5 c6

p d

Separating pattern p c4 d

state representation future

Counter Examples and Witnesses

slide-40
SLIDE 40
  • B. Steffen

Summer School CPS 2014 40

ΣC \ αold(c)

γold(αold(c)) c γ(α(p)) x d = γ(α(p)) c d αold(c)

ΣC

push‘ push

Alphabet Abstraction Refinement

slide-41
SLIDE 41
  • B. Steffen

Summer School CPS 2014 41

Bernhard Steffen | VMCAI 2011 @ Austin, Texas

Case Study

  • Biometric Passport
  • [Aarts et. al, 2010]

262 Concrete symbols, 256 x readFile(i).

‘read file(i)‘ aggregated according to the required authentication

  • 1 initial abstract symbols
  • 8 alphabet refinements,

to split readFile

  • 9 final abstract symbols
slide-42
SLIDE 42
  • B. Steffen

Summer School CPS 2014 42

  • Background
  • Manual Treatment of Data
  • Automated Alphabet Abstraction Refinement
  • Modelling Data Explicitly
  • Conclusions

Outline

42

slide-43
SLIDE 43
  • B. Steffen

Summer School CPS 2014 43

Data is crucial for modeling

  • Interface specifications
  • relate data in input to data in subsequent output
  • Communication protocols
  • sequence numbers, identifiers, ..

Extend automaton model

  • Data parameters in actions
  • State variables to remember parameter values

How to extend the learning techniques?

How to Extend with Data?

43

slide-44
SLIDE 44
  • B. Steffen

Summer School CPS 2014 44

Register Automata

slide-45
SLIDE 45
  • B. Steffen

Summer School CPS 2014 45

Relation: Data Languages

slide-46
SLIDE 46
  • B. Steffen

Summer School CPS 2014 46

The Impact of Register Automata

stack.push(51); stack.push(2012); stack.pop() true, false, null, 51, 2012 push(p)/OK, pop()/o(p), …

 L,  L

Query: push(p1)/OK push(p2)/OK pop()/p2

slide-47
SLIDE 47
  • B. Steffen

Summer School CPS 2014 47

A Data-Aware Nerode-Relation

slide-48
SLIDE 48
  • B. Steffen

Summer School CPS 2014 48

Reusing structure of L*

slide-49
SLIDE 49
  • B. Steffen

Summer School CPS 2014 50

  • Counterexample Analysis for inferring
  • New locations
  • New registers
  • New transitions

Analysis of Counterexamples III

50

slide-50
SLIDE 50
  • B. Steffen

Summer School CPS 2014 51

CE: New location

slide-51
SLIDE 51
  • B. Steffen

Summer School CPS 2014 52

CE: New location

slide-52
SLIDE 52
  • B. Steffen

Summer School CPS 2014 53

CE: New location

slide-53
SLIDE 53
  • B. Steffen

Summer School CPS 2014 54

CE: New location

slide-54
SLIDE 54
  • B. Steffen

Summer School CPS 2014 55

CE: New location

slide-55
SLIDE 55
  • B. Steffen

Summer School CPS 2014 56

CE: New location

slide-56
SLIDE 56
  • B. Steffen

Summer School CPS 2014 57

CE: New register

slide-57
SLIDE 57
  • B. Steffen

Summer School CPS 2014 58

CE: New register

slide-58
SLIDE 58
  • B. Steffen

Summer School CPS 2014 59

CE: New register

slide-59
SLIDE 59
  • B. Steffen

Summer School CPS 2014 60

CE: New register

slide-60
SLIDE 60
  • B. Steffen

Summer School CPS 2014 61

CE: New transition

slide-61
SLIDE 61
  • B. Steffen

Summer School CPS 2014 62

CE: New transition

slide-62
SLIDE 62
  • B. Steffen

Summer School CPS 2014 63

CE: New transition

slide-63
SLIDE 63
  • B. Steffen

Summer School CPS 2014 64

CE: New transition

slide-64
SLIDE 64
  • B. Steffen

Summer School CPS 2014 65

Experimental Evaluation

slide-65
SLIDE 65
  • B. Steffen

Summer School CPS 2014 66

Modeling Output explicitly: RMMs

  • Example: Stack of capacity 3
  • RA: output encoded as guarded transition
  • RMM: output with data for transitions

RA RMM

“… is in language” “… leads to output …”

slide-66
SLIDE 66
  • B. Steffen

Summer School CPS 2014 67

RMM: Explicit Output

slide-67
SLIDE 67
  • B. Steffen

Summer School CPS 2014 68

RMM: Explicit Output

push(p), pop() stack.push(51) stack.push(2012) stack.pop() OK, NOK , null, p

Query: push(p1)push(p2)pop() / p2

true, false, null, 51, 2012

slide-68
SLIDE 68
  • B. Steffen

Summer School CPS 2014 69

Inferring RMMs

  • Example: Nested stack of capacity 16
  • RMM: 781 locations, 45k MQ, 9 EQ, 20 sec.
  • Mealy, |D|=4: > 109 states
slide-69
SLIDE 69
  • B. Steffen

Summer School CPS 2014 70

Outline

  • Background
  • Manual Treatment of Data
  • Automated Alphabet Abstraction Refinement
  • Modelling Data Explicitly
  • Conclusions

70

slide-70
SLIDE 70
  • B. Steffen

Summer School CPS 2014 71

Conclusions and Perspectives

Main Practical Challenges are

  • Search for Counterexamples
  • Counterexample Analysis

Question: How much can counter examples tell about a system? We have seen scenarios for (beside the classical locations),

  • Optimal Alphabet Abstraction
  • Optimal Register Allocation
  • Optimal Transition Functions

We have seen how to get From DFA to Interface Programs or From Languages to Program Executions

71

slide-71
SLIDE 71
  • B. Steffen

Summer School CPS 2014 72

Conclusions and Perspectives

Beyond: Investigation of language extensions

  • Extended Guards
  • Actions with Effect
  • Procedural Structure?

Hybrid Approaches and Case Studies Experimental Evaluation and Performance Analysis

72

The RERS Greybox Challenge 2014