Active Automata Learning: From DFA to Interface Programs and Beyond - PowerPoint PPT Presentation

Active Automata Learning: From DFA to Interface Programs and Beyond or From Languages to Program Executions or (more technically) The Power of Counterexample Analysis Bernhard Steffen, Falk Howar, Malte Isberner TU Dortmund /CMU B. Steffen Summer School CPS 2014 1

Connect Scenario some service connector learn interrogate X try to use inform about new learner service and device look for known CONNECT interrogate B. Steffen Summer School CPS 2014 2 models environment

Data-Dependent Control Value-independent Data Dependencies B. Steffen Summer School CPS 2014 3

How to Extend w. Data? Data is crucial for modeling  Interface specifications • relate data in input to data in subsequent output  Communication protocols • sequence numbers, identifiers, .. (External) Mapper-Based Data Treatment Explicit Data Modelling B. Steffen Summer School CPS 2014 4

Outline  Background  Manual Treatment of Data  Automated Alphabet Abstraction Refinement  Modelling Data Explicitly  Conclusions 5 B. Steffen Summer School CPS 2014 5

Computer/Telephony Integrated Systems Model-Generator LAN Switch Application-PCs ISDN Network Application- Server B. Steffen Summer School CPS 2014 6

The Concrete Scenario Test Coordinator Rational Robot HTTP Hipermon Hipermon PCM CSTA II/III Application PCs HTTP Hipermon Hipermon PCM Application Server B. Steffen Summer School CPS 2014 7

Means of Observation Test Coordinator Rational Robot ^ ^ ^ HTTP Hipermon Hipermon PCM Application PCs CSTA II/III HTTP Hipermon Hipermon PCM Application Server ( small) learned models imposed major test suite optimizations B. Steffen Summer School CPS 2014 8

Moderated, Regular Extrapolation l Extrapolation Hypothesis Building beyond known facts l Regular Extrapolation-Universe: Extended Finite Automata l Moderated The Extrapolation Process requires targeted interaction Neither Correct nor Complete ! B. Steffen Summer School CPS 2014 9

Models in our Scenario Abstract representation of the protocol-level behaviour. Abstraction typically concerns { invokeID = 58391, replace ” symbolic names operation-value = 21 (cSTAEventReport), {obsEvent {eventSpecificInfo. ... .hookswitch • details l i no time stamps etc . {deviceId.dialingNumber = “500” deviceId = A1 hookswitchOnHook= TRUE, switchOnHook , ... ... timestamp = “20001010095551 ” } } } }}} B. Steffen Summer School CPS 2014 10

Sketch of the Model Structure Models comprise state changes as well as UPN- and CSTA-Observations. { Sys_Info {deviceId = A1 obs_CSTA upnOffHook hookswitchOnHook, ... } } obs_CSTA obs_CSTA device A1 obs_CSTA display(line 1, ...) Sys_Info LEDs: (1,on) (2,off) ... ... B. Steffen Summer School CPS 2014 11

Active Automata Learning Distinguishing Futures OT Reaching Words Lower Hypothesis Unknown System Automaton Transitions Closeness & Consistency Validation B. Steffen Summer School CPS 2014 12

Membership Queries  OT  1 Unknown System Abstract States a 1 Not closed! b 0 Transition Relation B. Steffen Summer School CPS 2014 13

Closure & Consistency  OT Unknown System  1 b 0 Closed & Consistent a a 1 ba 0 b bb 0 a,b B. Steffen Summer School CPS 2014 14

Equivalence Queries  OT  Unknown System 1 b 0 a 1 ab 1 a a 1 ba 0 b bb 0 a,b Counterexample: ab  L B. Steffen Summer School CPS 2014 15

Counter Example-Based Extension  OT Unknown System  1 b 0 a 1 a ab 1 ba 0 b bb 0 aa 0 a,b aba 0 Counterexample: ab  L abb 1 B. Steffen Summer School CPS 2014 16

Closure & Consistency  OT  Unknown System 1 b 0 a 1 ab 1 Not consistent: ba 0 row (  ) = row (a), but row (  a)  row (aa) bb 0 aa 0 aba 0 New Column: a abb 1 B. Steffen Summer School CPS 2014 17

Next Iteration  a OT Unknown System  1 1 b 0 0 a 1 0 ab 1 0 Closed & Consistent ba 0 0 bb 0 0 aa 0 0 aba 0 0 abb 1 0 B. Steffen Summer School CPS 2014 18

Next Iteration  a OT Unknown System  1 1 b 0 0 a 1 0 ab 1 0 a b ba 0 0 bb 0 0 b a aa 0 0 aba 0 0 a,b abb 1 0 Finished! B. Steffen Summer School CPS 2014 19

Active automata learning: L* Σ={a,b} a a b b b b MQ-Oracle aba  L? a a no a ? b a,b no, bb  L! EQ-Oracle B. Steffen Summer School CPS 2014 20

Summary of L* algorithm L* infers Finite State Machine from queries: 1. Pose membership queries until “saturation” 2. Construct Hypothesis from obtained information 3. Pose equivalence query 4. if no look at counterexample and goto 1 5. else return Hypothesis end  Has been used to learn large automata (≥100 kstates)  Adapted for Mealy Machines [Niese et al. 2003]  and for Interface Automata [Aarts et al. 2010]  Efficient Tool: LearnLib [TUDortmund] B. Steffen Summer School CPS 2014 21

Summary of L* algorithm L* infers Finite State Machine from queries: 1. Pose membership queries until “saturation” 2. Construct Hypothesis from obtained information 3. Pose equivalence query 4. if no look at counterexample and goto 1 5. else return Hypothesis end  Has been used to learn large automata (≥100 kstates)  Adapted for Mealy Machines [Niese et al. 2003]  and for Interface Automata [Aarts et al. 2010]  Efficient Tool: LearnLib [TUDortmund] B. Steffen Summer School CPS 2014 22

Analysis of Counterexamples I one essential suffix a b bb ε 0 0 a 1 1 b 1 1 All prefixes of bb 0 0 counterexample … bbb 0 0 aa 1 1 ab 1 1 ba 0 0 … … … B. Steffen Summer School CPS 2014 23

Analysis of Counterexamples I one essential suffix a b bb ε 0 0 a 1 1 b 1 1 All prefixes of bb 0 0 counterexample … bbb 0 0 aa 1 1 ab 1 1 Essential suffix ba 0 0 … … … B. Steffen Summer School CPS 2014 24

Effect: Reduced Observation Table Rivest and Shapire : Analyze counterexample separately (not in the table) Only add one ‚essential‘ suffix (i.e., witness), as column label to the table Consequence:  Guaranteed Consistency!  Improved worst case complexity BUT: Hypothesis Automata are no longer guaranteed to be minimal! (cf. Pnueli / Mahler‘s criticism) B. Steffen Summer School CPS 2014 25

Outline  Background  Manual Treatment of Data  Automated Alphabet Abstraction Refinement  Modelling Data Explicitly  Conclusions 26 B. Steffen Summer School CPS 2014 26

Simple Stack finite capacity B. Steffen Summer School CPS 2014 27

Mappers B. Steffen Summer School CPS 2014 28

Learning the stack as a language stack.push(1) stack.pop() push, pop  L,  L true, false, null, 1 B. Steffen Summer School CPS 2014 29

Introducing outputs: Mealy machines stack.push(1) stack.pop() push, pop OK, NOK , null, 1 true, false, null, 1 B. Steffen Summer School CPS 2014 30

Introducing outputs: Mealy machines stack.push(1) Stack.push(2) push1, push2, pop stack.pop() OK, NOK , null, 1, 2 true, false, null, 1, 2 B. Steffen Summer School CPS 2014 31

Outline  Background  Manual Treatment of Data  Automated Alphabet Abstraction Refinement  Modelling Data Explicitly  Conclusions 32 B. Steffen Summer School CPS 2014 32

Automated Alphabet Abstraction Refinement Learning setup in Practice <presence type=… /> Available <iq type= “result“ /> OK Test-driver LearnLib Static alphabet abstraction B. Steffen Summer School CPS 2014 33

Automated Alphabet Abstraction Refinement <presence type=… /> Available(type=avail…) <iq type= “result“ /> Test-driver OK LearnLib Available  Available(type=avail…) Non-det. Available‘  Available(type=unavail…) during EQ Test Learning relative to a given representation system CEGAR teacher <presence type=… /> Available <iq type= “result“ /> OK Test-driver LearnLib Static alphabet abstraction B. Steffen Summer School CPS 2014 34

The Mod-k Stack finite set of outputs, e.g.: odd / even stack.push(51); stack.push(2012); push, push’ , pop stack.pop() OK, NOK , null, odd, even true, false, null, 51, 2012 B. Steffen Summer School CPS 2014 35

The Mod-k Stack finite set of outputs, e.g.: odd / even push push pop / odd push push ’ pop / even stack.push(51); stack.push(2012); push, push’ , pop stack.pop() OK, NOK , null, odd, even true, false, null, 51, 2012 B. Steffen Summer School CPS 2014 36

Counter Examples and Witnesses c 1 c 2 c 3 c 4 c 5 c 6 γ ( α (c 1 )) γ ( α (c 2 )) γ ( α (c 3 )) γ ( α (c 4 )) γ ( α (c 5 )) γ ( α (c 6 )) Bern B. Steffen Summer School CPS 2014 37 hard Steff

Counter Examples and Witnesses c 1 c 2 c 3 c 4 c 5 c 6 c 5 c 6 c 4 γ ( α (c 1 )) γ ( α (c 2 )) γ ( α (c 3 )) γ ( α (c 4 )) c 5 c 6 γ ( α (c 1 )) γ ( α (c 2 )) γ ( α (c 3 )) γ ( α (c 4 )) γ ( α (c 5 )) γ ( α (c 6 )) Bern B. Steffen Summer School CPS 2014 38 hard Steff