learning to use learning in verification
play

Learning to Use Learning in Verification Jan K ret nsk y - PowerPoint PPT Presentation

Apr 20, 2023 •122 likes •651 views

Learning to Use Learning in Verification Jan K ret nsk y Technische Universit at M unchen, Germany joint work with T. Br azdil (Masaryk University Brno), K. Chatterjee, M. Chmel k, P . Daca, A. Fellner, T. Henzinger, T.

  1. Learning to Use Learning in Verification Jan Kˇ ret´ ınsk´ y Technische Universit¨ at M¨ unchen, Germany joint work with T. Br´ azdil (Masaryk University Brno), K. Chatterjee, M. Chmel´ ık, P . Daca, A. Fellner, T. Henzinger, T. Petrov (IST Austria), V. Forejt, M. Kwiatkowska, M. Ujma (Oxford University) D. Parker (University of Birmingham) published at ATVA 2014, CAV 2015, TACAS 2016 Mysore Park Workshop Trends and Challenges in Quantitative Verification February 3, 2016

  10. Approaches and their interaction 3/16 Formal methods ◮ precise ◮ scalability issues

  11. Approaches and their interaction 3/16 Formal methods ◮ precise ◮ scalability issues MEM-OUT

  12. Approaches and their interaction 3/16 Formal methods ◮ precise ◮ scalability issues

  13. Approaches and their interaction 3/16 Formal methods ACTION = rec ◮ precise N Y ◮ scalability issues l>0&b=1&ip mess=1 -> ACTION = b’=0&z’=0&n1’=min(n1+1,8)&ip mess’=0 Y N z ≤ 0 Y N

  14. Approaches and their interaction 3/16 Formal methods ◮ precise ◮ scalability issues

  15. Approaches and their interaction 3/16 Formal methods Learning ◮ precise ◮ weaker guarantees ◮ scalability issues ◮ scalable different objectives

  16. Approaches and their interaction 3/16 Formal methods Learning ◮ precise ◮ weaker guarantees ◮ scalability issues ◮ scalable

  17. Approaches and their interaction 3/16 Formal methods Learning ◮ precise ◮ weaker guarantees ◮ scalability issues ◮ scalable precise computation focus on important stuff

  18. What problems? 4/16 ◮ Verification ? ◮ ( ε )-optimality → PAC − ? ◮ hard guarantees → probably correct − ◮ Controller synthesis ◮ convergence is preferable ◮ at least probably correct? ◮ Synthesis

  19. Markov decision processes 5/16 ( S , s 0 ∈ S , A , ∆ : S → A → D ( S )) p . . . 1 a 1 up b 0 . 5 · · · v 1 s init down 0 . 01 0 . 5 0 . 99 c err t 1

  20. Markov decision processes 5/16 ( S , s 0 ∈ S , A , ∆ : S → A → D ( S )) p . . . 1 a 1 up b 0 . 5 · · · v 1 s init down 0 . 01 0 . 5 0 . 99 c err t 1 strategy σ P σ [ Reach err ] max

  21. Markov decision processes 5/16 ( S , s 0 ∈ S , A , ∆ : S → A → D ( S )) p . . . 1 a 1 up b b 0 . 5 0 . 5 · · · v 1 s init down down 0 . 01 0 . 01 0 . 5 0 . 5 0 . 99 0 . 99 c c err t 1 1 strategy σ P σ [ Reach err ] max

  22. Markov decision processes 5/16 ( S , s 0 ∈ S , A , ∆ : S → A → D ( S )) p . . . 1 a 1 up b b 0 . 5 0 . 5 · · · v 1 s init down down 0 . 01 0 . 01 0 . 5 0 . 5 0 . 99 0 . 99 c c err t 1 1 strategy σ P σ [ Reach err ] max

  23. Markov decision processes 5/16 ( S , s 0 ∈ S , A , ∆ : S → A → D ( S )) p . . . 1 a 1 up b b 0 . 5 0 . 5 · · · v 1 s init down down 0 . 01 0 . 01 0 . 5 0 . 5 0 . 99 0 . 99 c c err t 1 1 strategy σ P σ [ Reach err ] max

  24. Markov decision processes 5/16 ( S , s 0 ∈ S , A , ∆ : S → A → D ( S )) p . . . 1 a 1 up b 0 . 5 · · · v 1 s init down down 0 . 01 0 . 01 0 . 5 0 . 99 0 . 99 c err t 1 strategy σ P σ [ Reach err ] max

  25. Markov decision processes 5/16 ( S , s 0 ∈ S , A , ∆ : S → A → D ( S )) p . . . 1 a 1 up b 0 . 5 · · · v 1 s init down down 0 . 01 0 . 01 0 . 5 0 . 99 0 . 99 ACTION = down c err t 1 Y N strategy σ P σ [ Reach err ] max

  26. Ex.1: Computing strategies faster: How? 6/16 Fixed-point computation V ( s ) := max a ∈ ∆( s ) V ( s , a ) � ∆( s , a , s ′ ) · V ( s ′ ) V ( s , a ) := s ′ ∈ S

  27. Ex.1: Computing strategies faster: How? 6/16 Fixed-point computation V ( s ) := max a ∈ ∆( s ) V ( s , a ) � ∆( s , a , s ′ ) · V ( s ′ ) V ( s , a ) := s ′ ∈ S Order of evaluation?

  28. Ex.1: Computing strategies faster: How? 6/16 Fixed-point computation V ( s ) := max a ∈ ∆( s ) V ( s , a ) � ∆( s , a , s ′ ) · V ( s ′ ) V ( s , a ) := s ′ ∈ S Order of evaluation? [ATVA’14] More frequently evaluate those states that are visited more frequently

  29. Ex.1: Computing strategies faster: How? 6/16 Fixed-point computation V ( s ) := max a ∈ ∆( s ) V ( s , a ) � ∆( s , a , s ′ ) · V ( s ′ ) V ( s , a ) := s ′ ∈ S Order of evaluation? [ATVA’14] More frequently evaluate those states that are visited more frequently by reasonably good schedulers

  30. Ex.1: Computing strategies faster: How? 6/16 Reinforcement learning Fixed-point computation V ( s ) := max a ∈ ∆( s ) V ( s , a ) � ∆( s , a , s ′ ) · V ( s ′ ) V ( s , a ) := s ′ ∈ S Order of evaluation? [ATVA’14] More frequently evaluate those states that are visited more frequently by reasonably good schedulers

  31. Ex.1: Computing strategies faster: Algorithm 7/16 1: U ( · , · ) ← 1 , L ( · , · ) ← 0 2: L ( 1 , · ) ← 1 , U ( 0 , · ) ← 0 3: repeat 7: until U ( s 0 ) − L ( s 0 ) < ǫ

  32. Ex.1: Computing strategies faster: Algorithm 7/16 1: U ( · , · ) ← 1 , L ( · , · ) ← 0 2: L ( 1 , · ) ← 1 , U ( 0 , · ) ← 0 3: repeat sample a path from s 0 to { 1 , 0 } 4: U ( s , a ) ⊲ actions uniformly from arg max a ⊲ states according to ∆( s , a , s ′ ) 7: until U ( s 0 ) − L ( s 0 ) < ǫ

  33. Ex.1: Computing strategies faster: Algorithm 7/16 1: U ( · , · ) ← 1 , L ( · , · ) ← 0 2: L ( 1 , · ) ← 1 , U ( 0 , · ) ← 0 3: repeat sample a path from s 0 to { 1 , 0 } 4: U ( s , a ) ⊲ actions uniformly from arg max a ⊲ states according to ∆( s , a , s ′ ) for all visited transitions ( s , a , s ′ ) do 5: U pdate ( s , a , s ′ ) 6: 7: until U ( s 0 ) − L ( s 0 ) < ǫ

  34. Ex.1: Computing strategies faster: Algorithm 7/16 1: U ( · , · ) ← 1 , L ( · , · ) ← 0 2: L ( 1 , · ) ← 1 , U ( 0 , · ) ← 0 3: repeat sample a path from s 0 to { 1 , 0 } 4: U ( s , a ) ⊲ actions uniformly from arg max a ⊲ states according to ∆( s , a , s ′ ) for all visited transitions ( s , a , s ′ ) do 5: U pdate ( s , a , s ′ ) 6: 7: until U ( s 0 ) − L ( s 0 ) < ǫ —————————————————————————— 1: procedure U pdate ( s , a , · ) s ′ ∈ S ∆( s , a , s ′ ) · U ( s ′ ) U ( s , a ) := � 2: s ′ ∈ S ∆( s , a , s ′ ) · L ( s ′ ) L ( s , a ) := � 3:

  35. Ex.1: Computing strategies faster 8/16 important parts of the system Reinforcement Value Learning Iteration faster & sure updates Guaranteed upper & lower bounds at all times + practically fast convergence

  36. Ex.1: Computing strategies faster 8/16 important parts of the system Reinforcement Value Learning Iteration faster & sure updates Guaranteed upper & lower bounds at all times + practically fast convergence Remark: ◮ PAC SMC for MDP and (unbounded) LTL [ATVA’14]: | S | , p min ◮ practical PAC SMC for MC and (unbounded) LTL + mean payoff [TACAS’16]: p min

  37. Ex.1: Experimental results 9/16 Visited states Example PRISM BRTDP 3,001,911 760 zeroconf 4,427,159 977 5,477,150 1411 345,000 2018 wlan 1,295,218 2053 5,007,548 1995 6,719,773 26,508 firewire 13,366,666 25,214 19,213,802 32,214 17,722,564 1950 17,722,564 2902 mer 26,583,064 1950 26,583,064 2900

  38. Further examples on reinforcement learning 10/16 Sebastian Junges, Nils Jansen, Christian Dehnert, Ufuk Topcu, Joost-Pieter Katoen: Safety-Constrained Reinforcement Learning for MDPs. TACAS 2016. ◮ safe and cost-optimizing strategies ◮ (1) compute safe, permissive strategies ◮ (2) learn cost-optimal strategies (convergence) among them Alexandre David, Peter Gjl Jensen, Kim Guldstrand Larsen, Axel Legay, Didier Lime, Mathias Grund Srensen, Jakob Haahr Taankvist: On Time with Minimal Expected Cost! ATVA 2014. ◮ priced timed games → priced timed MDPs ◮ time-bounded cost-optimal (convergence) strategies ◮ (1) Uppaal TiGa for safe strategies ◮ (2) Uppaal SMC and learning for cost-optimal strategies

  39. Ex.2: Computing small strategies: Which decisions? 11/16 Importance of a node s with respect to � target and strategy σ : P σ [ � s ]

  40. Ex.2: Computing small strategies: Which decisions? 11/16 Importance of a node s with respect to � target and strategy σ : P σ [ � s | � target ]

  41. Ex.2: Computing small strategies: Which decisions? 11/16 Importance of a node s with respect to � target and strategy σ : P σ [ � s | � target ] Cut off states with zero importance (unreachable or useless)

  42. Ex.2: Computing small strategies: Which decisions? 11/16 Importance of a node s with respect to � target and strategy σ : P σ [ � s | � target ] Cut off states with zero importance (unreachable or useless) Cut off states with low importance (small error, ε -optimal strategy)

  43. Ex.2: Small strategies: Which representation? 12/16 How to make use of the exact importance?

  44. Ex.2: Small strategies: Which representation? 12/16 How to make use of the exact importance? Learn decisions in s in proportion to importance of s

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Synthesis, Verification, and Synthesis, Verification, and Inductive Learning Inductive Learning

Synthesis, Verification, and Synthesis, Verification, and Inductive Learning Inductive Learning

Synthesis, Verification, and Synthesis, Verification, and Inductive Learning Inductive Learning Sanjit A. Seshia EECS Department UC Berkeley Joint work with Susmit Jha (UTC) Dagstuhl Seminar Verified SW Working Group August 2014 July

661 views • 28 slides
Age Age Age Age E out ( h 1 ) E out ( h 2 ) E out ( h 3 ) E out ( h M )

Age Age Age Age E out ( h 1 ) E out ( h 2 ) E out ( h 3 ) E out ( h M )

recap: Verification h Income Learning From Data Lecture 4 Real Learning is Feasible Age Hoeffding: E out ( h ) E in ( h ) (with high probability) E out ( h ) Real Learning vs. Verification The Two Step Solution to Learning D Closer

240 views • 5 slides
Verification of Deep Learning Systems Xiaowei Huang, University of Liverpool December 25, 2017

Verification of Deep Learning Systems Xiaowei Huang, University of Liverpool December 25, 2017

Verification of Deep Learning Systems Xiaowei Huang, University of Liverpool December 25, 2017 Outline Background Challenges for Verification Deep Learning Verification [2] Feature-Guided Black-Box Testing [3] Conclusions and Future Works

973 views • 81 slides
Learning From Data Lecture 3 Is Learning Feasible? Outside the Data Probability to the Rescue

Learning From Data Lecture 3 Is Learning Feasible? Outside the Data Probability to the Rescue

Learning From Data Lecture 3 Is Learning Feasible? Outside the Data Probability to the Rescue Learning vs. Verification Selection Bias - A Cartoon M. Magdon-Ismail CSCI 4100/6100 recap: The Perceptron Learning Algorithm y = +1 y x

538 views • 27 slides
Verification of Agents learning through Reinforcement Shashank Pathak 12 Giorgio Metta 12 Luca

Verification of Agents learning through Reinforcement Shashank Pathak 12 Giorgio Metta 12 Luca

Reinforcement Learning Air hockey as case-study Air hockey as RL task Verification Repair Conclusion Verification of Agents learning through Reinforcement Shashank Pathak 12 Giorgio Metta 12 Luca Pulina 3 Armando Tacchella 2 Robotics, Brain

322 views • 16 slides
Learning is a never-ending process Tasks come and go, but learning is forever Learn more e ff

Learning is a never-ending process Tasks come and go, but learning is forever Learn more e ff

NeurIPS 2018 Tutorial on Automatic Machine Learning Learning to Learn Learning Learning Learning Learning Learning automl.org/events -&gt; AutoML Tutorial -&gt; Slides Frank Hutter Joaquin Vanschoren Eindhoven University of Technology

1.36k views • 40 slides
An end-to-end approach for the verification problem: learning the right distance Joo Monteiro

An end-to-end approach for the verification problem: learning the right distance Joo Monteiro

An end-to-end approach for the verification problem: learning the right distance Joo Monteiro 1,2 , Isabela Albuquerque 1 , Jahangir Alam 1,2 , R Devon Hjelm 3,4 , Tiago H. Falk 1 1-Institut National de la Recherche Scientifique (INRS-EMT)

650 views • 30 slides
Verification of Robotics and Autonomous Deep Learning Verification Systems Safety Definition

Verification of Robotics and Autonomous Deep Learning Verification Systems Safety Definition

Verification of Robotics and Autonomous Systems Xiaowei Huang Challenges Verification of Robotics and Autonomous Deep Learning Verification Systems Safety Definition Challenges Approaches Experimental Results Verification in Xiaowei

1.21k views • 99 slides
Machine Learning in Formal Verification Manish Pandey, PhD Chief Architect, New Technologies

Machine Learning in Formal Verification Manish Pandey, PhD Chief Architect, New Technologies

Machine Learning in Formal Verification Manish Pandey, PhD Chief Architect, New Technologies Synopsys, Inc. June 18, 2017 1 Build Better Formal Verification Tools? CAR BICYCLE DOG S oftware that learns from experience and enables

767 views • 40 slides
What is Action Learning? What is an ALC? 1 What is Action Learning? Learning by doing

What is Action Learning? What is an ALC? 1 What is Action Learning? Learning by doing

What is Action Learning? What is an ALC? 1 What is Action Learning? Learning by doing Process for bringing together a group of people with varied levels of skills and experience (but with influence, passion, etc. to impact an

191 views • 5 slides
Learning From Data Lecture 2 The Perceptron The Learning Setup A Simple Learning Algorithm: PLA

Learning From Data Lecture 2 The Perceptron The Learning Setup A Simple Learning Algorithm: PLA

Learning From Data Lecture 2 The Perceptron The Learning Setup A Simple Learning Algorithm: PLA Other Views of Learning Is Learning Feasible: A Puzzle M. Magdon-Ismail CSCI 4100/6100 recap: The Plan 1. What is Learning? 2. Can We do it?

613 views • 25 slides
Learning Loop Invariants for Program Verification Xujie Si*, Hanjun Dai*, Mukund Raghothaman,

Learning Loop Invariants for Program Verification Xujie Si*, Hanjun Dai*, Mukund Raghothaman,

Learning Loop Invariants for Program Verification Xujie Si*, Hanjun Dai*, Mukund Raghothaman, Mayur Naik, Le Song University of Pennsylvania Georgia Institute of Technology NeurIPS 2018 Code: https://github.com/PL-ML/code2inv * equal

386 views • 34 slides
Game Theoretic Learning for Verification and Control Sanjit A. Seshia Professor EECS, UC

Game Theoretic Learning for Verification and Control Sanjit A. Seshia Professor EECS, UC

Game Theoretic Learning for Verification and Control Sanjit A. Seshia Professor EECS, UC Berkeley Joint work with Dorsa Sadigh, Jon Kotker, Daniel Bundala, Anca Dragan, Alexander Rakhlin, S. Shankar Sastry Dagstuhl Seminar March 16, 2017 Two

615 views • 34 slides
4. Learning in MAS 4.1. What is learning? One definition: Bower, Hilgard: Theories of learning,

4. Learning in MAS 4.1. What is learning? One definition: Bower, Hilgard: Theories of learning,

4. Learning in MAS 4.1. What is learning? One definition: Bower, Hilgard: Theories of learning, Prentice-Hall, 1975: Learning refers to the change in a subjects behavior to a given situation brought about by his repeated experiences in

173 views • 15 slides
What are learning outcomes? Learning Outcomes Learning outcomes are statements that specify what

What are learning outcomes? Learning Outcomes Learning outcomes are statements that specify what

What are learning outcomes? Learning Outcomes Learning outcomes are statements that specify what learners will know or be able to Everybody Wins do as a result of a learning FYE 2007 activity. Dr. Carolyn Hopper Middle Tennessee State

344 views • 11 slides
Learning minimal separating DFA for Compositional Verification Karsten Fix February 23, 2017

Learning minimal separating DFA for Compositional Verification Karsten Fix February 23, 2017

Overview Definitions Algorithm References Learning minimal separating DFA for Compositional Verification Karsten Fix February 23, 2017 Karsten Fix Learning minimal separating DFA for Compositional Verification Overview Definitions

525 views • 35 slides
(Learning to) Learn to Control Jan K ret nsk y Technical University of Munich, Germany

(Learning to) Learn to Control Jan K ret nsk y Technical University of Munich, Germany

(Learning to) Learn to Control Jan K ret nsk y Technical University of Munich, Germany joint work with P . Ashok, T. Meggendorfer (TUM), T. Br azdil (Masaryk University Brno), K. Chatterjee, M. Chmel k, P . Daca, A.

544 views • 35 slides
Objectives Christensen-Stilman (RIM@GT) 8803-Lecture 1 January 6, 2008 4 / 24 Introduction

Objectives Christensen-Stilman (RIM@GT) 8803-Lecture 1 January 6, 2008 4 / 24 Introduction

Introduction Objectives Structure Administration 8803 - Mobile Manipulation Henrik I Christensen &amp; Mike Stilman Robotics &amp; Intelligent Machines @ GT Georgia Institute of Technology, Atlanta, GA 30332-0760 January 6, 2008

278 views • 6 slides
SFM-11:CONNECT Summer School, Bertinoro, June 2011 EU-FP7: CONNECT LSCITS/PSS

SFM-11:CONNECT Summer School, Bertinoro, June 2011 EU-FP7: CONNECT LSCITS/PSS

SFM-11:CONNECT Summer School, Bertinoro, June 2011 EU-FP7: CONNECT LSCITS/PSS VERIWARE Overview Lecture 1 (9am-11am) Introduction to Modelling and Quantitative Verification Marta Kwiatkowska Invited lecture: Christel

1.99k views • 173 slides
A Reusable Camera Interface for Rovers Daniel Clouse Jet Propulsion Laboratory California

A Reusable Camera Interface for Rovers Daniel Clouse Jet Propulsion Laboratory California

A Reusable Camera Interface for Rovers Daniel Clouse Jet Propulsion Laboratory California Institute of Technology Pasadena, California Workshop on Software Development and Integration into Robotics 14 April, 2007 Acknowledgements

561 views • 20 slides
Real-time Harris and Stephen implementation on Smart camera Merwan BIREM Franois BERRY 5-6

Real-time Harris and Stephen implementation on Smart camera Merwan BIREM Franois BERRY 5-6

Workshop on Architecture of Smart Camera Real-time Harris and Stephen implementation on Smart camera Merwan BIREM Franois BERRY 5-6 April 2012 Clermont-Ferrand, FRANCE 1 Summary 1 - Feature Extraction, 2 - Harris &amp; Stephen detector,

427 views • 17 slides
!System(on(Chip!Design! Data!Flow!Modeling! (Based!on!slides!at!ECE!522!at!UNM)! Hao$Zheng$

!System(on(Chip!Design! Data!Flow!Modeling! (Based!on!slides!at!ECE!522!at!UNM)! Hao$Zheng$

!System(on(Chip!Design! Data!Flow!Modeling! (Based!on!slides!at!ECE!522!at!UNM)! Hao$Zheng$ Comp$Sci$&amp;$Eng$ U$of$South$Florida$$ 1 HW/SW Codesign w/ FPGAs Data-Flow Modeling ECE 522 Data-Flow Modeling (A Practical Introduction to HW/SW

771 views • 29 slides
Wavescalar Assembly: Dataflow Winter 2006 CSE 548 - Dataflow Machines 1 Wavescalar Assembly:

Wavescalar Assembly: Dataflow Winter 2006 CSE 548 - Dataflow Machines 1 Wavescalar Assembly:

Wavescalar Assembly: Dataflow Winter 2006 CSE 548 - Dataflow Machines 1 Wavescalar Assembly: Format Wavescalar is an extension of the Alpha ISA RISC (more or less) Register to register becomes PE to PE

716 views • 19 slides
Introduction to Rules Engines Brian McCallister brianm@ninginc.com http://www.ninginc.com/

Introduction to Rules Engines Brian McCallister brianm@ninginc.com http://www.ninginc.com/

Introduction to Rules Engines Brian McCallister brianm@ninginc.com http://www.ninginc.com/ Objectives Understanding of forward-chaining rules engines Introduction to the Drools rules engine Insights into where rules engines are a

813 views • 45 slides

More recommend