efficient reinforcement learning with hierarchies of
play

Efficient Reinforcement Learning with Hierarchies of Machines by - PowerPoint PPT Presentation

Aug 23, 2023 •98 likes •273 views

Efficient Reinforcement Learning with Hierarchies of Machines by Leveraging Internal Transitions Aijun Bai* Stuart Russell UC Berkeley/Microsoft Research UC Berkeley

  1. Efficient Reinforcement Learning with Hierarchies of Machines by Leveraging Internal Transitions Aijun Bai* Stuart Russell UC Berkeley/Microsoft Research UC Berkeley

  2. Outline • Hierarchical RL with partial programs • Deterministic internal transitions • Results

  3. Hierarchical RL with partial programs [Parr & Russell, NIPS 97; Andre & Russell, NIPS 00, AAAI 02; Marthi et al, IJCAI 05] Partial program a Learning algorithm s,r Completion Hierarchically optimal for all terminating programs 3

  4. Partial Program – an Example repeat forever Choose({a1,a2,…})

  5. Partial Program – an Example Navigate(destination) while  At(destination,CurrentState()) Choose({N,S,E,W})

  6. Concurrent Partial Programs Top() for each p in Effectors() PlayKeep(p) PlayKeep(p) s  CurrentState() while  Terminal(s) if BallKickable(s) then Choose({Pass(),Hold()}) else if FastestToBall(s) then Intercept() else Choose(Stay(),Move()) Pass() KickTo(Choose(Effectors()\{self}),Choose({slow,fast}) …

  7. Technical development • Decisions based on internal state • Joint state ω = [s,m] environment state + program state (cf. [Russell & Wefald 1989] ) • MDP + partial program = SMDP over choice states in {ω}, learn Q(ω,c) for choices c • Additive decomposition of value functions • by subroutine structure [Dietterich 00, Andre & Russell 02] Q is a sum of sub-Q functions per subroutine • across concurrent threads [Russell & Zimdars 03] Q is a sum of sub-Q functions per thread, with decomposed reward signal 7

  8. Internal Transitions Top() • Transitions between for each p in Effectors() choice points with no PlayKeep(p) physical action intervening PlayKeep(p) s  CurrentState() • Internal transitions while  Terminal(s) take no (real) time and have zero reward if BallKickable(s) then Choose({Pass(),Hold()}) else if FastestToBall(s) then Intercept() • Internal transitions are else Choose(Stay(),Move()) deterministic Pass() KickTo(Choose(Effectors()\{self}),Choose({slow,fast}) …

  9. Idea 1 • Use internal transitions to shortcircuit the computations of Q values recursively if applicable • If (s, m, c) -> (s, m’) is an internal transition • Then, Q(s, m, c) = V(s, m’) = max c’ Q(s, m’, c’) • Cache internal transitions as <s, m, c, m’> tuples • No need for Q-learning on these

  10. Idea 2 • Identify weakest precondition P(s) for this internal transition to occur (cf EBL, chunking) • Cache internal transitions as <P, m, c, m’> tuples • Cache size independent of |S|, roughly proportional to size of partial program call graph

  11. The HAMQ-INT Algorithm • Track the set of predicates since last choice point • Save an abstracted rule of internal transition if qualified (τ = 0) in a dictionary ρ • Use the saved rules to shortcircuit the computation of Q values recursively whenever possible

  12. Experimental Result on Taxi

  13. 3 vs 2 Keepaway Comparisons • Option (Stone, 2005): • Each keeper learning separately • Learn a policy over Hold() and Pass(k, v) if ball kickable; otherwise, follow a fixed policy • Intercept() if fastest to the ball; otherwise, GetOpen() • GetOpen() is manually programmed for Option • Concurrent-Option: • Concurrent version of Option • One global Q function is learnt • Random: randomized version of Option • Concurrent-HAMQ • Learn its own version of GetOpen() by calling Stay() and Move(d, v) • Concurrent-HAMQ-INT

  14. Experimental Result on Keepaway

  15. Before and After Initial policy Converged policy

  16. Summary • HAMQ-INT algorithm • Automatically discovers internal transitions • Takes advantage of internal transitions for efficient learning • Outperforms the state of the art significantly on Taxi and RoboCup Keepaway • Future work • Scale up to full RoboCup task • More general integration of model-based and model-free reinforcement learning • More flexible forms of partial program (e.g., temporal logic)

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

SDRL: Interpretable and Data-efficient Deep Liu Reinforcement Learning Introduction Background

SDRL: Interpretable and Data-efficient Deep Liu Reinforcement Learning Introduction Background

SDRL: Symbolic Deep Reinforcement Learning SDRL: Interpretable and Data-efficient Deep Liu Reinforcement Learning Introduction Background Leveraging Symbolic Planning Method Experiment Conclusion Bo Liu and Future Work Auburn

1k views • 20 slides
RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

551 views • 35 slides
Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3.

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3.

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3. Policy Gradient and Gradient Estimators 4. Q-prop: Sample Efficient Policy Gradient and an Off-policy Critic 5. Model Based Planning in Discrete

769 views • 53 slides
Efficient Off-Policy Meta- Reinforcement Learning via Probabilistic Context Variables Rakelly,

Efficient Off-Policy Meta- Reinforcement Learning via Probabilistic Context Variables Rakelly,

Efficient Off-Policy Meta- Reinforcement Learning via Probabilistic Context Variables Rakelly, K., Zhou, A., Quillen, D., Finn, C., &amp; Levine, S ICML, 2019 Presented by: Egill Ian Gudmundsson Some Terminology On-policy learning: Only one

427 views • 16 slides
Efficient Planning 1 R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction

Efficient Planning 1 R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction

Efficient Planning 1 R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction Tuesday class summary: Planning: any computational process that uses a model to create or improve a policy Dyna framework: 2 R. S. Sutton and A. G.

557 views • 27 slides
Gaussian Processes for Sample Efficient Reinforcement Learning with RMAX-like Exploration Tobias

Gaussian Processes for Sample Efficient Reinforcement Learning with RMAX-like Exploration Tobias

Gaussian Processes for Sample Efficient Reinforcement Learning with RMAX-like Exploration Tobias Jung and Peter Stone Department of Computer Science University of Texas at Austin { tjung,pstone } @cs.utexas.edu Outline: 1. Motivation &amp;

349 views • 18 slides
Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine playing a new game whose rules you dont know; after a hundred or so moves your don t know; after a hundred or so moves, your opponent announces, You

511 views • 30 slides
Data-Efficient Hierarchical Reinforcement Learning Authors: Ofir Nachum, Shixiang Gu, Honglak

Data-Efficient Hierarchical Reinforcement Learning Authors: Ofir Nachum, Shixiang Gu, Honglak

Data-Efficient Hierarchical Reinforcement Learning Authors: Ofir Nachum, Shixiang Gu, Honglak Lee, Sergey Levine Presented by: Samuel Yigzaw 1 OUTLINE Introduction Background Main Contributions Related Work Experiments Conclusion PAGE 2

1.4k views • 26 slides
PEARL Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables Kate

PEARL Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables Kate

PEARL Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables Kate Rakelly*, Aurick Zhou*, Deirdre Quillen, Chelsea Finn, Sergey Levine Hula Beach, Never grow up, The Sled - by artist Matt Spangler,

603 views • 12 slides
SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning Marvin Zhang*,

SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning Marvin Zhang*,

SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning Marvin Zhang*, Sharad Vikram*, Laura Smith, Pieter Abbeel, Matthew J Johnson, Sergey Levine UC Berkeley, UC San Diego, Google Efficient reinforcement learning from

212 views • 20 slides
Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers (the class labels) Reinforcement learning: In reinforcement learning, the agent learns Markov Decision Processes based on a punishment/reward

89 views • 6 slides
Harnessing Wake Vortices for Efficient Collective Swimming via Deep Reinforcement Learning

Harnessing Wake Vortices for Efficient Collective Swimming via Deep Reinforcement Learning

Harnessing Wake Vortices for Efficient Collective Swimming via Deep Reinforcement Learning Siddhartha Verma With: Guido Novati and Petros Koumoutsakos CSE lab http://www.cse-lab.ethz.ch Collective Swimming Hydrodynamic benefit of

1.24k views • 27 slides
Examples of Reinforcement Learning Robocup Soccer Teams Stone &amp; Veloso, Reidmiller et al.

Examples of Reinforcement Learning Robocup Soccer Teams Stone &amp; Veloso, Reidmiller et al.

Reinforcement Learning Slides by Rich Sutton Mods by Dan Lizotte Refer to Reinforcement Learning: An Introduction by Sutton and Barto Alpaydin Chapter 16 Up until now we have been Supervised Learning Classifying, mostly

891 views • 31 slides
CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning Haarnoja, Tang et al. (2017) Reinforcement Learning with Deep Energy Based Policies, ICML . Haarnoja, Zhou et al. (2018) Soft Actor-Critic: Off-Policy

682 views • 24 slides
Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control Theory, you actually have some knowledge in RL. RL shares the overall goal with OCT: solving for a control policy such that the cumulative cost

610 views • 22 slides
Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still assume a Marko kov decision process (MDP): A se set o of st states s s S A se set o of a actions ( s (per st state) A A mo model

96 views • 5 slides
Jesus Pastor Youth

Jesus Pastor Youth

5/28/2015 Surrounding yourself with support and knowledge Jesus Pastor Youth Department Counselors / Social Workers Police Dept. CPS School Guidance

348 views • 7 slides
BVDFree England National BVD Elimination Scheme Break Free from BVD Contents Bovine Viral

BVDFree England National BVD Elimination Scheme Break Free from BVD Contents Bovine Viral

BVDFree England National BVD Elimination Scheme Break Free from BVD Contents Bovine Viral Diarrhoea Virus BVDFree Scheme What it is How it works Resources to support the scheme Break Free from BVD BVD What is it? A

299 views • 14 slides
Auto-scaling deadline- constrained workloads in containers in the cloud Jay Jay DesLauriers

Auto-scaling deadline- constrained workloads in containers in the cloud Jay Jay DesLauriers

Auto-scaling deadline- constrained workloads in containers in the cloud Jay Jay DesLauriers DesLauriers Research Associate, University of Westminster Project COLA Horizon 2020 33 months Completion September 2019 14

309 views • 17 slides
COVID-19 stimulus package update JobKeeper Payment and other measures Rahul Singh Doug McBirnie

COVID-19 stimulus package update JobKeeper Payment and other measures Rahul Singh Doug McBirnie

COVID-19 stimulus package update JobKeeper Payment and other measures Rahul Singh Doug McBirnie Technical Services Manager General Manager Accurium Challenger l Job Keeper payment What is it? Legislated on 8 th April Still waiting

526 views • 25 slides
Parameterized Programs CS256/Spring 2008 Lecture #09 Zohar Manna 0 : loop

Parameterized Programs CS256/Spring 2008 Lecture #09 Zohar Manna 0 : loop

Parameterized Programs CS256/Spring 2008 Lecture #09 Zohar Manna 0 : loop forever do Chapter 2 1 : noncritical Invariance: Applications 2 : request y

150 views • 14 slides
F OOT F ORM Decomposed: Using primitive constraints in OT Jason Eisner, University of

F OOT F ORM Decomposed: Using primitive constraints in OT Jason Eisner, University of

F OOT F ORM Decomposed: Using primitive constraints in OT Jason Eisner, University of Pennsylvania 1. Goals of the work Hayes (1995) makes an extensive study of metrical stress systems, within a unifyingtypologicalframework. The

509 views • 29 slides
PhD Thesis proposal Supporting Conceptual Modelling in ORM by Reasoning Francesco Sportelli

PhD Thesis proposal Supporting Conceptual Modelling in ORM by Reasoning Francesco Sportelli

PhD Thesis proposal Supporting Conceptual Modelling in ORM by Reasoning Francesco Sportelli Otto von Guericke Universit at Magdeburg Abstract Object-Role Modelling (ORM) is a framework for modelling and querying information at the conceptual

414 views • 17 slides
Investor In estor Pr Prese esentation ntation rd Quart 3 rd arter er 2017 Filed by Peoples

Investor In estor Pr Prese esentation ntation rd Quart 3 rd arter er 2017 Filed by Peoples

Nasdaq: PEBO Investor In estor Pr Prese esentation ntation rd Quart 3 rd arter er 2017 Filed by Peoples Bancorp Inc. Pursuant to Rule 425 under the Securities Act of 1933 and deemed filed pursuant to Rule 14a-12 under the Securities

884 views • 42 slides

More recommend