survey leveraging human guidance for deep reinforcement
play

Survey: Leveraging Human Guidance for Deep Reinforcement Learning - PowerPoint PPT Presentation

Jun 19, 2023 •45 likes •465 views

Survey: Leveraging Human Guidance for Deep Reinforcement Learning Tasks Ruohan Zhang, Faraz Torabi, Lin Guan, Dana H. Ballard, Peter Stone University of Texas at Austin Presented by Lin Guan Lin Guan (UT Austin) Paper#10921 1 / 33 A

  1. Survey: Leveraging Human Guidance for Deep Reinforcement Learning Tasks Ruohan Zhang, Faraz Torabi, Lin Guan, Dana H. Ballard, Peter Stone University of Texas at Austin Presented by Lin Guan Lin Guan (UT Austin) Paper#10921 1 / 33

  2. A Reinforcement Learning Problem: Montezuma’s Revenge Lin Guan (UT Austin) Paper#10921 2 / 33

  3. A Reinforcement Learning Problem: Montezuma’s Revenge Lin Guan (UT Austin) Paper#10921 3 / 33

  4. A Reinforcement Learning Problem: Montezuma’s Revenge Lin Guan (UT Austin) Paper#10921 4 / 33

  5. Learning Objective Find an optimal policy , i.e., the action to take in an observed state that maximizes expected longterm reward Lin Guan (UT Austin) Paper#10921 5 / 33

  6. Montezuma’s Revenge: Imitation Learning Lin Guan (UT Austin) Paper#10921 6 / 33

  7. Survey Scope 64 papers, 5 types of human guidance that... Lin Guan (UT Austin) Paper#10921 7 / 33

  8. Survey Scope 64 papers, 5 types of human guidance that... Are beyond conventional step-by-step action demonstrations Lin Guan (UT Austin) Paper#10921 7 / 33

  9. Survey Scope 64 papers, 5 types of human guidance that... Are beyond conventional step-by-step action demonstrations Have shown promising results in training agents to solve deep reinforcement learning tasks Lin Guan (UT Austin) Paper#10921 7 / 33

  10. Introduction 1 Learning from Human Evaluative Feedback 2 Learning from Human Preference 3 Hierarchical Imitation 4 Imitation from Observation 5 Learning Attention from Human 6 Conclusion 7 Lin Guan (UT Austin) Paper#10921 8 / 33

  11. Montezuma’s Revenge: Evaluative Feedback Lin Guan (UT Austin) Paper#10921 9 / 33

  12. Motivation While the true reward is delayed and sparse, human evaluative feedback is immediate and dense. Lin Guan (UT Austin) Paper#10921 10 / 33

  13. Representative Works Interpreting human feedback as: Reward function, replacing reward provided by the environment TAMER: Training an agent manually via evaluative reinforcement [Knox and Stone, 2009, Warnell et al., 2018] Lin Guan (UT Austin) Paper#10921 11 / 33

  14. Representative Works Interpreting human feedback as: Direct policy labels Advise [Griffith et al., 2013, Cederborg et al., 2015] Lin Guan (UT Austin) Paper#10921 12 / 33

  15. Representative Works Interpreting human feedback as: Direct policy labels Advise [Griffith et al., 2013, Cederborg et al., 2015] Advantage function COACH: Convergent actor-critic by humans [MacGlashan et al., 2017] This interpretation explains human feedback behaviors better in several tasks Still an unresolved issue that requires carefully designed human studies Lin Guan (UT Austin) Paper#10921 12 / 33

  16. Introduction 1 Learning from Human Evaluative Feedback 2 Learning from Human Preference 3 Hierarchical Imitation 4 Imitation from Observation 5 Learning Attention from Human 6 Conclusion 7 Lin Guan (UT Austin) Paper#10921 13 / 33

  17. Montezuma’s Revenge: Human Preference Lin Guan (UT Austin) Paper#10921 14 / 33

  18. Motivation Ranking behaviors is easier than rating them. And sometimes the ranking can only be provided at the end of a behavior trajectory. Lin Guan (UT Austin) Paper#10921 15 / 33

  19. Representative Works [Christiano et al., 2017]: As an inverse reinforcement learning problem, i.e., learn human reward function from human preference rather than from demonstration Lin Guan (UT Austin) Paper#10921 16 / 33

  20. Representative Works [Christiano et al., 2017]: As an inverse reinforcement learning problem, i.e., learn human reward function from human preference rather than from demonstration Query selection? Preference elicitation [Zintgraf et al., 2018] Lin Guan (UT Austin) Paper#10921 16 / 33

  21. Representative Works [Christiano et al., 2017]: As an inverse reinforcement learning problem, i.e., learn human reward function from human preference rather than from demonstration Query selection? Preference elicitation [Zintgraf et al., 2018] Many good works on preference-based reinforcement learning [Wirth et al., 2017] Lin Guan (UT Austin) Paper#10921 16 / 33

  22. Introduction 1 Learning from Human Evaluative Feedback 2 Learning from Human Preference 3 Hierarchical Imitation 4 Imitation from Observation 5 Learning Attention from Human 6 Conclusion 7 Lin Guan (UT Austin) Paper#10921 17 / 33

  23. Montezuma’s Revenge: Hierarchical Imitation Lin Guan (UT Austin) Paper#10921 18 / 33

  24. Motivation Human is good at specifying high-level abstract goals while the agent is good at performing low-level fine-grained controls. Lin Guan (UT Austin) Paper#10921 19 / 33

  25. Representative Works High-level+low-level demonstrations [Le et al., 2018] Lin Guan (UT Austin) Paper#10921 20 / 33

  26. Representative Works High-level+low-level demonstrations [Le et al., 2018] High-level demonstrations only [Andreas et al., 2017] Lin Guan (UT Austin) Paper#10921 20 / 33

  27. Representative Works High-level+low-level demonstrations [Le et al., 2018] High-level demonstrations only [Andreas et al., 2017] A promising combination: High-level: Imitation learning, e.g., DAgger [Ross et al., 2011] Low-level: Reinforcement learning, e.g., DQN [Mnih et al., 2015] Lin Guan (UT Austin) Paper#10921 20 / 33

  28. Introduction 1 Learning from Human Evaluative Feedback 2 Learning from Human Preference 3 Hierarchical Imitation 4 Imitation from Observation 5 Learning Attention from Human 6 Conclusion 7 Lin Guan (UT Austin) Paper#10921 21 / 33

  29. Montezuma’s Revenge: Imitation from Observation Lin Guan (UT Austin) Paper#10921 22 / 33

  30. Motivation To utilize a large amount of human demonstration data that do not have action labels, e.g., YouTube videos Lin Guan (UT Austin) Paper#10921 23 / 33

  31. Representative Works Challenge 1: Perception Viewpoint [Liu et al., 2018, Stadie et al., 2017] Embodiment [Gupta et al., 2018, Sermanet et al., 2018] Lin Guan (UT Austin) Paper#10921 24 / 33

  32. Representative Works Challenge 1: Perception Viewpoint [Liu et al., 2018, Stadie et al., 2017] Embodiment [Gupta et al., 2018, Sermanet et al., 2018] Challenge 2: Control Model-based: Infer the missing action given a state transitions ( s , s ′ ) by learning an inverse dynamics model [Nair et al., 2017, Torabi et al., 2018a] Model-free: e.g., bring the state distribution of the imitator closer to that of the trainer using generative adversarial learning [Merel et al., 2017, Torabi et al., 2018b] Lin Guan (UT Austin) Paper#10921 24 / 33

  33. Representative Works Challenge 1: Perception Viewpoint [Liu et al., 2018, Stadie et al., 2017] Embodiment [Gupta et al., 2018, Sermanet et al., 2018] Challenge 2: Control Model-based: Infer the missing action given a state transitions ( s , s ′ ) by learning an inverse dynamics model [Nair et al., 2017, Torabi et al., 2018a] Model-free: e.g., bring the state distribution of the imitator closer to that of the trainer using generative adversarial learning [Merel et al., 2017, Torabi et al., 2018b] Please see paper#10945: Recent Advances in Imitation Learning from Observation [Torabi et al., 2019] Lin Guan (UT Austin) Paper#10921 24 / 33

  34. Introduction 1 Learning from Human Evaluative Feedback 2 Learning from Human Preference 3 Hierarchical Imitation 4 Imitation from Observation 5 Learning Attention from Human 6 Conclusion 7 Lin Guan (UT Austin) Paper#10921 25 / 33

  35. Montezuma’s Revenge: Human Attention Lin Guan (UT Austin) Paper#10921 26 / 33

  36. Motivation Human visual attention provides additional information on why a particular decision is made, e.g., by indicating the current object of interest. Lin Guan (UT Austin) Paper#10921 27 / 33

  37. Representative Works AGIL: Attention-guided imitation learning [Zhang et al., 2018] Including attention does lead to higher accuracy in imitating human actions Lin Guan (UT Austin) Paper#10921 28 / 33

  38. Representative Works (a) Cooking [Li et al., 2018] (b) Driving [Palazzi et al., 2018, Xia et al., 2019] Lin Guan (UT Austin) Paper#10921 29 / 33

  39. Survey Scope An agent can learn... From human evaluative feedback From human preference From high-level goals specified by humans By observing human performing the task From human visual attention Lin Guan (UT Austin) Paper#10921 30 / 33

  40. Future Directions Shared datasets and reproducibility Understanding human trainers’ behaviors, e.g.,[Thomaz and Breazeal, 2008] A unified lifelong learning framework [Abel et al., 2017] Lin Guan (UT Austin) Paper#10921 31 / 33

  41. Survey: Leveraging Human Guidance for Deep Reinforcement Learning Tasks Ruohan Zhang, Faraz Torabi, Lin Guan, Dana H. Ballard, Peter Stone University of Texas at Austin Presented by Lin Guan Thank You! Lin Guan (UT Austin) Paper#10921 32 / 33

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature 2015] CS 486/686 University of Waterloo Lecture 20: July 10, 2017 Outline Value Function Approximation Linear approximation Neural

706 views • 19 slides
DeepMind Self-Learning Atari Agent Human - level control through deep reinforcement learning

DeepMind Self-Learning Atari Agent Human - level control through deep reinforcement learning

DeepMind Self-Learning Atari Agent Human - level control through deep reinforcement learning Nature Vol 518, Feb 26, 2015 The Deep Mind of Demis Hassabis Backchannel / Medium.com interview with David Levy Advanced Topics:

989 views • 68 slides
SDRL: Interpretable and Data-efficient Deep Liu Reinforcement Learning Introduction Background

SDRL: Interpretable and Data-efficient Deep Liu Reinforcement Learning Introduction Background

SDRL: Symbolic Deep Reinforcement Learning SDRL: Interpretable and Data-efficient Deep Liu Reinforcement Learning Introduction Background Leveraging Symbolic Planning Method Experiment Conclusion Bo Liu and Future Work Auburn

1k views • 20 slides
Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

540 views • 15 slides
1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement learning? Agent/Actor + Action + Environment + State + Reward How does reinforcement learning work?

792 views • 31 slides
Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and Courville [chapt. 6,7,8]; AIMA [sect. 21.1-21.3]; Sutton and Barto, Reinforcement Learning: an

527 views • 35 slides
Human-level control through deep reinforcement learning Volodymyr Mnih, Koray Kavukcuoglu, David

Human-level control through deep reinforcement learning Volodymyr Mnih, Koray Kavukcuoglu, David

Human-level control through deep reinforcement learning Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles

515 views • 39 slides
Human-level control through deep reinforcement Liia Butler But first... A quote "The

Human-level control through deep reinforcement Liia Butler But first... A quote "The

Human-level control through deep reinforcement Liia Butler But first... A quote "The question of whether machines can think... is about as relevant as the question of whether submarines can swim" Edsger W. Dijkstra Overview 1.

338 views • 21 slides
Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning Methods) Peter Henderson Statistical Society of Canada Annual Meeting 2018 Contributors Riashat Islam Phil Bachman Doina Precup David Meger Joelle

850 views • 37 slides
Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning

Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning

Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning arXiv:1702.08165 Reinforcement Learning with Deep Energy-Based Policies 1 / 25 Reinforcement Learning Environment Action Reward Interpreter State

531 views • 25 slides
Deep Reinforcement Learning for Robotics:

Deep Reinforcement Learning for Robotics:

Deep Reinforcement Learning for Robotics: Frontiers and Beyond Shixiang (Shane) Gu ( ) 2018.5.27 01 Deep RL: successes and limitations Computation-Constrained Data-Constrained

634 views • 21 slides
Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1.

Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1.

Introduction to Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1. Homework 1 is due next Wednesday! Remember that Monday is a holiday, so no office hours 2. Remember to start forming final project

877 views • 48 slides
Assessing Generalization in Deep Reinforcement Learning Soo Jung Jang Background Before (ex:

Assessing Generalization in Deep Reinforcement Learning Soo Jung Jang Background Before (ex:

Assessing Generalization in Deep Reinforcement Learning Soo Jung Jang Background Before (ex: factory robot) Now (ex: human-like intelligence) focus on one environment apply to multiple environment generalization is not considered

142 views • 12 slides
Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey

Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey

Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1. Homework 3 is extended by one week, to Wednesday after next Todays Lecture 1. Managing overfitting in model-based RL

686 views • 40 slides
Introduction to Deep Reinforcement Learning and Control Spring 2019, CMU 10-403 Katerina

Introduction to Deep Reinforcement Learning and Control Spring 2019, CMU 10-403 Katerina

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Introduction to Deep Reinforcement Learning and Control Spring 2019, CMU 10-403 Katerina Fragkiadaki Course Logistics Course website : all you need to

1.13k views • 88 slides
Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays

Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays

Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays Lecture 1. So far: manually design reward function to define a task 2. What if we want to learn the reward function from observing an expert, and

650 views • 33 slides
MICE Update J. Pasternak 03/12/2014, SLAC, MAP meeting Outline Introduction Preparations

MICE Update J. Pasternak 03/12/2014, SLAC, MAP meeting Outline Introduction Preparations

MICE Update J. Pasternak 03/12/2014, SLAC, MAP meeting Outline Introduction Preparations for Step IV MICE Demonstration of Ionization Cooling (MDIC) Summary 03/12/2014, SLAC, MAP meeting Basics of ionization cooling Muons

793 views • 41 slides
Mixed Precision Neural Architecture Search for Energy Efficient Deep Learning Chengyue Gong* 1 ,

Mixed Precision Neural Architecture Search for Energy Efficient Deep Learning Chengyue Gong* 1 ,

Mixed Precision Neural Architecture Search for Energy Efficient Deep Learning Chengyue Gong* 1 , Zixuan Jiang * 2 , Dilin Wang 1 , Yibo Lin 2 , Qiang Liu 1 , and David Z. Pan 2 1 CS Department, 2 ECE Department The University of Texas at Austin

329 views • 31 slides
Opportunities Andy Macdonald 6 th December 2016 Agenda 1. Offshore Renewable Energy Catapult

Opportunities Andy Macdonald 6 th December 2016 Agenda 1. Offshore Renewable Energy Catapult

Offshore Wind Meets Opportunities Andy Macdonald 6 th December 2016 Agenda 1. Offshore Renewable Energy Catapult 2. Opportunities 3. Challenges 4. Examples 5. Ideas # here ore.catapult.org.uk @ORECatapult ORE Catapult Mission:

286 views • 24 slides
The Relational Model of Data 5DV119 Introduction to Database Management Ume a University

The Relational Model of Data 5DV119 Introduction to Database Management Ume a University

The Relational Model of Data 5DV119 Introduction to Database Management Ume a University Department of Computing Science Jan Erik Mostr om jem+idb@cs.umu.se http://www.cs.umu.se/~jem Slides originally by Stephen J. Hegner

258 views • 25 slides
Nonlinear Modulational Instability of Dispersive PDE Models Jiayin Jin, Shasha Liao, and Zhiwu

Nonlinear Modulational Instability of Dispersive PDE Models Jiayin Jin, Shasha Liao, and Zhiwu

Nonlinear Modulational Instability of Dispersive PDE Models Jiayin Jin, Shasha Liao, and Zhiwu Lin Georgia Tech ICERM workshop on water waves, 4/28/2017 Jiayin Jin, Shasha Liao, and Zhiwu Lin Georgia Tech (ICERM workshop on water waves,

471 views • 29 slides
Machine Learning Software: Design and Practical Use Chih-Jen Lin National Taiwan University

Machine Learning Software: Design and Practical Use Chih-Jen Lin National Taiwan University

Machine Learning Software: Design and Practical Use Chih-Jen Lin National Taiwan University eBay Research Labs Talk at Machine Learning Summer School, Kyoto, August 31, 2012 Chih-Jen Lin (National Taiwan Univ.) 1 / 102 Machine Learning

1.3k views • 104 slides
Smaller, more accurate regression forests using tree alternating optimization Arman

Smaller, more accurate regression forests using tree alternating optimization Arman

Smaller, more accurate regression forests using tree alternating optimization Arman Zharmagambetov and Miguel A. Carreira-Perpi n an Dept. of Computer Science and Engineering University of California, Merced ICML, July 2020 Smaller,

315 views • 16 slides
Lecture 1 08/24/15 Instructor: Yu-San Lin yusan@psu.edu

Lecture 1 08/24/15 Instructor: Yu-San Lin yusan@psu.edu

CMPSC431W: Database Management Systems Lecture 1 08/24/15 Instructor: Yu-San Lin yusan@psu.edu Course Website: hCp://www.cse.psu.edu/~yul189/cmpsc431w Slides based

730 views • 14 slides

More recommend