improving imitation learning with reinforcement learning
play

Improving Imitation Learning with Reinforcement Learning Niklas - PowerPoint PPT Presentation

Mar 25, 2024 •570 likes •905 views

MIN Faculty Department of Informatics Improving Imitation Learning with Reinforcement Learning Niklas Fiedler University of Hamburg Faculty of Mathematics, Informatics and Natural Sciences Department of Informatics Technical Aspects of

  1. MIN Faculty Department of Informatics Improving Imitation Learning with Reinforcement Learning Niklas Fiedler University of Hamburg Faculty of Mathematics, Informatics and Natural Sciences Department of Informatics Technical Aspects of Multimodal Systems November 26, 2019 N. Fiedler – Improving Imitation Learning with Reinforcement Learning 1 / 23

  2. Outline Introduction Imitation Learning Combining RL and IL Conclusion 1. Introduction Motivation 2. Imitation Learning Demonstration Methods Behavioral Cloning Inverse Reinforcement Learning 3. Combining Reinforcement Learning and Imitation Learning BC Application IRL Application 4. Conclusion N. Fiedler – Improving Imitation Learning with Reinforcement Learning 2 / 23

  3. Goal Introduction Imitation Learning Combining RL and IL Conclusion ◮ Imitate expert behavior ◮ Improve learning by including knowledge given by demonstration ◮ Learn expert policies → Make use of expert demonstrations N. Fiedler – Improving Imitation Learning with Reinforcement Learning 3 / 23

  4. Motivation Humans are Awesome Introduction Imitation Learning Combining RL and IL Conclusion https://rejectedprincesses.tumblr.com/post/150495232038/ chynara-madinkulova-long-hair-and-aida-akmatova N. Fiedler – Improving Imitation Learning with Reinforcement Learning 4 / 23

  5. Motivation Learning from Demonstration Introduction Imitation Learning Combining RL and IL Conclusion Learning from experts is natural behavior [Haw50], https://www.wakecounseling.com/therapy-blog/play-therapy N. Fiedler – Improving Imitation Learning with Reinforcement Learning 5 / 23

  6. Imitation Learning Introduction Imitation Learning Combining RL and IL Conclusion Method to learn a behavior based on a demonstration Various forms of demonstration. Two prominent methods of implementation: 1. Behavioral Cloning 2. Inverse Reinforcement Learning N. Fiedler – Improving Imitation Learning with Reinforcement Learning 6 / 23

  7. Imitation Learning Introduction Imitation Learning Combining RL and IL Conclusion Method to learn a behavior based on a demonstration Various forms of demonstration. Two prominent methods of implementation: 1. Behavioral Cloning 2. Inverse Reinforcement Learning N. Fiedler – Improving Imitation Learning with Reinforcement Learning 6 / 23

  8. Imitation Learning Introduction Imitation Learning Combining RL and IL Conclusion Method to learn a behavior based on a demonstration Various forms of demonstration. Two prominent methods of implementation: 1. Behavioral Cloning 2. Inverse Reinforcement Learning N. Fiedler – Improving Imitation Learning with Reinforcement Learning 6 / 23

  9. Demonstration Methods Introduction Imitation Learning Combining RL and IL Conclusion Virtual/Augumented Reality Tracking of Human Motions Video Stream Teleoperation s3.ap-south-1.amazonaws.com/kidobotikz.sprw/master/assets/images/blog/blog-2018110811630.jpg siamagazin.com/bimanual-teleoperation-of-a-compliant-whole-body-controlled-humanoid-robot/ https://ar-tracking.com/applications/motion-capture/ https://www.youtube.com/watch?v=5BTIE_fhReo N. Fiedler – Improving Imitation Learning with Reinforcement Learning 7 / 23

  10. Demonstration Methods Introduction Imitation Learning Combining RL and IL Conclusion Virtual/Augumented Reality Tracking of Human Motions Video Stream Teleoperation s3.ap-south-1.amazonaws.com/kidobotikz.sprw/master/assets/images/blog/blog-2018110811630.jpg siamagazin.com/bimanual-teleoperation-of-a-compliant-whole-body-controlled-humanoid-robot/ https://ar-tracking.com/applications/motion-capture/ https://www.youtube.com/watch?v=5BTIE_fhReo N. Fiedler – Improving Imitation Learning with Reinforcement Learning 7 / 23

  11. Demonstration Methods Introduction Imitation Learning Combining RL and IL Conclusion Virtual/Augumented Reality Tracking of Human Motions Video Stream Teleoperation s3.ap-south-1.amazonaws.com/kidobotikz.sprw/master/assets/images/blog/blog-2018110811630.jpg siamagazin.com/bimanual-teleoperation-of-a-compliant-whole-body-controlled-humanoid-robot/ https://ar-tracking.com/applications/motion-capture/ https://www.youtube.com/watch?v=5BTIE_fhReo N. Fiedler – Improving Imitation Learning with Reinforcement Learning 7 / 23

  12. Demonstration Methods Introduction Imitation Learning Combining RL and IL Conclusion Virtual/Augumented Reality Tracking of Human Motions Video Stream Teleoperation s3.ap-south-1.amazonaws.com/kidobotikz.sprw/master/assets/images/blog/blog-2018110811630.jpg siamagazin.com/bimanual-teleoperation-of-a-compliant-whole-body-controlled-humanoid-robot/ https://ar-tracking.com/applications/motion-capture/ https://www.youtube.com/watch?v=5BTIE_fhReo N. Fiedler – Improving Imitation Learning with Reinforcement Learning 7 / 23

  13. Behavioral Cloning Introduction Imitation Learning Combining RL and IL Conclusion ◮ Training a direct link between demonstrated input and output ◮ Large amounts of training data necessary ◮ Poor generalization N. Fiedler – Improving Imitation Learning with Reinforcement Learning 8 / 23

  14. Behavioral Cloning Video Introduction Imitation Learning Combining RL and IL Conclusion https://www.youtube.com/watch?v=5BTIE_fhReo N. Fiedler – Improving Imitation Learning with Reinforcement Learning 9 / 23

  15. Inverse Reinforcement Learning Reinforcement Learning Introduction Imitation Learning Combining RL and IL Conclusion N. Fiedler – Improving Imitation Learning with Reinforcement Learning 10 / 23

  16. Inverse Reinforcement Learning Reinforcement Learning vs. Inversed Reinforcement Learning Introduction Imitation Learning Combining RL and IL Conclusion RL IRL (partially observed) policy π or history given reward function R sampled from that policy reward function R optimal policy π searching for which given behavior for given reward is optimal https://thinkingwires.com/posts/2018-02-13-irl-tutorial-1.html N. Fiedler – Improving Imitation Learning with Reinforcement Learning 11 / 23

  17. Inverse Reinforcement Learning Introduction Imitation Learning Combining RL and IL Conclusion https://medium.com/@sanketgujar95/generative-adversarial-imitation-learning-266f45634e60 N. Fiedler – Improving Imitation Learning with Reinforcement Learning 12 / 23

  18. Imitation Learning Behavioral Cloning vs. Inversed Reinforcement Learning Introduction Imitation Learning Combining RL and IL Conclusion Inversed Reinforcement Behavioral Cloning Learning ◮ Weak generalization ◮ Strong generalization ◮ Relatively low ◮ Large computational effort computational effort ◮ Complex structure N. Fiedler – Improving Imitation Learning with Reinforcement Learning 13 / 23

  19. Combining Reinforcement Learning and Imitation Learning Introduction Imitation Learning Combining RL and IL Conclusion ◮ Reducing the impact of shortcomings of both methods ◮ Applications should outperform demonstrators after RL applications ◮ Accelerated training process ◮ Extending the capabilities learned with imitation learning N. Fiedler – Improving Imitation Learning with Reinforcement Learning 14 / 23

  20. BC Application Introduction Imitation Learning Combining RL and IL Conclusion Overcoming Exploration in Reinforcement Learning with Demonstrations Ashvin Nair 12 , Bob McGrew 1 , Marcin Andrychowicz 1 , Wojciech Zaremba 1 and Pieter Abbeel 12 2018 IEEE International Conference on Robotics and Automation (ICRA) 1 OpenAI 2 University of California, Berkeley N. Fiedler – Improving Imitation Learning with Reinforcement Learning 15 / 23

  21. BC Application Goal Introduction Imitation Learning Combining RL and IL Conclusion Pushing Sliding Pick and Place [NMA + 18] N. Fiedler – Improving Imitation Learning with Reinforcement Learning 16 / 23

  22. BC Application Results Introduction Imitation Learning Combining RL and IL Conclusion [NMA + 18] N. Fiedler – Improving Imitation Learning with Reinforcement Learning 17 / 23

  23. IRL Application Introduction Imitation Learning Combining RL and IL Conclusion Reinforcement and Imitation Learning for Diverse Visuomotor Skills Yuke Zhu 1 , Ziyu Wang 2 , Josh Merel 2 , Andrei Rusu 2 , Tom Erez 2 , Serkan Cabi 2 , Saran Tunyasuvunakool 2 , Janos Kramar 2 , Raia Hadsell 2 , Nando de Freitas 2 and Nicolas Heess 2 1 Computer Science Department, Stanford University 2 OpenAI N. Fiedler – Improving Imitation Learning with Reinforcement Learning 18 / 23

  24. IRL Application Goal Introduction Imitation Learning Combining RL and IL Conclusion [ZWM + 18] N. Fiedler – Improving Imitation Learning with Reinforcement Learning 19 / 23

  25. IRL Application Method Introduction Imitation Learning Combining RL and IL Conclusion [ZWM + 18] N. Fiedler – Improving Imitation Learning with Reinforcement Learning 20 / 23

  26. IRL Example Results - Block stacking Introduction Imitation Learning Combining RL and IL Conclusion [ZWM + 18] N. Fiedler – Improving Imitation Learning with Reinforcement Learning 21 / 23

  27. Combining Reinforcement Learning and Imitation Learning Comparison Introduction Imitation Learning Combining RL and IL Conclusion IRL Approach BC Approach ◮ Inversed Reinforcement ◮ Behavioral Cloning Learning ◮ Simulation only ◮ Policies transferred to real robot ◮ Goal : improve training ◮ Goal : improve result performance and task complexity performance and task complexity N. Fiedler – Improving Imitation Learning with Reinforcement Learning 22 / 23

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Learning to Optimize as Policy Learning Yisong Yue Policy Learning (Reinforcement &

Learning to Optimize as Policy Learning Yisong Yue Policy Learning (Reinforcement &

Learning to Optimize as Policy Learning Yisong Yue Policy Learning (Reinforcement & Imitation) Goal: Find Optimal Policy State/Context s t Agent Imitation Learning: Optimize imitation loss Reinforcement Learning: Optimize

547 views • 53 slides
Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent

Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent state reward action S t R t A t R t+ 1 Environment S t+ 1

1.07k views • 52 slides
Table of Contents Behavioral Cloning 1 Inverse Reinforcement Learning 2 Apprenticeship

Table of Contents Behavioral Cloning 1 Inverse Reinforcement Learning 2 Apprenticeship

Lecture 7: Imitation Learning 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 2 With slides from Katerina Fragkiadaki and Pieter Abbeel Lecture 7: Imitation Learning 3 Emma Brunskill (CS234 Reinforcement Learning. ) Winter 2018 1 /

1.27k views • 45 slides
Maximum Entropy Inverse RL, Adversarial imitation learning Katerina Fragkiadaki Reinforcement

Maximum Entropy Inverse RL, Adversarial imitation learning Katerina Fragkiadaki Reinforcement

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Maximum Entropy Inverse RL, Adversarial imitation learning Katerina Fragkiadaki Reinforcement Learning Probability Dynamics distribution over next Model T

608 views • 56 slides
Maximum Entropy Inverse RL, Adversarial imitation learning Katerina Fragkiadaki Reinforcement

Maximum Entropy Inverse RL, Adversarial imitation learning Katerina Fragkiadaki Reinforcement

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Maximum Entropy Inverse RL, Adversarial imitation learning Katerina Fragkiadaki Reinforcement Learning Probability Dynamics distribution over next Model T

911 views • 66 slides
CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 2: Supervised & Imitation

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 2: Supervised & Imitation

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 2: Supervised & Imitation Learning Instructor: Animesh Garg TA: Dylan Turpin & Tingwu Wang Agenda Invitation to Imitation DAGGER: Dataset Aggregation

904 views • 41 slides
Implicit Imitation in Multiagent Reinforcement Learning Bob Price and Craig Boutilier Slide 1

Implicit Imitation in Multiagent Reinforcement Learning Bob Price and Craig Boutilier Slide 1

Implicit Imitation in Multiagent Reinforcement Learning Bob Price and Craig Boutilier Slide 1 ICML-99 Slides: Dana Dahlstrom CSE 254, UCSD 2002.04.23 Overview Learning by imitation entails watching a mentor perform a task. Slide 2

414 views • 14 slides
10703 Deep Reinforcement Learning Imitation Learning - 1 Tom Mitchell November 4, 2018

10703 Deep Reinforcement Learning Imitation Learning - 1 Tom Mitchell November 4, 2018

10703 Deep Reinforcement Learning Imitation Learning - 1 Tom Mitchell November 4, 2018 Recommended readings: Used Materials Much of the material and slides for this lecture were borrowed from Katerina Fragkiadaki, and Ruslan

1.25k views • 49 slides
CS885 Reinforcement Learning Module 3: July 5, 2020 Imitation Learning Torabi, F., Warnell, G.,

CS885 Reinforcement Learning Module 3: July 5, 2020 Imitation Learning Torabi, F., Warnell, G.,

CS885 Reinforcement Learning Module 3: July 5, 2020 Imitation Learning Torabi, F., Warnell, G., & Stone, P. (2018). Behavioral cloning from observation. In IJCAI (pp. 4950-4957). Ho, J., & Ermon, S. (2016). Generative adversarial

464 views • 14 slides
Implicit Imitation in Multiagent Reinforcement Learning Bob Price and Craig Boutilier Slides:

Implicit Imitation in Multiagent Reinforcement Learning Bob Price and Craig Boutilier Slides:

Implicit Imitation in Multiagent Reinforcement Learning Bob Price and Craig Boutilier Slides: Dana Dahlstrom CSE 254, UCSD 2002.04.23 1 Overview In imitation, a learner observes a mentor in action. The approach proposed in this paper

440 views • 30 slides
Table of Contents Behavioral Cloning 1 Inverse Reinforcement Learning 2 Apprenticeship

Table of Contents Behavioral Cloning 1 Inverse Reinforcement Learning 2 Apprenticeship

Lecture 7: Imitation Learning in Large State Spaces 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2019 2 With slides from Katerina Fragkiadaki and Pieter Abbeel Lecture 7: Imitation Learning in Large State Spaces 3 Emma Brunskill (CS234

924 views • 47 slides
Refresh Your Knowledge 6 Experience replay in deep Q-learning (select all): Involves using a bank

Refresh Your Knowledge 6 Experience replay in deep Q-learning (select all): Involves using a bank

Lecture 7: Imitation Learning in Large State Spaces 1 Emma Brunskill CS234 Reinforcement Learning. Winter 2020 1 With slides from Katerina Fragkiadaki and Pieter Abbeel Lecture 7: Imitation Learning in Large State Spaces 1 Emma Brunskill (CS234

526 views • 52 slides
Refresh Your Knowledge 6 Experience replay in deep Q-learning (select all): Involves using a bank

Refresh Your Knowledge 6 Experience replay in deep Q-learning (select all): Involves using a bank

Lecture 7: Imitation Learning in Large State Spaces 1 Emma Brunskill CS234 Reinforcement Learning. Winter 2020 1 With slides from Katerina Fragkiadaki and Pieter Abbeel Lecture 7: Imitation Learning in Large State Spaces 1 Emma Brunskill (CS234

797 views • 57 slides
RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

551 views • 35 slides
Gen enerativ erative e Adver ersaria sarial l Im Imitation itation Le Learning arning

Gen enerativ erative e Adver ersaria sarial l Im Imitation itation Le Learning arning

Gen enerativ erative e Adver ersaria sarial l Im Imitation itation Le Learning arning Stefan efano o Ermon on Joint work with Jayesh Gupta, Jonathan Ho, Yunzhu Li, Hongyu Ren, and Jiaming Song Reinforcement Learning Goal: Learn

530 views • 40 slides
Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine playing a new game whose rules you dont know; after a hundred or so moves your don t know; after a hundred or so moves, your opponent announces, You

511 views • 30 slides
License Introduction to Version Control with Git Andreas Skielboe 1 All images adapted from Pro

License Introduction to Version Control with Git Andreas Skielboe 1 All images adapted from Pro

License Introduction to Version Control with Git Andreas Skielboe 1 All images adapted from Pro Git by Scott Chacon and released Adapted by Dr. Andrew Vardy 2 under license Creative Commons BY-NC-SA 3.0. 1. Dark Cosmology Centre See

335 views • 7 slides
Imitating Latent Policies from Observation Ashley D. Edwards, Himanshu Sahni, Yannick Schroecker,

Imitating Latent Policies from Observation Ashley D. Edwards, Himanshu Sahni, Yannick Schroecker,

Imitating Latent Policies from Observation Ashley D. Edwards, Himanshu Sahni, Yannick Schroecker, Charles L. Isbell Georgia Institute of Technology Introduction Imitation from Observation enables learning from state sequences Typical

553 views • 10 slides
MODEL-BASED API TESTING FOR SMT SOLVERS Aina Niemetz , Mathias Preiner , Armin

MODEL-BASED API TESTING FOR SMT SOLVERS Aina Niemetz , Mathias Preiner , Armin

MODEL-BASED API TESTING FOR SMT SOLVERS Aina Niemetz , Mathias Preiner , Armin Biere Johannes Kepler University, Linz, Austria Stanford University, USA SMT Workshop 2017, July 22 23 Heidelberg, Germany SMT Solvers

701 views • 27 slides
Schema validation and evolution for PGs Eugenia Oshurko (ENS Lyon) 7 March 2019 Main ideas

Schema validation and evolution for PGs Eugenia Oshurko (ENS Lyon) 7 March 2019 Main ideas

Schema validation and evolution for PGs Eugenia Oshurko (ENS Lyon) 7 March 2019 Main ideas Schema as a PG Schema validation via graph homomorphisms Schema nodes define node types Schema relations define relations allowed between

340 views • 11 slides
Coarse Classification of Binary Minimal Clones Zarathustra Brady Minimal clones A clone C is

Coarse Classification of Binary Minimal Clones Zarathustra Brady Minimal clones A clone C is

Coarse Classification of Binary Minimal Clones Zarathustra Brady Minimal clones A clone C is minimal if f C nontrivial implies C = Clo( f ). Minimal clones A clone C is minimal if f C nontrivial implies C = Clo( f ). If Clo( f

1.52k views • 96 slides
Rcpp classes and vectors Romain Franois Consulting Datactive, ThinkR DataCamp Optimizing R

Rcpp classes and vectors Romain Franois Consulting Datactive, ThinkR DataCamp Optimizing R

DataCamp Optimizing R Code with Rcpp OPTIMIZING R CODE WITH RCPP Rcpp classes and vectors Romain Franois Consulting Datactive, ThinkR DataCamp Optimizing R Code with Rcpp Previously on this course Create C++ functions Write loops

572 views • 35 slides
Scalable Detection of Semantic Clones Mark Gabel Lingxiao Jiang Zhendong Su Motivation

Scalable Detection of Semantic Clones Mark Gabel Lingxiao Jiang Zhendong Su Motivation

Scalable Detection of Semantic Clones Mark Gabel Lingxiao Jiang Zhendong Su Motivation Maintenance problem Refactoring Automated procedure extraction Aspect mining Program understanding Copy/paste bugs 2 Clone

421 views • 27 slides
Reverse engineering Reverse engineer Did anyone analyze f1 something similar A binary file f2

Reverse engineering Reverse engineer Did anyone analyze f1 something similar A binary file f2

Asm2Vec : Boosting Static Representation Robustness for Binary Clone Search against Code Obfuscation and Compiler Optimization Steven H. H. Ding Benjamin C. M. Fung Philippe Charland Data Mining and Security Lab Mission Critical Cyber Security

625 views • 23 slides

More recommend