assessing generalization in deep reinforcement learning
play

Assessing Generalization in Deep Reinforcement Learning Soo Jung - PowerPoint PPT Presentation

Dec 13, 2023 •22 likes •142 views

Assessing Generalization in Deep Reinforcement Learning Soo Jung Jang Background Before (ex: factory robot) Now (ex: human-like intelligence) focus on one environment apply to multiple environment generalization is not considered

  1. Assessing Generalization in Deep Reinforcement Learning Soo Jung Jang

  2. Background Before (ex: factory robot) Now (ex: human-like intelligence) focus on one environment apply to multiple environment generalization is not considered generalization is important ● Paper’s Goal: Empirical study of generalization in deep RL with different (1) algorithms, (2) environments, and (3) metrics

  3. Algorithms ● Vanilla (Baseline) Algorithms ○ A2C: Actor-Critic Family ○ PPO: Policy-Gradient Family ● Generalization-Tackling Algorithms ○ EPOpt: Robust Approach ○ RL2: Adapt Approach ● 6 Algorithms Total A2C, PPO, EPOpt-A2C, EPOpt-PPO, RL2-A2C, RL2-PPO

  4. Algorithms - Vanilla ● A2C / Actor-Critic Family ○ Critic: learns a value function ○ Actor: uses the function to learn a policy that maximizes expected reward ● PPO / Policy-Gradient Family ○ Learn sequence of improving policies ○ Maximize surrogate for the expected reward via gradient ascent

  5. Algorithms - Generalization-Tackling ● EPOpt / Robust Approach ○ Maximize expected reward over subset of environments with lowest expected reward (Maximize conditional value at risk) ● RL2 / Adapt Approach ○ Learn environment embedding at test time “on-the-fly” ○ RNN with current trajectory as input / hidden states = embeddings

  6. Algorithms - Network Architecture ● Feed Forward (FF) ○ Multi-layer perceptron (MLP) ● Recurrent (RC) ○ LSTM on top of MLP ● 4 Non-RL2 Algorithms → Test on both FF and RC ● 2 RL2 Algorithms → Test only on RC

  7. Environments ● 6 Environments (OpenAI) CartPole MountainCar AcroBot Pendulum HalfCheetah Hopper

  8. Metrics - Environment Parameters ● Deterministic (D) ○ fixed at default value (fixed environment) ● Random (R) ○ uniformly sampled from d -dimensional box (feasible environment) ● Extreme (E) ○ uniformly sampled from union of 2 intervals that straddle corresponding interval in R (edge cases) Schematic ( d =2 and 4 samples)

  9. Metrics - Evaluation ● 3 Evaluation Metrics From 3x3 train-test pairs of (D/R/E) 1. Default: DD 2. Interpolation: RR 3. Extrapolation: Mean of DR, DE, RE ● Metric Value (Performance) ○ Success Rate (%) of episodes where a certain goal is completed

  10. Experiment ● Compare performance of: ○ 10 algo combinations (6 algorithms / 2 architectures) ○ 6 environments ○ 3 metrics (default, interpolation, extrapolation) ● Methodology Train 15000 episodes / Test 1000 episodes ● Fairness No memory of previous episode Several sweeps of hyperparameters Success rate instead of reward itself

  11. Results ● Default > Inter > Extrapolation ● FF > RC Architecture ● Vanilla > Generalization-Tackling ● RL2 variants do not work ● EPOpt-PPO works well in continuous action space (Pendulum, ½ Cheetah, Hopper)

  12. Discussion Questions ● Generalization-tackling algorithms tested in this paper failed. What would be a potential strategy that makes generalization work? How would you solve this RL generalization problem? ● Why do you think generalization-tackling algorithms and recurrent (RC) architectures perform worse than Vanilla and feed forward (FF)? When would you expect generalized-tackling algorithms and recurrent (RC) architectures to work better? ● Do you think the paper’s experiment methodology is fair? Is there a better way to evaluate generalization on different algorithms and architectures?

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Structured Losses Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning

Structured Losses Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning

Structured Losses Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning Junhyuk Oh, Satinder Singh, Honglak Lee, Pushmeet Kohli Oh et al. 2017 https://arxiv.org/abs/1706.05064 Presented by Beln Saldas belen@mit.edu

820 views • 51 slides
Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

540 views • 15 slides
1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement learning? Agent/Actor + Action + Environment + State + Reward How does reinforcement learning work?

792 views • 31 slides
Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and Courville [chapt. 6,7,8]; AIMA [sect. 21.1-21.3]; Sutton and Barto, Reinforcement Learning: an

527 views • 35 slides
Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature 2015] CS 486/686 University of Waterloo Lecture 20: July 10, 2017 Outline Value Function Approximation Linear approximation Neural

706 views • 19 slides
Deep learning: Challenges in learning and generalization Tomas Mikolov, Facebook AI What is

Deep learning: Challenges in learning and generalization Tomas Mikolov, Facebook AI What is

Deep learning: Challenges in learning and generalization Tomas Mikolov, Facebook AI What is generalization? Memorization to remember all training examples (Universal Approximator) 2 + 3 = 5, 3 + 2 = 5, ... Generalization to

281 views • 14 slides
Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning Methods) Peter Henderson Statistical Society of Canada Annual Meeting 2018 Contributors Riashat Islam Phil Bachman Doina Precup David Meger Joelle

850 views • 37 slides
Towards a Foundation of Deep Learning: SGD, Overparametrization, and Generalization Jason D. Lee

Towards a Foundation of Deep Learning: SGD, Overparametrization, and Generalization Jason D. Lee

Towards a Foundation of Deep Learning: SGD, Overparametrization, and Generalization Jason D. Lee University of Southern California January 29, 2019 Jason Lee Successes of Deep Learning Game-playing (AlphaGo, DOTA, King of Glory) Computer

665 views • 48 slides
Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning

Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning

Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning arXiv:1702.08165 Reinforcement Learning with Deep Energy-Based Policies 1 / 25 Reinforcement Learning Environment Action Reward Interpreter State

531 views • 25 slides
Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1.

Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1.

Introduction to Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1. Homework 1 is due next Wednesday! Remember that Monday is a holiday, so no office hours 2. Remember to start forming final project

877 views • 48 slides
Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey

Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey

Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1. Homework 3 is extended by one week, to Wednesday after next Todays Lecture 1. Managing overfitting in model-based RL

686 views • 40 slides
Deep Reinforcement Learning and Complex Environments Raia Hadsell End-to-end Deep Learning

Deep Reinforcement Learning and Complex Environments Raia Hadsell End-to-end Deep Learning

Deep Reinforcement Learning and Complex Environments Raia Hadsell End-to-end Deep Learning for robots? slide from V. Vanhoucke End-to-end Deep Learning for robots? 2010 : Speech Recognition Audio Acoustic Model Phonetic Model

1.05k views • 65 slides
Generalization of Deep Learning 1 Yuan YAO HKUST Some Theories are limited but help:

Generalization of Deep Learning 1 Yuan YAO HKUST Some Theories are limited but help:

Generalization of Deep Learning 1 Yuan YAO HKUST Some Theories are limited but help: Approximation Theory and Harmonic Analysis : What functions are represented well by deep neural networks, without suffering the curse of dimensionality and

568 views • 15 slides
Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays

Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays

Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays Lecture 1. So far: manually design reward function to define a task 2. What if we want to learn the reward function from observing an expert, and

650 views • 33 slides
CMP784 DEEP LEARNING Lecture #12 Deep Reinforcement Learning Aykut Erdem // Hacettepe

CMP784 DEEP LEARNING Lecture #12 Deep Reinforcement Learning Aykut Erdem // Hacettepe

DeepLoco by X. B. Peng, G. Berseth & M. van de Panne CMP784 DEEP LEARNING Lecture #12 Deep Reinforcement Learning Aykut Erdem // Hacettepe University // Spring 2018 Neural Face by Taehoon Kim Previously on CMP784 Generative

895 views • 71 slides
Deep Reinforcement Learning for Robotics:

Deep Reinforcement Learning for Robotics:

Deep Reinforcement Learning for Robotics: Frontiers and Beyond Shixiang (Shane) Gu ( ) 2018.5.27 01 Deep RL: successes and limitations Computation-Constrained Data-Constrained

634 views • 21 slides
0-6 hour Weather Forecast Guidance at The Weather Company Steven Honey, Joseph Koval, Cathryn

0-6 hour Weather Forecast Guidance at The Weather Company Steven Honey, Joseph Koval, Cathryn

1 0-6 hour Weather Forecast Guidance at The Weather Company Steven Honey, Joseph Koval, Cathryn Meyer, Peter Neilley The Weather Company 2 TWC Forecasts: Widespread Adoption 3 0-6 Hour Forecast Details Global coverage 15-minute

336 views • 19 slides
Regularized Nonlinear Acceleration. Alexandre dAspremont , CNRS & D.I. Ecole Normale Sup

Regularized Nonlinear Acceleration. Alexandre dAspremont , CNRS & D.I. Ecole Normale Sup

Regularized Nonlinear Acceleration. Alexandre dAspremont , CNRS & D.I. Ecole Normale Sup erieure . with Damien Scieur & Francis Bach. Support from ERC SIPA and ITN SpaRTaN. Alex dAspremont Huatulco, January 2018. 1/30

766 views • 31 slides
Landsat Calibration: Interpolation, Extrapolation, and Reflection L DCM Sc ie nc e T e a m Me

Landsat Calibration: Interpolation, Extrapolation, and Reflection L DCM Sc ie nc e T e a m Me

Landsat Calibration: Interpolation, Extrapolation, and Reflection L DCM Sc ie nc e T e a m Me e ting USGS E ROS Aug ust 16-18, 2011 De nnis He lde r, Da ve Aa ro n And the I P L a b c re w Outline I nte rpo la tio n: Wha t ha

428 views • 22 slides
Evaluation for Stability data Q1E Sumie Yoshioka, Ph. D. MHLW National Institute of Health

Evaluation for Stability data Q1E Sumie Yoshioka, Ph. D. MHLW National Institute of Health

Evaluation for Stability data Q1E Sumie Yoshioka, Ph. D. MHLW National Institute of Health Sciences Q1E provides recommendations on : How to use stability data generated according to Q1AR When and how a retest period or a shelf life

344 views • 22 slides
Presentation to ARC Technical Coordinating Committee June 13, 2019 Fayette County 2016

Presentation to ARC Technical Coordinating Committee June 13, 2019 Fayette County 2016

Presentation to ARC Technical Coordinating Committee June 13, 2019 Fayette County 2016 Population: 109,495 Major Municipalities Peachtree City Fayetteville Tyrone No Interstate million vehicles passing into and out of

752 views • 18 slides
Biosimilars in Oncology The Patient Perspective Opportunities and Challenges Jenn Gordon

Biosimilars in Oncology The Patient Perspective Opportunities and Challenges Jenn Gordon

Biosimilars in Oncology The Patient Perspective Opportunities and Challenges Jenn Gordon Director of Operations Canadian Breast Cancer Network Disclosure: CBCN receives educational grants and funding from the following industry

592 views • 8 slides
The human dimension of the energy transition Linda Steg University of Groningen, Department of

The human dimension of the energy transition Linda Steg University of Groningen, Department of

faculty of behavioural environmental psychology 3-Feb-20 | 1 and social sciences 3-Feb-20 | 1 The human dimension of the energy transition Linda Steg University of Groningen, Department of Psychology 3-Feb-20 | 2 Psychology and

219 views • 20 slides
Team Building - New Management Principles to Motivate Teams Dr. Waffa Karkukly, PMP, ACP, CMP

Team Building - New Management Principles to Motivate Teams Dr. Waffa Karkukly, PMP, ACP, CMP

Team Building - New Management Principles to Motivate Teams Dr. Waffa Karkukly, PMP, ACP, CMP karkuklyw@yahoo.com March 1, 2014 Outline About us New Management Principles Empower Teams Grow Structure Energize People

918 views • 30 slides

More recommend