hierarchical reinforcement learning and human behavior
play

Hierarchical Reinforcement Learning and Human Behavior Matthew - PowerPoint PPT Presentation

Dec 08, 2023 •138 likes •845 views

Hierarchical Reinforcement Learning and Human Behavior Matthew Botvinick Princeton Neuroscience Institute and Department of Psychology Princeton University a a a * v v v Knutson et al., NeuroReport , 2001 Schultz et al.,

  1. Hierarchical Reinforcement Learning and Human Behavior Matthew Botvinick Princeton Neuroscience Institute and Department of Psychology Princeton University

  2. a a a * v v v � � �

  3. Knutson et al., NeuroReport , 2001 Schultz et al., Science , 1997

  4. Matsumoto & Hikosaka, Nature , 2007 Gehring & Willoughby, Science , 2002

  5. From Glasher, Daw, Dayan & O’Doherty, 2010 From Niv, Joel & Dayan, TICS, 2006 (artwork by B. Balleine)

  6. 0.8%

  7. The Curse of Dimensionality

  8. 15.6% The Blessing of Abstraction

  9. ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! s ( t =1) s ( t =2) s ( t =3) s ( t =4) s ( t =5) s ( t =6) Botvinick, Niv & Barto, Cognition , 2009

  10. W W S W W P G W W W W After Sutton, Precup & Singh, 1999 Botvinick, Niv & Barto, Cognition , 2009

  11. ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! s ( t =1) s ( t =2) s ( t =3) s ( t =4) s ( t =5) s ( t =6) Botvinick, Niv & Barto, Cognition , 2009

  13. Botvinick & Weinstein, Trans. Royal Society, 2014

  14. Hamilton & Grafton, J Neurosci , 2006 Humpheys & Forde, Cog. Neuropsych., 2001

  15. A C Actor Actor π ( s ) DLS Critic Critic state ( s ) state ( s ) V ( s ) VS action action R ( s ) δ DA HT+ Environment Environment B D Actor Actor 1 1 π ( s ) DLS DLS DLS o Critic Critic VS state ( s ) o state ( s ) DLPFC V ( s ) o + OFC action action R ( s ) δ HT+ DA o Environment Environment Botvinick, Niv & Barto, Cognition , 2009

  16. From Curtis & D’Esposito, TICS , 2003

  17. White & Wise, Exp Br Res, 1999

  18. Miller & Cohen, Ann. Rev. Neurosci , 2001

  19. From Badre, TICS , 2008

  20. A C Actor Actor π ( s ) DLS Critic Critic state ( s ) state ( s ) V ( s ) VS action action R ( s ) δ DA HT+ 2 2 Environment Environment B D Actor Actor π ( s ) DLS DLS DLS o Critic Critic VS state ( s ) o state ( s ) DLPFC V ( s ) o + OFC action action R ( s ) δ HT+ DA o Environment Environment Botvinick, Niv & Barto, Cognition , 2009

  21. O’Reilly & Frank, Neural Computation, 2006

  22. O’Reilly & Frank, Neural Computation, 2006 Bonini et al., J. Neurosci ., 2011

  23. A C Actor Actor π ( s ) DLS Critic Critic state ( s ) state ( s ) V ( s ) VS action action R ( s ) δ DA HT+ Environment Environment 3 3 B D Actor Actor π ( s ) DLS DLS DLS o Critic Critic VS state ( s ) o state ( s ) DLPFC V ( s ) o + OFC action action R ( s ) δ HT+ DA o Environment Environment Botvinick, Niv & Barto, Cognition , 2009

  24. Schoenbaum, et al. J Neurosci . 1999

  25. A C Actor Actor π ( s ) DLS Critic Critic state ( s ) state ( s ) V ( s ) VS action action R ( s ) δ DA HT+ Environment Environment B D Actor Actor π ( s ) DLS DLS DLS o Critic Critic VS state ( s ) o state ( s ) DLPFC V ( s ) o + OFC action action R ( s ) δ HT+ DA o Environment Environment 4 4 Botvinick, Niv & Barto, Cognition , 2009

  26. ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! s ( t =1) s ( t =2) s ( t =3) s ( t =4) s ( t =5) s ( t =6)

  27. Carlos Diuk

  28. Carlos Diuk Diuk, et al. , J Neurosci, 2013

  29. “RPE” ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! s ( t =1) s ( t =2) s ( t =3) s ( t =4) s ( t =5) s ( t =6) “PPE”

  30. Jose Fernandes Alec Solway Ribas-Fernandes et al., Neuron , 2011

  31. Jose Fernandes Alec Solway Standard RL Hierarchical RL PPE 1 RPE RPE A 0 -1 B D A C E D B C E Timestep Timestep Timestep Ribas-Fernandes et al., Neuron , 2011

  32. Jose Fernandes Alec Solway ! From Yeung, et al., 2005 Ribas-Fernandes et al., Neuron , 2011

  33. Jose Fernandes Alec Solway Ribas-Fernandes et al., Neuron , 2011

  34. Search Time A 4 Log Solution Time 3 Model Evidenc 2 1 1 100 200 Episode Botvinick, Niv & Barto, Cognition , 2009

  35. The Burden of Abstraction

  36. � 1. What should be learned? � � 2. Do people learn it? � � 3. How? �

  37. Alec Solway Carlos Diuk Solway et al., PLoS Comp. Biol., 2014

  38. Alec Solway Carlos Diuk Solway et al., PLoS Comp. Biol., 2014

  39. Alec Solway Carlos Diuk Solway et al., PLoS Comp. Biol., 2014

  40. Alec Solway Carlos Diuk !!!!!!!!!!!!!!! Pr !"#" !"#$% = Solway et al., PLoS Comp. Biol., 2014

  41. !!!!!!!!!!!!!!! Pr !"#" !"#$% = Pr !"#" !"#$% , ! Pr ! ! !"#$% , ! ∈ ! Codelength Search Time !!!!!!!!!!!!!!! Pr !"#" !"#$% = Model Evidence Solway et al., PLoS Comp. Biol., 2014

  42. Solway et al., PLoS Comp. Biol., 2014

  43. Zachary’s karate club Santa Fe Institute Lusseau’s bottlenose dolphins collaborations Fortunato, Physics Reports , 2010

  44. Simsek, Wolfe & Barto, 2005

  45. � 1. What should be learned? � � 2. Do people learn it? � � 3. How? �

  46. Carlos Diuk DebbieYee Solway et al., PLoS Comp. Biol., 2014

  47. Carlos Diuk DebbieYee Solway et al., PLoS Comp. Biol., 2014

  48. Carlos Diuk DebbieYee Solway et al., PLoS Comp. Biol., 2014

  49. Carlos Diuk DebbieYee B 2900 2800 G S ������������� 2700 2600 2500 2400 2300 2200 ���� Reject Solway et al., PLoS Comp. Biol., 2014

  50. � 1. What should be learned? � � 2. Do people learn it? � � 3. How? �

  51. , !!!!!!!!!!!!!!! Pr !"#" !"#$% = Pr !"#" !"#$% , ! Pr ! ! !"#$% ! ∈ !

  52. Anna Schapiro Schapiro et al., Nature Neurosci, 2013

  53. 1.00 0.66 -0.36 0.66 1.00 -0.36 -0.36 -0.36 1.00 Schapiro et al., Nature Neurosci, 2013

  54. Schapiro et al., Nature Neurosci, 2013

  55. Schapiro et al., Nature Neurosci, 2013

  56. Time Schapiro et al., Nature Neurosci, 2013

  57. Experiment 1 Experiment 2 All trials 0.4 0.4 Hamiltonian Probability of parse paths 0.3 0.3 0.2 0.2 0.1 0.1 0 0 Cluster transition Cluster transition Other parse Other parse parse parse Schapiro et al., Nature Neurosci, 2013

  58. + HC Schapiro et al., Nature Neurosci, 2013

  59. Carlos Diuk 1.0 Successor Representation 0.9 0.8 Correlation 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 0.38 0.36 0.34 Pattern Correlation 0.32 0.30 0.28 0.26 0.24 0.22 0.20 0.18 + HC Diuk et al., in prep.

  60. Current Stimulus Next Stimulus Schapiro et al., 2013.; Rogers & McClelland, 2003

  61. Codelength Search Time !!!!!!!!!!!!!!! Pr !"#" !"#$% = Model Evidence Solway et al., PLoS Comp. Biol., 2014

  62. cf. Dayan, 1993

  63. Rosvall & Bergstrom, PNAS, 2008

  64. Mahadevan & Maggioni, 2005

  65. Stachenfeld, Botvinick & Gershman, NIPS, 2014

  66. Olshausen & Field, Nature, 1996

  67. Botvinick & Plaut, Psych Review, 2004

  68. Conclusions � • The scaling problem in RL • Hierarchy can help • Model-free versus model-based HRL • HRL in the brain • The need for good representations • Task decomposition, bottlenecks, community detection • Prospective coding and structure discovery Codelength • Hierarchy as compression

  69. Collaborators Lab Contributors Carlos Diuk (Facebook) Jose Ribas-Fernandes (U. Victoria) Andy Barto (UMass) Anna Schapiro Yael Niv (Princeton) Alec Solway (V. Tech / UCL) Tim Rogers (Wisconsin) Kim Stachenfeld Nick Turk-Browne (Princeton) Ari Weinstein Debbie Yee (Wash. U.)

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 11: Hierarchical Reinforcement

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 11: Hierarchical Reinforcement

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 11: Hierarchical Reinforcement Learning Animesh Garg Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning Richard S. Sutton , Doina

2.18k views • 29 slides
FeUdal Networks for Hierarchical Reinforcement Learning Alexander Sasha Vezhnevets, Simon

FeUdal Networks for Hierarchical Reinforcement Learning Alexander Sasha Vezhnevets, Simon

FeUdal Networks for Hierarchical Reinforcement Learning Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, Koray Kavukcuoglu Topic: Hierarchical RL Presenter: Thophile Gaudin Why Hierarchical

921 views • 22 slides
Language as an Abstraction for Hierarchical Deep Reinforcement Learning Paper Authors: Yiding

Language as an Abstraction for Hierarchical Deep Reinforcement Learning Paper Authors: Yiding

Language as an Abstraction for Hierarchical Deep Reinforcement Learning Paper Authors: Yiding Jiang, Shixiang Gu, Kevin Murphy, Chelsea Finn Problem Overview Learning a variety of compositional , long horizon skills while being able to

364 views • 34 slides
Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature 2015] CS 486/686 University of Waterloo Lecture 20: July 10, 2017 Outline Value Function Approximation Linear approximation Neural

706 views • 19 slides
Device Placement Optimization with Reinforcement Learning A Hierarchical Model for Device

Device Placement Optimization with Reinforcement Learning A Hierarchical Model for Device

Device Placement Optimization with Reinforcement Learning A Hierarchical Model for Device Placement A. Mirhoseini, Hieu Pham, A. Goldie et al November 2019 Problem Background Tensorflow allows user to place operators on different devices to

371 views • 15 slides
Subgoals in Hierarchical Reinforcement Learning Tianren Tang Tian Tan Shangqi Guo Xiaolin Hu

Subgoals in Hierarchical Reinforcement Learning Tianren Tang Tian Tan Shangqi Guo Xiaolin Hu

Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning Tianren Tang Tian Tan Shangqi Guo Xiaolin Hu Feng Chen Background Goal-Conditional HRL High policy suffers from non-stationary problem From MARL's

330 views • 13 slides
Hierarchical Bayesian Methods for Reinforcement Learning David Wingate wingated@mit.edu Joint

Hierarchical Bayesian Methods for Reinforcement Learning David Wingate wingated@mit.edu Joint

Hierarchical Bayesian Methods for Reinforcement Learning David Wingate wingated@mit.edu Joint work with Noah Goodman, Dan Roy, Leslie Kaelbling and Joshua Tenenbaum My Research: Agents Rich sensory data Structured prior knowledge Reasonable

1.04k views • 90 slides
FeUdal Networks for Hierarchical Reinforcement Learning Alexander Sasha Vezhnevets, Simon

FeUdal Networks for Hierarchical Reinforcement Learning Alexander Sasha Vezhnevets, Simon

FeUdal Networks for Hierarchical Reinforcement Learning Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, Koray Kavukcuoglu: DeepMind Rene Bidart (rbbidart@uwaterloo.ca) CS885 June 22, 2018

653 views • 27 slides
Data-Efficient Hierarchical Reinforcement Learning Authors: Ofir Nachum, Shixiang Gu, Honglak

Data-Efficient Hierarchical Reinforcement Learning Authors: Ofir Nachum, Shixiang Gu, Honglak

Data-Efficient Hierarchical Reinforcement Learning Authors: Ofir Nachum, Shixiang Gu, Honglak Lee, Sergey Levine Presented by: Samuel Yigzaw 1 OUTLINE Introduction Background Main Contributions Related Work Experiments Conclusion PAGE 2

1.4k views • 26 slides
Video Captioning via Hierarchical Reinforcement Learning Xin Wang, Wenhu Chen, Jiawei Wi,

Video Captioning via Hierarchical Reinforcement Learning Xin Wang, Wenhu Chen, Jiawei Wi,

Video Captioning via Hierarchical Reinforcement Learning Xin Wang, Wenhu Chen, Jiawei Wi, Tuan-Fang Wang, William Yang Wang Publishing year: 2018 Presenter: David Radke CS885 University of Waterloo July, 2020 Overview Problem:

493 views • 24 slides
Interactive Reinforcement Learning Human Generated Reward Presentation for Summer Camp 2015

Interactive Reinforcement Learning Human Generated Reward Presentation for Summer Camp 2015

Interactive Reinforcement Learning Human Generated Reward Presentation for Summer Camp 2015 May 25 2015 Reinforcement Learning Trial and error learning Explore and exploit Sutton and Barto 1988 Represent, predict and control

252 views • 8 slides
CS 285 Instructor: Sergey Levine UC Berkeley Todays Lecture 1. Does reinforcement learning

CS 285 Instructor: Sergey Levine UC Berkeley Todays Lecture 1. Does reinforcement learning

Reframing Control as an Inference Problem CS 285 Instructor: Sergey Levine UC Berkeley Todays Lecture 1. Does reinforcement learning and optimal control provide a reasonable model of human behavior? 2. Is there a better explanation? 3.

782 views • 45 slides
RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

551 views • 35 slides
Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine playing a new game whose rules you dont know; after a hundred or so moves your don t know; after a hundred or so moves, your opponent announces, You

511 views • 30 slides
DeepMind Self-Learning Atari Agent Human - level control through deep reinforcement learning

DeepMind Self-Learning Atari Agent Human - level control through deep reinforcement learning

DeepMind Self-Learning Atari Agent Human - level control through deep reinforcement learning Nature Vol 518, Feb 26, 2015 The Deep Mind of Demis Hassabis Backchannel / Medium.com interview with David Levy Advanced Topics:

989 views • 68 slides
DualDICE Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections Ofir

DualDICE Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections Ofir

DualDICE Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections Ofir Nachum ,* Yinlam Chow,* Bo Dai, Lihong Li Google Research *Equal contribution Reinforcement Learning Reinforcement Learning A policy acts on an

575 views • 33 slides
Signaling Networks The word is the shadow of the deed. - Democritus Democritus The Laughing

Signaling Networks The word is the shadow of the deed. - Democritus Democritus The Laughing

from Democritus to Signaling Networks The word is the shadow of the deed. - Democritus Democritus The Laughing Philosopher 400BC Proclus states that Pythagoras and Epicurus agree with Cratylus, but Democritus and Aristotle agree with

953 views • 60 slides
De-ghosting for Gigapixel Snapshot Processing Alexandros-Stavros Iliopoulos 1 Jun Hu 1 Nikos

De-ghosting for Gigapixel Snapshot Processing Alexandros-Stavros Iliopoulos 1 Jun Hu 1 Nikos

De-ghosting for Gigapixel Snapshot Processing Alexandros-Stavros Iliopoulos 1 Jun Hu 1 Nikos Pitsianis 2 , 1 Xiaobai Sun 1 Mike Gehm 3 David Brady 1 1 Duke University 2 Aristotle University of Thessaloniki 3 University of Arizona March 20, 2013

736 views • 37 slides
Porting Nouveau to Tegra K1 How NVIDIA became a Nouveau contributor Alexandre Courbot, NVIDIA

Porting Nouveau to Tegra K1 How NVIDIA became a Nouveau contributor Alexandre Courbot, NVIDIA

Porting Nouveau to Tegra K1 How NVIDIA became a Nouveau contributor Alexandre Courbot, NVIDIA FOSDEM 2015 The Story So Far... In 2014 NVIDIA released the Tegra K1 SoC 32 bit quad-core or 64-bit dual core ARM 192-cores low-power

488 views • 23 slides
Generating Top-N Recommendations from Binary Profile Data Michael Hahsler Marketing Research and

Generating Top-N Recommendations from Binary Profile Data Michael Hahsler Marketing Research and

Generating Top-N Recommendations from Binary Profile Data Michael Hahsler Marketing Research and e-Business Adviser Hall Financial Group, Frisco, Texas, USA Hall Wines, St. Helena, California, USA Berufungsvortrag Wirtschaftsinformatik,

721 views • 38 slides
Robotics Engineering Prof. Michael Gennert Robotics Engineering Program Director Fall 2016

Robotics Engineering Prof. Michael Gennert Robotics Engineering Program Director Fall 2016

Robotics Engineering Prof. Michael Gennert Robotics Engineering Program Director Fall 2016 Robotics Education Gap Robotics Research PhD @ Large Research Robotics University Engineering Industrial Robotics Technology AA, AS @ Making

945 views • 22 slides
BLOGS ARE ECHO CHAMBERS: BLOGS ARE ECHO CHAMBERS Eric Gilbert | Tony Bergstrom | Karrie Karahalios

BLOGS ARE ECHO CHAMBERS: BLOGS ARE ECHO CHAMBERS Eric Gilbert | Tony Bergstrom | Karrie Karahalios

BLOGS ARE ECHO CHAMBERS: BLOGS ARE ECHO CHAMBERS Eric Gilbert | Tony Bergstrom | Karrie Karahalios | University of Illinois Are blogs echo chambers? Do blogs cut readers off from dissenting opinions? Negroponte 1996 Sunstein Sunstein 2002

406 views • 28 slides
An Extension of Systems Factorial Technology (SFT) to Arbitrary Numbers of Processes James

An Extension of Systems Factorial Technology (SFT) to Arbitrary Numbers of Processes James

An Extension of Systems Factorial Technology (SFT) to Arbitrary Numbers of Processes James T. Townsend 1 , Haiyuan Yang 1 , Mario Fific 2 1 Indiana University Bloomington 2 Grand Valley State University Theory Driven Methodology SFT is a

405 views • 36 slides
Marketing PostgreSQL brand where to start Valeria Kaplan dataegret.com About me PostgreSQL:

Marketing PostgreSQL brand where to start Valeria Kaplan dataegret.com About me PostgreSQL:

Marketing PostgreSQL brand where to start Valeria Kaplan dataegret.com About me PostgreSQL: 24/7 Support Consulting Training dataegret.com github.com/dataegret pgcenter.org facebook.com/dataegret @dataegret

732 views • 57 slides

More recommend