PDF Editor
PDF Converter
Image Converter
Video Converter
Audio Converter
File Compressor
deep reinforcement learning building blocks

Deep Reinforcement Learning Building Blocks Arjun Chandra Research - PowerPoint PPT Presentation

Feb 11, 2023 •176 likes •1.68k views

Deep Reinforcement Learning Building Blocks Arjun Chandra Research Scientist Telenor Research / Telenor-NTNU AI Lab arjun.chandra@telenor.com @boelger 8 November 2017 https://join.slack.com/t/deep-rl-tutorial/signup The Plan The Problem

  1. our toy problem lookup table N S E W 1 0 0 0 0 2 0 0 0 0 3 0 0 0 0 4 0 0 0 0 5 0 0 0 0 6 0 0 0 0 home 7 0 0 0 0 8 0 0 0 0 9 0 0 0 0

  2. our toy problem lookup table 0 0 0 1 2 3 0 0 0 0 0 0 0 0 0 0 0 0 4 5 6 0 0 0 0 0 0 home 0 0 0 0 0 0 7 8 9 0 0 0 0 0 0 home 0 0 0

  3. reward structure? 0 0 0 1 2 3 0 0 0 0 0 0 0 0 0 0 0 0 0 4 5 6 0 0 0 0 0 0 0 0 home 0 0 0 0 0 0 0 7 7 8 9 0 0 0 0 0 0 home 0 0 0 move… to 7/home: out of bounds: to 5: to any cell except 5 and 7: 10 -5 -10 -1

  4. let’s fix 𝛽 = 0.1, 𝛿 = 0.5 0 0 0 1 2 3 0 0 0 0 0 0 0 0 0 0 0 0 4 5 6 0 0 0 0 0 0 home 0 0 0 0 0 0 7 8 9 0 0 0 0 0 0 0 0 0

  5. a + γ max a ' Q ( s ', a ') − Q ( s , a )) Q ( s , a ) ← Q ( s , a ) + α ( r s 0 0 0 1 2 3 0 0 0 0 0 0 0 0 0 𝛽 = 0.1 0 0 0 𝛿 = 0.5 4 5 6 0 0 0 0 0 0 0 0 0 0 0 0 7 8 9 0 0 0 0 0 0 home 0 0 0 say 𝜁 -greedy policy… episode 1 begins...

  6. a + γ max a ' Q ( s ', a ') − Q ( s , a )) Q ( s , a ) ← Q ( s , a ) + α ( r s -1 0 0 0 1 2 3 0 0 ? 0 0 0 0 0 0 𝛽 = 0.1 0 0 0 𝛿 = 0.5 4 5 6 0 0 0 0 0 0 0 0 0 0 0 0 7 8 9 0 0 0 0 0 0 home 0 0 0

  7. a + γ max a ' Q ( s ', a ') − Q ( s , a )) Q ( s , a ) ← Q ( s , a ) + α ( r s 0 0 0 1 2 3 0 0 -0.1 0 0 0 0 0 0 𝛽 = 0.1 0 0 0 𝛿 = 0.5 4 5 6 0 0 0 0 0 0 0 0 0 0 0 0 7 8 9 0 0 0 0 0 0 home 0 0 0

  8. a + γ max a ' Q ( s ', a ') − Q ( s , a )) Q ( s , a ) ← Q ( s , a ) + α ( r s 0 0 0 1 2 3 0 0 -0.1 0 0 0 0 0 0 𝛽 = 0.1 0 0 0 𝛿 = 0.5 4 5 6 0 0 0 0 0 0 0 0 0 0 0 0 7 8 9 0 0 0 0 0 0 home 0 0 0

  9. a + γ max a ' Q ( s ', a ') − Q ( s , a )) Q ( s , a ) ← Q ( s , a ) + α ( r s -5 ? 0 0 1 2 3 0 0 -0.1 0 0 0 0 0 0 𝛽 = 0.1 0 0 0 𝛿 = 0.5 4 5 6 0 0 0 0 0 0 0 0 0 0 0 0 7 8 9 0 0 0 0 0 0 home 0 0 0

  10. a + γ max a ' Q ( s ', a ') − Q ( s , a )) Q ( s , a ) ← Q ( s , a ) + α ( r s -0.5 0 0 1 2 3 0 0 -0.1 0 0 0 0 0 0 𝛽 = 0.1 0 0 0 𝛿 = 0.5 4 5 6 0 0 0 0 0 0 0 0 0 0 0 0 7 8 9 0 0 0 0 0 0 home 0 0 0

  11. a + γ max a ' Q ( s ', a ') − Q ( s , a )) Q ( s , a ) ← Q ( s , a ) + α ( r s -0.5 0 0 1 2 3 0 0 -0.1 0 0 0 0 0 0 𝛽 = 0.1 0 0 0 𝛿 = 0.5 4 5 6 0 0 0 0 0 0 0 0 0 0 0 0 7 8 9 0 0 0 0 0 0 home 0 0 0

  12. a + γ max a ' Q ( s ', a ') − Q ( s , a )) Q ( s , a ) ← Q ( s , a ) + α ( r s -0.5 0 0 1 2 3 0 0 -0.1 0 0 0 -1 ? 0 0 𝛽 = 0.1 0 0 0 𝛿 = 0.5 4 5 6 0 0 0 0 0 0 0 0 0 0 0 0 7 8 9 0 0 0 0 0 0 home 0 0 0

  13. a + γ max a ' Q ( s ', a ') − Q ( s , a )) Q ( s , a ) ← Q ( s , a ) + α ( r s -0.5 0 0 1 2 3 0 0 -0.1 0 0 0 -0.1 0 0 𝛽 = 0.1 0 0 0 𝛿 = 0.5 4 5 6 0 0 0 0 0 0 0 0 0 0 0 0 7 8 9 0 0 0 0 0 0 home 0 0 0

  14. a + γ max a ' Q ( s ', a ') − Q ( s , a )) Q ( s , a ) ← Q ( s , a ) + α ( r s -0.5 0 0 1 2 3 0 0 -0.1 0 0 0 -0.1 0 0 𝛽 = 0.1 0 0 0 𝛿 = 0.5 4 5 6 0 0 0 0 0 0 0 0 0 0 0 0 7 8 9 0 0 0 0 0 0 home 0 0 0

  15. a + γ max a ' Q ( s ', a ') − Q ( s , a )) Q ( s , a ) ← Q ( s , a ) + α ( r s -0.5 0 0 1 2 3 0 0 -0.1 0 0 0 -10 -0.1 0 0 𝛽 = 0.1 0 0 0 𝛿 = 0.5 4 5 6 0 ? 0 0 0 0 0 0 0 0 0 0 7 8 9 0 0 0 0 0 0 home 0 0 0

  16. a + γ max a ' Q ( s ', a ') − Q ( s , a )) Q ( s , a ) ← Q ( s , a ) + α ( r s -0.5 0 0 1 2 3 0 0 -0.1 0 0 0 -0.1 0 0 𝛽 = 0.1 0 0 0 𝛿 = 0.5 4 5 6 0 -1 0 0 0 0 0 0 0 0 0 0 7 8 9 0 0 0 0 0 0 home 0 0 0

  17. a + γ max a ' Q ( s ', a ') − Q ( s , a )) Q ( s , a ) ← Q ( s , a ) + α ( r s -0.5 0 0 1 2 3 0 0 -0.1 0 0 0 -0.1 0 0 𝛽 = 0.1 0 0 0 𝛿 = 0.5 4 5 6 0 -1 0 0 0 0 0 0 0 0 0 0 7 8 9 0 0 0 0 0 0 home 0 0 0

  18. a + γ max a ' Q ( s ', a ') − Q ( s , a )) Q ( s , a ) ← Q ( s , a ) + α ( r s -0.5 0 0 1 2 3 0 0 -0.1 0 0 0 -0.1 0 0 𝛽 = 0.1 0 0 0 𝛿 = 0.5 4 5 6 0 -1 0 0 0 0 -1 0 ? 0 0 0 0 7 8 9 0 0 0 0 0 0 home 0 0 0

  19. a + γ max a ' Q ( s ', a ') − Q ( s , a )) Q ( s , a ) ← Q ( s , a ) + α ( r s -0.5 0 0 1 2 3 0 0 -0.1 0 0 0 -0.1 0 0 𝛽 = 0.1 0 0 0 𝛿 = 0.5 4 5 6 0 -1 0 0 0 0 0 -0.1 0 0 0 0 7 8 9 0 0 0 0 0 0 home 0 0 0

  20. a + γ max a ' Q ( s ', a ') − Q ( s , a )) Q ( s , a ) ← Q ( s , a ) + α ( r s -0.5 0 0 1 2 3 0 0 -0.1 0 0 0 -0.1 0 0 𝛽 = 0.1 0 0 0 𝛿 = 0.5 4 5 6 0 -1 0 0 0 0 0 -0.1 0 0 0 0 7 8 9 0 0 0 0 0 0 home 0 0 0

  21. a + γ max a ' Q ( s ', a ') − Q ( s , a )) Q ( s , a ) ← Q ( s , a ) + α ( r s -0.5 0 0 1 2 3 0 0 -0.1 0 0 0 -0.1 0 0 𝛽 = 0.1 0 0 0 𝛿 = 0.5 4 5 6 0 -1 0 0 0 0 0 -0.1 0 0 0 0 10 7 8 9 0 0 ? 0 0 0 home 0 0 0

  22. a + γ max a ' Q ( s ', a ') − Q ( s , a )) Q ( s , a ) ← Q ( s , a ) + α ( r s -0.5 0 0 1 2 3 0 0 -0.1 0 0 0 -0.1 0 0 𝛽 = 0.1 0 0 0 𝛿 = 0.5 4 5 6 0 -1 0 0 0 0 0 -0.1 0 0 0 0 7 8 9 0 0 1 0 0 0 home 0 0 0 episode 1 ends.

  23. let’s work out the next episode, star<ng at state 4 -0.5 0 0 1 2 3 0 0 -0.1 0 0 0 -0.1 0 0 𝛽 = 0.1 0 0 0 𝛿 = 0.5 4 5 6 0 -1 0 0 0 0 0 -0.1 0 0 0 0 7 8 9 0 0 1 0 0 0 home 0 0 0 go WEST and then SOUTH how does the table change?

  24. -0.5 0 0 1 2 3 0 0 -0.1 0 0 0 -0.1 0 0 𝛽 = 0.1 0 0 0 𝛿 = 0.5 4 5 6 -0.5 -1 0 0 0 0 1 -0.1 0 0 0 0 7 8 9 0 0 1 0 0 0 0 0 0

  25. and the next episode, star<ng at state 3 go WEST -> SOUTH -> WEST -> SOUTH

  26. -0.5 0 0 1 2 3 0 0 -0.1 0 -0.1 0 -0.1 -1 0 𝛽 = 0.1 0 0 0 𝛿 = 0.5 4 5 6 -0.5 -1 -0.05 0 0 0 1.9 -0.1 0 0 0 0 7 8 9 0 0 1 0 0 0 0 0 0 over <me, values will converge to op<mal!

  27. what we just saw was some episodes of Q-learning values update towards value of op+mal policy : target comes from value of assumed next best ac+on off-policy learning

  28. SARSA-learning? values update towards value of current policy : target comes from value of the actual next ac+on on-policy learning

  29. Q SARSA By Andreas Tille (Own work) [GFDL (www.gnu.org/copylei/fdl.html) or CC-BY-SA-3.0-2.5-2.0-1.0 (www.creaCvecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons data generated by data not generated by target policy target policy 𝜁 : 0.1 𝛿 : 1.0 SARSA Q Example credit Travis DeWolf : https://studywolf.wordpress.com/ and https://git.io/vFBvv

  30. Problem Decomposition nested sub-problems solution to sub-problem informs solution to whole problem

  31. Bellman Expectation Backup system of linear equations solution: value of policy a, q(s, a) s, v(s) r s’ a r a’, q(s’, a’) s’, v(s’) Value of = P(path) * Value(path) Value of = P(path) * Value(path) ⎛ ⎞ a + γ a + γ a v π ( s ') ∑ ∑ ∑ ∑ v π ( s ) = π ( a | s ) r s q π ( s , a ) = r s π ( a '| s ') q π ( s ', a ') P a P ⎜ ⎟ ⎝ ⎠ ss ' ss ' a s ' s ' a ' Bellman expectation equations under a given policy

  32. Bellman Optimality Backup system of non-linear equations solution: value of optimal policy a, q(s, a) s, v(s) r s’ a r a’, q(s’, a’) s’, v(s’) Value of = P(path) * Value(path) Value of = P(path) * Value(path) ⎛ ⎞ a + γ a + γ a v * ( s ') ∑ ∑ q * ( s , a ) = r s a v * ( s ) = max a r s P max a ' q * ( s ', a ') P ⎜ ⎟ ⎝ ⎠ ss ' ss ' s ' s ' Bellman optimality equations under optimal policy

  33. Value Based

  34. Dynamic Programming …using Bellman equations as iterative updates 1 2 what’s best to do? -5 -5 N E -1 W 1 2 -1 -5 -5 3 4 S -10 10 3 4 home

  35. Dynamic Programming …using Bellman equations as iterative updates 1 2 what’s best to do? -5 -5 N E -1 W 1 2 -1 -5 -5 3 4 S -10 10 3 4 home

Recommend

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and Courville [chapt. 6,7,8]; AIMA [sect. 21.1-21.3]; Sutton and Barto, Reinforcement Learning: an

528 views • 35 slides
Reinforcement Learning Timothy Chou Charlie Tong Vincent Zhuang April 19, 2016 Reinforcement

Reinforcement Learning Timothy Chou Charlie Tong Vincent Zhuang April 19, 2016 Reinforcement

Reinforcement Learning Q-Learning Deep Q-Learning on Atari Reinforcement Learning Timothy Chou Charlie Tong Vincent Zhuang April 19, 2016 Reinforcement Learning Q-Learning Deep Q-Learning on Atari Table of Contents Reinforcement Learning

939 views • 63 slides
Blocks What is syntax (delimiters) Where can blocks be used Scope and blocks Do

Blocks What is syntax (delimiters) Where can blocks be used Scope and blocks Do

Blocks What is syntax (delimiters) Where can blocks be used Scope and blocks Do blocks return a value (use in expressions) Entry point(s) to blocks Exit point(s) from blocks Block syntax and use Typically need start/stop

463 views • 5 slides
Reinforcement Learning AIMA Chapters: 21.1, 21.2, 21.3. Sutton and Barto, Reinforcement Learning:

Reinforcement Learning AIMA Chapters: 21.1, 21.2, 21.3. Sutton and Barto, Reinforcement Learning:

Reinforcement Learning Reinforcement Learning AIMA Chapters: 21.1, 21.2, 21.3. Sutton and Barto, Reinforcement Learning: an Introduction, 2nd Edition: Chapters 6 (6.1 6.5) Outline Reinforcement Learning Reinforcement Learning: the

589 views • 27 slides
1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement learning? Agent/Actor + Action + Environment + State + Reward How does reinforcement learning work?

793 views • 31 slides
RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

552 views • 35 slides
Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

541 views • 15 slides
Reinforcement Learning UMaine COS 470/570 Introduction to AI Why reinforcement learning?

Reinforcement Learning UMaine COS 470/570 Introduction to AI Why reinforcement learning?

Reinforcement Learning&lt;br/&gt;&lt;br/&gt; 4/25/19, 8*06 PM Reinforcement Learning&lt;br/&gt;&lt;br/&gt; 4/25/19, 8*06 PM Reinforcement Learning UMaine COS 470/570 Introduction to AI Why reinforcement learning? Spring 2019 Created:

371 views • 15 slides
Reinforcement Learning and Simulation-Based Search David Silver Reinforcement Learning and

Reinforcement Learning and Simulation-Based Search David Silver Reinforcement Learning and

Reinforcement Learning and Simulation-Based Search Reinforcement Learning and Simulation-Based Search David Silver Reinforcement Learning and Simulation-Based Search Outline 1 Reinforcement Learning 2 Simulation-Based Search 3 Planning Under

425 views • 20 slides
Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine playing a new game whose rules you dont know; after a hundred or so moves your don t know; after a hundred or so moves, your opponent announces, You

512 views • 30 slides
Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest

Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest

Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest Lecture May 24, 2017 Lecture overview What makes a reinforcement learning algorithm safe ? Notation Creating a safe reinforcement learning

1.42k views • 88 slides
Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature 2015] CS 486/686 University of Waterloo Lecture 20: July 10, 2017 Outline Value Function Approximation Linear approximation Neural

706 views • 19 slides
Deep learning Deep reinforcement learning Hamid Beigy Sharif university of technology December

Deep learning Deep reinforcement learning Hamid Beigy Sharif university of technology December

Deep learning Deep learning Deep reinforcement learning Hamid Beigy Sharif university of technology December 25, 2018 Hamid Beigy | Sharif university of technology | December 25, 2018 1 / 65 Deep learning Table of contents 1 Introduction 2

836 views • 65 slides
FBPQ and building blocks FBPQ and building blocks Mark Drye Director of Asset Management

FBPQ and building blocks FBPQ and building blocks Mark Drye Director of Asset Management

FBPQ and building blocks FBPQ and building blocks Mark Drye Director of Asset Management Director of Asset Management CE Electric UK The building blocks of investment planning Investment plans are built from a consideration of the motives

344 views • 6 slides
CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning Haarnoja, Tang et al. (2017) Reinforcement Learning with Deep Energy Based Policies, ICML . Haarnoja, Zhou et al. (2018) Soft Actor-Critic: Off-Policy

684 views • 24 slides
Deep Reinforcement Learning Philipp Koehn 21 April 2020 Philipp Koehn Artificial Intelligence:

Deep Reinforcement Learning Philipp Koehn 21 April 2020 Philipp Koehn Artificial Intelligence:

Deep Reinforcement Learning Philipp Koehn 21 April 2020 Philipp Koehn Artificial Intelligence: Deep Reinforcement Learning 21 April 2020 Reinforcement Learning 1 Sequence of actions moves in chess driving controls in car

815 views • 63 slides
Global polarization signals from hot, dense and whirly QCD matter Malena Tejeda-Yeomans

Global polarization signals from hot, dense and whirly QCD matter Malena Tejeda-Yeomans

Global polarization signals from hot, dense and whirly QCD matter Malena Tejeda-Yeomans Facultad de Ciencias Universidad de Colima exico a M a (MexNICA Collaboration) A. Ayala, M. Ayala, E. Cuautle, I. Dom nguez, M. Fontaine, I.

902 views • 35 slides
SAP cyber Slapping A Penetra2on Testers Guide CALL TRANSACTION SUIM Dave

SAP cyber Slapping A Penetra2on Testers Guide CALL TRANSACTION SUIM Dave

DeepSec Vienna 2012 SAP cyber Slapping A Penetra2on Testers Guide CALL TRANSACTION SUIM Dave Hartley ( @nmonkee ). Principal Security Consultant @MWRInfoSecurity / @MWRLabs .

1.25k views • 113 slides
On the status of flavor anomalies Diego Guadagnoli LAPTh Annecy (France) Recap of flavor

On the status of flavor anomalies Diego Guadagnoli LAPTh Annecy (France) Recap of flavor

On the status of flavor anomalies Diego Guadagnoli LAPTh Annecy (France) Recap of flavor anomalies: b s LHCb and B factories measured several key b s and b c modes. Agreement with the SM is less than perfect. D. Guadagnoli,

802 views • 76 slides
Quick context UNSW top 100 research intensive, 59,000 students and 7,000 staff. New

Quick context UNSW top 100 research intensive, 59,000 students and 7,000 staff. New

Back to the future 22nd June to 1st July using a pen in STEM e-exams Mathew Hillier now at Macquarie University, Sydney, Australia Pilot ran Aug to Dec 2019 Quick context UNSW top 100 research intensive, 59,000 students and

321 views • 11 slides
Robust mixture modeling using multivariate skew t distributions Tsung-I Lin Department of

Robust mixture modeling using multivariate skew t distributions Tsung-I Lin Department of

Robust mixture modeling using multivariate skew t distributions Tsung-I Lin Department of Applied Mathematics and Institute of Statistics National Chung Hsing University, Taiwan August 24, 2010 ( NCHU ) 1 / 15 T.I. Lin National Chung Hsing

159 views • 15 slides
EPD BLUP Daniela Lourenco Keith Bertrand Heather Bradford,

EPD BLUP Daniela Lourenco Keith Bertrand Heather Bradford,

Daniela Lourenco, Un. of Georgia 6/1/17 Traditional evaluation Performance The promise of Pedigree Progeny genomics for beef improvement EPD BLUP Daniela Lourenco Keith

478 views • 8 slides
Seguridad de las contraseas Este trabajo fue originalmente desarrollado por

Seguridad de las contraseas Este trabajo fue originalmente desarrollado por

Seguridad de las contraseas Este trabajo fue originalmente desarrollado por CERT.br, parte de NIC.br, con el obje&lt;vo de promover la concien&lt;zacin sobre el uso

697 views • 27 slides
Conformality and Q harmonicity in sub-Riemannian manifolds Joint work L.C., Enrico Le Donne

Conformality and Q harmonicity in sub-Riemannian manifolds Joint work L.C., Enrico Le Donne

Conformality and Q harmonicity in sub-Riemannian manifolds Joint work L.C., Enrico Le Donne (Jyv askyl a) and Alessandro Ottazzi (CIRM, Trento) Paris, September 29th October 3rd, 2014 Geometric Analysis on sub-Riemannian manifolds

627 views • 31 slides

More recommend

Logo Sambuz

Unleash a World of Digital Possibilities—Browse, Share, and Explore Content Without Boundaries

Useful Links

About Us Privacy Services FAQ's Blogs Contact

Newsletter

Stay Informed & Inspired—Subscribe for the Latest Updates, Tips, and Exclusive Content Directly to Your Inbox.

Mail Us

mail@sambuz.com

2025 SAMBUZ, All right reserved.