approximate q learning update
play

Approximate Q-Learning Update Initialize weight for each feature to - PowerPoint PPT Presentation

Dec 23, 2022 •130 likes •249 views

<latexit

  1. <latexit sha1_base64="cCR4OZ9l/OETB0Xei2CIfbgOzM=">ACDHicbVDLSsNAFJ3UV62vqEs3g1WoICURQV0Uim5ctmBsoY1hMp20QyeTMDNRSsgPuPFX3LhQcesHuPNvnLZaOuBC4dz7uXe/yYUaks69soLCwuLa8UV0tr6xubW+b2zq2MEoGJgyMWibaPJGUE0dRxUg7FgSFPiMtf3g19lv3REga8Rs1iokboj6nAcVIackzD5oVeYyOYA12ZRJ6Ka3Z2V3KMxh4dOo8eNQzy1bVmgDOEzsnZCj4Zlf3V6Ek5BwhRmSsmNbsXJTJBTFjGSlbiJjPAQ9UlHU45CIt108k0GD7XSg0EkdHEFJ+rviRSFUo5CX3eGSA3krDcW/M6iQrO3ZTyOFGE4+miIGFQRXAcDexRQbBiI0QFlTfCvEACYSVDrCkQ7BnX54nzkn1omo3T8v1yzyNItgD+6ACbHAG6uAaNIADMHgEz+AVvBlPxovxbnxMWwtGPrML/sD4/AG0/pmn</latexit> <latexit sha1_base64="cCR4OZ9l/OETB0Xei2CIfbgOzM=">ACDHicbVDLSsNAFJ3UV62vqEs3g1WoICURQV0Uim5ctmBsoY1hMp20QyeTMDNRSsgPuPFX3LhQcesHuPNvnLZaOuBC4dz7uXe/yYUaks69soLCwuLa8UV0tr6xubW+b2zq2MEoGJgyMWibaPJGUE0dRxUg7FgSFPiMtf3g19lv3REga8Rs1iokboj6nAcVIackzD5oVeYyOYA12ZRJ6Ka3Z2V3KMxh4dOo8eNQzy1bVmgDOEzsnZCj4Zlf3V6Ek5BwhRmSsmNbsXJTJBTFjGSlbiJjPAQ9UlHU45CIt108k0GD7XSg0EkdHEFJ+rviRSFUo5CX3eGSA3krDcW/M6iQrO3ZTyOFGE4+miIGFQRXAcDexRQbBiI0QFlTfCvEACYSVDrCkQ7BnX54nzkn1omo3T8v1yzyNItgD+6ACbHAG6uAaNIADMHgEz+AVvBlPxovxbnxMWwtGPrML/sD4/AG0/pmn</latexit> <latexit sha1_base64="cCR4OZ9l/OETB0Xei2CIfbgOzM=">ACDHicbVDLSsNAFJ3UV62vqEs3g1WoICURQV0Uim5ctmBsoY1hMp20QyeTMDNRSsgPuPFX3LhQcesHuPNvnLZaOuBC4dz7uXe/yYUaks69soLCwuLa8UV0tr6xubW+b2zq2MEoGJgyMWibaPJGUE0dRxUg7FgSFPiMtf3g19lv3REga8Rs1iokboj6nAcVIackzD5oVeYyOYA12ZRJ6Ka3Z2V3KMxh4dOo8eNQzy1bVmgDOEzsnZCj4Zlf3V6Ek5BwhRmSsmNbsXJTJBTFjGSlbiJjPAQ9UlHU45CIt108k0GD7XSg0EkdHEFJ+rviRSFUo5CX3eGSA3krDcW/M6iQrO3ZTyOFGE4+miIGFQRXAcDexRQbBiI0QFlTfCvEACYSVDrCkQ7BnX54nzkn1omo3T8v1yzyNItgD+6ACbHAG6uAaNIADMHgEz+AVvBlPxovxbnxMWwtGPrML/sD4/AG0/pmn</latexit> <latexit sha1_base64="cCR4OZ9l/OETB0Xei2CIfbgOzM=">ACDHicbVDLSsNAFJ3UV62vqEs3g1WoICURQV0Uim5ctmBsoY1hMp20QyeTMDNRSsgPuPFX3LhQcesHuPNvnLZaOuBC4dz7uXe/yYUaks69soLCwuLa8UV0tr6xubW+b2zq2MEoGJgyMWibaPJGUE0dRxUg7FgSFPiMtf3g19lv3REga8Rs1iokboj6nAcVIackzD5oVeYyOYA12ZRJ6Ka3Z2V3KMxh4dOo8eNQzy1bVmgDOEzsnZCj4Zlf3V6Ek5BwhRmSsmNbsXJTJBTFjGSlbiJjPAQ9UlHU45CIt108k0GD7XSg0EkdHEFJ+rviRSFUo5CX3eGSA3krDcW/M6iQrO3ZTyOFGE4+miIGFQRXAcDexRQbBiI0QFlTfCvEACYSVDrCkQ7BnX54nzkn1omo3T8v1yzyNItgD+6ACbHAG6uAaNIADMHgEz+AVvBlPxovxbnxMWwtGPrML/sD4/AG0/pmn</latexit> Approximate Q-Learning Update Initialize weight for each feature to 0. Every time we take an action, perform this update: The Q-value estimate for (s,a) is the weighted sum of its features: n X Q ( s, a ) = f i ( s, a ) w i i =1

  2. <latexit sha1_base64="DA+nU6WuIiNWRNEhvmCSzynrSI=">ACI3icbZDNSsNAFIUn9a/Wv6pLN4NFqCAlEUFCkU3LluwtDUMJlO2qGTSZiZtJSQh3Hjq7hxocWNC9/FSdqFtl4YOJzvXu7c4aMSmWaX0ZuZXVtfSO/Wdja3tndK+4fPMogEpg0cAC0XaRJIxy0lRUMdIOBUG+y0jLHd6lvDUiQtKAP6hJSLo+6nPqUYyUtpziTaMsz9AprEJbRr4T06qVPMU8gZ5DZ2Ts0JSOCI7HCbRxL1DQy5BTLJkVMyu4LKy5KIF51Z3i1O4FOPIJV5ghKTuWGapujISimJGkYEeShAgPUZ90tOTIJ7IbZ0cm8EQ7PegFQj+uYOb+noiRL+XEd3Wnj9RALrLU/I91IuVdWPKw0gRjmeLvIhBFcA0MdijgmDFJlogLKj+K8QDJBWOteCDsFaPHlZNM8r1xWrcVGq3c7TyIMjcAzKwAKXoAbuQR0AQbP4BW8gw/jxXgzpsbnrDVnzGcOwZ8yvn8AzemiEA=</latexit> <latexit sha1_base64="DA+nU6WuIiNWRNEhvmCSzynrSI=">ACI3icbZDNSsNAFIUn9a/Wv6pLN4NFqCAlEUFCkU3LluwtDUMJlO2qGTSZiZtJSQh3Hjq7hxocWNC9/FSdqFtl4YOJzvXu7c4aMSmWaX0ZuZXVtfSO/Wdja3tndK+4fPMogEpg0cAC0XaRJIxy0lRUMdIOBUG+y0jLHd6lvDUiQtKAP6hJSLo+6nPqUYyUtpziTaMsz9AprEJbRr4T06qVPMU8gZ5DZ2Ts0JSOCI7HCbRxL1DQy5BTLJkVMyu4LKy5KIF51Z3i1O4FOPIJV5ghKTuWGapujISimJGkYEeShAgPUZ90tOTIJ7IbZ0cm8EQ7PegFQj+uYOb+noiRL+XEd3Wnj9RALrLU/I91IuVdWPKw0gRjmeLvIhBFcA0MdijgmDFJlogLKj+K8QDJBWOteCDsFaPHlZNM8r1xWrcVGq3c7TyIMjcAzKwAKXoAbuQR0AQbP4BW8gw/jxXgzpsbnrDVnzGcOwZ8yvn8AzemiEA=</latexit> <latexit sha1_base64="DA+nU6WuIiNWRNEhvmCSzynrSI=">ACI3icbZDNSsNAFIUn9a/Wv6pLN4NFqCAlEUFCkU3LluwtDUMJlO2qGTSZiZtJSQh3Hjq7hxocWNC9/FSdqFtl4YOJzvXu7c4aMSmWaX0ZuZXVtfSO/Wdja3tndK+4fPMogEpg0cAC0XaRJIxy0lRUMdIOBUG+y0jLHd6lvDUiQtKAP6hJSLo+6nPqUYyUtpziTaMsz9AprEJbRr4T06qVPMU8gZ5DZ2Ts0JSOCI7HCbRxL1DQy5BTLJkVMyu4LKy5KIF51Z3i1O4FOPIJV5ghKTuWGapujISimJGkYEeShAgPUZ90tOTIJ7IbZ0cm8EQ7PegFQj+uYOb+noiRL+XEd3Wnj9RALrLU/I91IuVdWPKw0gRjmeLvIhBFcA0MdijgmDFJlogLKj+K8QDJBWOteCDsFaPHlZNM8r1xWrcVGq3c7TyIMjcAzKwAKXoAbuQR0AQbP4BW8gw/jxXgzpsbnrDVnzGcOwZ8yvn8AzemiEA=</latexit> <latexit sha1_base64="DA+nU6WuIiNWRNEhvmCSzynrSI=">ACI3icbZDNSsNAFIUn9a/Wv6pLN4NFqCAlEUFCkU3LluwtDUMJlO2qGTSZiZtJSQh3Hjq7hxocWNC9/FSdqFtl4YOJzvXu7c4aMSmWaX0ZuZXVtfSO/Wdja3tndK+4fPMogEpg0cAC0XaRJIxy0lRUMdIOBUG+y0jLHd6lvDUiQtKAP6hJSLo+6nPqUYyUtpziTaMsz9AprEJbRr4T06qVPMU8gZ5DZ2Ts0JSOCI7HCbRxL1DQy5BTLJkVMyu4LKy5KIF51Z3i1O4FOPIJV5ghKTuWGapujISimJGkYEeShAgPUZ90tOTIJ7IbZ0cm8EQ7PegFQj+uYOb+noiRL+XEd3Wnj9RALrLU/I91IuVdWPKw0gRjmeLvIhBFcA0MdijgmDFJlogLKj+K8QDJBWOteCDsFaPHlZNM8r1xWrcVGq3c7TyIMjcAzKwAKXoAbuQR0AQbP4BW8gw/jxXgzpsbnrDVnzGcOwZ8yvn8AzemiEA=</latexit> AQL Update Details • The weighted sum of features is equivalent to a dot product between the feature and weight vectors: n X Q ( s, a ) = f i ( s, a ) w i = ~ w · f ( s, a ) i =1 • The correction term is the same for all features. • The correction to each feature is weighted by how “active” that feature was.

  3. Exercise: Feature Q-Update Reward eating food: discount: .95 +10 learning rate: .3 Reward for losing: -500 • Suppose PacMan is considering the up action. Old Feature Values: w bias = 1 w ghosts = -20 w food = 2 Features: w eats = 4 • closest-food • bias • eats-food • #-of-ghosts-1-step-away

  4. Notes on Approximate Q-Learning • Learns weights for a tiny number of features. • Every feature’s value is update every step. • No longer tracking values for individual (s,a) pairs. • (s,a) value estimates are calculated from features. • The weight update is a form of gradient descent. • We’ve seen this before. • We’re performing a variant of linear regression. • Feature extraction is a type of basis change. • We’ll see these again.

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Approximate Q-Learning 3/23/18 On-Policy Learning (SARSA) Instead of updating based on the best

Approximate Q-Learning 3/23/18 On-Policy Learning (SARSA) Instead of updating based on the best

Approximate Q-Learning 3/23/18 On-Policy Learning (SARSA) Instead of updating based on the best action from the next state, update based on the action your current policy actually takes from the next state. SARSA update: When would this be

572 views • 19 slides
Sensors Approximate computing approximate edge detection Machine learning hidden

Sensors Approximate computing approximate edge detection Machine learning hidden

Uncertain&lt; T &gt; A First-Order Type for Uncertain Data James Bornholt Australian National University Todd Mytkowicz Microsoft Research Kathryn S. McKinley Microsoft Research Sensors Approximate computing

1.19k views • 97 slides
Approximate Q-Learning 2/24/17 State Value V(s) vs. Action Value Q(s,a) &quot; # X t r t

Approximate Q-Learning 2/24/17 State Value V(s) vs. Action Value Q(s,a) &quot; # X t r t

Approximate Q-Learning 2/24/17 State Value V(s) vs. Action Value Q(s,a) &quot; # X t r t V = E t =0 Either way, value is the sum of future discounted rewards (assuming the agent behaves optimally): V(s) : after being in state s

404 views • 17 slides
Approximate Inference Part 1 of 2 Tom Minka Microsoft Research, Cambridge, UK Machine Learning

Approximate Inference Part 1 of 2 Tom Minka Microsoft Research, Cambridge, UK Machine Learning

Approximate Inference Part 1 of 2 Tom Minka Microsoft Research, Cambridge, UK Machine Learning Summer School 2009 http://mlg.eng.cam.ac.uk/mlss09/ 1 Bayesian paradigm Consistent use of probability theory for representing unknowns

1.05k views • 81 slides
Time Series Forecasting With a Learning Algorithm: An Approximate Dynamic Programming Approach

Time Series Forecasting With a Learning Algorithm: An Approximate Dynamic Programming Approach

Time Series Forecasting With a Learning Algorithm: An Approximate Dynamic Programming Approach Ricardo Collado 1 an Creamer 1 Germ 1 School of Business Stevens Institute of Technology Hoboken, New Jersey R. Collado (Stevens) Learning Time

891 views • 71 slides
Reinforcement Learning: Approximate Dynamic Programming Decision Making Under Uncertainty, Chapter

Reinforcement Learning: Approximate Dynamic Programming Decision Making Under Uncertainty, Chapter

Reinforcement Learning: Approximate Dynamic Programming Decision Making Under Uncertainty, Chapter 10 Christos Dimitrakakis Chalmers November 21, 2013 Christos Dimitrakakis (Chalmers) Reinforcement Learning: Approximate Dynamic Programming

587 views • 22 slides
Probabilistic &amp; Unsupervised Learning Convex Algorithms in Approximate Inference Maneesh

Probabilistic &amp; Unsupervised Learning Convex Algorithms in Approximate Inference Maneesh

Probabilistic &amp; Unsupervised Learning Convex Algorithms in Approximate Inference Maneesh Sahani maneesh@gatsby.ucl.ac.uk Gatsby Computational Neuroscience Unit, and MSc ML/CSML, Dept Computer Science University College London Term 1,

861 views • 41 slides
Backward Analysis via Over-Approximate Abstraction and Under-Approximate Subtraction Alexey

Backward Analysis via Over-Approximate Abstraction and Under-Approximate Subtraction Alexey

Backward Analysis via Over-Approximate Abstraction and Under-Approximate Subtraction Alexey Bakhirkin 1 Josh Berdine 2 Nir Piterman 1 1 University of Leicester, Department of Computer Science 2 Microsoft Research Goal A backwards analysis

733 views • 23 slides
Approximate Kernel Methods and Learning on Aggregates Dino Sejdinovic joint work with Leon Law,

Approximate Kernel Methods and Learning on Aggregates Dino Sejdinovic joint work with Leon Law,

Approximate Kernel Methods and Learning on Aggregates Dino Sejdinovic joint work with Leon Law, Seth Flaxman, Dougal Sutherland, Kenji Fukumizu, Ewan Cameron, Tim Lucas, Katherine Battle (and many others) Department of Statistics University of

580 views • 34 slides
Multiple-Step Greedy Policies in Online and Approximate Reinforcement Learning Neural Information

Multiple-Step Greedy Policies in Online and Approximate Reinforcement Learning Neural Information

... Multiple-Step Greedy Policies in Online and Approximate Reinforcement Learning Neural Information Processing Systems, December 18 Yonathan Efroni 1 Gal Dalal 1 Bruno Scherrer 2 Shie Mannor 1 1 Department of Electrical Engineering,

716 views • 36 slides
Approximate Q-Learning 3-25-16 Exploration policy vs. optimal policy Where do the exploration

Approximate Q-Learning 3-25-16 Exploration policy vs. optimal policy Where do the exploration

Approximate Q-Learning 3-25-16 Exploration policy vs. optimal policy Where do the exploration traces come from? We need some policy for acting in the environment before we understand it. Wed like to get decent rewards while

264 views • 14 slides
Approximate MPC based on machine learning and probabilistic verification Sergio Lucia Technische

Approximate MPC based on machine learning and probabilistic verification Sergio Lucia Technische

Approximate MPC based on machine learning and probabilistic verification Sergio Lucia Technische Universitt Berlin Einstein Center Digital Future www.iot.tu-berlin.de Motivation Solving NMPC problems in real time is still challenging:

1.03k views • 36 slides
Partially Exchangeable Networks and Architectures for Learning Summary Statistics in Approximate

Partially Exchangeable Networks and Architectures for Learning Summary Statistics in Approximate

Partially Exchangeable Networks and Architectures for Learning Summary Statistics in Approximate Bayesian Computation ICML 2019 Samuel Wiqvist Centre for Mathematical Sciences, Lund University, Sweden samuel wiqvist June 13, 2019 Joint

685 views • 24 slides
Cycle-Consistent Adversarial Learning as Approximate Bayesian Inference Louis C. Tiao 1 Edwin V.

Cycle-Consistent Adversarial Learning as Approximate Bayesian Inference Louis C. Tiao 1 Edwin V.

Cycle-Consistent Adversarial Learning as Approximate Bayesian Inference Louis C. Tiao 1 Edwin V. Bonilla 2 Fabio Ramos 1 July 22, 2018 1 University of Sydney, 2 University of New South Wales Motivation: Unpaired Image-to-Image Translation

329 views • 22 slides
Approximate Homomorphic Encryption and Privacy Preserving Machine Learning Jung Hee Cheon (SNU,

Approximate Homomorphic Encryption and Privacy Preserving Machine Learning Jung Hee Cheon (SNU,

Approximate Homomorphic Encryption and Privacy Preserving Machine Learning Jung Hee Cheon (SNU, CryptoLab) Thanks to YongSoo Song, Kiwoo Lee, Andrey KIM Outline 1. Homomorphic Encryption 2. HEAAN 3. Bootstrapping of HEAAN 4. Toolkit for

981 views • 50 slides
Learning in extended and approximate Rational Speech Acts models Christopher Potts Stanford

Learning in extended and approximate Rational Speech Acts models Christopher Potts Stanford

A Gricean ideal Implicatures RSA Learned RSA Neural RSA Language and action Prospects Learning in extended and approximate Rational Speech Acts models Christopher Potts Stanford Linguistics EMNLP 2016 Will Monroe 1 / 56 A Gricean ideal

2.04k views • 162 slides
Beyond X-ray timing I: Using reprocessing to map the geometry of X-ray binaries Teo

Beyond X-ray timing I: Using reprocessing to map the geometry of X-ray binaries Teo

Beyond X-ray timing I: Using reprocessing to map the geometry of X-ray binaries Teo Muoz-Darias INAF-OSSERVATORIO ASTRONOMICO DI BRERA (OAB) University of Southampton, Black hole variability 2010 Monday, June 7, 2010 OUTLINE 1. Why could

630 views • 44 slides
Observations of jets in X-ray binaries Tom Maccarone (University of Southampton) ING

Observations of jets in X-ray binaries Tom Maccarone (University of Southampton) ING

Observations of jets in X-ray binaries Tom Maccarone (University of Southampton) ING archive/Nick Szymanek Russell et al. 2007 Stirling et al. 2001 Black Hole X-ray binaries: key sources for understanding accretion-ejection phenomenology

347 views • 21 slides
Measuring Geometrical Distances to Cepheids with II Jeter Hall Fermilab Center for Particle

Measuring Geometrical Distances to Cepheids with II Jeter Hall Fermilab Center for Particle

Measuring Geometrical Distances to Cepheids with II Jeter Hall Fermilab Center for Particle Astrophysics Workshop on Stellar Intensity January 29-30, 2009 Interferometry in Salt Lake City Cosmic Distance Distance Ladder Redshift Variable

251 views • 9 slides
X-ray and gamma-ray binaries with the ANTARES telescope A. Snchez-Losa D. Dornic A. Colerio

X-ray and gamma-ray binaries with the ANTARES telescope A. Snchez-Losa D. Dornic A. Colerio

Time-dependent search of neutrino emission from X-ray and gamma-ray binaries with the ANTARES telescope A. Snchez-Losa D. Dornic A. Colerio (IFIC) (INFN Sezione di Bari) (CPPM) agustin.sanchez@infn.ba.it dornic@cppm.in2p3.fr

588 views • 13 slides
Constraints on Compact Star Radii and the Equation of State From Gravitational Waves, Pulsars and

Constraints on Compact Star Radii and the Equation of State From Gravitational Waves, Pulsars and

Constraints on Compact Star Radii and the Equation of State From Gravitational Waves, Pulsars and Supernovae J. M. Lattimer Department of Physics &amp; Astronomy Stony Brook University September 13, 2016 Collaborators: E. Brown (MSU), C.

744 views • 48 slides
Impolitely Requesting Awareness ling.upenn.edu/~cahern/ Christopher Ahern and Jason Quinley

Impolitely Requesting Awareness ling.upenn.edu/~cahern/ Christopher Ahern and Jason Quinley

Introduction Politeness Signaling Awareness Conclusions Impolitely Requesting Awareness ling.upenn.edu/~cahern/ Christopher Ahern and Jason Quinley University of Pennsylvania and University of T ubingen July 26, 2013 Ahern (UPENN)

670 views • 30 slides
Planning and Optimization E1. Critical Path Heuristics: h m Malte Helmert and Gabriele R oger

Planning and Optimization E1. Critical Path Heuristics: h m Malte Helmert and Gabriele R oger

Planning and Optimization E1. Critical Path Heuristics: h m Malte Helmert and Gabriele R oger Universit at Basel November 20, 2017 Set Representation Perfect Regression Heuristic Critical Path Heuristics Computation Summary Content of

834 views • 38 slides
Efficacy of Spiritual Integration Past research has demonstrated that religiously congruent

Efficacy of Spiritual Integration Past research has demonstrated that religiously congruent

10/29/2020 Traditional Islamically Integrated Psychotherapy (TIIP) Hooman Keshavarzi, Psy.D Khalil Center Efficacy of Spiritual Integration Past research has demonstrated that religiously congruent therapists have a significant positive

515 views • 27 slides

More recommend