learning action representations for reinforcement learning
play

Learning Action Representations for Reinforcement Learning Georgios - PowerPoint PPT Presentation

Dec 11, 2023 •287 likes •611 views

Learning Action Representations for Reinforcement Learning Georgios Scott Yash James Philip Theocharous Jordan Chandak Kostas Thomas Reinforcement Learning Problem Statement Thousands of possible actions! Problem Statement Thousands of

  1. Learning Action Representations for Reinforcement Learning Georgios Scott Yash James Philip Theocharous Jordan Chandak Kostas Thomas

  2. Reinforcement Learning

  3. Problem Statement Thousands of possible actions!

  4. Problem Statement Thousands of possible actions! ● Personalized tutoring systems

  5. Problem Statement Thousands of possible actions! ● Personalized tutoring systems ● Advertisement/marketing

  6. Problem Statement Thousands of possible actions! ● Personalized tutoring systems ● Advertisement/marketing ● Medical treatment - drug prescription

  7. Problem Statement Thousands of possible actions! ● Personalized tutoring systems ● Advertisement/marketing ● Medical treatment - drug prescription ● Portfolio management

  8. Problem Statement Thousands of possible actions! ● Personalized tutoring systems ● Advertisement/marketing ● Medical treatment - drug prescription ● Portfolio management ● Video/Songs recommendation

  9. Problem Statement Thousands of possible actions! ● Personalized tutoring systems ● Advertisement/marketing ● Medical treatment - drug prescription ● Portfolio management ● Video/Songs recommendation ● … ● … ● Option selection

  10. Problem Statement Thousands of possible actions! ● Personalized tutoring systems ● Advertisement/marketing ● Medical treatment - drug prescription ● Portfolio management ● Video/Songs recommendation ● … ● … ● Option selection

  11. Key Insights - Actions are not independent discrete quantities.

  12. Key Insights - Actions are not independent discrete quantities. - There is a low dimensional structure underlying their behavior pattern.

  13. Key Insights - Actions are not independent discrete quantities. - There is a low dimensional structure underlying their behavior pattern. - This structure can be learned independent of the reward .

  14. Key Insights - Actions are not independent discrete quantities. - There is a low dimensional structure underlying their behavior pattern. - This structure can be learned independent of the reward . - Instead of raw actions, agent can act in this space of behavior and feedback can be generalized to similar actions.

  15. Proposed Method

  16. Algorithm (a) Supervised learning of action representations.

  17. Algorithm (a) Supervised learning of action representations. (b) Learning internal policy with policy gradients.

  19. Results

  20. Results

  21. Results

  22. Results

  23. Real-world Applications at Adobe Photoshop HelpX Actions = 1498 tutorials Actions = 1843 tools

  24. Poster #112 Today

  26. Results (Action representations) Actual behavior of 2 12 Maze Learned representations of 2 12 actions domain actions

  27. Policy decomposition

  28. Case 1: Action representations are known - The internal policy acts in the space of action representations - Any existing policy gradient algorithm can be used to improve its local performance, independent of the mapping function.

  29. Case 2: Learning action representations - P(a|e) required to map representation to action can be learned by satisfying the earlier assumption: - We parameterize P(a|e) and P(e|s,s’) with learnable functions f and g , respectively. - Observed transition tuples are from the required distribution. - Parameters can be learned by minimizing the stochastic KL divergence. - Procedure is independent of reward .

  30. Experiments Toy Maze: - Agent in continuous state with n actuators. 2 n actions. Exponentially large action space. - - Long horizon and single goal reward. Adobe Datasets : - N-gram based multi-time step user behavior model from passive data. - Rewards defined using a surrogate objective. - Photoshop tool recommendation ( 1843 tools) - HelpX tutorial recommendation ( 1498 tutorials)

  31. Advantages - Exploits structure in space of actions. - Quick generalization of feedback to similar actions. - Less parameters updated using high variance policy gradients. - Drop-in extension for existing policy gradient algorithms.

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B.

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B.

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B. Temporal Difference Reinforcement Learning C. PVLV Model D. Cerebellum and Error-driven Learning 2/23/18 COSC 494/594 CCN 2 Sensory-Motor Loop

790 views • 56 slides
Foundations of Machine Learning Reinforcement Learning Reinforcement Learning Agent exploring

Foundations of Machine Learning Reinforcement Learning Reinforcement Learning Agent exploring

Foundations of Machine Learning Reinforcement Learning Reinforcement Learning Agent exploring environment. Interactions with environment: action state Agent Environment reward Problem: find action policy that maximizes cumulative reward

828 views • 66 slides
SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning Marvin Zhang*,

SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning Marvin Zhang*,

SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning Marvin Zhang*, Sharad Vikram*, Laura Smith, Pieter Abbeel, Matthew J Johnson, Sergey Levine UC Berkeley, UC San Diego, Google Efficient reinforcement learning from

212 views • 20 slides
Reinforcement learning with restrictions on the action set Mario Bravo Universidad de Chile

Reinforcement learning with restrictions on the action set Mario Bravo Universidad de Chile

Reinforcement learning with restrictions on the action set Reinforcement learning with restrictions on the action set Mario Bravo Universidad de Chile Joint work with Mathieu Faure (AMSE-GREQAM) Reinforcement learning with restrictions on the

976 views • 68 slides
Reinforcement Learning Framework Reinforcement Learning Rewards, Returns Lectures 4 and 5

Reinforcement Learning Framework Reinforcement Learning Rewards, Returns Lectures 4 and 5

1 Reinforcement Learning Framework Reinforcement Learning Rewards, Returns Lectures 4 and 5 Environment Dynamics Gillian Hayes Components of a Problem 18th January 2007 Values and Action Values, V and Q Optimal Policies

109 views • 7 slides
Reinforcement Learning: How Does It Work? We detect a state Reinforcement Learning We choose an

Reinforcement Learning: How Does It Work? We detect a state Reinforcement Learning We choose an

1 Reinforcement Learning: How Does It Work? We detect a state Reinforcement Learning We choose an action Lecture 2 We get a reward Gillian Hayes Our aim is to learn a policy what action to choose in what state to get maximum reward 11th

641 views • 5 slides
RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

551 views • 35 slides
Global plan Reinforcement learning I: prediction classical conditioning

Global plan Reinforcement learning I: prediction classical conditioning

Global plan Reinforcement learning I: prediction classical conditioning dopamine Reinforcement learning II: Reinforcement learning II: dynamic programming; action selection Pavlovian misbehaviour vigor

584 views • 46 slides
Reinforcement Learning for Continuous State and Action Spaces Gradient Methods 1 MACHINE LEARNING

Reinforcement Learning for Continuous State and Action Spaces Gradient Methods 1 MACHINE LEARNING

MACHINE LEARNING 2012 MACHINE LEARNING TECHNIQUES AND APPLICATIONS Reinforcement Learning for Continuous State and Action Spaces Gradient Methods 1 MACHINE LEARNING 2012 Reinforcement Learning (RL) Supervised Learning Unsupervised

777 views • 51 slides
Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent

Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent state reward action S t R t A t R t+ 1 Environment S t+ 1

1.07k views • 52 slides
1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement learning? Agent/Actor + Action + Environment + State + Reward How does reinforcement learning work?

792 views • 31 slides
Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine playing a new game whose rules you dont know; after a hundred or so moves your don t know; after a hundred or so moves, your opponent announces, You

511 views • 30 slides
A Bayesian Approach to Finding Compact Representations for Reinforcement Learning Special thanks

A Bayesian Approach to Finding Compact Representations for Reinforcement Learning Special thanks

A Bayesian Approach to Finding Compact Representations for Reinforcement Learning Special thanks to Joelle Pineau for presenting our paper - July, 2012 1 Authors Alborz Geramifard Stefanie Tellex David Wingate Nicholas Roy Jonathan How 2

491 views • 25 slides
Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers (the class labels) Reinforcement learning: In reinforcement learning, the agent learns Markov Decision Processes based on a punishment/reward

89 views • 6 slides
What is Action Learning? What is an ALC? 1 What is Action Learning? Learning by doing

What is Action Learning? What is an ALC? 1 What is Action Learning? Learning by doing

What is Action Learning? What is an ALC? 1 What is Action Learning? Learning by doing Process for bringing together a group of people with varied levels of skills and experience (but with influence, passion, etc. to impact an

191 views • 5 slides
Examples of Reinforcement Learning Robocup Soccer Teams Stone & Veloso, Reidmiller et al.

Examples of Reinforcement Learning Robocup Soccer Teams Stone & Veloso, Reidmiller et al.

Reinforcement Learning Slides by Rich Sutton Mods by Dan Lizotte Refer to Reinforcement Learning: An Introduction by Sutton and Barto Alpaydin Chapter 16 Up until now we have been Supervised Learning Classifying, mostly

891 views • 31 slides
Question How is HTE information typically presented in prescription drug labeling, and are there

Question How is HTE information typically presented in prescription drug labeling, and are there

Question How is HTE information typically presented in prescription drug labeling, and are there opportunities available for enhancement? Iris Masucci, PharmD Special Assistant for Labeling, Office of Medical Policy, CDER, FDA Prescription

88 views • 7 slides
PRESCRIPTION CHANGES DURING GERIATRIC CARE EPISODES A trend analysis Marianne Reimers

PRESCRIPTION CHANGES DURING GERIATRIC CARE EPISODES A trend analysis Marianne Reimers

PRESCRIPTION CHANGES DURING GERIATRIC CARE EPISODES A trend analysis Marianne Reimers Karolinska Institutet 2005, 2010, 2015 Prescription changes during care episodes and over time Factors potentially contributing to such changes

341 views • 19 slides
Threshold resummation for Drell-Yan production: theory and phenomenology Marco Bonvini

Threshold resummation for Drell-Yan production: theory and phenomenology Marco Bonvini

Threshold resummation for Drell-Yan production: theory and phenomenology Marco Bonvini Dipartimento di Fisica, Universit` a di Genova & INFN, sezione di Genova HP 2 .3rd GGI, Firenze, September 1417, 2010 In collaboration with:

1.3k views • 112 slides
Spending Challenges for Prescription Drugs: Part B and Part D Reform Stacie B. Dusetzina, PhD

Spending Challenges for Prescription Drugs: Part B and Part D Reform Stacie B. Dusetzina, PhD

Spending Challenges for Prescription Drugs: Part B and Part D Reform Stacie B. Dusetzina, PhD Associate Professor of Health Policy Ingram Associate Professor of Cancer Research Prescription Drugs and Spending Trends Recent Proposals for

351 views • 8 slides
The Craft of XML Text Encoding in historical and humanistic context Wendell Piez JADH 2015

The Craft of XML Text Encoding in historical and humanistic context Wendell Piez JADH 2015

The Craft of XML Text Encoding in historical and humanistic context Wendell Piez JADH 2015 University of Kyoto Kyoto, Japan September 2 2015 An Ukiyo-e woodblock print depicting a woodblock printing shop. (Is the scene realistic, or

500 views • 14 slides
Graph-Processing Systems (focusing on GraphChi) Recall: PageRank in MapReduce (Hadoop) (a,[c])

Graph-Processing Systems (focusing on GraphChi) Recall: PageRank in MapReduce (Hadoop) (a,[c])

Graph-Processing Systems (focusing on GraphChi) Recall: PageRank in MapReduce (Hadoop) (a,[c]) (c,PR(a) / out (a)), (a,[c]) ((a,PR(a)/out(a)) PR(a) = 1-l/N + l* sum(PR(y)/out(y)) Input: H (a,PR(b) / out (b)), adjacency D (b,[a])

389 views • 21 slides
Graph Ordering Lecture 16 CSCI 4974/6971 27 Oct 2016 1 / 12 Todays Biz 1. Reminders 2.

Graph Ordering Lecture 16 CSCI 4974/6971 27 Oct 2016 1 / 12 Todays Biz 1. Reminders 2.

Graph Ordering Lecture 16 CSCI 4974/6971 27 Oct 2016 1 / 12 Todays Biz 1. Reminders 2. Review 3. Distributed Graph Processing 2 / 12 Reminders Project Update Presentation: In class November 3rd Assignment 4: due date TBD (early

326 views • 20 slides
Coresets Meet EDCS: Algorithms for Matching and Vertex Cover on Massive Graphs Sepehr Assadi

Coresets Meet EDCS: Algorithms for Matching and Vertex Cover on Massive Graphs Sepehr Assadi

Coresets Meet EDCS: Algorithms for Matching and Vertex Cover on Massive Graphs Sepehr Assadi University of Pennsylvania Joint work with MohammadHossein Bateni (Google), Aaron Bernstein (Rutgers), Vahab Mirrokni (Google), and Cliff Stein

1.11k views • 93 slides

More recommend