Towards TempoRL: Learning When to Act Andr Biedenkapp, Raghu Rajan, - PowerPoint PPT Presentation

Oct 02, 2023 •110 likes •277 views

Towards TempoRL: Learning When to Act Andr Biedenkapp, Raghu Rajan, Frank Hutter & Marius Lindauer Biedenkapp, Rajan, Hutter and Lindauer Towards TempoRL BIG@ICML 2020 In a Nutshell 1. We propose a proactive way of doing RL 2. We

Towards TempoRL: Learning When to Act André Biedenkapp, Raghu Rajan, Frank Hutter & Marius Lindauer Biedenkapp, Rajan, Hutter and Lindauer Towards TempoRL BIG@ICML 2020
In a Nutshell 1. We propose a proactive way of doing RL 2. We introduce skip-connections into MDPs ○ through action repetition ○ allows for faster propagation of rewards 3. We propose a novel algorithm using skip-connections ○ learn what action to take & when to make new decisions ○ condition when on what 4. We evaluate our approach with tabular Q-learning on small grid worlds Biedenkapp, Rajan, Hutter and Lindauer Towards TempoRL BIG@ICML 2020
Motivation r = 0 r = 1 Biedenkapp, Rajan, Hutter and Lindauer Towards TempoRL BIG@ICML 2020
Motivation r = 0 r = -1 Biedenkapp, Rajan, Hutter and Lindauer Towards TempoRL BIG@ICML 2020
Optimal Policies Biedenkapp, Rajan, Hutter and Lindauer Towards TempoRL BIG@ICML 2020
Optimal Policies: When do we need to act? # Steps: 16 # Decisions: 16 Biedenkapp, Rajan, Hutter and Lindauer Towards TempoRL BIG@ICML 2020
Optimal Policies: When do we need to act? # Steps: 16 # Decisions: 5 Biedenkapp, Rajan, Hutter and Lindauer Towards TempoRL BIG@ICML 2020
Optimal Policies: When do we need to act? # Steps: 16 # Decisions: 4 Biedenkapp, Rajan, Hutter and Lindauer Towards TempoRL BIG@ICML 2020
Optimal Policies: When do we need to act? # Steps: 16 # Decisions: 3 Biedenkapp, Rajan, Hutter and Lindauer Towards TempoRL BIG@ICML 2020
Proactive Decision Making # Steps: 16 # Steps: 16 # Decisions: 16 # Decisions: 3 ~80% fewer Decision points Biedenkapp, Rajan, Hutter and Lindauer Towards TempoRL BIG@ICML 2020
Skip MDPs Biedenkapp, Rajan, Hutter and Lindauer Towards TempoRL BIG@ICML 2020
Flat Hierarchy 1. Use standard Q-learning to determine the behaviour 2. Condition skips on the chosen action. 3. Play action for the next steps The action Q-function The skip Q-function can be learned using n-step updates Biedenkapp, Rajan, Hutter and Lindauer Towards TempoRL BIG@ICML 2020
Experimental Evaluation Biedenkapp, Rajan, Hutter and Lindauer Towards TempoRL BIG@ICML 2020
Experimental Evaluation Biedenkapp, Rajan, Hutter and Lindauer Towards TempoRL BIG@ICML 2020
Wrap-Up Code & Data available: https://github.com/automl/TabularTempoRL Future work: - Use deep function approximation - Different exploration mechanisms for skip and behaviour policies Biedenkapp, Rajan, Hutter and Lindauer Towards TempoRL BIG@ICML 2020

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Learning is a never-ending process Tasks come and go, but learning is forever Learn more e ff

NeurIPS 2018 Tutorial on Automatic Machine Learning Learning to Learn Learning Learning Learning Learning Learning automl.org/events -> AutoML Tutorial -> Slides Frank Hutter Joaquin Vanschoren Eindhoven University of Technology

1.36k views • 40 slides

What is Action Learning? What is an ALC? 1 What is Action Learning? Learning by doing

What is Action Learning? What is an ALC? 1 What is Action Learning? Learning by doing Process for bringing together a group of people with varied levels of skills and experience (but with influence, passion, etc. to impact an

191 views • 5 slides

Learning From Data Lecture 2 The Perceptron The Learning Setup A Simple Learning Algorithm: PLA

Learning From Data Lecture 2 The Perceptron The Learning Setup A Simple Learning Algorithm: PLA Other Views of Learning Is Learning Feasible: A Puzzle M. Magdon-Ismail CSCI 4100/6100 recap: The Plan 1. What is Learning? 2. Can We do it?

613 views • 25 slides

4. Learning in MAS 4.1. What is learning? One definition: Bower, Hilgard: Theories of learning,

4. Learning in MAS 4.1. What is learning? One definition: Bower, Hilgard: Theories of learning, Prentice-Hall, 1975: Learning refers to the change in a subjects behavior to a given situation brought about by his repeated experiences in

173 views • 15 slides

What are learning outcomes? Learning Outcomes Learning outcomes are statements that specify what

What are learning outcomes? Learning Outcomes Learning outcomes are statements that specify what learners will know or be able to Everybody Wins do as a result of a learning FYE 2007 activity. Dr. Carolyn Hopper Middle Tennessee State

344 views • 11 slides

Machine Learning 11 AI Slides (6e) c Lin Zuoquan@PKU 1998-2020 11 1 11 Machine Learning

Machine Learning 11 AI Slides (6e) c Lin Zuoquan@PKU 1998-2020 11 1 11 Machine Learning 11.1 Learning agents 11.2 Inductive learning 11.3 Deep learning 11.4 Statistical learning 11.5 Reinforcement learning 11.6 Transfer learning

1.7k views • 159 slides

04 FOCUS ON LEARNING STUDENTS REFLECTING ON THEIR LEARNING In this unit we are learning to

workshop 04 FOCUS ON LEARNING STUDENTS REFLECTING ON THEIR LEARNING In this unit we are learning to . discuss the role of reflection as an important part of learning identify strategies that will help students to reflect on their

539 views • 14 slides

The JOY of LEARNING Think of a topic you are passionate about Think about your learning

The JOY of LEARNING Think of a topic you are passionate about Think about your learning process How much time was spent? How did you practice? To experience research-based learning tools: Focus/Diffuse modes of thinking

1.03k views • 18 slides

Complex learning example: curve fitting t = sin(2 x ) + noise t n t 1 y ( x n , w ) 0 1

Artificial Intelligence: Representation and Problem Solving 15-381 April 10, 2007 Introduction to Learning & Decision Trees Learning and Decision Trees to learning aka: regression What is learning? pattern recognition

462 views • 17 slides

Further thoughts How does e-Learning relate to instruction? How does eLearning change

Ideas from group What is the difference between learning and training? What is the difference between learning and education? Where does learning take place? Who learns? What is the role of instruction in learning? Further

800 views • 47 slides

Learning Sciences: Impact on Learning Technologies & Learning Activities Phillip D. Long,

Learning Sciences: Impact on Learning Technologies & Learning Activities Phillip D. Long, Ph.D. 12-May, 2015 Narrative Arc ( 5 2) Conclusion: What do we want? Learning learning |lrniNG | noun the acquisition of knowledge or

653 views • 36 slides

The Learning Tree Workshop: The Learning Tree Workshop: Regulating Sensation Series on Learning

The Learning Tree Workshop: The Learning Tree Workshop: Regulating Sensation Series on Learning Differences, Learning Challenges, and Learning Strengths Transcript by Stanley I. Greenspan M.D. Narration by Brian MacDonald Sensory Modulation

453 views • 16 slides

Foundations of AI Why learning works 1 6 . Statistical Machine Learning Bayesian Learning and

Contents Statistical learning Foundations of AI Why learning works 1 6 . Statistical Machine Learning Bayesian Learning and Why Learning Works W olfram Burgard, Bernhard Nebel, and Andreas Karw ath 10/ 1 10/ 2 Statistical Learning

305 views • 5 slides

Welcome to Welcome to The Learning Tree Workshop Series on Learning Differences, Learning

Welcome to Welcome to The Learning Tree Workshop Series on Learning Differences, Learning Challenges, and Learning Strengths Strengths As a companion to Dr. Greenspans The Learning Tree : g Overcoming Learning Disabilities From the

367 views • 16 slides

Active Learning Passive Learning Active Learning 1. Think 1. Acquisition of knowledge Ability

Active Learning Passive Learning Active Learning 1. Think 1. Acquisition of knowledge Ability 2. Decision making 2. Application of knowledge 3. Explain/communicate 1. Deep learning Approach to Uni-directional 2. Interactive Learning

282 views • 5 slides

Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning

GCT634/AI613: Musical Applications of Machine Learning (Fall 2020) Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning Pipeline in Classification Tasks A set of hand-designed audio features are selected

398 views • 26 slides

What Does the CARES Act Mean for my Small Business? Answering your Questions with Congressman

What Does the CARES Act Mean for my Small Business? Answering your Questions with Congressman Paul Mitchell What is the CARES Act? The Coronavirus Aid, Relief, and Economic Security (CARES) Act is the third Coronavirus relief package

437 views • 16 slides

Agenda Wha hat t is A is ACT District CT District Testing esting? Why d hy do A o

ACT CT, the , the Compa Company ny The he ACT CT Colle College ge and and Car Career eer Read eadiness iness Benchma Ben hmarks Agenda Wha hat t is A is ACT District CT District Testing esting? Why d hy do

422 views • 27 slides

Mix Mixin ing P g Patterns i in S Social N l Networks Leto Peel Universit catholique de

Mix Mixin ing P g Patterns i in S Social N l Networks Leto Peel Universit catholique de Louvain @PiratePeel Bird irds s of of a a feat feather. her... fl floc ock tog oget ether her Visi Visibi bilit ity y an and ran

1.04k views • 48 slides

Table of Contents Convolutional Neural Nets (CNNs) 1 Deep Q Learning 2 Lecture 6: CNNs and Deep

Lecture 6: CNNs and Deep Q Learning 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 2 With many slides for DQN from David Silver and Ruslan Salakhutdinov and some vision slides from Gianni Di Caro and images from Stanford CS231n,

850 views • 67 slides

Barry Whaley, MS Project Director Southeast ADA Center Burton Blatt Institute Syracuse

Barry Whaley, MS Project Director Southeast ADA Center Burton Blatt Institute Syracuse University College of Law Syracuse, NY The Americans with Disabilities Act: Disclosure and Reasonable Accommodations in Employment September 25, 2020 The

542 views • 15 slides

The CARES Act What businesses need to know 1 The CARES Act Panelist Click to edit Master title

Click to edit Master title style B U S I N E S S S T R A T E G I E S : C O P I N G W I T H C O V I D - 1 9 The CARES Act What businesses need to know 1 The CARES Act Panelist Click to edit Master title style What businesses need to know

882 views • 40 slides

CARES Act Town Hall: Part II MAY 13, 2020 9:00 10:30AM ZOOM FACILITATORS Erica Orians,

CARES Act Town Hall: Part II MAY 13, 2020 9:00 10:30AM ZOOM FACILITATORS Erica Orians, Michigan Center for Student Success Precious Miller, Michigan Center for Student Success Priya Chaplot, National Center for Inquiry and Improvement

537 views • 21 slides

Saving Active Management A self-guided exploration : Ron Surz Click on any picture to open a

Saving Active Management A self-guided exploration : Ron Surz Click on any picture to open a related article. Ron@PPCA-inc.com (949)488-8339 Active Managers Have Lost $500 Billion in Last 3 Years S&P, Morningstar, Vanguard, etc. say

494 views • 12 slides