Reinforcement Learning with a Corrupted Reward Channel Tom Everitt, - PowerPoint PPT Presentation

Mar 16, 2023 •158 likes •275 views

Reinforcement Learning with a Corrupted Reward Channel Tom Everitt, Victoria Krakovna, Laurent Orseau, Marcus Hutter, Shane Legg Australian National University Google DeepMind IJCAI 17 and arXiv Motivation We will need to control

Reinforcement Learning with a Corrupted Reward Channel Tom Everitt, Victoria Krakovna, Laurent Orseau, Marcus Hutter, Shane Legg Australian National University Google DeepMind IJCAI 17 and arXiv
Motivation ● We will need to control Human-Level+ AI ● By identifying problems with various AI-paradigms, we can focus research on – the right paradigms – crucial problems within promising paradigms
The Wireheading Problem ● Future RL agent hijacks reward signal (wireheading) ● CoastRunners agent drives in small circle (misspecified reward function) ● RL agent shortcuts reward sensor (sensory error) ● Cooperative Inverse RL agent misperceives human action (adversarial counterexample)
Formalisation ● Reinforcement Learning is traditionally modeled with Markov Decision Process (MDP): ● This fails to model situations where there is a difference between – True reward – Observed reward ● Can be modeled with Corrupt Reward MDP:
Simplifying assumptions
Good intentions ● Natural optimise true reward using observed reward as evidence ● Theorem: Will still suffer near-maximal regret ● Good intentions is not enough!
Avoiding Over-Optimisation ● Quantilising agent randomly picks a state/policy where reward above threshold ● Theorem: For q corrupt states, exists s.t. has average regret at most ● Avoiding over-optimisation helps!
Richer Information Reward Observation Graphs ● RL: ● Decoupled RL: – States “self-estimate” – Cooperative IRL their reward – Learning values from stories – Learning from Human Preferences
Learning true reward Majority vote Safe state – Cooperative Inverse RL – Learning from Human – Learning values from Preferences stories ● Richer information helps!
Experiments ● AIXIjs: http://aslanides.io/aixijs/demo.html True reward Observed reward
Key Takeaways ● Wireheading: observed reward true reward ● Good intentions is not enough ● Either: – Avoid over-optimisation – Give the agent rich data to learn from (CIRL, stories, human preferences) ● Experiments available online

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Reinforcement Learning with a Corrupted Reward Channel Tom Everitt, Victoria Krakovna, Laurent

Reinforcement Learning with a Corrupted Reward Channel Tom Everitt, Victoria Krakovna, Laurent Orseau, Marcus Hutter, Shane Legg IJCAI 2017 and arXiv (slides adapted from Tom's IJCAI talk) Motivation Want to give RL agents good incentives

390 views • 20 slides

Reward Shaping in Episodic Reinforcement Learning Marek Grze s Canterbury, UK AAMAS 2017

Reward Shaping in Episodic Reinforcement Learning Marek Grze s Canterbury, UK AAMAS 2017 S ao Paulo, May 812 Motivating Reward Shaping Reinforcement Learning Agent reward state action r t a t s t r t+ 1 Environment s t+ 1 [Sutt

1.41k views • 16 slides

Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays

Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays Lecture 1. So far: manually design reward function to define a task 2. What if we want to learn the reward function from observing an expert, and

650 views • 33 slides

Foundations of Machine Learning Reinforcement Learning Reinforcement Learning Agent exploring

Foundations of Machine Learning Reinforcement Learning Reinforcement Learning Agent exploring environment. Interactions with environment: action state Agent Environment reward Problem: find action policy that maximizes cumulative reward

828 views • 66 slides

Reinforcement Learning: How Does It Work? We detect a state Reinforcement Learning We choose an

1 Reinforcement Learning: How Does It Work? We detect a state Reinforcement Learning We choose an action Lecture 2 We get a reward Gillian Hayes Our aim is to learn a policy what action to choose in what state to get maximum reward 11th

641 views • 5 slides

Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers (the class labels) Reinforcement learning: In reinforcement learning, the agent learns Markov Decision Processes based on a punishment/reward

89 views • 6 slides

Where's The Reward? Where's The Reward? A Review of Reinforcement Learning for Instructional

Where's The Reward? Where's The Reward? A Review of Reinforcement Learning for Instructional Sequencing Shayan Doroudi 1 2 2 Research Question Research Question Over the past 50 years, how Over the past 50 years, how successful has RL

1.92k views • 163 slides

Interactive Reinforcement Learning Human Generated Reward Presentation for Summer Camp 2015

Interactive Reinforcement Learning Human Generated Reward Presentation for Summer Camp 2015 May 25 2015 Reinforcement Learning Trial and error learning Explore and exploit Sutton and Barto 1988 Represent, predict and control

252 views • 8 slides

Using Natural Language for Reward Shaping in Reinforcement Learning Prasoon Goyal , Scott Niekum

Using Natural Language for Reward Shaping in Reinforcement Learning Prasoon Goyal , Scott Niekum and Raymond J. Mooney The University of Texas at Austin Motivation 2 Motivation In sparse reward settings, random exploration has very

936 views • 47 slides

Learning frameworks Associative reinforcement learning Given input, learn to produce output

Learning frameworks Associative reinforcement learning Given input, learn to produce output (action) that maximizes immediate reward Supervised learning Modified Associative reward-penalty ( A R - P ) Assumes environment specifies correct output

241 views • 5 slides

1 Video of Demo Q-Learning Auto Cliff Grid Exploration vs. Exploitation How to Explore? Video

Reinforcement Learning Reinforcement Learning II We still assume an MDP: A set of states s S A set of actions (per state) A A model T(s,a,s) A reward function R(s,a,s) Still looking for a policy (s) New

358 views • 6 slides

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement learning? Agent/Actor + Action + Environment + State + Reward How does reinforcement learning work?

792 views • 31 slides

Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent state reward action S t R t A t R t+ 1 Environment S t+ 1

1.07k views • 52 slides

Reinforcement Learning A (almost) quick (and very incomplete) introduction Slides from David

Reinforcement Learning A (almost) quick (and very incomplete) introduction Slides from David Silver, Dan Klein, Mausam, Dan Weld Reinforcement Learning At each time step t : Agent executes an action A t Environment emits a reward Rt

1.1k views • 71 slides

Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning

Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning arXiv:1702.08165 Reinforcement Learning with Deep Energy-Based Policies 1 / 25 Reinforcement Learning Environment Action Reward Interpreter State

531 views • 25 slides

Introduction to Reinforcement Learning Abdeslam Boularias Monday, June 25, 2018 1 / 93 What is

Machine Learning Summer School in Algiers Introduction to Reinforcement Learning Abdeslam Boularias Monday, June 25, 2018 1 / 93 What is reinforcement learning? a way of programming agents by reward and punishment without needing to specify

1.15k views • 93 slides

Probability and Statistics for Computer Science A major use of probability in sta4s4cal

Probability and Statistics for Computer Science A major use of probability in sta4s4cal inference is the upda4ng of probabili4es when certain events are observed Prof. M.H. DeGroot Credit: wikipedia Hongye Liu, Teaching

670 views • 51 slides

NAG : Motivating Deployment of Networked Systems Mohit Lad UCLA Deployment of Networked Systems

NAG : Motivating Deployment of Networked Systems Mohit Lad UCLA Deployment of Networked Systems Goal: How do you convince users to deploy applications? Prior work: Lottery System (Sigcomm 07) Probabilistic incentive to motivate

448 views • 7 slides

Networks Used As Episodic Memory for An Autonomous Robot Outline

Fusion Adaptive Resonance Theory Networks Used As Episodic Memory for An Autonomous Robot Outline 2 Goals for autonomous robots IRL-1 3 Role of episodic memory 4 Proposed model: EM-ART 5

392 views • 22 slides

Gen enerativ erative e Adver ersaria sarial l Im Imitation itation Le Learning arning

Gen enerativ erative e Adver ersaria sarial l Im Imitation itation Le Learning arning Stefan efano o Ermon on Joint work with Jayesh Gupta, Jonathan Ho, Yunzhu Li, Hongyu Ren, and Jiaming Song Reinforcement Learning Goal: Learn

530 views • 40 slides

Tor Metrics Ecosystem Data Collection, Archive, Analysis and Visualisation Iain R. Learmonth

Tor Metrics Ecosystem Data Collection, Archive, Analysis and Visualisation Iain R. Learmonth (irl) September 17, 2018 Tor Project $ whoami Tor Metrics Team Member Background in Internet @iainlearmonth Measurement @irl@mastodon.technology

426 views • 41 slides

Probability 2.3 Independence Anna Karlin Most slides by Alex Tsun Agenda Chain Rule

Probability 2.3 Independence Anna Karlin Most slides by Alex Tsun Agenda Chain Rule Independence Conditional Independence in class not Chain Rule (Idea) Have a Standard 52-Card Deck. 4 Suits (Clubs, Diamonds, Hearts,

678 views • 23 slides

CALIFORNIA ELECTRIC SYSTEMS FOR THE 21 ST CENTURY (CES-21) NOVEMBER, 12, 2014 DOUG RHOADES CHIEF

ANNUAL INDUSTRY WORKSHOP NOVEMBER 12-13, 2014 CALIFORNIA ELECTRIC SYSTEMS FOR THE 21 ST CENTURY (CES-21) NOVEMBER, 12, 2014 DOUG RHOADES CHIEF ENGINEER, CYBERSECURITY, SOUTHERN CALIFORNIA EDISON TRUSTWORTHY CYBER INFRASTRUCTURE FOR THE POWER

195 views • 7 slides

Formalizing Theatrical Performances Using Multi-Agent Organizations Andreas Schmidt Jensen ,

Formalizing Theatrical Performances Using Multi-Agent Organizations Andreas Schmidt Jensen , Johannes Svante Spurkeland and Jrgen Villadsen Department of Applied Mathematics and Computer Science Technical University of Denmark November 21,

600 views • 42 slides