reinforcement learning
play

Reinforcement learning Fredrik D. Johansson Clinical ML @ MIT - PowerPoint PPT Presentation

Oct 17, 2022 β€’346 likes β€’1.06k views

Reinforcement learning Fredrik D. Johansson Clinical ML @ MIT 6.S897/HST.956: Machine Learning for Healthcare, 2019 Reminder: Causal effects Potential outcomes under treatment and control, 1 , 0 Covariates and treatment, ,

  1. Reinforcement learning Fredrik D. Johansson Clinical ML @ MIT 6.S897/HST.956: Machine Learning for Healthcare, 2019

  2. Reminder: Causal effects β–Ί Potential outcomes under treatment and control, 𝑍 1 , 𝑍 0 β–Ί Covariates and treatment, π‘Œ, π‘ˆ π‘ˆ π‘Œ β–Ί Conditional average treatment effect (CATE) π·π΅π‘ˆπΉ π‘Œ = 𝔽 𝑍 1 βˆ’ 𝑍 0 ∣ π‘Œ 𝑍 Potential outcomes Features

  3. Today: Treatment policies/regimes β–Ί A policy 𝝆 assigns treatments to patients (typically depending on their medical history/state) β–Ί Example: For a patient with medical history 𝑦 , 𝜌(𝑦) = 𝕁[π·π΅π‘ˆπΉ 𝑦 > 0] β€œTreat if effect is positive” β–Ί Today we focus on policies guided by clinical outcomes (as opposed to legislation, monetary cost or side-effects)

  4. Example: Sepsis management β–Ί Sepsis is a complication of an infection which can lead to massive organ failure and death β–Ί One of the leading causes of death in the ICU β–Ί The primary treatment target is the infection β–Ί Other symptoms need management: breathing difficulties, low blood pressure, …

  5. Recall: Potential outcomes Septic patient with breathing difficulties 𝑍(0) Unobserved responses Blood 1. Should the patient be put on oxygen mechanical ventilation? π‘Œ Observed decisions 𝑍(1) & response π‘ˆ Mechanical ventilation? Sedation? Vasopressors? Time

  6. Today: Sequential decision making β–Ί Many clinical decisions are made in sequence β–Ί Choices early may rule out actions later β–Ί Can we optimize the policy by which actions are made? 𝐡 9 𝑆 8 𝑆 9 𝑆 : 𝑒 𝑇 8 𝑇 9 𝑇 : 𝑒 8 𝑒 9 𝑒 :

  7. Recall: Potential outcomes Septic patient with breathing difficulties Unobserved responses 1. Should the patient be put on mechanical ventilation? Observed decisions & response Mechanical ventilation? Sedation? Vasopressors? Time

  8. Example: Sepsis management Septic patient with breathing difficulties Unobserved 2. Should the patient be responses sedated? (To alleviate discomfort due Observed decisions to mech. ventilation) & response Mechanical ventilation? Sedation? Vasopressors? Time

  9. Example: Sepsis management Septic patient with breathing difficulties 3. Should we Unobserved responses artificially raise blood pressure? (Which may have Observed decisions dropped due to & response sedation) Mechanical ventilation? Sedation? Vasopressors? Time

  10. Example: Sepsis management Septic patient with breathing difficulties Observed decisions & response Mechanical ventilation? Sedation? Vasopressors? Time

  11. Finding optimal policies β–Ί How can we treat patients so that their outcomes are as good as possible ? Outcome β–Ί What are good outcomes ? β–Ί Which policies should we consider? Mechanical ventilation? Sedation? Vasopressors?

  12. Success stories in popular press β–Ί AlphaStar β–Ί AlphaGo β–Ί DQN Atari β–Ί Open AI Five

  13. Reinforcement learning Game state 𝑇 8 β–Ί Maximize reward! Possible actions 𝐡 8 Next state 𝑇 9 Reward 𝑆 9 (Loss) Figure by Tim Wheeler, tim.hibal.org

  14. Great! Now let’s treat patients β–Ί Patient state at time 𝑇 = is like the game board β–Ί Medical treatments 𝐡 = are like the actions β–Ί Outcomes 𝑆 = are the rewards in the game 𝐡 9 β–Ί What could possibly go wrong? 𝑆 8 𝑆 9 𝑆 : 𝑒 𝑇 8 𝑇 9 𝑇 : 𝑒 8 𝑒 9 𝑒 :

  15. 1. Decision processes 2. Reinforcement learning 3. Learning from batch (off-policy) data 4. Reinforcement learning in healthcare

  16. Decision processes β–Ί An agent repeatedly, at times 𝑒 takes actions 𝐡 = Agent to receive rewards 𝑆 = Reward 𝑆 = Action 𝐡 = from an environment , the state 𝑇 = of which is (partially) observed Environment State 𝑇 =

  17. Decision process: Mechanical ventilation @FG= HII + 𝑆 = @A=BCD + 𝑆 = @FG= HG 𝑆 = = 𝑆 = Agent Reward $ # Action " # Environment State % # 𝑇 8 𝐡 8 𝐡 9 𝑇 ? , 𝑆 ? 𝐡 ? 𝑆 : 𝑇 9 , 𝑆 9 Mechanical ventilation? Sedation? Spontaneous breathing trial? Time

  18. Decision process: Mechanical ventilation β–Ί State 𝑇 = includes demographics, physiological measurements, ventilator settings, level of consciousness, dosage of 𝑇 8 sedatives, time to 𝑇 ? ventilation, number of 𝑇 9 intubations

  19. Decision process: Mechanical ventilation β–Ί Actions 𝐡 = include intubation and extubation, as well as administration and dosages of sedatives 𝐡 8 𝐡 ? 𝐡 9

  20. Decision processes β–Ί A decision process specifies how states 𝑇 = , actions 𝐡 = , and rewards 𝑆 = are distributed : π‘ž(𝑇 8 , … , 𝑇 : , 𝐡 8 , … , 𝐡 : , 𝑆 8 , … , 𝑆 : ) β–Ί The agent interacts with the environment according to a behavior policy 𝜈 = π‘ž(𝐡 = ∣ β‹― ) * * The … depends on the type of agent

  21. Markov Decision Processes β–Ί Markov decision processes (MDPs) are a special case β–Ί Markov transitions : π‘ž 𝑇 = 𝑇 8 , … , 𝑇 =N9 , 𝐡 8 , … , 𝐡 =N9 = π‘ž(𝑇 = ∣ 𝑇 =N9 , 𝐡 =N9 ) β–Ί Markov reward function: π‘ž 𝑆 = 𝑇 = , 𝐡 = = π‘ž 𝑆 = 𝑇 8 , … , 𝑇 =N9 , 𝐡 8 , … , 𝐡 =N9 β–Ί Markov action policy 𝜈 = π‘ž(𝐡 = ∣ 𝑇 = ) = π‘ž 𝐡 = 𝑇 8 , … , 𝑇 =N9 , 𝐡 8 , … , 𝐡 =N9

  22. Markov assumption β–Ί State transitions, actions and reward depend only on most recent state-action pair 𝐡 8 𝐡 : … 𝑇 8 𝑇 9 𝑇 : 𝑆 8 𝑆 :

  23. Contextual bandits (special case)* β–Ί States are independent: π‘ž 𝑇 = 𝑇 =N9 , 𝐡 =N9 = π‘ž(𝑇 = ) β–Ί Equivalent to single-step case : potential outcomes! 𝐡 8 𝐡 : … 𝑇 8 𝑇 9 𝑇 : 𝑆 8 𝑆 : * The term β€œcontextual bandits” has connotations of efficient exploration, which is not addressed here

  24. Contextual bandits & potential outcomes β–Ί Think of each state 𝑇 A as an i.i.d. patient, the actions 𝐡 A as the treatment group indicators and 𝑆 A as the outcomes 𝐡 8 𝐡 : … 𝑇 8 𝑇 : 𝑆 8 𝑆 :

  25. Goal of RL β–Ί Like previously with causal effect estimation, we are interested in the effects of actions 𝐡 = on future rewards 𝐡 8 𝐡 : … 𝑇 8 𝑇 9 𝑇 : 𝑆 8 𝑆 :

  26. Value maximization β–Ί The goal of most RL algorithms is to maximize the expected cumulative rewardβ€”the value π‘Š P of its policy 𝜌 : β–Ί Return : 𝐻 = = βˆ‘ 𝑆 D Sum of future rewards DS= β–Ί Value: π‘Š P = 𝔽 T U ∼P 𝐻 8 Expected sum of rewards under policy 𝜌 β–Ί The expectation is taken with respect to scenarios acted out according to the learned policy 𝜌

  27. Example Value G P β‰ˆ 1 π‘œ [ 𝐻 G π‘Š β–Ί Let’s say that we have data from a policy 𝜌 AS9 9 = 0 𝑏 ? Return 9 = 1 𝑏 W 9 9 = 1 𝑆 9 𝑏 9 9 + 𝑆 ? 9 + 𝑆 W 𝐻 9 = 𝑆 9 9 9 𝑆 W 9 𝑆 ? Patient 1 ? = 1 Patient 2 𝑏 W ? = 0 ? + 𝑆 ? ? + 𝑆 W 𝐻 ? = 𝑆 9 ? = 1 𝑏 9 ? 𝑏 ? ? 𝑆 W ? 𝑆 9 ? 𝑆 ? Patient 3 W 𝑆 9 W = 0 W = 0 𝑏 ? 𝑏 9 W W = 0 𝑆 ? 𝑏 W W 𝑆 W 𝐻 W = 𝑆 9 W + 𝑆 ? W + 𝑆 W W

  28. Robot in a room β–Ί Stochastic actions π‘ž Move up 𝐡 = β€π‘£π‘žβ€ = 0.8 + 1 Available non-opposite moves have uniform probability βˆ’1 β–Ί Rewards: +1 at [4,3] (terminal state) Start -1 at [4,2] (terminal) -0.04 per step Slide from Peter Bodik

  29. Robot in a room What is the optimal policy? β–Ί Stochastic actions π‘ž Move up 𝐡 = β€π‘£π‘žβ€ = 0.8 + 1 ? ? ? Available non-opposite moves have uniform probability ? ? βˆ’1 β–Ί Rewards: +1 at [4,3] (terminal state) ? ? ? ? -1 at [4,2] (terminal) -0.04 per step Slide from Peter Bodik

  30. Robot in a room β–Ί The following is the optimal policy/trajectory under + 1 deterministic transitions βˆ’1 β–Ί Not achievable in our stochastic transition model Slide from Peter Bodik

  31. Robot in a room β–Ί Optimal policy + 1 β–Ί How can we learn this? βˆ’1 Slide from Peter Bodik

  32. 1. Decision processes 2. Reinforcement learning 3. Learning from batch (off-policy) data 4. Reinforcement learning in healthcare

  33. Paradigms* Model-based RL Value-based RL Policy-based RL Transitions Value/return Policy π‘ž 𝑇 = 𝑇 =N9 , 𝐡 =N9 π‘ž 𝐻 = 𝑇 = , 𝐡 = π‘ž(𝐡 = ∣ 𝑇 = ) G-computation Q-learning REINFORCE MDP estimation G-estimation Marginal structural models *We focus on off-policy RL here

  34. Paradigms* Model-based RL Value-based RL Policy-based RL Transitions Value/return Policy π‘ž 𝑇 = 𝑇 =N9 , 𝐡 =N9 π‘ž 𝐻 = 𝑇 = , 𝐡 = π‘ž(𝐡 = ∣ 𝑇 = ) G-computation Q-learning REINFORCE MDP estimation G-estimation Marginal structural models *We focus on off-policy RL here

  35. Dynamic programming β–Ί Assume that we know how good a state-action pair is + 1 [3,1] [4,3] β–Ί Q: Which end state is the βˆ’1 best? A: [4,3] Start β–Ί Q: What is the best way to get there? A: Only [3,1] Slide from Peter Bodik

  36. Dynamic programming β–Ί [2,1] is slightly better than [3,2] because of the risk of + 1 transitioning to [4,2] from [3,2] [2,1] βˆ’1 β–Ί Which is the best way to [2,1]? [3,2] [4,2] Start Slide from Peter Bodik

  37. Dynamic programming β–Ί The idea of dynamic programming for + 1 reinforcement learning is to recursively learn the best βˆ’1 action/value in a previous state given the best action/value in future states Slide from Peter Bodik

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

551 views β€’ 35 slides
Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine playing a new game whose rules you dont know; after a hundred or so moves your don t know; after a hundred or so moves, your opponent announces, You

511 views β€’ 30 slides
Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers (the class labels) Reinforcement learning: In reinforcement learning, the agent learns Markov Decision Processes based on a punishment/reward

89 views β€’ 6 slides
Examples of Reinforcement Learning Robocup Soccer Teams Stone & Veloso, Reidmiller et al.

Examples of Reinforcement Learning Robocup Soccer Teams Stone & Veloso, Reidmiller et al.

Reinforcement Learning Slides by Rich Sutton Mods by Dan Lizotte Refer to Reinforcement Learning: An Introduction by Sutton and Barto Alpaydin Chapter 16 Up until now we have been Supervised Learning Classifying, mostly

891 views β€’ 31 slides
CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning Haarnoja, Tang et al. (2017) Reinforcement Learning with Deep Energy Based Policies, ICML . Haarnoja, Zhou et al. (2018) Soft Actor-Critic: Off-Policy

682 views β€’ 24 slides
Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control Theory, you actually have some knowledge in RL. RL shares the overall goal with OCT: solving for a control policy such that the cumulative cost

610 views β€’ 22 slides
Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still assume a Marko kov decision process (MDP): A se set o of st states s s S A se set o of a actions ( s (per st state) A A mo model

96 views β€’ 5 slides
R i f R i f Reinforcement Learning III Reinforcement Learning III t L t L i i III III Dec

R i f R i f Reinforcement Learning III Reinforcement Learning III t L t L i i III III Dec

R i f R i f Reinforcement Learning III Reinforcement Learning III t L t L i i III III Dec 03 2008 1 Large State Spaces h When a problem has a large state space we can not longer represent the U or Q functions as explicit tables explicit

436 views β€’ 18 slides
Reinforcement Learning UMaine COS 470/570 Introduction to AI Why reinforcement learning?

Reinforcement Learning UMaine COS 470/570 Introduction to AI Why reinforcement learning?

Reinforcement Learning<br/><br/> 4/25/19, 8*06 PM Reinforcement Learning<br/><br/> 4/25/19, 8*06 PM Reinforcement Learning UMaine COS 470/570 Introduction to AI Why reinforcement learning? Spring 2019 Created:

371 views β€’ 15 slides
7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B.

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B.

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B. Temporal Difference Reinforcement Learning C. PVLV Model D. Cerebellum and Error-driven Learning 2/23/18 COSC 494/594 CCN 2 Sensory-Motor Loop

790 views β€’ 56 slides
Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

540 views β€’ 15 slides
Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning Ja Jan-Wi

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning Ja Jan-Wi

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning Ja Jan-Wi Willem van de Meent Northeastern University [These slides were created by Dan Klein and Pieter Abbeel for CS188 Intro to AI at UC Berkeley. All CS188

145 views β€’ 10 slides
Reinforcement Learning AIMA Chapters: 21.1, 21.2, 21.3. Sutton and Barto, Reinforcement Learning:

Reinforcement Learning AIMA Chapters: 21.1, 21.2, 21.3. Sutton and Barto, Reinforcement Learning:

Reinforcement Learning Reinforcement Learning AIMA Chapters: 21.1, 21.2, 21.3. Sutton and Barto, Reinforcement Learning: an Introduction, 2nd Edition: Chapters 6 (6.1 6.5) Outline Reinforcement Learning Reinforcement Learning: the

587 views β€’ 27 slides
Module 11 Introduction to Reinforcement Learning CS 886 Sequential Decision Making and

Module 11 Introduction to Reinforcement Learning CS 886 Sequential Decision Making and

Module 11 Introduction to Reinforcement Learning CS 886 Sequential Decision Making and Reinforcement Learning University of Waterloo Machine Learning Supervised Learning Teacher tells learner what to remember Reinforcement Learning

401 views β€’ 29 slides
1 Pure Reinforcement Learning vs. Reinforcement Learning Monte-Carlo Planning No knowledge

1 Pure Reinforcement Learning vs. Reinforcement Learning Monte-Carlo Planning No knowledge

CSE 573: Artificial Intelligence Todo Reinforcement Learning Add simulations from 473 Dan Weld Add UCB bound (cut bolzman & constant epsilon Add snazzy videos (pendulum, zico kolter See http://www See http://www-

462 views β€’ 11 slides
Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest

Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest

Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest Lecture May 24, 2017 Lecture overview What makes a reinforcement learning algorithm safe ? Notation Creating a safe reinforcement learning

1.42k views β€’ 88 slides
#sepsis 1 Method Hannah Shotton 2 Study Advisory Group Study proposal Study Advisory

#sepsis 1 Method Hannah Shotton 2 Study Advisory Group Study proposal Study Advisory

@NCEPOD #sepsis 1 Method Hannah Shotton 2 Study Advisory Group Study proposal Study Advisory Group Study design: key themes, method, questionnaire Acute medicine Emergency medicine General practice Surgery

1.71k views β€’ 122 slides
R ELIABLE A PPLICATION OF THE S EPSIS B UNDLE E LEMENTS North Shore-LIJ Health System Martin E.

R ELIABLE A PPLICATION OF THE S EPSIS B UNDLE E LEMENTS North Shore-LIJ Health System Martin E.

R ELIABLE A PPLICATION OF THE S EPSIS B UNDLE E LEMENTS North Shore-LIJ Health System Martin E. Doerfler, MD SVP, Clinical Strategy and Development Pr Pres esent entation ation Ou Outline ne The NSLIJ HS High Reliability in Health

663 views β€’ 43 slides
Malaysian Healthy Ageing Society The Impact of Age on Severe Sepsis and Septic Shock Survivors

Malaysian Healthy Ageing Society The Impact of Age on Severe Sepsis and Septic Shock Survivors

Organised by: Co-Sponsored: Malaysian Healthy Ageing Society The Impact of Age on Severe Sepsis and Septic Shock Survivors Outcomes and Performance Status one Year after Hospital Discharge Alkhalaf M 1 , Abd. Aziz N 2 , Arabi Y 3 , B.

567 views β€’ 18 slides
MHA Keystone Center MICAH QN Meeting May 18, 2018 Agenda ADEs due to Opioids measure

MHA Keystone Center MICAH QN Meeting May 18, 2018 Agenda ADEs due to Opioids measure

MHA Keystone Center MICAH QN Meeting May 18, 2018 Agenda ADEs due to Opioids measure Sepsis measures a) Post-Operative Sepsis (PSI-13) b) Sepsis Mortality Rate Falls with Injury measure Person & Family Engagement HRM

594 views β€’ 46 slides
IHI Expedition Antibiotic Stewardship Session 5: Focus on: 72 Hour Antibiotic Time - out

IHI Expedition Antibiotic Stewardship Session 5: Focus on: 72 Hour Antibiotic Time - out

May 15, 2014 These presenters have nothing to disclose IHI Expedition Antibiotic Stewardship Session 5: Focus on: 72 Hour Antibiotic Time - out Scott Flanders, MD Jeff Rohde, MD Megan Mack, MD Matt Tupps, PharmD, MHA Diane Jacobsen,

810 views β€’ 42 slides
Introducing PCORnet and the Greater Plains Collaborative: The National Patient-Centered Clinical

Introducing PCORnet and the Greater Plains Collaborative: The National Patient-Centered Clinical

Introducing PCORnet and the Greater Plains Collaborative: The National Patient-Centered Clinical Research Network and Our Role Russ Waitman, University of Kansas Medical Center Marshfield Clinic, January 22, 2014 Outline PCORnet standard

388 views β€’ 35 slides
VBP Bootcamp Managed Long Term Care October 2017 2 10/13/2017 October 2017 2 Agenda Area

VBP Bootcamp Managed Long Term Care October 2017 2 10/13/2017 October 2017 2 Agenda Area

10/13/2017 VBP Bootcamp Managed Long Term Care October 2017 2 10/13/2017 October 2017 2 Agenda Area Details Timing One, 1-hour class Class 1 Smaller classroom setting with pauses in the presentation to: Setting a) Field questions

427 views β€’ 24 slides
Disclosures CODE SEPSIS I have no disclosures (not nowmaybe later) David Shimabukuro, MDCM

Disclosures CODE SEPSIS I have no disclosures (not nowmaybe later) David Shimabukuro, MDCM

5/13/2014 Disclosures CODE SEPSIS I have no disclosures (not nowmaybe later) David Shimabukuro, MDCM Associate Professor Medical Director, 13 ICU Physician Lead, UCSF DSRIP Sepsis Project Agenda Agenda Epidemiology Epidemiology

592 views β€’ 12 slides

More recommend