Policy Certificates: Towards Accountable Reinforcement Learning - PowerPoint PPT Presentation

Policy Certificates: Towards Accountable Reinforcement Learning Christoph Dann, Lihong Li, Wei Wei, Emma Brunskill CMU Google Research Stanford University

Minimax-Optimal PAC Bounds Key contribution: new algorithm for episodic tabular MDPs with PAC Bound High-prob. Regret Bound S: #states, A: #actions, H: episode length, T: #episodes, ϵ: accuracy

Minimax-Optimal PAC Bounds Key contribution: new algorithm for episodic tabular MDPs with PAC Bound High-prob. Regret Bound First minimax-optimal! (for small ϵ) S: #states, A: #actions, H: episode length, T: #episodes, ϵ: accuracy Prior work: [DLB ‘17] [DB ‘15]

<latexit sha1_base64="30xMli8LGXUeEmLEMSjzuAbUbRA=">ACX3icbVHLTgIxFC3jG1+jroybRmJigiEzg0aXqAtZ+gAlgYF0StGzsP2jpFM5v8BpeuXLnVrR3ARMGbtjk59x721MvElyBZb3mjJnZufmFxaX8srq2rq5sXmrwlhSVqehCGXDI4oJHrA6cBCsEUlGfE+wO69/nuXvnphUPAxqMIiY65P7gPc4JaCpjkla6lFCcnNabTu1FBdxC9gzDPs2ry/O3MSxrAOcrTQZSavtci1Np3T2kRb9HGlSxDdtJ2uadsyCVbKGgaeBPQYFNI7Ljvne6oY09lkAVBClmrYVgZsQCZwKluZbsWIRoX1yz5oaBsRnyk2GV0nxnma6uBdKvQPAQ/Z3RUJ8pQa+p5U+gQc1mcvI/3OU6gupifHQO3ETHkQxsICOpvdigSHEmdm4yWjIAYaECq5fgCmD0QSCvpL8toZe9KHaXDrlOxybk6LFQqY48W0Q7aRfvIRseogqroEtURS/oA32ir9ybsWCsGeZIauTGNVvoTxjb34GntLY=</latexit> Minimax-Optimal PAC Bounds Key contribution: new algorithm for episodic tabular MDPs with PAC Bound High-prob. Regret Bound First minimax-optimal! (for small ϵ) Matches existing + improves for large H S: #states, A: #actions, H: episode length, T: #episodes, ϵ: accuracy Prior work: √ √ H 3 T + S 2 AH 2 SAH 2 T + [AOM ‘17] [DLB ‘17] [DB ‘15]

Motivation: Need for Accountability in Online RL current episode Even with PAC + regret bounds: expected return in next episode during learning unknown

Motivation: Need for Accountability in Online RL How good will my treatment be? Is it the best possible?

Our Proposal: Algorithms output policy certificates before each episode

Algorithms with policy certificates Natural extension of model-based optimistic algorithms 1. UCB on optimal value function 2. Greedy Policy 3. LCB on value function of current policy 4. Output certificate 0

Algorithms with Policy Certificates Natural extension of model-based optimistic algorithms 1. UCB on optimal value function 2. Greedy Policy 3. LCB on value function of current policy 4. Output certificate 0

Symbiosis of Optimism and Certificates Certificates: • Challenge: random • Insight from optimism: at known rate

Symbiosis of Optimism and Certificates Certificates: Optimism: • Challenge: random • Challenge: exploration bonus depends on • Insight from optimism: • Insight from certificates: at known rate bound by

Symbiosis of Optimism and Certificates Certificates: Optimism: • Challenge: random • Challenge: exploration bonus depends on • Insight from optimism: • Insight from certificates: at known rate bound by More accountable algorithms Better exploration bonuses yield through accurate policy certificates minimax-optimal PAC & regret bounds

Policy Certificates: Towards Accountable Reinforcement Learning - PowerPoint PPT Presentation

Policy Certificates: Towards Accountable Reinforcement Learning Christoph Dann, Lihong Li, Wei Wei, Emma Brunskill CMU Google Research Stanford University Minimax-Optimal PAC

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3.

Refresh Your Knowledge. Policy Gradient Policy gradient algorithms change the policy parameters

Reinforcement Learning Policy Op5miza5on Pieter Abbeel UC Berkeley EECS Policy

Learning to Optimize as Policy Learning Yisong Yue Policy Learning (Reinforcement &

Reinforcement Learning and Model Predictive Control RL : optimizes policy for given cost

Online Trust and Digital Certificates: The Policy Landscape Stephen Schultze Associate Director

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Module 7 Policy Iteration CS 886 Sequential Decision Making and Reinforcement Learning

Who Owns Renewable Energy Certificates: An Exploration of Policy Options and Practice Edward A.

The Provable E ff ectiveness of Policy Gradient Methods in Reinforcement Learning Sham Kakade

Accountable Care Systems Andrew Bland SPG, 26th January 2018 Development of Accountable

Jeffrey Levi Professor of Health Policy and Management September 30, 2016 What is

Efficient Off-Policy Meta- Reinforcement Learning via Probabilistic Context Variables Rakelly,

Offline Policy-search in Bayesian Reinforcement Learning Castronovo Michael University of Li`

DCSO: Dynamic Combination of Detector Scores for Outlier Ensembles Yue Zhao Maciej K.

Cold atoms in 2D optical lattices under staggered rotation Cristiane MORAIS SMITH Institute for

Audience Outlook Monitor: Museums and Galleries Tandi Palmer Williams June 2020 Many people

Opportunities and Challenges IP-SOC DAYS Shanghai - Sep 13 th 2018 www.allegrodvt.com Allegro DVT

Robust Configuration Management <draft-cole-netconf-robust-config-00.txt> Robert G. Cole 1

Our Ability to Improve The 22nd Princeton Conference, Session VII Quality: Has the health care

WebKit, HTML5 media and GStreamer on multiple platforms Spreading GStreamer awesome in WebKit

HOM Transfer Function Measurements and other topics Larry Doolittle, LBNL ICFA Workshop on High

Policy Certificates: Towards Accountable Reinforcement Learning - PowerPoint PPT Presentation

Policy Certificates: Towards Accountable Reinforcement Learning Christoph Dann, Lihong Li, Wei Wei, Emma Brunskill CMU Google Research Stanford University Minimax-Optimal PAC

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3.

Refresh Your Knowledge. Policy Gradient Policy gradient algorithms change the policy parameters

Reinforcement Learning Policy Op5miza5on Pieter Abbeel UC Berkeley EECS Policy

Learning to Optimize as Policy Learning Yisong Yue Policy Learning (Reinforcement &amp;

Reinforcement Learning and Model Predictive Control RL : optimizes policy for given cost

Online Trust and Digital Certificates: The Policy Landscape Stephen Schultze Associate Director

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Module 7 Policy Iteration CS 886 Sequential Decision Making and Reinforcement Learning

Who Owns Renewable Energy Certificates: An Exploration of Policy Options and Practice Edward A.

The Provable E ff ectiveness of Policy Gradient Methods in Reinforcement Learning Sham Kakade

Accountable Care Systems Andrew Bland SPG, 26th January 2018 Development of Accountable

Jeffrey Levi Professor of Health Policy and Management September 30, 2016 What is

Efficient Off-Policy Meta- Reinforcement Learning via Probabilistic Context Variables Rakelly,

Offline Policy-search in Bayesian Reinforcement Learning Castronovo Michael University of Li`

DCSO: Dynamic Combination of Detector Scores for Outlier Ensembles Yue Zhao Maciej K.

Cold atoms in 2D optical lattices under staggered rotation Cristiane MORAIS SMITH Institute for

Audience Outlook Monitor: Museums and Galleries Tandi Palmer Williams June 2020 Many people

Opportunities and Challenges IP-SOC DAYS Shanghai - Sep 13 th 2018 www.allegrodvt.com Allegro DVT

Robust Configuration Management &lt;draft-cole-netconf-robust-config-00.txt&gt; Robert G. Cole 1

Our Ability to Improve The 22nd Princeton Conference, Session VII Quality: Has the health care

WebKit, HTML5 media and GStreamer on multiple platforms Spreading GStreamer awesome in WebKit

HOM Transfer Function Measurements and other topics Larry Doolittle, LBNL ICFA Workshop on High

Learning to Optimize as Policy Learning Yisong Yue Policy Learning (Reinforcement &

Robust Configuration Management <draft-cole-netconf-robust-config-00.txt> Robert G. Cole 1