Fingerprint Policy Optimisation for Robust Reinforcement Learning - PowerPoint PPT Presentation

Poster #50 Fingerprint Policy Optimisation for Robust Reinforcement Learning Supratik Paul, Michael A. Osborne, Shimon Whiteson This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreements \#637713)

Motivation 2

Motivation • Environment variable (EV) • E.g. wind conditions • Controllable during learning but not during execution 2

Motivation • Environment variable (EV) • E.g. wind conditions • Controllable during learning but not during execution • Objective: Find 𝜌 ∗ = 𝑏𝑠𝑕𝑛𝑏𝑦 𝜌 𝐾 𝜌 = 𝑏𝑠𝑕𝑛𝑏𝑦 𝜌 𝔽 𝐹𝑊~𝑞(𝐹𝑊) [𝑆 𝜌 ] 2

Motivation • Environment variable (EV) • E.g. wind conditions • Controllable during learning but not during execution • Objective: Find 𝜌 ∗ = 𝑏𝑠𝑕𝑛𝑏𝑦 𝜌 𝐾 𝜌 = 𝑏𝑠𝑕𝑛𝑏𝑦 𝜌 𝔽 𝐹𝑊~𝑞(𝐹𝑊) [𝑆 𝜌 ] • Need to account for rare events • E.g. rare wind conditions leading to a crash 2

Naïve application of f policy gradients 3

Naïve application of f policy gradients Trajectories ~ 𝜌 3

Naïve application of f policy gradients Rare Events Trajectories ~ 𝜌 4

Naïve application of f policy gradients Rare Events Trajectories ~ 𝜌 • Monte Carlo estimate of the Policy Gradient has very high variance ⟹ Doomed to failure 4

Fingerprint Policy Optimisation (F (FPO) 5

Fingerprint Policy Optimisation (F (FPO) At each iteration, select parameters 𝜔 of 𝑟 𝜔 (𝐹𝑊) such that it maximises one-step expected return 5

Fingerprint Policy Optimisation (F (FPO) 6

Fingerprint Policy Optimisation (F (FPO) • 𝜌 ′ = 𝜌 + α𝛼𝐾 𝜌 • 𝐾 𝜌 ′ = f(𝜌, 𝜔) 6

Fingerprint Policy Optimisation (F (FPO) • 𝜌 ′ = 𝜌 + α𝛼𝐾 𝜌 • 𝐾 𝜌 ′ = f(𝜌, 𝜔) • Model 𝐾 𝜌 ′ as a Gaussian Process with inputs (𝜌, 𝜔) • Use Bayesian Optimisation to select 𝜔|𝜌 = argmax 𝜔 f(𝜌, 𝜔) 6

Fingerprint Policy Optimisation (F (FPO) • 𝜌 ′ = 𝜌 + α𝛼𝐾 𝜌 • 𝐾 𝜌 ′ = f(𝜌, 𝜔) • Model 𝐾 𝜌 ′ as a Gaussian Process with inputs (𝜌, 𝜔) • Use Bayesian Optimisation to select 𝜔|𝜌 = argmax 𝜔 f(𝜌, 𝜔) Low dimensional representation 𝜌 is high dimensional “Fingerprint” 6

Policy fi fingerprints 7

Policy fi fingerprints • Disambiguation, not accurate representation 7

Policy fi fingerprints • Disambiguation, not accurate representation • State/Action fingerprints: Gaussians fitted to the stationary state/action distribution induced by 𝜌 7

Policy fi fingerprints • Disambiguation, not accurate representation • State/Action fingerprints: Gaussians fitted to the stationary state/action distribution induced by 𝜌 • Gross simplification, but good at disambiguating between policies 7

Results Half Cheetah Ant • Velocity target = 2 with probability • Reward proportional to velocity 98% and ‘normal’ reward • 5% chance that velocity > 2 leads to • Velocity target = 4 with probability 2% joint damage with large negative with significantly high reward reward 8

Poster #50 Fingerprint Policy Optimisation for Robust Reinforcement Learning Supratik Paul, Michael A. Osborne, Shimon Whiteson This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreements \#637713)

Fingerprint Policy Optimisation for Robust Reinforcement Learning - PowerPoint PPT Presentation

Poster #50 Fingerprint Policy Optimisation for Robust Reinforcement Learning Supratik Paul, Michael A. Osborne, Shimon Whiteson This project has received funding from the European Research Council (ERC) under the European Unions Horizon 2020

Action Robust Reinforcement Learning and Applications in Continuous Control Chen Tessler *,

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3.

Refresh Your Knowledge. Policy Gradient Policy gradient algorithms change the policy parameters

Kernel-based Reinforcement Learning in Robust Markov Decision Processes Shiau Hong Lim, Arnaud

Reproducible, Reusable, and Robust Reinforcement Learning Joelle Pineau Facebook AI Research,

WE MAKE INNOVATION HAPPEN Pry-ID The cable fingerprint Pry-ID is the fingerprint for your

Reinforcement Learning Policy Op5miza5on Pieter Abbeel UC Berkeley EECS Policy

Learning to Optimize as Policy Learning Yisong Yue Policy Learning (Reinforcement &

Reinforcement Learning and Model Predictive Control RL : optimizes policy for given cost

A Whorlwind Tour A Guide To Expert Fingerprint Evidence Presented by Philip Gilhooley

DactyMatch Green Bit Green Bit Fingerprint Recognition Recognition Fingerprint SDK v.2.2

COSEC DOOR FMX High Performance Door Controller with Multispectral Fingerprint Reader Demanding

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

Optimisation Constraint Problems Combinatorial Optimisation Modelling (in MiniZinc) Solving

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

VIDEN VIDEN At Attacker Identification on In-Vehicle Networks Kyong-Tak Cho and Kang G. Shin

Private Fingerprint Matching Siamak F Shahandashti Reihaneh Safavi-Naini Philip Ogunbona Uni of

The Calculation of Molecular Similarity: Principles and Practice Peter Willett, University of

Web-enabled Biometric Software (WEBS) Mr. William A. Thum Accessions Suitability Office ARNG-HRR-O

Deriving intelligence from USB stack interactions Andy Davis, Research Director NCC Group Image

1 Introduction There are three fundamental principles of There are three fundamental

Br Browser fi fingerprinting Nataliia Bielova @nataliabielova February 12

Fingerprinting ECUs for Vehicle Intrusion Detection Kyong-Tak Cho, Kang G. Shin, University of

Fingerprint Policy Optimisation for Robust Reinforcement Learning - PowerPoint PPT Presentation

Poster #50 Fingerprint Policy Optimisation for Robust Reinforcement Learning Supratik Paul, Michael A. Osborne, Shimon Whiteson This project has received funding from the European Research Council (ERC) under the European Unions Horizon 2020

Action Robust Reinforcement Learning and Applications in Continuous Control Chen Tessler *,

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3.

Refresh Your Knowledge. Policy Gradient Policy gradient algorithms change the policy parameters

Kernel-based Reinforcement Learning in Robust Markov Decision Processes Shiau Hong Lim, Arnaud

Reproducible, Reusable, and Robust Reinforcement Learning Joelle Pineau Facebook AI Research,

WE MAKE INNOVATION HAPPEN Pry-ID The cable fingerprint Pry-ID is the fingerprint for your

Reinforcement Learning Policy Op5miza5on Pieter Abbeel UC Berkeley EECS Policy

Learning to Optimize as Policy Learning Yisong Yue Policy Learning (Reinforcement &amp;

Reinforcement Learning and Model Predictive Control RL : optimizes policy for given cost

A Whorlwind Tour A Guide To Expert Fingerprint Evidence Presented by Philip Gilhooley

DactyMatch Green Bit Green Bit Fingerprint Recognition Recognition Fingerprint SDK v.2.2

COSEC DOOR FMX High Performance Door Controller with Multispectral Fingerprint Reader Demanding

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

Optimisation Constraint Problems Combinatorial Optimisation Modelling (in MiniZinc) Solving

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

VIDEN VIDEN At Attacker Identification on In-Vehicle Networks Kyong-Tak Cho and Kang G. Shin

Private Fingerprint Matching Siamak F Shahandashti Reihaneh Safavi-Naini Philip Ogunbona Uni of

The Calculation of Molecular Similarity: Principles and Practice Peter Willett, University of

Web-enabled Biometric Software (WEBS) Mr. William A. Thum Accessions Suitability Office ARNG-HRR-O

Deriving intelligence from USB stack interactions Andy Davis, Research Director NCC Group Image

1 Introduction There are three fundamental principles of There are three fundamental

Br Browser fi fingerprinting Nataliia Bielova @nataliabielova February 12

Fingerprinting ECUs for Vehicle Intrusion Detection Kyong-Tak Cho, Kang G. Shin, University of

Learning to Optimize as Policy Learning Yisong Yue Policy Learning (Reinforcement &