DS595/CS525 Reinforcement Learning Prof. Yanhua Li Time: 6:00pm - - PowerPoint PPT Presentation

DS595/CS525 Reinforcement Learning

• Prof. Yanhua Li

Welcome to

Time: 6:00pm –8:50pm R Zoom Lecture Fall 2020 This lecture will be recorded!!!

No Quiz Today

3

Project 3 due today

4

Next Thursday: No class Happy Thanksgiving

5

Project 4 is available Starts 10/29 Thursday

v https://github.com/yingxue-

zhang/DS595CS525-RL- Projects/tree/master/Project4

v Important Dates: v Project Proposal: Thursday 11/12/2020 v Progressive report: Thursday 11/26/2020 v Final Project:

§ Tuesday 12/8/2020 team project report is due § Thursday 12/10/2020 Virtual Poster Session

This Lecture

v Actor-Critic methods

§ A2C § A3C

§ Pathwise Derivative Policy Gradient

§ Advanced RL Techniques

§ Advanced techniques for DQN

• Multi-step DQN, Noisy net DQN
• Distributional DQN
• DQN for continuous action space

§ Sparse Reward

• Reward shaping, Curiosity module
• Curriculum learning, Hierarchical RL

Reinforcement Learning Single Agent Multiple Agents Inverse Reinforcement Learning

Tabular representation of reward Model-based control Model-free control (MC, SARSA, Q-Learning) Function representation of reward

• 1. Linear value function approx

(MC, SARSA, Q-Learning)

• 2. Value function approximation

(Deep Q-Learning, Double DQN, prioritized DQN, Dueling DQN)

• 3. Policy function approximation

(Policy gradient, PPO, TRPO)

• 4. Actor-Critic methods (A2C,

A3C, Pathwise Derivative PG) Advanced topics in RL （Sparse Rewards） Review of Deep Learning As bases for non-linear function approximation (used in 2-4). Linear reward function learning Imitation learning Apprenticeship learning Inverse reinforcement learning MaxEnt IRL MaxCausalEnt IRL MaxRelEnt IRL Non-linear reward function learning Generative adversarial imitation learning (GAIL) Adversarial inverse reinforcement learning (AIRL) Review of Generative Adversarial nets As bases for non-linear IRL Multi-Agent Reinforcement Learning Multi-agent Actor-Critic etc. Multi-Agent Inverse Reinforcement Learning MA-GAIL MA-AIRL AMA-GAIL

Applications

This Lecture

v Actor-Critic methods

§ A2C § A3C § Pathwise Derivative Policy Gradient

§ Advanced RL Techniques

§ Advanced techniques for DQN

• Multi-step DQN, Noisy net DQN
• Distributional DQN
• DQN for continuous action space

§ Sparse Reward

• Reward shaping, Curiosity module
• Curriculum learning, Hierarchical RL

§ Project #4 progress update

v Value-based v Policy-based

§ Actor-Critic

Model-free RL Algorithms

(Learned Value Function) (Learned Policy Function) (Learned both Value and Policy Functions)

v Value-based

vDeep Q-Learning (DQN) vDouble DQN vDueling DQN vPrioritized DQN

v Policy-based

§ Basic Policy Gradient Algorithm § REINFORCE § Vanilla, PPO, TRPO, PPO2

§ Actor-Critic

§ A2C § A3C § Pathwise Derivative Policy Gradient

Model-free RL Algorithms

(Learned Value Function) (Learned Policy Function) (Learned both Value and Policy Functions)

Basic Policy Gradient Algorithm

Update Model Data Collection

• nly used once

Unbiased estimator

Epsilon Greedy Boltzmann Exploration

(A2C algorithm) Value function Approximation Policy Gradient

Asynchronous Advantage Actor-Critic (A3C)

From A2C to A3C

Pathwise Derivative Policy Gradient

David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, Martin Riedmiller, “Deterministic Policy Gradient Algorithms”, ICML, 2014 Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, Daan Wierstra, “CONTINUOUS CONTROL WITH DEEP REINFORCEMENT LEARNING”, ICLR, 2016

Replaced ε-greedy policy with πnetwork.

v Value-based

vDeep Q-Learning (DQN) vDouble DQN vDueling DQN vPrioritized DQN

v Policy-based

§ Basic Policy Gradient Algorithm § REINFORCE § Vanilla, PPO, TRPO, PPO2

§ Actor-Critic

§ A2C § A3C § Pathwise Derivative Policy Gradient

Model-free RL Algorithms

(Learned Value Function) (Learned Policy Function) (Learned both Value and Policy Functions)

This Lecture

v Actor-Critic methods

§ A2C § A3C § Pathwise Derivative Policy Gradient

§ Advanced RL Techniques

§ Advanced techniques for DQN

• Multi-step DQN, Noisy net DQN
• Distributional DQN
• DQN for continuous action space

§ Sparse Reward

• Reward shaping, Curiosity module
• Curriculum learning, Hierarchical RL

Multi-step DQN

Experience Buffer

Balance between MC and TD

Noisy Net for DQN

v Noise on Action (Epsilon Greedy) v Noise on Parameters

https://arxiv.org/abs/1706.01905 https://arxiv.org/abs/1706.10295

The noise would NOT change in an episode. Inject noise into the parameters of Q- function at the beginning of each episode Add noise

Noisy Net

Random exploration Systematic exploration

Demo

https://blog.openai.com/better- exploration-with-parameter-noise/ Which one is action noise vs parameter noise？

Distributional Q-function

Different distributions can have the same values.

• 10

10

• 10

10

Distributional Q-function

s A network with 3 outputs A network with 15 outputs (each action has 5 bins) s

Demo

https://youtu.be/yFBwyPuO2Vg

Rainbow

https://arxiv.org/abs/1710.02298

Continuous Actions

Solution 1 Solution 2 Using gradient ascent to solve the

• ptimization problem.

See which action can obtain the largest Q value

P r

• s

a n d C

• n

s ? ? ?

Continuous Actions

Solution 3 Design a network to make the optimization easy. s vector matrix scalar

Continuous Actions

Solution 4 Don’t use Q-learning

Policy-based Value-based Learning an Actor Learning a Critic Actor + Critic

https://www.youtube.com/watch?v=ZhsEKTo7V04

This Lecture

v Actor-Critic methods

§ A2C § A3C § Pathwise Derivative Policy Gradient

§ Advanced RL Techniques

§ Advanced techniques for DQN

• Multi-step DQN, Noisy net DQN
• Distributional DQN
• DQN for continuous action space

§ Sparse Reward

• Reward shaping, Curiosity module
• Curriculum learning, Hierarchical RL

§ Project #4 progress update

Sparse Reward

Reward Shaping

Reward Shaping

Reward Shaping

VizDoom

https://openreview.net/forum?id=Hk3 mPK5gg&noteId=Hk3mPK5gg

Reward Shaping

https://openreview.net/pdf?id=Hk3mPK5gg

Get reward, when closer Need domain knowledge

Curiosity

https://arxiv.org/abs/1705.05363

Actor Env Env Actor Env

… …

Reward Reward updated updated

ICM = intrinsic curiosity module

ICM ICM

Intrinsic Curiosity Module

Network 1 diff

Encourage exploration Some states is hard to predict, but not important. Trivial events

Intrinsic Curiosity Module

Network 1 diff

Encourage exploration Some states is hard to predict, but not important. Trivial events

Intrinsic Curiosity Module

Network 1 Feature Ext Feature Ext Network 2 diff

Sparse Reward

Curriculum Learning

Curriculum Learning

v Starting from simple training examples, and

then becoming harder and harder.

VizDoom

Sparse Reward

Hierarchical Reinforcement Learning

https://arxiv.org/abs/1805.08180

Reinforcement Learning Single Agent Multiple Agents Inverse Reinforcement Learning

Tabular representation of reward Model-based control Model-free control (MC, SARSA, Q-Learning) Function representation of reward

• 1. Linear value function approx

(MC, SARSA, Q-Learning)

• 2. Value function approximation

(Deep Q-Learning, Double DQN, prioritized DQN, Dueling DQN)

• 3. Policy function approximation

(Policy gradient, PPO, TRPO)

• 4. Actor-Critic methods (A2C,

A3C, Pathwise Derivative PG) Advanced topics in RL （Sparse Rewards） Review of Deep Learning As bases for non-linear function approximation (used in 2-4). Linear reward function learning Imitation learning Apprenticeship learning Inverse reinforcement learning MaxEnt IRL MaxCausalEnt IRL MaxRelEnt IRL Non-linear reward function learning Generative adversarial imitation learning (GAIL) Adversarial inverse reinforcement learning (AIRL) Review of Generative Adversarial nets As bases for non-linear IRL Multi-Agent Reinforcement Learning Multi-agent Actor-Critic etc. Multi-Agent Inverse Reinforcement Learning MA-GAIL MA-AIRL AMA-GAIL

Applications

Questions?