CMP784 DEEP LEARNING Lecture #12 Deep Reinforcement Learning - - PowerPoint PPT Presentation

Lecture #12 – Deep Reinforcement Learning

Aykut Erdem // Hacettepe University // Spring 2018

CMP784

DEEP LEARNING

DeepLoco by X. B. Peng, G. Berseth & M. van de Panne

Previously on CMP784

• Generative Adversarial Networks

(GANs)

• How do GANs work
• Conditional GAN
• Tips and Tricks
• Applications

Neural Face by Taehoon Kim

Lecture overview

• What is Reinforcement Learning?
• Components of a RL problem
• Markov Decision Processes
• Value-Based Deep RL
• Policy-Based Deep RL
• Model-Based Deep RL

Disclaimer: r: Much of the material and slides for this lecture were borrowed from

—John Schulman’s talk on “Deep Reinforcement Learning: Policy Gradients and Q-Learning” —David Silver’s tutorial on “Deep Reinforcement Learning” —Lex Fridman’s MIT 6.S094 Deep Learning for Self-Driving Cars class

What is Reinforcement Learning?

What is Reinforcement Learning?

Supervised Learning Unsupervised Learning Reinforcement Learning

Slide credit: Razvan Pascanu

What is Reinforcement Learning?

• Branch of machine learning concerned with taking sequences of

actions

• Usually described in terms of agent interacting with a previously

unknown environment, trying to maximize cumulative reward

Agent Environment action

• bservation, reward

Motor Control and Robotics

Robotics:

• Observations: camera images, joint angles
• Actions: joint torques
• Rewards: stay balanced, navigate to target locations, serve and protect humans

Business Operations

Inventory Management:

• Observations: current inventory levels
• Actions: number of units of each item to purchase
• Rewards: profit

Image Captioning

Hard Attention for Image Captioning:

• Observations: current image window
• Actions: where to look
• Rewards: classification

Games

A different kind of optimization problem (min-max) but still considered to be RL.

• Go (complete information, deterministic) – AlphaGo
• Backgammon (complete information, stochastic)

– TD-Gammon

• Stratego (incomplete information, deterministic)
• Poker (incomplete information, stochastic)

Matej Moravčík et al. DeepStack: Expert-level artificial intelligence in heads-up no-limit poker. Science 02 Mar 2017 David Silver, Aja Huang, et al. “Mastering the game of Go with deep neural networks and tree search”. In: Nature 529.7587 (2016), pp. 484–489. Gerald Tesauro. “Temporal difference learning and TD-Gammon”. In: Communications of the ACM 38.3 (1995), pp. 58–68. Game tree complexity = bd Brute force search intractable: 1. Search space is huge 2. “Impossible” for computers to evaluate who is winning

Why is Go hard for computers to play?

How does RL relate to Supervised Learning?

• Supervised learning:

— Environment samples input-output pair — Agent predicts — Agent receives loss — Environment asks agent a question, and then tells her the right answer

pair (xt, yt) ∼ ⇢

ts ˆ yt = f (xt)

ss `(yt, ˆ yt)

How does RL relate to Supervised Learning?

• Reinforcement learning:

− Environment samples input-output pair

§ Input depends on your previous actions!

− Agent predicts − Agent receives cost where P a probability distribution unknown to the agent.

ts ˆ yt = f (xt)

input xt ∼ P(xt | xt−1, yt−1)

st ct ∼ P(ct | xt, ˆ yt) w

Reinforcement Learning in a nutshell

RL is a general-purpose framework for decision-making

• RL is for an ag

agen ent with the capacity to ac act

• Each ac

action influences the agent’s future st state

• Success is measured by a scalar re

reward rd signal

• Goal: se

select ct act ctions

• ns to
• maximize fut

uture ure re reward rd

Deep Learning in a nutshell

RL is a general-purpose framework for decision-making

• Given an obj
• bject

ctive

• Learn re

repre rese sentatio ion that is required to achieve objective

• Directly from ra

raw in inputs

• Using minimal domain knowledge

Deep Reinforcement Learning: AI = RL + DL

We seek a single agent which can solve any human-level task

• RL defines the objective
• DL gives the mechanism
• RL + DL = general intelligence
• Examples:

− Pla Play games: Atari, poker, Go, ... − Ex Explor plore worlds: 3D worlds, Labyrinth, ... − Co Control physical systems: manipulate, walk, swim, ... − Interact ct with users: recommend, optimize, personalize, ...

Agent and Environment

• At each step t the agent:
• Executes action at
• Receives observation ot
• Receives scalar reward rt
• The environment:
• Receives action at
• Emits observations ot+1
• Emits scalar reward rt+1

• bservation

reward action at rt

• t

Example Reinforcement Learning Problem

• An agent operates in an environment: At

Atari Breakout

• ut
• An agent has the capacity to ac

act

• Each action influences the agent’s fu

futu ture sta tate te

• Success is measured by a re

reward rd signal

• Go

Goal is to select actions to maximize future reward

• Each action influences the agent’s
• Each action influences the agent’s

State

• Experience is a sequence of observations, actions, rewards
• 1, r1, a1, ..., at-1, ot , rt
• The state is a summary of experience

st = f(o1,r1,a1,...,at-1,ot,rt)

• In a fully observed environment

st =f(ot)

Major Components of an RL Agent

• An RL agent may include one or more of these components:

− Policy: Agent’s behavior function − Value function: How good is each state and/or action − Model: Agent’s representation of the environment

Policy

• A policy is the agent’s behavior
• It is a map from state to action:

− Deterministic policy: − Stochastic policy:

: a = π(s) P

: π(a|s) = P [a|s]

Value Function

• A value function is a prediction of future reward

− “How much reward will I get from action a in state s?”

• Q-value function gives expected total reward

− from state s and action a − under policy − with discount factor

• Value functions decompose into a Bellman equation

y π

• r γ

⇥

Qπ(s, a) = E ⇥ rt+1 + γrt+2 + γ2rt+3 + ... | s, a ⇤

Qπ(s, a) = Es0,a0 ⇥ r + γQπ(s0, a0) | s, a ⇤

Q⇤(s, a) = Es0  r + γ max

a0

Q⇤(s0, a0) | s, a

• Optimal Value Functions
• An optimal value function is the maximum achievable value
• Once we have Q* we can act optimally,
• Optimal value maximizes over all decisions. Informally:
• Formally, optimal values decompose into a Bellman equation

Q⇤(s, a) = max

π

Qπ(s, a) = Qπ⇤(s, a)

π⇤(s) = argmax

a

Q⇤(s, a)

Q⇤(s, a) = rt+1 + γ max

at+1 rt+2 + γ2 max at+2 rt+3 + ...

= rt+1 + γ max

at+1 Q⇤(st+1, at+1)

Model

• bservation

reward action at rt

• t

Model

• Model is learnt from experience
• Acts as proxy for environment
• Planner interacts with model
• e.g. using lookahead search

Approaches To Reinforcement Learning

• Value-based RL

− Estimate the optimal value function Q*(s,a) − This is the maximum value achievable under any policy

• Policy-based RL

− Search directly for the optimal policy − This is the policy achieving maximum future reward

• Model-based RL

− Build a model of the environment − Plan (e.g. by lookahead) using model

y π∗

Deep Reinforcement Learning

• Use deep neural networks to represent

− Value function − Policy − Model

• Optimize loss function by stochastic gradient descent

Value-Based Deep RL

Q-Networks

• Represent value function by Q-network with weights w

Q(s, a, w) ≈ Q∗(s, a)

s s a Q(s,a,w) Q(s,a1,w) Q(s,am,w) … w w

l = ✓ r + γ max

a0

Q(s0, a0, w) − Q(s, a, w) ◆2

Q⇤(s, a) = Es0  r + γ max

a0

Q⇤(s0, a0) | s, a

• Q-Learning
• Optimal Q-values should obey Bellman equation
• Treat right-handside

as a target

• Minimize MSE loss by stochastic gradient descent
• Converges to Q* using table lookup representation
• But diverges using neural networks due to:

− Correlations between samples − Non-stationary targets

e r + γ max

a0

Q(s0, a0, w)

Deep Q-Networks (DQN): Experience Replay

• To remove correlations, build data-set from agent’s own experience
• Sample experiences from data-set and apply update
• To deal with non-stationarity, target parameters w — are held fixed

s1, a1, r2, s2 s2, a2, r3, s3 → s, a, r, s0 s3, a3, r4, s4 ... st, at, rt+1, st+1 → st, at, rt+1, st+1

l = ✓ r + γ max

a0

Q(s0, a0, w) − Q(s, a, w) ◆2

Deep Reinforcement Learning in Atari

state reward action at rt st

Human level control through deep reinforcement learning, V. Mnih et al. Nature 518:529-533, 2015.

DQN in Atari

• End-to-end learning of values Q(s,a) from pixels s
• Input state s is stack of raw pixels from last 4 frames
• Output is Q(s,a) for 18 joystick/button positions
• Reward is change in score for that step

Network architecture and hyperparameters fixed across all games

DQN Results in Atari

V

Two-dimensional t-SNE embedding of the representations in the last hidden layer assigned by DQN to game states experienced while playing Space Invaders.

Improvements since Nature DQN

• Do

Double DQ DQN: Remove upward bias caused by

− Current Q-network w is used to select actions − Older Q-network w — is used to evaluate actions

• Pri

riori riti tized re replay: Weight experience according to surprise

− Store experience in priority queue according to DQN error

• Du

Duelling netw twork rk: Split Q-network into two channels

− Action-independent value function V(s,v) − Action-dependent advantage function A(s, a, w)

Combined algorithm: 3x mean Atari score vs Nature DQN

y max

a

Q(s, a, w)

l = ✓ r + γQ(s0, argmax

a0

Q(s0, a0, w), w) − Q(s, a, w) ◆2

• r + γ max

a0

Q(s0, a0, w) − Q(s, a, w)

• Q(s, a) = V (s, v) + A(s, a, w)

Gorila

• 10x faster than Nature DQN on 38 out of 49 Atari games
• Applied to recommender systems within Google

Asynchronous Reinforcement Learning

• Exploits multithreading of standard CPU
• Execute many instances of agent in parallel
• Network parameters shared between threads
• Parallelism decorrelates data

− Viable alternative to experience replay

• Similar speedup to Gorila - on a single machine!

Policy-Based Deep RL

Deep Policy Networks

• Represent policy by deep network with weights u
• Define objective function as total discounted reward
• Optimize objective end-to-end by SGD
• i.e. Adjust policy parameters u to achieve more reward

a = π(a|s, u) or a = π(s, u)

L(u) = E ⇥ r1 + γr2 + γ2r3 + ... | π(·, u) ⇤

Policy Gradients

How to make high-value actions more likely:

• The gradient of a stochastic policy is given by
• The gradient of a deterministic policy is given by

if a is continuous and Q is differentiable

∂L(u) ∂u = E ∂log π(a|s, u) ∂u Qπ(s, a)

• cy π(a|s, u)

∂L(u) ∂u = E ∂Qπ(s, a) ∂a ∂a ∂u

• y a = π(s)

Actor-Critic Algorithm

• Estimate value function
• Update policy parameters u by stochastic gradient ascent
• r

• n Q(s, a, w) ≈ Qπ(s, a)

∂l ∂u = ∂log π(a|s, u) ∂u Q(s, a, w)

∂l ∂u = ∂Q(s, a, w) ∂a ∂a ∂u

Asynchronous Advantage Actor-Critic (A3C)

• Estimate state-value function
• Q-value estimated by an n-step sample
• Actor is updated towards target
• Critic is updated to minimize MSE w.r.t. target
• 4x mean Atari score vs Nature DQN

V (s, v) ≈ E [rt+1 + γrt+2 + ...|s]

qt = rt+1 + γrt+2... + γn−1rt+n + γnV (st+n, v) ∂lu ∂u = ∂log π(at|st, u) ∂u (qt − V (st, v))

lv = (qt − V (st, v))2

Deep Reinforcement Learning in Labyrinth

A3C in Labyrinth

st st+1 st-1

• t-1
• t
• t+1

π(a|st-1) π(a|st) π(a|st+1) V(st-1) V(st) V(st-1)

y π(a|st)

s π(a|s)

• End-to-end learning of

softmax policy from pixels

• Observations ot are raw

pixels from current frame

• State st =f (o1,...,ot ) is a

recurrent neural network (LSTM)

• Outputs both value V(s) and

softmax over actions

• Task is to collect apples (+1

reward) and escape (+10 reward)

Deep Reinforcement Learning with Continuous Actions

How can we deal with high-dimensional continuous action spaces?

• Can’t easily compute

− Actor-critic algorithms learn without max

• Q-values are differentiable w.r.t a
• Deterministic policy gradients exploit knowledge of

e max

a

Q(s, a)

• f ∂Q

∂a

Deep DPG

DPG is the continuous analogue of DQN

• Experience replay: build data-set from agent’s experience
• Critic estimates value of current policy by DQN

To deal with non-stationarity, targets u — ,w — are held fixed

• Actor updates policy in direction that improves Q
• In other words critic provides loss function for actor

lw = ✓ r + γQ(s0, π(s0, u), w) − Q(s, a, w) ◆2

∂lu ∂u = ∂Q(s, a, w) ∂a ∂a ∂u

DPG in Simulated Physics

• Physics domains are simulated in

MuJoCo

• End-to-end learning of control

policy from raw pixels s

• Input state s is stack of raw pixels

from last 4 frames

• Two separate convnets are used

for Q and

• Policy is adjusted in direction

that most improves Q

Q(s,a)

π(s) a

and π and π

A3C in Simulated Physics Demo

• Asynchronous RL is viable alternative to experience replay
• Train a hierarchical, recurrent locomotion controller
• Retrain controller on more challenging tasks

Model-Based Deep RL

Learning Models of the Environment

• Challenging to plan due to compounding errors

− Errors in the transition model compound over the trajectory − Planning trajectories differ from executed trajectories − At end of long, unusual trajectory, rewards are totally wrong

Learning Models of the Environment

• Challenging to plan due to compounding errors

− Errors in the transition model compound over the trajectory − Planning trajectories differ from executed trajectories − At end of long, unusual trajectory, rewards are totally wrong

Case Study: AlphaGo

Why is Go hard for computers to play?

Game tree complexity = bd Brute force search intractable:

• 1. Search space is huge
• 2. “Impossible” for computers

to evaluate who is winning

Game tree complexity = bd Brute force search intractable: 1. Search space is huge 2. “Impossible” for computers to evaluate who is winning

Why is Go hard for computers to play?

Convolutional Neural Network

Convolutional neural network

Value Network

Position Move probabilities

Policy Network

Policy network

Move probabilities Position

s p (a|s)

• Position

Move probabilities

Neural Network Training Pipeline

Neural network training pipeline

Human expert positions Supervised Learning policy network Self-play data Value network Reinforcement Learning policy network

Supervised Learning of Policy Networks

• Pol

Policy cy ne networ

• rk:

k: 12 layer convolutional neural network

• Tr

Traini ning ng da data: 30M positions from human expert games (KGS 5+ dan)

• Tr

Traini ning ng algor gorithm hm: maximize likelihood by SGD

• Tr

Traini ning ng time: 4 weeks on 50 GPUs using Google Cloud

• Res

Results ts: 57% accuracy on held out test data (state-of-the art was 44%)

Policy network: 12 layer convolutional neural network Training data: games of self-play between policy network Training algorithm: maximise wins z by policy gradient reinforcement learning Training time: 1 week on 50 GPUs using Google Cloud Results: 80% vs supervised learning. Raw network ~3 amateur dan.

Reinforcement learning of policy networks

Policy network: 12 layer convolutional neural network Training data: 30M positions from human expert games (KGS 5+ dan) Training algorithm: maximise likelihood by stochastic gradient descent Training time: 4 weeks on 50 GPUs using Google Cloud Results: 57% accuracy on held out test data (state-of-the art was 44%)

Supervised learning of policy networks

Reinforcement Learning of Policy Networks

• Pol

Policy cy ne networ

• rk:

k: 12 layer convolutional neural network

• Tr

Traini ning ng da data: games of self-play between policy network

• Tr

Traini ning ng algor gorithm hm: maximize wins z by policy gradient reinforcement learning

• Tr

Traini ning ng time: 1 week on 50 GPUs using Google Cloud

• Res

Results ts: 80% vs supervised learning. Raw network ~3 amateur dan.

Policy network: 12 layer convolutional neural network Training data: games of self-play between policy network Training algorithm: maximise wins z by policy gradient reinforcement learning Training time: 1 week on 50 GPUs using Google Cloud Results: 80% vs supervised learning. Raw network ~3 amateur dan.

Reinforcement learning of policy networks

Policy network: 12 layer convolutional neural network Training data: games of self-play between policy network Training algorithm: maximise wins z by policy gradient reinforcement learning Training time: 1 week on 50 GPUs using Google Cloud Results: 80% vs supervised learning. Raw network ~3 amateur dan.

Reinforcement learning of policy networks

Reinforcement Learning of Value Networks

• Val

Value e netw etwork: 12 layer convolutional neural network

• Tr

Traini ning ng da data: 30 million games of self-play

• Tr

Traini ning ng algor gorithm hm: minimize MSE by stochastic gradient descent

• Tr

Traini ning ng time: 1 week on 50 GPUs using Google Cloud

• Res

Results ts: First strong position evaluation function - previously thought impossible

Value network: 12 layer convolutional neural network Training data: 30 million games of self-play Training algorithm: minimise MSE by stochastic gradient descent Training time: 1 week on 50 GPUs using Google Cloud Results: First strong position evaluation function - previously thought impossible

Reinforcement learning of value networks

Value network: 12 layer convolutional neural network Training data: 30 million games of self-play Training algorithm: minimise MSE by stochastic gradient descent Training time: 1 week on 50 GPUs using Google Cloud Results: First strong position evaluation function - previously thought impossible

Reinforcement learning of value networks

Exhaustive Search

Exhaustive search

Reducing Depth with Value Network

Reducing Breadth with Policy Network

Evaluating AlphaGo Against Computers

Evaluating AlphaGo against computers

Zen Crazy Stone 3000 2500 2000 1500 1000 500 3500 AlphaGo (Nature v13) Pachi Fuego Gnu Go 4000 4500 AlphaGo (Seoul v18)

9p 7p 5p 3p 1p 9d 7d 5d 3d 1d 1k 3k 5k 7k

Professional dan (p) Amateur dan (d) Beginner kyu (k)

AlphaGo (Mar 2016) AlphaGo (Oct 2015) Crazy Stone and Zen Lee Sedol (9p) Top player of past decade Fan Hui (2p) 3-times reigning Euro Champion Amateur humans Computer Programs Calibration Human Players Beats Beats Nature match KGS DeepMind challenge match Beats Beats 5-0 4-1