Introduction to policy search in Reinforcement Learning Djalel - - PowerPoint PPT Presentation

Introduction to policy search in Reinforcement Learning

Djalel Benbouzid — Data:Lab Munich, Volkswagen Group June 27th 2018

quick outline

introduction and some reminders introduce a different way to RL, wrt. first lecture gradients-based methods gradients-free methods

machine

function

• utput

data

machine

data function

• utput 


(supervision)

machine learning “traditional” programming

• Supervised Learning
• Unsupervised Learning
• Reinforcement Learning

x

f(x)

2bservations Prediction

Machine Learning paradigms

source: Bengio et. al

what about deep learning?

it’s all about inferring functions
 and abstracting problems away

supervised learning: 
 function mapping, I/O unsupervised learning: learning densities, and/or representations reinforcement learning: learn a series of actions, a policy

π(s) 7! a1, a2, a3, . . .

p(x, z)

p(x)

f : X → Y

f ∼ p(y | x)

• r

Agent (policy)

action

at

state

st

Environment

new state

st+1

new reward

rt+1

reward

rt

the RL paradigm

max E "X

t

rt #

The goal Unknown

st+1 ∼ P(st+1 | st, at)

rt ∼ P(rt | st, at)

labels

unsupervised supervised

evaluative

immediate delayed

causality data

I.I.D.

changing & 
 correlated

what makes RL harder than SL?

exploration vs exploitation credit assignment learning stability

two main approaches for Reinforcement Learning

Value-based

π?(s) = arg max

a

Q?(s, a)

V (s) = Et[Rt | St = s]

Q(s, a) = Et[Rt | St = s, At = a]

• dynamic programming
• temporal-difference (TD) learning
• indirect method in some sense
• not for large / continuous action space


(until recently) Policy-based

πθ(s, a) = P(a | s, θ)

• stochastic policies
• handles continuous action space
• but… suffers from local optima 


and high variance

• ptimization problem

Loss function directly search in space J(θ)

θ

policy search (or optimization)

J(θ) =

Find that maximizes

θ

Objective function

J(θ) = Epθ(s1:t,a1:t) "X

t

r(st, at) #

Parametrized policy

a ∼ πθ(s) = P(a | s, θ)

• r

∼ | a = πθ(s) = f(a | s, θ)

Policy Gradient

policy gradient: general formulation

a ∼ πθ(s) = P(a | s, θ)

actions

st+1 ∼ p(st+1 | st, a)

transitions

pθ(τ) = p(s1)

T

Y

t=1

πθ(at | st)p(st+1 | st, at)

τ = (s1, a1, . . . , sT , aT )

trajectories

r(τ) = X

t

r(st, at)

cumulative reward

θ? = arg max

✓

E⌧∼pθ(⌧) "X

t

r(st, at) #

Objective function J(θ) = Eτ∼pθ(τ)[r(τ)]

= Z r(τ)pθ(τ) dτ

non-difgerentiable objective function!

➠ likelihood-ratio trick

rθJ(θ) = rθ Z r(τ)pθ(τ) dτ = Z r(τ)rθpθ(τ) dτ = Z r(τ)rθpθ(τ) ✓pθ(τ) pθ(τ) ◆ dτ = Z r(τ)pθ(τ) ✓rθpθ(τ) pθ(τ) ◆ dτ = Z r(τ)pθ(τ)rθ log pθ(τ) dτ = Eτ∼pθ(τ)[r(τ)rθ log pθ(τ)]

scale by the trajectory evaluation and follow the gradient of the trajectories logprop
 (in expectation) many names in the literature: score function estimator, likelihood-ratio trick, etc.

1 - interact with the environment (generate trajectories) 2 - compute their cumulative reward 3 - compute an estimate of loss gradient 4 - update the policy parameters

N

g ⇡ rθJ(θ)

θ ← θ + α × g

rθJ(θ) = Eτ∼pθ(τ) " T X

t=1

r(st, at)

T

X

t=1

rθ log πθ(at | st) # ⇡ 1 N

N

X

i=1

T X

t=1

r(s(i)

t , a(i) t ) T

X

t=1

rθ log πθ ⇣ a(i)

t

• s(i)

t

⌘!

rθJ(θ) = Eτ∼pθ(τ)[r(τ)rθ log pθ(τ)]

pθ(τ) = p(s1)

T

Y

t=1

πθ(at | st)p(st+1 | st, at) log pθ(τ) = log p(s1) +

T

X

t=1

πθ(at | st) + p(st+1 | st, at)

rθ

➠ the REINFORCE algorithm

Williams, Ronald J. "Simple statistical gradient-following algorithms for connectionist reinforcement learning." Reinforcement Learning. Springer, Boston, MA, 1992. Sutton, Richard S., et al. "Policy gradient methods for reinforcement learning with function approximation." Advances in neural information processing systems. 2000.

unbiased estimator

REINFORCE, A Monte-Carlo Policy-Gradient Method (episodic) Input: a differentiable policy parameterization π(a|s, θ), 8a 2 A, s 2 S, θ 2 Rd Initialize policy parameter θ Repeat forever: Generate an episode S0, A0, R1, . . . , ST1, AT1, RT , following π(·|·, θ) For each step of the episode t = 0, . . . , T 1: G return from step t θ θ + αγtGr

θ log π(At|St, θ)

source: Sutton & Barto

Gaussian policies Softmax policies

• fuen a neural network

rθ log πθ(at | st) = 1 2Σ−1(at µ(st))rθµ(st)

πθ(at | st) ∼ N(µ(st), Σ)

πθ(at | st) = ef(st,at) P

a0

t ef(st,a0 t)

rθ log πθ(at | st) = f(st, at) Eπθ[f(st, ·)]

1 - interact with the environment (generate trajectories) 2 - compute their cumulative reward 3 - compute an estimate of loss gradient 4 - update the policy parameters

N

g ⇡ rθJ(θ)

θ ← θ + α × g

rθJ(θ) = Eτ∼pθ(τ) " T X

t=1

r(st, at)

T

X

t=1

rθ log πθ(at | st) # ⇡ 1 N

N

X

i=1

T X

t=1

r(s(i)

t , a(i) t ) T

X

t=1

rθ log πθ ⇣ a(i)

t

• s(i)

t

⌘!

rθJ(θ) = Eτ∼pθ(τ)[r(τ)rθ log pθ(τ)]

pθ(τ) = p(s1)

T

Y

t=1

πθ(at | st)p(st+1 | st, at) log pθ(τ) = log p(s1) +

T

X

t=1

πθ(at | st) + p(st+1 | st, at)

rθ

high variance! unbiased estimator

➠ the REINFORCE algorithm

Williams, Ronald J. "Simple statistical gradient-following algorithms for connectionist reinforcement learning." Reinforcement Learning. Springer, Boston, MA, 1992. Sutton, Richard S., et al. "Policy gradient methods for reinforcement learning with function approximation." Advances in neural information processing systems. 2000.

reducing the variance with the temporal structure

rθJ(θ) = Eτ∼pθ(τ) " T X

t=1

rθ log πθ(at | st)

T

X

t0=t

r(st0 , at0 ) #

actions should only be influenced by future rewards

X X = Eτ∼pθ(τ) " T X

t=1

rθ log πθ(at | st) Qπθ(st, at) #

the return is a sample of the Q value function Rt =

T

X

t0=t

r(st0 , at0 ) ∼ Qπθ(st, at)

Rt

reducing the variance with a baseline

Eτ∼pθ(τ)[rθ log pθ(τ)B] = Z pθ(τ)rθ log pθ(τ)B dτ = Z pθ(τ)rθpθ(τ) pθ(τ) B dτ = Z rθpθ(τ)B dτ = rθ Z pθ(τ)B dτ = B rθ Z pθ(τ) dτ = B ⇥ 0

➠ still unbiased estimator

rθJ(θ) = Eτ∼pθ(τ) " T X

t=1

rθ log pθ(τ) T X

t0=t

r

• st0 , at0
• b(st)

!#

which baseline?

More substance in
 Greensmith, Evan, Peter L. Bartlett, and Jonathan Baxter. "Variance reduction techniques for gradient estimates in reinforcement learning." Journal of Machine Learning Research, 2004

b(st) = b ≈ 1 N X

t

r(st, at)

constant, or moving average

b(st) = ˆ Vν(st) = Et[Rt | St = s]

estimate the value function intuition: only increase probability of actions that improve over the average

let’s recap

θ? = arg max

✓

E⌧∼pθ(⌧) "X

t

r(st, at) #

rθJ(θ) =

X X = Eτ∼pθ(τ) " T X

t=1

rθ log πθ(at | st) Qπθ(st, at) #

rθJ(θ) = Eτ∼pθ(τ) " T X

t=1

rθ log πθ(at | st) Qπθ(st, at) V πθ(st) #

rθJ(θ) = Eτ∼pθ(τ) " T X

t=1

r(st, at)

T

X

t=1

rθ log πθ(at | st) # !

actor-critic algorithms

déjà-vu?

rθJ(θ) = Eτ∼pθ(τ) " T X

t=1

rθ log πθ(at | st) Qπθ(st, at) #

instead of sampling

• ne can approximate it

Qπθ(st, at)

ˆ Qω(st, at)

we now have two parameters to fit

• for the policy (the actor)
• for the value function (the critic)

θ ω value-based policy-based actor-critic

⇡ Eτ∼pθ(τ) " T X

t=1

rθ log πθ(at | st) ˆ Qω(st, at) #

Q-learning!

One-step Actor–Critic (episodic) Input: a differentiable policy parameterization π(a|s, θ), 8a 2 A, s 2 S, θ 2 Rd Input: a differentiable state-value parameterization ˆ v(s,w), 8s 2 S, w 2 Rm Parameters: step sizes α > 0, β > 0 Initialize policy parameter θ and state-value weights w Repeat forever: Initialize S (first state of episode) I 1 While S is not terminal: A ⇠ π(·|S, θ) Take action A, observe S0, R δ R + γ ˆ v(S0,w) ˆ v(S,w) (if S0 is terminal, then ˆ v(S0,w) . = 0) w w + βδr

wˆ

v(S,w) θ θ + αI δr

θ log π(A|S, θ)

I γI S S0

source: Sutton & Barto

actor-critic algorithms with a baseline

rθJ(θ) = Eτ∼pθ(τ) " T X

t=1

rθ log πθ(at | st) Qπθ(st, at) V πθ(st) #

t t − t

| {z }

Aπθ (st,at)

advantage function is an unbiased sample of the Advantage function

δπθ

t

= rt + γV πθ(st+1) − V πθ(st, )

TD error

δπθ

t

∼ Aπθ(st, at)

family of policy gradient algorithms

from David Silver’s online course

rθ log πθ(at | st) Rt vanilla policy gradient rθ log πθ(at | st) (Rt ˆ Vν) REINFORCE rθ log πθ(at | st) ˆ Qω(st, at) actor-critic rθ log πθ(at | st) ˆ Aω(st, at) advantage actor-critic rθ log πθ(at | st) δt TD actor-critic

example: Asynchronous Advantage Actor-Critic (A3C)

Algorithm S3 Asynchronous advantage actor-critic - pseudocode for each actor-learner thread.

// Assume global shared parameter vectors θ and θv and global shared counter T = 0 // Assume thread-specific parameter vectors θ0 and θ0

v

Initialize thread step counter t 1 repeat Reset gradients: dθ 0 and dθv 0. Synchronize thread-specific parameters θ0 = θ and θ0

v = θv

tstart = t Get state st repeat Perform at according to policy π(at|st; θ0) Receive reward rt and new state st+1 t t + 1 T T + 1 until terminal st or t tstart == tmax R = ⇢ for terminal st V (st, θ0

v)

for non-terminal st// Bootstrap from last state for i 2 {t 1, . . . , tstart} do R ri + γR Accumulate gradients wrt θ0: dθ dθ + rθ0 log π(ai|si; θ0)(R V (si; θ0

v))

Accumulate gradients wrt θ0

v: dθv dθv + ∂ (R V (si; θ0 v))2/∂θ0 v

end for Perform asynchronous update of θ using dθ and of θv using dθv. until T > Tmax

Mnih, Volodymyr, et al. "Asynchronous methods for deep reinforcement learning." International Conference on Machine Learning. 2016.

using the MuJoCo library

gradient-free methods

• approximate the gradient with numerical methods
• perturb the component of by a small step size
• sample (eg. from a standard Gaussian distribution)
• finite differences

∂J(θ) ∂θk ≈ J(θ + νδ) − J(θ − νδ) 2ν

kth

θ ν δ

Algorithm 1 Basic Random Search (BRS)

1: Hyperparameters: step-size α, number of directions sampled per iteration N, standard devi-

ation of the exploration noise ν

2: Initialize: θ0 = 0, and j = 0. 3: while ending condition not satisfied do 4:

Sample δ1, δ2, . . . , δN of the same size as θj, with i.i.d. standard normal entries.

5:

Collect 2N rollouts of horizon H and their corresponding rewards using the policies πj,k,+(x) = πθj+νδk(x) and πj,k,−(x) = πθj−νδk(x), with k ∈ {1, 2, . . . , N}.

6:

Make the update step: θj+1 = θj + α

N N

X

k=1

[r(πj,k,+) − r(πj,k,−)] δk .

7:

j ← j + 1.

8: end while

. 1803.07055-Simple random search provides a competitive approach to reinforcement learning

augmented random search

1: Hyperparameters: step-size α, number of directions sampled per iteration N, standard devi-

ation of the exploration noise ν, number of top-performing directions to use b (b < N is allowed

• nly for V1-t and V2-t)

2: Initialize: M0 = 0 ∈ Rp×n, µ0 = 0 ∈ Rn, and Σ0 = In ∈ Rn×n, j = 0. 3: while ending condition not satisfied do 4:

Sample δ1, δ2, . . . , δN in Rp×n with i.i.d. standard normal entries.

5:

Collect 2N rollouts of horizon H and their corresponding rewards using the 2N policies V1: ( πj,k,+(x) = (Mj + νδk)x πj,k,−(x) = (Mj − νδk)x V2: ( πj,k,+(x) = (Mj + νδk) diag (Σj)−1/

2 (x − µj)

πj,k,−(x) = (Mj − νδk) diag(Σj)−1/

2(x − µj)

for k ∈ {1, 2, . . . , N}.

6:

Sort the directions δk by max{r(πj,k,+), r(πj,k,−)}, denote by δ(k) the k-th largest direction, and by πj,(k),+ and πj,(k),− the corresponding policies.

7:

Make the update step: Mj+1 = Mj +

α bσR b

X

k=1

⇥ r(πj,(k),+) − r(πj,(k),−) ⇤ δ(k), where σR is the standard deviation of the 2b rewards used in the update step.

8:

V2 : Set µj+1, Σj+1 to be the mean and covariance of the 2NH(j + 1) states encountered from the start of training.2

9:

j ← j + 1

10: end while

. 1803.07055-Simple random search provides a competitive approach to reinforcement learning

augmented random search

cross-entropy method

• generate random trajectories according to
• rank them according to their performance (eg. sum of rewards)
• use the best performing ones to improve the sampling process

P(· | θ)

CEM: for iter i = 1, 2, … for populaRon member e = 1, 2, ... sample execute roll-outs under store endfor where indexes over top p % endfor

θ(e) ∼ Pµ(i)(θ)

πθ(e)

µ(i+1) = arg max

µ

X

¯ e

log Pµ(θ(¯

e))

¯ e

(θ(e), U(e))

source: Pieter Abbeel’s slides

bottomline of gradient-free methods

• easy to implement
• good to have aside, as a toolbox for comparison
• not always suitable for huge parameter vectors
• don’t bootstrap, so they need to know the full trajectory
• no particular assumptions, ie., blackbox functions, except fast evaluation of the rewards
• common sense states: when you have a gradient, exploit it! (do you agree?)

closing

model-based reinforcement learning efficient reinforcement learning exploration and intrinsic motivation hierarchical reinforcement learning

(some) research questions

going further

• Reinforcement Learning: An Introduction. Richard S. Sutton & Andrew G. Barto


http://incompleteideas.net/book/the-book-2nd.html (PDF freely available)

• Algorithms for Reinforcement Learning. Csaba Szepesvári


https://sites.ualberta.ca/~szepesva/RLBook.html (PDF freely available)

• Approximate Dynamic Programming: Solving the curses of dimensionality . Warren B. Powell


http://adp.princeton.edu/

• David Silver’s excellent course (on video) 


http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html

• Pieter Abbeel & John Schulman’s tutorial @NIPS2016 https://nips.cc/Conferences/2016/Schedule?

showEvent=6198

libraries & frameworks

• OpenAI Gym https://gym.openai.com/
• Keras-RL https://github.com/matthiasplappert/keras-rl
• TensorForce https://reinforce.io/blog/introduction-to-tensorforce/
• OpenAI Baselines https://github.com/openai/baselines
• Coach http://coach.nervanasys.com/
• Showed during the talk
• javascript library https://cs.stanford.edu/people/karpathy/reinforcejs/gridworld_dp.html

questions?