CS 4803 / 7643: Deep Learning Topics: Policy Gradients Actor - - PowerPoint PPT Presentation

CS 4803 / 7643: Deep Learning

Ashwin Kalyan Georgia Tech

Topics:

– Policy Gradients – Actor Critic

2

Topics we’ll cover

• Overview of RL
• RL vs other forms of learning
• RL “API”
• Applications
• Framework: Markov Decision Processes (MDP’s)
• Definitions and notations
• Policies and Value Functions
• Solving MDP’s
• Value Iteration (recap)
• Q-Value Iteration (new)
• Policy Iteration
• Reinforcement learning
• Value-based RL (Q-learning, Deep-Q Learning)
• Policy-based RL (Policy gradients)
• Actor-Critic

3

• Markov Decision Processes (MDP):
• States:
• Actions:
• Rewards:
• Transition Function:
• Discount Factor:

Recap: MDPs

S

A

R(s, a, s0)

T(s, a, s0) = p(s0|s, a)

γ

4

Recap: Optimal Value Function

The optimal Q-value function at state s and action a, is the expected cumulative reward from taking action a in state s and acting optimally thereafter

5

Recap: Optimal Value Function

The optimal Q-value function at state s and action a, is the expected cumulative reward from taking action a in state s and acting optimally thereafter Optimal policy:

Recap: Learning Based Methods

• Typically, we don’t know the environment
• unknown, how actions affect the environment.
• unknown, what/when are the good actions?

6

Recap: Learning Based Methods

• Typically, we don’t know the environment
• unknown, how actions affect the environment.
• unknown, what/when are the good actions?
• But, we can learn by trial and error.
• Gather experience (data) by performing actions.
• Approximate unknown quantities from data.

7

• Collect a dataset
• Loss for a single data point:
• Act according optimally according to the learnt Q function:

Recap: Deep Q-Learning

8

Target Q-Value Predicted Q-Value

π(s) = arg max

a∈A Q(s, a)

Pick action with best Q value

9

Transition function and reward function

Getting to the optimal policy

T

R

Use value / policy iteration known Obtain “optimal” policy

10

Transition function and reward function

Getting to the optimal policy

T

R

Use value / policy iteration known Estimate Q values From data Obtain “optimal” policy

Previous class: Q - learning

unknown

11

Transition function and reward function

Getting to the optimal policy

T

R

Use value / policy iteration known Obtain “optimal” policy Estimate and from data

T

R

Estimate Q values From data unknown

Homework!

12

Transition function and reward function

Getting to the optimal policy

T

R

Use value / policy iteration known Obtain “optimal” policy Estimate and from data

T

R

Estimate Q values From data unknown unknown

This class!

13

• Class of policies defined by parameters
• Eg: can be parameters of linear transformation, deep network, etc.

Learning the optimal policy

πθ(a|s) : S → A

θ

θ

14

• Class of policies defined by parameters
• Eg: can be parameters of linear transformation, deep network, etc.
• Want to maximize:
• In other words,

Learning the optimal policy

πθ(a|s) : S → A

θ

θ

J(π) = E " T X

t=1

R(st, at) #

π∗ = arg max

π:S→A E

" T X

t=1

R(st, at) #

θ∗ = arg max

θ

E " T X

t=1

R(st, at) #

15

• Class of policies defined by parameters
• Eg: can be parameters of linear transformation, deep network, etc.
• Want to maximize:
• In other words,

Learning the optimal policy

πθ(a|s) : S → A

θ

θ

J(π) = E " T X

t=1

R(st, at) #

π∗ = arg max

π:S→A E

" T X

t=1

R(st, at) #

θ∗ = arg max

θ

E " T X

t=1

R(st, at) #

16

Learning the optimal policy

• Slightly rewriting the notation:
• Let , the trajectory

τ = (s0, a0, . . . sT , aT )

pθ(τ) = pθ(s0, a0, . . . sT , aT )

=

T

Y

t=0

pθ(at | st) · p(st+1 | st, at)

arg max

θ

Eτ∼pθ(τ) [R(τ)]

17

Learning the optimal policy

= Eat∼π(·|st),st+1∼p(·|st,at) " T X

t=0

R(st, at) #

Sample a few trajectories by acting according to πθ

J(θ) = Eτ∼pθ(τ) [R(τ)]

≈ 1 N

N

X

i=1 T

X

t=1

r(si

t, ai t)

{τi}N

i=1

18

• 1. Sample trajectories by acting according to
• 2. Compute policy gradient as
• 3. Update policy

REINFORCE

πθ

τi = {s1, a1, . . . sT , aT }i

θ θ + αrθJ(θ)

Run the policy and sample trajectories Compute policy gradient Update policy Slide credit: Sergey Levine

rθJ(θ) ⇡ X

i

" T X

t=1

rθ log πθ(ai

t | si t) · T

X

t=1

R(si

t | ai t)

#

19

= rθ Z πθ(τ)R(τ)dτ

Policy Gradients

rθJ(θ) = rθEτ∼pθ(τ)[R(τ)]

Expand expectation

20

= rθ Z πθ(τ)R(τ)dτ

Policy Gradients

rθJ(θ) = rθEτ∼pθ(τ)[R(τ)]

= Z rθπθ(τ)R(τ)dτ

Expand expectation Exchange integration and expectation

21

= rθ Z πθ(τ)R(τ)dτ

Policy Gradients

rθJ(θ) = rθEτ∼pθ(τ)[R(τ)]

= Z rθπθ(τ)R(τ)dτ

= Z rθπθ(τ) · πθ(τ) πθ(τ) · R(τ)dτ

Expand expectation Exchange integration and expectation

22

= rθ Z πθ(τ)R(τ)dτ

Policy Gradients

rθJ(θ) = rθEτ∼pθ(τ)[R(τ)]

= Z rθπθ(τ)R(τ)dτ

= Z rθπθ(τ) · πθ(τ) πθ(τ) · R(τ)dτ

Expand expectation Exchange integration and expectation

rθ log π(τ) = rθπ(τ) π(τ)

23

= rθ Z πθ(τ)R(τ)dτ

Policy Gradients

rθJ(θ) = rθEτ∼pθ(τ)[R(τ)]

= Z rθπθ(τ)R(τ)dτ

= Z rθπθ(τ) · πθ(τ) πθ(τ) · R(τ)dτ

= Z πθ(τ)rθ log πθ(τ)R(τ)dτ

Expand expectation Exchange integration and expectation

rθ log π(τ) = rθπ(τ) π(τ)

24

= rθ Z πθ(τ)R(τ)dτ

Policy Gradients

rθJ(θ) = rθEτ∼pθ(τ)[R(τ)]

= Z rθπθ(τ)R(τ)dτ

= Z rθπθ(τ) · πθ(τ) πθ(τ) · R(τ)dτ

= Z πθ(τ)rθ log πθ(τ)R(τ)dτ

= Eτ∼pθ(τ)[rθ log πθ(τ)R(τ)]

Expand expectation Exchange integration and expectation

rθ log π(τ) = rθπ(τ) π(τ)

25

Policy Gradients

rθJ(θ) = Eτ∼pθ(τ)[rθ log πθ(τ)R(τ)]

rθ " log p(s0) +

T

X

t=1

log πθ(at|st) +

T

X

t=1

log p(st+1 | st, at) #

pθ(τ) = pθ(s0, a0, . . . sT , aT )

=

T

Y

t=0

pθ(at | st) · p(st+1 | st, at)

26

Policy Gradients

rθJ(θ) = Eτ∼pθ(τ)[rθ log πθ(τ)R(τ)]

rθ " log p(s0) +

T

X

t=1

log πθ(at|st) +

T

X

t=1

log p(st+1 | st, at) #

Doesn’t depend on Transition probabilities!

27

Policy Gradients

rθJ(θ) = Eτ∼pθ(τ)[rθ log πθ(τ)R(τ)]

rθ " log p(s0) +

T

X

t=1

log πθ(at|st) +

T

X

t=1

log p(st+1 | st, at) #

= Eτ∼pθ(τ) " T X

t=1

rθ log πθ(at|st) ·

T

X

t=1

R(st, at) #

28

Policy Gradients

rθJ(θ) = Eτ∼pθ(τ)[rθ log πθ(τ)R(τ)]

rθ " log p(s0) +

T

X

t=1

log πθ(at|st) +

T

X

t=1

log p(st+1 | st, at) #

= Eτ∼pθ(τ) " T X

t=1

rθ log πθ(at|st) ·

T

X

t=1

R(st, at) #

29

• 1. Sample trajectories by acting according to
• 2. Compute policy gradient as
• 3. Update policy

REINFORCE

πθ

τi = {s1, a1, . . . sT , aT }i

θ θ + αrθJ(θ)

Run the policy and sample trajectories Compute policy gradient Update policy Slide credit: Sergey Levine

rθJ(θ) ⇡ X

i

" T X

t=1

rθ log πθ(ai

t | si t) · T

X

t=1

R(si

t | ai t)

#

Pong from pixels

30

Image Credit: http://karpathy.github.io/2016/05/31/rl/

Pong from pixels

31

Image Credit: http://karpathy.github.io/2016/05/31/rl/

Pong from pixels

32

Image Credit: http://karpathy.github.io/2016/05/31/rl/

Intuition

(C) Dhruv Batra 33

34

Policy Gradients

rθJ(θ) = Eτ∼pθ(τ)[rθ log πθ(τ)R(τ)]

rθ " log p(s0) +

T

X

t=1

log πθ(at|st) +

T

X

t=1

log p(st+1 | st, at) #

= Eτ∼pθ(τ) " T X

t=1

rθ log πθ(at|st) ·

T

X

t=1

R(st, at) #

Formalizes notion of “trial and error”:

• If reward is high, probability of actions seen is increased
• If reward is low, probability of actions seen is reduced

35

Issues with Policy Gradients

• Credit assignment is hard!
• Which specific action led to increase in reward
• Suffers from high variance à leading to unstable training

36

Issues with Policy Gradients

• Credit assignment is hard!
• Which specific action led to increase in reward
• Suffers from high variance à leading to unstable training
• How to reduce the variance?
• Subtract a constant from the reward!

rθJ(θ) = Eτ∼pθ(τ) " T X

t=1

rθ log πθ(at|st) ·

T

X

t=1

R(st, at) b #

37

Issues with Policy Gradients

• Credit assignment is hard!
• Which specific action led to increase in reward
• Suffers from high variance à leading to unstable training
• How to reduce the variance?
• Subtract a constant from the reward!
• Why does it work?
• What is the best choice of b?

rθJ(θ) = Eτ∼pθ(τ) " T X

t=1

rθ log πθ(at|st) ·

T

X

t=1

R(st, at) b #

Homework!

38

Taking a step back

rθJ(θ) = Eτ∼pθ(τ) " T X

t=1

rθ log πθ(at|st) ·

T

X

t=1

R(st, at) #

Policy Evaluation (Recall Policy iteration)

• REINFORCE: Evaluate and update policy based on Monte-Carlo estimates of

the total reward – very noisy!

• Other ways of policy evaluation?
• If we had the Q function, we could have used it!

39

• Learn both policy and Q function
• Use the “actor” to sample trajectories
• Use the Q function to “evaluate” or “critic” the policy

Actor-Critic

40

• Learn both policy and Q function
• Use the “actor” to sample trajectories
• Use the Q function to “evaluate” or “critic” the policy
• REINFORCE:
• Actor-critic:

Actor-Critic

rθJ(πθ) = Ea∼πθ [rθ log πθ(a|s)R(s, a)]

rθJ(πθ) = Ea∼πθ [rθ log πθ(a|s)Qπθ(s, a)]

41

• Learn both policy and Q function
• Use the “actor” to sample trajectories
• Use the Q function to “evaluate” or “critic” the policy
• REINFORCE:
• Actor-critic:
• Q function is unknown too! Update using

Actor-Critic

rθJ(πθ) = Ea∼πθ [rθ log πθ(a|s)R(s, a)]

rθJ(πθ) = Ea∼πθ [rθ log πθ(a|s)Qπθ(s, a)]

R(s, a)

42

• Initialize s, (policy network) and (Q network)

Actor-Critic

θ

β

43

• Initialize s, (policy network) and (Q network)
• sample action

Actor-Critic

θ

β

a ∼ πθ(·|s)

44

• Initialize s, (policy network) and (Q network)
• sample action
• For each step:
• Sample reward and next state

Actor-Critic

θ

R(s, a)

s0 ∼ p(s0|s, a)

β

a ∼ πθ(·|s)

45

• Initialize s, (policy network) and (Q network)
• sample action
• For each step:
• Sample reward and next state
• evaluate “actor” using “critic”

Actor-Critic

θ

R(s, a)

s0 ∼ p(s0|s, a)

Qβ(s, a)

β

a ∼ πθ(·|s)

46

• Initialize s, (policy network) and (Q network)
• sample action
• For each step:
• Sample reward and next state
• evaluate “actor” using “critic” and update policy:

Actor-Critic

θ

R(s, a)

s0 ∼ p(s0|s, a)

Qβ(s, a)

β

a ∼ πθ(·|s)

θ θ + αrθ log πθ(a | s)Qβ(s, a)

47

• Initialize s, (policy network) and (Q network)
• sample action
• For each step:
• Sample reward and next state
• evaluate “actor” using “critic” and update policy:
• Update “critic”:
• Recall Q-learning

Actor-Critic

θ

R(s, a)

s0 ∼ p(s0|s, a)

Qβ(s, a)

β

a ∼ πθ(·|s)

θ θ + αrθ log πθ(a | s)Qβ(s, a)

48

• Initialize s, (policy network) and (Q network)
• sample action
• For each step:
• Sample reward and next state
• evaluate “actor” using “critic” and update policy:
• Update “critic”:
• Recall Q-learning
• Update Accordingly
• Actor-Critic

θ

R(s, a)

s0 ∼ p(s0|s, a)

Qβ(s, a)

β

a ∼ πθ(·|s)

β

a ← a0, s ← s0

θ θ + αrθ log πθ(a | s)Qβ(s, a)

49

Actor-critic

rθJ(πθ) = Ea∼πθ [rθ log πθ(a|s)R(s, a)]

rθJ(πθ) = Ea∼πθ [rθ log πθ(a|s)Qπθ(s, a)]

• In general, replacing the policy evaluation or the “critic” leads to

different flavors of the actor-critic

• REINFORCE:
• Q – Actor Critic

50

Actor-critic

rθJ(πθ) = Ea∼πθ [rθ log πθ(a|s)R(s, a)]

rθJ(πθ) = Ea∼πθ [rθ log πθ(a|s)Qπθ(s, a)]

• In general, replacing the policy evaluation or the “critic” leads to

different flavors of the actor-critic

• REINFORCE:
• Q – Actor Critic
• Advantage Actor Critic:

rθJ(πθ) = Ea∼πθ [rθ log πθ(a|s)Aπθ(s, a)]

= Qπθ(s, a) − V πθ(s)

“how much better is an action than expected?

51

• Policy Learning:
• Policy gradients
• REINFORCE
• Reducing Variance (Homework!)
• Actor-Critic:
• Other ways of performing “policy evaluation”
• Variants of Actor-critic

Summary