CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy - - PowerPoint PPT Presentation

CS885 Reinforcement Learning Module 2: June 6, 2020

Maximum Entropy Reinforcement Learning Haarnoja, Tang et al. (2017) Reinforcement Learning with Deep Energy Based Policies, ICML. Haarnoja, Zhou et al. (2018) Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, ICML.

CS885 Spring 2020 Pascal Poupart 1 University of Waterloo

CS885 Spring 2020 Pascal Poupart 2

Maximum Entropy RL

• Why do several implementations of important RL

baselines (e.g., A2C, PPO) add an entropy regularizer?

• Why is maximizing entropy desirable in RL?
• What is the Soft Actor Critic algorithm?

University of Waterloo

Reinforcement Learning

Deterministic Policies

• There always exists an
• ptimal deterministic policy
• Search space is smaller for

deterministic than stochastic policies

• Practitioners prefer

deterministic policies

Stochastic Policies

• Search space is continuous

for stochastic policies (helps with gradient descent)

• More robust (less likely to
• verfit)
• Naturally incorporate

exploration

• Facilitate transfer learning
• Mitigate local optima

University of Waterloo CS885 Spring 2020 Pascal Poupart 3

Encouraging Stochasticity

Standard MDP

• States: 𝑇
• Actions: 𝐵
• Reward: 𝑆(𝑡, 𝑏)
• Transition: Pr(𝑡!|𝑡, 𝑏)
• Discount: 𝛿

Soft MDP

• States: 𝑇
• Actions: 𝐵
• Reward: 𝑆 𝑡, 𝑏 + 𝜇𝐼 𝜌 ⋅ 𝑡
• Transition: Pr(𝑡!|𝑡, 𝑏)
• Discount: 𝛿

University of Waterloo CS885 Spring 2020 Pascal Poupart 4

CS885 Spring 2020 Pascal Poupart 5

Entropy

• Entropy: measure of uncertainty

– Information theory: expected #

• f bits needed to communicate

the result of a sample

𝐼 𝑞 = − ∑! 𝑞 𝑦 log 𝑞(𝑦)

University of Waterloo

𝐼(𝑞) 𝑞(𝑦)

CS885 Spring 2020 Pascal Poupart 6

Optimal Policy

• Standard MDP

𝜌∗ = argmax

"

)

#$% &

𝛿#𝐹'!,)!|" 𝑆 𝑡#, 𝑏#

• Soft MDP

𝜌'+,-

∗

= argmax

"

)

#$% &

𝛿#𝐹'!,)!|" 𝑆 𝑡#, 𝑏# + 𝜇𝐼 𝜌 ⋅ 𝑡#

University of Waterloo

Maximum entropy policy Entropy regularized policy

CS885 Spring 2020 Pascal Poupart 7

Q-function

• Standard MDP

𝑅" 𝑡%, 𝑏% = 𝑆 𝑡%, 𝑏% + )

#$. /

𝛿#𝐹'!,)!|'",)","[𝑆 𝑡#, 𝑏# ]

• Soft MDP

𝑅'+,-

"

𝑡%, 𝑏% = 𝑆 𝑡%, 𝑏% + )

#$. /

𝛿#𝐹'!,)!|'",)"," 𝑆 𝑡#, 𝑏# + 𝜇𝐼 𝜌 ⋅ 𝑡#

University of Waterloo

NB: No entropy with first reward term since action is not chosen according to 𝜌

CS885 Spring 2020 Pascal Poupart 8

Greedy Policy

• Standard MDP (deterministic policy)

𝜌'())*+(𝑡) = argmax

,

𝑅(𝑡, 𝑏)

• Soft MDP (stochastic policy)

𝜌'())*+ ⋅ 𝑡 = argmax

• ∑, 𝜌 𝑏|𝑡 𝑅 𝑡, 𝑏 + 𝜇𝐼 𝜌 ⋅ 𝑡

=

./0 1 2,⋅ /6 ∑0 ./0 1 2,, /6 = 𝑡𝑝𝑔𝑢𝑛𝑏𝑦(𝑅 𝑡,⋅ /𝜇)

when 𝜇 → 0 then 𝑡𝑝𝑔𝑢𝑛𝑏𝑦 becomes regular max

University of Waterloo

CS885 Spring 2020 Pascal Poupart 9

Derivation

• Concave objective (can find global maximum)

𝐾(𝜌, 𝑅) = ∑, 𝜌 𝑏|𝑡 𝑅 𝑡, 𝑏 + 𝜇𝐼 𝜌 ⋅ 𝑡 = ∑, 𝜌 𝑏 𝑡 [𝑅 𝑡, 𝑏 − 𝜇 log 𝜌 𝑏 𝑡 ]

• Partial derivative

𝜖𝐾 𝜖𝜌 𝑏 𝑡 = 𝑅 𝑡, 𝑏 − 𝜇 log 𝜌 𝑏 𝑡 + 1

• Setting the derivative to 0 and isolating 𝜌 𝑏 𝑡 yields

𝜌 𝑏 𝑡 = exp 𝑅 𝑡, 𝑏 /𝜇 − 1 ∝ exp(𝑅 𝑡, 𝑏 /𝜇)

• Hence 𝜌'())*+ ⋅ 𝑡 =

./0 1 2,⋅ /6 ∑0 ./0 1 2,, /6 = 𝑡𝑝𝑔𝑢𝑛𝑏𝑦(𝑅 𝑡,⋅ /𝜇)

University of Waterloo

CS885 Spring 2020 Pascal Poupart 10

Greedy Value function

• What is the value function induced by the greedy policy?
• Standard MDP: 𝑊 𝑡 = max

!

𝑅(𝑡, 𝑏)

• Soft MDP:

𝑊

!"#$ 𝑡 = 𝜇𝐼 𝜌%&''() ⋅ 𝑡

+ ∑* 𝜌%&''() 𝑏 𝑡 𝑅!"#$ 𝑡, 𝑏 = 𝜇 log ∑* exp

+!"#$ !,*

• = 3

𝑛𝑏𝑦-

*

𝑅!"#$(𝑡, 𝑏) when 𝜇 → 0 then 3 𝑛𝑏𝑦- becomes regular max

University of Waterloo

CS885 Spring 2020 Pascal Poupart 11

Derivation

𝑊

#$%& 𝑡

= 𝜇𝐼 𝜌'())*+ ⋅ 𝑡 + ∑, 𝜌'())*+ 𝑏 𝑡 𝑅#$%& 𝑡, 𝑏 = 𝜇𝐼 𝜌'())*+ ⋅ 𝑡 + ∑, 𝜌'())*+ 𝑏 𝑡 𝜇 log 𝜌'())*+ 𝑏 𝑡 + log ∑,! exp

• "#$% #,,!

/

= 𝜇𝐼 𝜌'())*+ ⋅ 𝑡 + 𝜇 ∑, 𝜌'())*+ 𝑏 𝑡 log 𝜌'())*+ 𝑏 𝑡 + 𝜇 log ∑,! exp

• "#$% #,,!

/

= 𝜇𝐼 𝜌'())*+ ⋅ 𝑡 − 𝜇𝐼 𝜌'())*+ ⋅ 𝑡 + 𝜇 log ∑,! exp

• "#$% #,,!

/

= 𝜇 log ∑,! exp

• "#$% #,,!

/

= @ max/

,

𝑅#$%&(𝑡, 𝑏)

University of Waterloo

since 𝜌%&''() 𝑏 𝑡 =

*+, -"#$% .,0 /2 ∑&! *+, -"#$% .,0! /2

CS885 Spring 2020 Pascal Poupart 12

Soft Q-Value Iteration

SoftQValueIteration(MDP, 𝜇) Initialize 𝜌% to any policy 𝑗 ← 0 Repeat 𝑅'+,-

• 12. 𝑡, 𝑏 ← 𝑆 𝑡, 𝑏 + 𝛿 ∑'0 Pr(𝑡3|𝑡, 𝑏) ?

max4

)0

𝑅'+,-

1

(𝑡′, 𝑏′) 𝑗 ← 𝑗 + 1 Until 𝑅'+,-

1

𝑡, 𝑏 − 𝑅'+,-

• 15. 𝑡, 𝑏

/ ≤ 𝜗

Extract policy: 𝜌67889: ⋅ 𝑡 = 𝑡𝑝𝑔𝑢𝑛𝑏𝑦(𝑅'+,-

1

𝑡,⋅ /𝜇)

University of Waterloo

Soft Bellman equation: 𝑅'+,-

∗

𝑡, 𝑏 = 𝑆 𝑡, 𝑏 + 𝛿 )

'0

Pr(𝑡3|𝑡, 𝑏) ? max4

)0

𝑅'+,-

∗

(𝑡′, 𝑏′)

CS885 Spring 2020 Pascal Poupart 13

Soft Q-learning

• Q-learning based on Soft Q-Value Iteration
• Replace expectations by samples
• Represent Q-function by a function approximator

(e.g., neural network)

• Do gradient updates based on temporal differences

University of Waterloo

CS885 Spring 2020 Pascal Poupart 14

Soft Q-learning (soft variant of DQN)

University of Waterloo

Initialize weights 𝒙 and = 𝒙 at random in [−1,1] Observe current state 𝑡 Loop Select action 𝑏 and execute it Receive immediate reward 𝑠 Observe new state 𝑡’ Add (𝑡, 𝑏, 𝑡4, 𝑠) to experience buffer Sample mini-batch of experiences from buffer For each experience ̂ 𝑡, E 𝑏, ̂ 𝑡4, ̂ 𝑠 in mini-batch Gradient:

56&& 5𝒙 = 𝑅𝒙 .89:

̂ 𝑡, E 𝑏 − ̂ 𝑠 − 𝛿 H max2

; 0!

𝑅𝒙

.89:( ̂

𝑡′, E 𝑏′)

5-𝒙

"#$%

̂ ., ; 5𝒙

Update weights: 𝒙 ← 𝒙 − 𝛽

56&& 5𝒙

Update state: 𝑡 ← 𝑡’ Every 𝑑 steps, update target: = 𝒙 ← 𝒙

CS885 Spring 2020 Pascal Poupart 15

Soft Actor Critic

• In practice, actor critic techniques tend to perform

better than Q-learning.

• Can we derive a soft actor-critic algorithm?
• Yes, idea:

– Critic: soft Q-function – Actor: (greedy) softmax policy

University of Waterloo

CS885 Spring 2020 Pascal Poupart 16

Soft Policy Iteration

SoftPolicyIteration(MDP, 𝜇) Initialize 𝜌% to any policy 𝑗 ← 0 Repeat Policy evaluation: Repeat until convergence 𝑅'+,-

"1

𝑡, 𝑏 ← 𝑆 𝑡, 𝑏 +𝛿 ∑'0 Pr 𝑡3 𝑡, 𝑏 ∑)0 𝜌1 𝑏′ 𝑡′ 𝑅'+,-

"1

𝑡′, 𝑏′ + 𝜇𝐼 𝜌1 ⋅ 𝑡′ ∀𝑡, 𝑏 Policy improvement: 𝜌12. 𝑏 𝑡 ← 𝑡𝑝𝑔𝑢𝑛𝑏𝑦 𝑅'+,-

"1

𝑡, 𝑏 /𝜇 =

;<= >2345

61

',) /4 ∑70 ;<= >2345

61

',)0 /4

∀𝑡, 𝑏 𝑗 ← 𝑗 + 1 Until 𝑅'+,-

"1

𝑡, 𝑏 − 𝑅'+,-

"189 𝑡, 𝑏 /

≤ 𝜗

University of Waterloo

CS885 Spring 2020 Pascal Poupart 17

Policy improvement

Theorem 1: Let 𝑅LMNO

PE

(𝑡, 𝑏) be the Q-function of 𝜌Q Let 𝜌QRS 𝑏 𝑡 = 𝑡𝑝𝑔𝑢𝑛𝑏𝑦 𝑅LMNO

PE

𝑡, 𝑏 /𝜇 Then 𝑅LMNO

PEFG 𝑡, 𝑏 ≥ 𝑅LMNO PE

𝑡, 𝑏 ∀𝑡, 𝑏 Proof: first show that

∑, 𝜌P 𝑏 𝑡 𝑅2QRS

• A

(𝑡, 𝑏) + 𝜇𝐼 𝜌P ⋅ 𝑡 ≤ ∑, 𝜌PTU 𝑏 𝑡 𝑅2QRS

• A

𝑡, 𝑏 + 𝜇𝐼(𝜌PTU ⋅ 𝑡 )

then use this inequality to show that

𝑅2QRS

• ABC 𝑡, 𝑏 ≥ 𝑅2QRS
• A

𝑡, 𝑏 ∀𝑡, 𝑏

University of Waterloo

CS885 Spring 2020 Pascal Poupart 18

Inequality derivation

∑! 𝜌" 𝑏 𝑡 𝑅#$%&

'!

(𝑡, 𝑏) + 𝜇𝐼 𝜌" ⋅ 𝑡 = ∑! 𝜌" 𝑏 𝑡 𝑅#$%&

'!

𝑡, 𝑏 − 𝜇 log 𝜌" 𝑏 𝑡 = ∑! 𝜌" 𝑏 𝑡 [ 𝜇 log 𝜌"() 𝑏 𝑡 − 𝜇 log ∑!" exp(𝑅#$%&

'!

𝑡, 𝑏* /𝜇) − 𝜇 log 𝜌" 𝑏 𝑡 ] = 𝜇 ∑! 𝜌" 𝑏 𝑡 [ log

'!#$ 𝑏 𝑡 '! 𝑏 𝑡

+ log ∑!" exp(𝑅#$%&

'!

𝑡, 𝑏′ /𝜇)] = −𝜇𝐿𝑀(𝜌"()| 𝜌" + 𝜇 ∑! 𝜌" 𝑏 𝑡 log ∑!" exp(𝑅#$%&

'!

𝑡, 𝑏′ /𝜇) ≤ 𝜇 ∑! 𝜌" 𝑏 𝑡 log ∑!" exp(𝑅#$%&

'!

𝑡, 𝑏′ /𝜇) = ∑! 𝜌"() 𝑏 𝑡 𝜇 log ∑!" exp(𝑅#$%&

'!

𝑡, 𝑏′ /𝜇) = ∑! 𝜌"() 𝑏 𝑡 𝑅#$%&

'!

𝑡, 𝑏 − 𝜇 log 𝜌"() 𝑡, 𝑏 = ∑! 𝜌"() 𝑏 𝑡 𝑅#$%&

'!

𝑡, 𝑏 + 𝜇𝐼 𝜌"() ⋅ 𝑡

University of Waterloo

since 𝜌=>? 𝑏 𝑡 =

*+, -"#$%

()

.,0 /2 ∑&! *+, -"#$%

()

.,0! /2

since 𝜌=>? 𝑏 𝑡 =

*+, -() .,0 /2 ∑&! *+, -() .,0! /2

CS885 Spring 2020 Pascal Poupart 19

Proof derivation

𝑅2QRS

• A

𝑡, 𝑏 = 𝑆 𝑡, 𝑏 + 𝛿𝐹2D 𝐹,D~-A 𝑅2QRS

• A

𝑡!, 𝑏! + 𝜇𝐼 𝜌P ⋅ 𝑡′ ≤ 𝑆 𝑡, 𝑏 + 𝛿𝐹2D 𝐹,D~-ABC 𝑅2QRS

• A

𝑡!, 𝑏! + 𝜇𝐼 𝜌PTU ⋅ 𝑡′ ≤ ⋯ ≤ ⋯ ≤ 𝑅2QRS

• ABC(𝑡, 𝑏)

University of Waterloo

since 𝐹0!~A) 𝑅.89:

A)

𝑡4, 𝑏4 + 𝜇𝐼 𝜌= ⋅ 𝑡′ ≤ 𝐹0!~A)*+ 𝑅.89:

A)

𝑡4, 𝑏4 + 𝜇𝐼 𝜌=>? ⋅ 𝑡′ repeatedly apply QBCDE

A, (s′, a′) ≤ 𝑆 𝑡′, 𝑏′ + 𝛿𝐹.!! 𝐹0!!~A)*+ 𝑅.89: A)

𝑡44, 𝑏44 + 𝜇𝐼 𝜌=>? ⋅ 𝑡′′

CS885 Spring 2020 Pascal Poupart 20

Convergence to Optimal 𝑅,-./

∗

and 𝜌,-./

∗

• Theorem 2: When 𝜗 = 0, soft policy iteration converges to
• ptimal 𝑅2QRS

∗

and 𝜌2QRS

∗

.

• Proof:

– We know that 𝑅"1:9 𝑡, 𝑏 ≥ 𝑅"1 𝑡, 𝑏 ∀𝑡, 𝑏 according to Theorem 1 – Since the Q-functions are upper bounded by (max

',) 𝑆 𝑡, 𝑏 + 𝐼 𝑣𝑜𝑗𝑔𝑝𝑠𝑛 )/(1 − 𝛿)

then soft policy iteration converges – At convergence, 𝑅"189 = 𝑅"1 and therefore the Q-function satisfies Bellman’s equation:

𝑅.89:

A)-+ 𝑡, 𝑏 = 𝑅.89: A)

𝑡, 𝑏 = 𝑆 𝑡, 𝑏 + 𝛿 _

.!

Pr(𝑡4|𝑡, 𝑏) H max2

0!

𝑅.89:

A)-+(𝑡′, 𝑏′)

University of Waterloo

CS885 Spring 2020 Pascal Poupart 21

Soft Actor-Critic

• RL version of soft policy iteration
• Use neural networks to represent policy and value

function

• At each policy improvement step, project new policy

in the space of parameterized neural nets

University of Waterloo

CS885 Spring 2020 Pascal Poupart 22

Soft Actor Critic (SAC)

University of Waterloo

Initialize weights 𝒙, I 𝒙, 𝜄 at random in [−1,1] Observe current state 𝑡 Loop Sample action 𝑏~𝜌;(⋅ |𝑡) and execute it Receive immediate reward 𝑠 Observe new state 𝑡’ Add (𝑡, 𝑏, 𝑡<, 𝑠) to experience buffer Sample mini-batch of experiences from buffer For each experience ̂ 𝑡, S 𝑏, ̂ 𝑡<, ̂ 𝑠 in mini-batch Sample S 𝑏′~𝜌;(⋅ | ̂ 𝑡′) Gradient:

=>(( =𝒙 = 𝑅𝒙 #$%&

̂ 𝑡, S 𝑏 − ̂ 𝑠 − 𝛿[Q @

𝐱 #$%&

̂ 𝑡<, S 𝑏< + 𝜇𝐼 𝜌; ⋅ ̂ 𝑡< )

=-𝒙

"#$%

̂ #, C , =𝒙

Update weights: 𝒙 ← 𝒙 − 𝛽

=>(( =𝒙

Update policy: 𝜄 ← 𝜄 − 𝛽

=DE 𝜌; 𝑡𝑝𝑔𝑢𝑛𝑏𝑦 𝑅@ F #$%&/𝜇 =;

Update state: 𝑡 ← 𝑡’ Every 𝑑 steps, update target: I 𝒙 ← 𝒙

CS885 Spring 2020 Pascal Poupart 23

Empirical Results

• Comparison on several robotics tasks

University of Waterloo From Haarnoja, Zhou et al. (2018)

CS885 Spring 2020 Pascal Poupart 24

Robustness to Environment Changes

University of Waterloo

https://youtu.be/KOObeIjzXTY

Check out this video

Using Soft Actor Critic (SAC), Minotaur learns to walk quickly and to generalize to environments with challenges that it was not trained to deal with!