Separating value functions across time-scales Joshua Romoff* 1,2 , - - PowerPoint PPT Presentation

Separating value functions across time-scales

Joshua Romoff*1,2, Peter Henderson*3, Ahmed Touati2,4, Emma Brunskill3, Joelle Pineau1,2 , Yann Ollivier2

1MILA-McGill University, 2Facebook AI Research, 3Stanford University, 4MILA-Université

de Montréal *Equal Contribution

RL Background

• Monte-Carlo Return / Target:
• Value function:
• 𝐻𝑢 ∶ =

∞

∑

𝑗=0

𝛿𝑗𝑠𝑢+𝑗 𝑊(𝑡) ∶ = 𝔽[𝐻𝑢| 𝑡𝑢 = 𝑡, 𝜌]

Separating value functions across time- scales

RL Background - bootstrapping

• Multi-step returns:
• – returns :
• 𝐻𝑙

𝑢 ∶ = 𝑙−1

∑

𝑗=0

𝛿𝑗𝑠𝑢+𝑗+ 𝛿𝑙 𝑊(𝑡𝑢+𝑙)

𝜇

𝐻𝜇

𝑢 ∶ = (1 − 𝜇) ∞

∑

𝑙=1

𝜇𝑙−1𝐻

𝑙 𝑢

Separating value functions across time- scales

Learning - Problems with large 𝑈𝐸

𝛿

• Part of the problem formulation:
• When

training is difficult

𝐻𝑢 ∶ =

∞

∑

𝑗=0

𝛿𝑗𝑠𝑢+𝑗

𝛿 → 1 𝑊𝛿

Separating value functions across time- scales

• Our Solution

𝑈𝐸( ∆ )

• Define a sequence of s:
• , …,
• with

𝛿 ∆ ∶ = (𝛿0, 𝛿1

𝛿𝑎)

𝛿𝑗 ≤ 𝛿𝑗+1 ∀ 𝑗

Separating value functions across time- scales

• Our Solution

𝑈𝐸( ∆ )

• Define a sequence of s:
• , …,
• with
• Learn

and

𝛿 ∆ ∶ = (𝛿0, 𝛿1

𝛿𝑎)

𝛿𝑗 ≤ 𝛿𝑗+1 ∀ 𝑗 𝑋0 ∶ = 𝑊𝛿0 𝑋𝑗 ∶ = 𝑊𝛿𝑗 − 𝑊𝛿𝑗−1

Separating value functions across time- scales

• Our Solution

𝑈𝐸( ∆ )

• Define a sequence of s:
• , …,
• with
• Learn

and

• Recompose:

𝛿 ∆ ∶ = (𝛿0, 𝛿1

𝛿𝑎)

𝛿𝑗 ≤ 𝛿𝑗+1 ∀ 𝑗 𝑋0 ∶ = 𝑊𝛿0 𝑋𝑗 ∶ = 𝑊𝛿𝑗 − 𝑊𝛿𝑗−1

𝑎

∑

𝑗=0

𝑋𝑗 = 𝑊𝛿𝑎

Separating value functions across time- scales

• Bellman Equation

𝑈𝐸( ∆ )

• We can use Bellman Equations:
• We extend it to multi-step

and

𝑋0 ∶ 𝑠𝑢+ 𝛿0 𝑋0(𝑇𝑢+1) 𝑋𝑗>0 ∶ (𝛿𝑗 − 𝛿𝑗−1)𝑊𝛿𝑗−1(𝑇𝑢+1) + 𝛿𝑗 𝑋𝑗 (𝑇𝑢+1) 𝑈𝐸 𝑈𝐸(𝜇)

Separating value functions across time- scales

• Bellman Equation

𝑈𝐸( ∆ )

Separating value functions across time- scales

• Equivalence results

𝑈𝐸( ∆ )

• Equivalence to standard

𝑈𝐸(𝜇)

Separating value functions across time- scales

We did it! Wait…

• Equivalence results

𝑈𝐸( ∆ )

• Equivalence to standard

𝑈𝐸(𝜇)

Separating value functions across time- scales

We did it! Wait…

• Equivalence conditions

𝑈𝐸( ∆ )

• Linear function approximation
• Same learning rates for each W
• Same K-step /

for each W 


𝛿𝜇

Separating value functions across time- scales

– Benefits more tuning 𝑈𝐸( ∆ )

• We don’t have to be equivalent!
• Change the learning rates
• -step / return

𝑙

𝜇

Separating value functions across time- scales

– Benefits more tuning 𝑈𝐸( ∆ )

• What will this get us?
• Let’s turn to a slightly different setting to

get more insight. 


Separating value functions across time- scales

– Benefits more tuning 𝑈𝐸( ∆ )

• “Phasic” updates for standard TD

Separating value functions across time- scales

– Benefits more tuning 𝑈𝐸( ∆ )

Get an error bound using large deviation analysis with bias and variance components! Dependent on k-steps and discount (also size of samples). 


Separating value functions across time- scales

Small Note: Kearns & Singh have a slightly different variance term constant, the proof was excluded from the 2000 paper, so we instead used Hoeffding inequality to reach this constant (see our supplemental).

– Benefits more tuning 𝑈𝐸( ∆ )

Separating value functions across time- scales

If we do the same for our method, get a bias-variance tradeoff.

– Little tuning required 𝑈𝐸( ∆ )

• 1. Adaptive optimizer handle the learning rate
• 2. Set
• r

, 1)

• 3. Set to be double the horizon of

𝑙𝑗 = 1 1 − 𝛿𝑗 𝜇𝑗 = min(𝛿𝑎 𝜇𝑎 𝛿𝑗 𝛿𝑗 𝛿𝑗−1

Separating value functions across time- scales

– for actor critic algorithms 𝑈𝐸( ∆ )

• 1. Train the Ws as described
• 2. Use the sum of Ws instead of V in policy

update We apply it to PPO and test it on Atari

Separating value functions across time- scales

– for actor critic algorithms 𝑈𝐸( ∆ )

Separating value functions across time- scales

– for actor critic algorithms 𝑈𝐸( ∆ )

Separating value functions across time- scales

– for actor critic algorithms 𝑈𝐸( ∆ )

Separating value functions across time- scales

– for actor critic algorithms 𝑈𝐸( ∆ )

Separating value functions across time- scales

• Atari Experiments

𝑈𝐸( ∆ )

Separating value functions across time- scales

• Atari Experiments

𝑈𝐸( ∆ )

Separating value functions across time- scales

• Atari Experiments

𝑈𝐸( ∆ )

Separating value functions across time- scales

• What does it learn? (Atari)

𝑈𝐸( ∆ )

Separating value functions across time- scales

• What does it learn? (Atari)

𝑈𝐸( ∆ )

Separating value functions across time- scales

• Benefits

𝑈𝐸( ∆ )

Separating value functions across time- scales

More knobs to tune bias-variance trade-off! :) More insight into value of policy at different time-scales! Bellman update for learning separated value functions, allows for some theoretical insights. Natural splitting for distributed computing.

• Downsides

𝑈𝐸( ∆ )

Separating value functions across time- scales

More knobs to tune bias-variance trade-off! :( Somewhat more compute intensive.

Meets Reward Estimation 𝑈𝐸( ∆ )

Separating value functions across time- scales

Joshua Romoff*, Peter Henderson*, Alexandre Piche, Vincent Francois-Lavet, and Joelle Pineau. "Reward Estimation for Variance Reduction in Deep Reinforcement Learning." In Conference on Robot Learning, pp. 674-699. 2018.

Previously demonstrated simple property that by using a learned estimation

• f the reward, we can reduce variance in learning especially in noisy

environments.

Meets Reward Estimation 𝑈𝐸( ∆ )

Separating value functions across time- scales

Here, we are using many estimators, looking at a similar bias-variance trade-

• ff.

An interesting future investigation would look into whether separating value functions across many estimators has similar natural benefits in the cases of noisy rewards as in our reward estimation work.

Joshua Romoff*, Peter Henderson*, Alexandre Piche, Vincent Francois-Lavet, and Joelle Pineau. "Reward Estimation for Variance Reduction in Deep Reinforcement Learning." In Conference on Robot Learning, pp. 674-699. 2018.

Other Extensions

• Adding more Ws to move discount factor to 1
• Q-learning extension
• Use the natural time-scale split for

distributed computing updates

Separating value functions across time- scales

Thanks! More Questions?