Trust Region Policy Optimization John Schulman, Sergey Levine, - - PowerPoint PPT Presentation

Trust Region Policy Optimization

John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, Pieter Abbeel @ICML 2015 Presenter: Shivam Kalra Shivam.kalra@uwaterloo.ca CS 885 (Reinforcement Learning)

• Prof. Pascal Poupart

June 20th 2018

Reinforcement Learning

Action Value Function Q-Learning Policy Gradients TRPO Actor Critic PPO A3C ACKTR

Ref: https://www.youtube.com/watch?v=CKaN5PgkSBc

Policy Gradient

For i=1,2,… Collect N trajectories for policy 𝜌𝜄 Estimate advantage function 𝐵 Compute policy gradient 𝑕 Update policy parameter 𝜄 = 𝜄𝑝𝑚𝑒 + 𝛽𝑕

Problems of Policy Gradient

For i=1,2,… Collect N trajectories for policy 𝜌𝜄 Estimate advantage function 𝐵 Compute policy gradient 𝑕 Update policy parameter 𝜄 = 𝜄𝑝𝑚𝑒 + 𝛽𝑕

Non stationary input data due to changing policy and reward distributions change

Problems of Policy Gradient

For i=1,2,… Collect N trajectories for policy 𝜌𝜄 Estimate advantage function 𝐵 Compute policy gradient 𝑕 Update policy parameter 𝜄 = 𝜄𝑝𝑚𝑒 + 𝛽𝑕

cv Advantage is very random initially Advantage Policy

You’re bad

Problems of Policy Gradient

For i=1,2,… Collect N trajectories for policy 𝜌𝜄 Estimate advantage function 𝐵 Compute policy gradient 𝑕 Update policy parameter 𝜄 = 𝜄𝑝𝑚𝑒 + 𝛽𝑕

We need more carefully crafted policy update We want improvement and not degradation Idea: We can update old policy 𝜌𝑝𝑚𝑒 to a new policy ෤ 𝜌 such that they are “trusted” distance apart. Such conservative policy update allows improvement instead

• f degradation.

RL to Optimization

• Most of ML is optimization
• Supervised learning is reducing training loss
• RL: what is policy gradient optimizing?
• Favoring (𝑡, 𝑏) that gave more advantage 𝐵.
• Can we write down optimization problem that allows to do small update on a

policy 𝜌 based on data sampled from 𝜌 (on-policy data)

Ref: https://www.youtube.com/watch?v=xvRrgxcpaHY (6:40)

What loss to optimize?

• Optimize 𝜃(𝜌) i.e., expected return of a policy 𝜌
• We collect data with 𝜌𝑝𝑚𝑒 and optimize the objective to get a new

policy ෤ 𝜌.

𝜃 𝜌 = 𝔽𝑡0~𝜍0,𝑏𝑢~𝜌 . 𝑡𝑢 ෍

𝑢=0 ∞

𝛿𝑢𝑠

𝑢

What loss to optimize?

• We can express 𝜃 ෤

𝜌 in terms of the advantage over the original policy1.

𝜃 ෤ 𝜌 = 𝜃 𝜌𝑝𝑚𝑒 + 𝔽𝜐~෥

𝜌 ෍ 𝑢=0 ∞

𝛿𝑢𝐵𝜌𝑝𝑚𝑒(𝑡𝑢, 𝑏𝑢)

[1] Kakade, Sham, and John Langford. "Approximately optimal approximate reinforcement learning." ICML. Vol. 2. 2002.

Expected return of new policy Expected return of

• ld policy

Sample from new policy

What loss to optimize?

• Previous equation can be rewritten as1:

𝜃 ෤ 𝜌 = 𝜃 𝜌𝑝𝑚𝑒 + ෍

𝑡

𝜍෥

𝜌(𝑡) ෍ 𝑏

෤ 𝜌 𝑏 𝑡 𝐵𝜌(𝑡, 𝑏)

[1] Schulman, John, et al. "Trust region policy optimization." International Conference on Machine Learning. 2015.

Expected return of new policy Expected return of

• ld policy

Discounted visitation frequency 𝜍𝜌 𝑡 = 𝑄 𝑡0 = 𝑡 + 𝛿𝑄 𝑡1 = 𝑡 + 𝛿2𝑄 + ⋯

𝜃 ෤ 𝜌 = 𝜃 𝜌𝑝𝑚𝑒 + ෍

𝑡

𝜍෥

𝜌(𝑡) ෍ 𝑏

෤ 𝜌 𝑏 𝑡 𝐵𝜌(𝑡, 𝑏)

What loss to optimize?

New Expected Return Old Expected Return

≥ 𝟏

𝜃 ෤ 𝜌 = 𝜃 𝜌𝑝𝑚𝑒 + ෍

𝑡

𝜍෥

𝜌(𝑡) ෍ 𝑏

෤ 𝜌 𝑏 𝑡 𝐵𝜌(𝑡, 𝑏)

New Expected Return Old Expected Return

>

What loss to optimize?

≥ 𝟏

Guaranteed Improvement from 𝜌𝑝𝑚𝑒 → ෤ 𝜌

𝜃 ෤ 𝜌 = 𝜃 𝜌𝑝𝑚𝑒 + ෍

𝑡

𝜍෥

𝜌(𝑡) ෍ 𝑏

෤ 𝜌 𝑏 𝑡 𝐵𝜌(𝑡, 𝑏)

New State Visitation is Difficult

State visitation based on new policy New policy “Complex dependency of 𝜍෥

𝜌(𝑡) on

෤ 𝜌 makes the equation difficult to

• ptimize directly.” [1]

[1] Schulman, John, et al. "Trust region policy optimization." International Conference on Machine Learning. 2015.

𝜃 ෤ 𝜌 = 𝜃 𝜌𝑝𝑚𝑒 + ෍

𝑡

𝜍෥

𝜌(𝑡) ෍ 𝑏

෤ 𝜌 𝑏 𝑡 𝐵𝜌(𝑡, 𝑏)

New State Visitation is Difficult

[1] Schulman, John, et al. "Trust region policy optimization." International Conference on Machine Learning. 2015.

𝑀 ෤ 𝜌 = 𝜃 𝜌𝑝𝑚𝑒 + ෍

𝑡

𝜍𝜌(𝑡) ෍

𝑏

෤ 𝜌 𝑏 𝑡 𝐵𝜌(𝑡, 𝑏)

Local approximation of 𝜽(෥ 𝝆)

Local approximation of 𝜃(෤ 𝜌)

[1] Schulman, John, et al. "Trust region policy optimization." International Conference on Machine Learning. 2015.

𝑀 ෤ 𝜌 = 𝜃 𝜌𝑝𝑚𝑒 + ෍

𝑡

𝜍𝜌(𝑡) ෍

𝑏

෤ 𝜌 𝑏 𝑡 𝐵𝜌𝑝𝑚𝑒(𝑡, 𝑏)

The approximation is accurate within step size 𝜀 (trust region) Monotonic improvement guaranteed 𝜄 𝜄′ 𝜌𝜄′ 𝑡 𝑏 does not change dramatically. Trust region

• Monotonically improving policies can be generated by:
• The following bound holds:

Local approximation of 𝜃(෤ 𝜌)

𝜃 ෤ 𝜌 ≥ 𝑀 ෤ 𝜌 − 𝐷𝐸𝐿𝑀

𝑛𝑏𝑦(𝜌, ෤

𝜌) Where, 𝐷 =

4𝜗𝛿 1−𝛿 2

𝜌 = arg max

𝜌

[𝑀 ෤ 𝜌 − 𝐷𝐸𝐿𝑀

𝑛𝑏𝑦 𝜌, ෤

𝜌 ] Where, 𝐷 =

4𝜗𝛿 1−𝛿 2

Minorization Maximization (MM) algorithm

Actual function 𝜃(𝜌) Surrogate function 𝑀 𝜌 − 𝐷𝐸𝐿𝑀

𝑛𝑏𝑦(𝜌, ෤

𝜌)

Optimization of Parameterized Policies

• Now policies are parameterized 𝜌𝜄 𝑏 𝑡 with parameters 𝜄
• Accordingly surrogate function changes to

arg max

𝜄

[𝑀 𝜄 − 𝐷𝐸𝐿𝑀

𝑛𝑏𝑦 𝜄𝑝𝑚𝑒, 𝜄 ]

Optimization of Parameterized Policies

arg max

𝜄

[𝑀 𝜄 − 𝐷𝐸𝐿𝑀

𝑛𝑏𝑦 𝜄𝑝𝑚𝑒, 𝜄 ]

In practice 𝐷 results in very small step sizes One way to take larger step size is to constraint KL divergence between the new policy and the old policy, i.e., a trust region constraint: 𝒏𝒃𝒚𝒋𝒏𝒋𝒜𝒇

𝜾

𝑴𝜾(𝜾) subject to, 𝑬𝑳𝑴

𝒏𝒃𝒚 𝜾𝒑𝒎𝒆, 𝜾 ≤ 𝜺

Solving KL-Penalized Problem

• max𝑗𝑛𝑗𝑨𝑓𝜄 𝑀 𝜄 − 𝐷. 𝐸𝐿𝑀

𝑛𝑏𝑦(𝜄𝑝𝑚𝑒, 𝜄)

• Use mean KL divergence instead of max.
• i.e., max𝑗𝑛𝑗𝑨𝑓

𝜄

𝑀 𝜄 − 𝐷. 𝐸𝐿𝑀(𝜄𝑝𝑚𝑒, 𝜄)

• Make linear approximation to 𝑀 and quadratic to KL term:

max𝑗𝑛𝑗𝑨𝑓

𝜄

𝑕 . 𝜄 − 𝜄𝑝𝑚𝑒 − 𝑑 2 𝜄 − 𝜄𝑝𝑚𝑒 𝑈𝐺(𝜄 − 𝜄𝑝𝑚𝑒) where, 𝑕 =

𝜖 𝜖𝜄 𝑀 𝜄 ȁ𝜄=𝜄𝑝𝑚𝑒,

𝐺 =

𝜖2 𝜖2𝜄 𝐸𝐿𝑀 𝜄𝑝𝑚𝑒, 𝜄 ȁ𝜄=𝜄𝑝𝑚𝑒

Solving KL-Penalized Problem

• Make linear approximation to 𝑀 and quadratic to KL term:
• Solution: 𝜄 − 𝜄𝑝𝑚𝑒 =

1 𝑑 𝐺−1𝑕. Don’t want to form full Hessian matrix

𝐺 =

𝜖2 𝜖2𝜄 𝐸𝐿𝑀 𝜄𝑝𝑚𝑒, 𝜄 ȁ𝜄=𝜄𝑝𝑚𝑒.

• Can compute 𝐺−1𝑕 approximately using conjugate gradient

algorithm without forming 𝐺 explicitly.

max𝑗𝑛𝑗𝑨𝑓

𝜄

𝑕 . 𝜄 − 𝜄𝑝𝑚𝑒 − 𝑑 2 𝜄 − 𝜄𝑝𝑚𝑒 𝑈𝐺(𝜄 − 𝜄𝑝𝑚𝑒) where, 𝑕 =

𝜖 𝜖𝜄 𝑀 𝜄 ȁ𝜄=𝜄𝑝𝑚𝑒,

𝐺 = 𝜖2

𝜖2𝜄 𝐸𝐿𝑀 𝜄𝑝𝑚𝑒, 𝜄 ȁ𝜄=𝜄𝑝𝑚𝑒

Conjugate Gradient (CG)

• Conjugate gradient algorithm approximately solves for 𝑦 = 𝐵−1𝑐

without explicitly forming matrix 𝐵

• After 𝑙 iterations, CG has minimized

1 2 𝑦𝑈𝐵𝑦 − 𝑐𝑦

TRPO: KL-Constrained

• Unconstrained problem: max𝑗𝑛𝑗𝑨𝑓

𝜄

𝑀 𝜄 − 𝐷. 𝐸𝐿𝑀(𝜄𝑝𝑚𝑒, 𝜄)

• Constrained problem: max𝑗𝑛𝑗𝑨𝑓

𝜄

𝑀 𝜄 subject to 𝐷. 𝐸𝐿𝑀 𝜄𝑝𝑚𝑒, 𝜄 ≤ 𝜀

• 𝜀 is a hyper-parameter, remains fixed over whole learning process
• Solve constrained quadratic problem: compute 𝐺−1𝑕 and then rescale step

to get correct KL

• max𝑗𝑛𝑗𝑨𝑓

𝜄

𝑕 . 𝜄 − 𝜄𝑝𝑚𝑒 subject to 1

2 𝜄 − 𝜄𝑝𝑚𝑒 𝑈𝐺 𝜄 − 𝜄𝑝𝑚𝑒 ≤ 𝜀

• Lagrangian: ℒ 𝜄, 𝜇 = 𝑕 .

𝜄 − 𝜄𝑝𝑚𝑒 − 𝜇

2 [ 𝜄 − 𝜄𝑝𝑚𝑒 𝑈𝐺 𝜄 − 𝜄𝑝𝑚𝑒 − 𝜀]

• Differentiate wrt 𝜄 and get 𝜄 − 𝜄𝑝𝑚𝑒 = 1

𝜇 𝐺−1𝑕

• We want 1

2 𝑡𝑈𝐺𝑡 = 𝜀

• Given candidate step 𝑡𝑣𝑜𝑡𝑑𝑏𝑚𝑓𝑒 rescale to 𝑡 =

2𝜀 𝑡𝑣𝑜𝑡𝑑𝑏𝑚𝑓𝑒.(𝐺𝑡𝑣𝑜𝑡𝑑𝑏𝑚𝑓𝑒) 𝑡𝑣𝑜𝑡𝑑𝑏𝑚𝑓𝑒

TRPO Algorithm

For i=1,2,… Collect N trajectories for policy 𝜌𝜄 Estimate advantage function 𝐵 Compute policy gradient 𝑕 Use CG to compute 𝐼−1𝑕 Compute rescaled step 𝑡 = 𝛽𝐼−1𝑕 with rescaling and line search Apply update: 𝜄 = 𝜄𝑝𝑚𝑒 + 𝛽𝐼−1𝑕 max𝑗𝑛𝑗𝑨𝑓

𝜄

𝑀 𝜄 subject to 𝐷. 𝐸𝐿𝑀 𝜄𝑝𝑚𝑒, 𝜄 ≤ 𝜀

Questions?