DAC: The Double Actor-Critic Architecture for Learning Options - - PowerPoint PPT Presentation

DAC: The Double Actor-Critic Architecture for Learning Options

NeurIPS 2019

Shangtong Zhang, Shimon Whiteson

Presenter: Ehsan Mehralian March 17, 2020

Outline

• Problem statement
• Option Critic
• Double Actor Critic

Problem statement

• Temporal abstraction is a key component in RL:
• Better exploration
• Faster learning
• Better generalization
• Transfer learning
• MDP + Temporal Abstract actions = SMDP
• SMDP algorithms are data inefficient —> The option framework (Sutton et al., 1999)
• Rises two problems:
• Learning options
• Learning a master policy

3

Previous works

• Based on finding subgoals:
• Difficult to scale up
• Can be as expensive as the entire task
• Using value-based methods:
• Can’t cope with large action spaces
• Policy based methods have better convergence

properties with function approximation

4

The Option Critic framework (PL Bacon et.al, 2017)

• Blurs the line between discovering options and learning
• ptions
• The first scalable end-to-end approach
• No slow down within a single task
• Faster convergence in transfer learning

5

Background

• MDP
• Goal:
• Policy Gradient

M ≡ {S, A, R(s, a), P(s0|s, a), P0(s), γ}

π∗ = arg max

θ

ρ(πθ) = arg max

θ

Eτ∼pθ(τ)[

∞

X

t=1

γt−1rt|s0, πθ]

∂ρ ∂θ = X

s

dπ(s) X

a

∂π(s, a) ∂θ Qπ(s, a)

dπ(s) =

∞

X

t=0

γtPr(st = s|s0, π)

6

The Options Framework

• A Markovian Option is a triple:
• Initiation set
• Intra-option policy
• Termination function
• Let denote the intra-option policy of option

parametrized by θ and , the termination function of parameterized by

ω ∈ Ω

(Iω, πω, βω)

πω,θ

βω,ϑ

ϑ

7

Iω ⊂ S

βω : S → [0, 1]

βω

πω

Iω

ω

ω

The Option Critic framework

• Discounted return

ρ(Ω, θ, ϑ, s0, w0) = EΩ,θ,ω[

∞

X

t=0

γtrt+1|s0, ω0]

8

The Option Critic framework

• Discounted return
• Option-value function

ρ(Ω, θ, ϑ, s0, w0) = EΩ,θ,ω[

∞

X

t=0

γtrt+1|s0, ω0]

QΩ(s, w) = X

a

πω,θ(a|s)QU(s, ω, a)

9

The Option Critic framework

• Discounted return
• Option-value function
• Value of executing an action in the context of a state-option pair

ρ(Ω, θ, ϑ, s0, w0) = EΩ,θ,ω[

∞

X

t=0

γtrt+1|s0, ω0]

QΩ(s, w) = X

a

πω,θ(a|s)QU(s, ω, a)

QU(s, ω, a) = r(s, a) + γ X

s0

P(s0|s, a)U(ω, s0)

10

The Option Critic framework

• Discounted return
• Option-value function
• Value of executing an action in the context of a state-option pair
• Option-value function upon arrival

ρ(Ω, θ, ϑ, s0, w0) = EΩ,θ,ω[

∞

X

t=0

γtrt+1|s0, ω0]

QΩ(s, w) = X

a

πω,θ(a|s)QU(s, ω, a)

QU(s, ω, a) = r(s, a) + γ X

s0

P(s0|s, a)U(ω, s0)

U(ω, s0) = (1 − βω,ϑ(s0))QΩ(s0, w) + βω,ϑ(s0)VΩ(s0)

11

The Option Critic framework (cont.)

• pairs lead to an augmented state space
• One step probability transition

12

(s, ω)

P(st+1, ωt+1|st, ωt) = X

a

πωt,θ(a|st)P(st+1|st, at)(

(1 − βω,ϑ(st+1))1wt=wt+1 + βω,ϑ(st+1)πΩ(ωt+1|st+1))

The option Critic framework (cont.)

• Theorem 1 (Intra-Option Policy Gradient Theorem)
• Theorem 2 (Termination Gradient Theorem)

∂ρ ∂θ = X

s,ω

µΩ(s, ω|s0, ω0) X

a

∂πω,θ(a|s) ∂θ QU(s, w, a)

µΩ(s, ω|s0, ω0) =

∞

X

t=0

γtP(st = s, ωt = ω|s0, ω0)

13

∂ρ ∂ϑ = − X

s0,ω

µΩ(s0, ω|s1, ω0)∂βω,ϑ(s0) ∂ϑ AΩ(s0, w)

Results

• Works great!
• But:
• Cannot directly leverage recent advances in gradient-based

policy optimization from MDPs

• Solution:
• DAC (The Double Actor-Critic Architecture)

14

The Double Actor-Critic Architecture for Learning Options (DAC)

• Reformulate the option framework as two parallel

augmented MDPs

• A policy based method
• All policy optimization algorithms can be used off the

shelf (advantage over Option Critic)

• Apply an actor-critic algorithm on each augmented MDP

(double actor critic)

• Show that one critic is enough.

15

Two Augmented MDPs

• The high-MDP:
• The agent makes high-level decisions (i.e., option

selection) in according to and thus

• ptimizes
• The low-MDP:
• The agent makes low-level decisions (i.e., action

selection) in according to and thus optimizes

M H

M L

π, {βω}

M H

π, {βω}

{πω}

M L

{πω}

16

High MDP

• Define a dummy option # and
• Interpret a state-option pair as new state and an option as a new action.
• Define Markov Policy

Ω+ = Ω ∪ {#}

17

Low MDP

• Interpret state-option pair as state and leave actions unchanged.
• Define Markov policy:

18

How to sample from these MDPs?

19

• Consider trajectories with non zero probability:
• Where
• Define functions:

ΩL = {τ L|p(τ L|πL, M L) > 0}

ΩH = {τ H|p(τ H|πH, M H) > 0}

Ω = {τ|p(τ|π, O, M) > 0}

τ = {S0, O0, S1, O1, ..., ST }

f H : Ω → ΩH

f L : Ω → ΩL

How to sample from these MDPs?

20

• Lemma 1: is a bijection and
• Lemma 2: is a bijection and
• So, Sampling from is equivalent to sampling

from and

p(τ|π, O, M) = p(τ L|πL, M L), r(τ) = r(τ L)

{π, O, M}

{πH, M H}

{πL, M L}

p(τ|π, O, M) = p(τ H|πH, M H), r(τ) = r(τ H)

f H : Ω → ΩH

f L : Ω → ΩL

• Proposition:
• Optimizing in M is equivalent to optimizing in

and optimizing in

J = Z r(τ)p(τ|π, O, M)dτ = Z r(τ H)p(τ H|πH, M H)dτ H = Z r(τ L)p(τ L|πL, M L)dτ L

How to optimize and in these MDPs?

πH

πL

π, O

πH

M H

πL

M L

How to optimize and in these MDPs?

22

• Observation1: MH depends on while depends on and
• When we keep the intra-option policies fixed and optimize , we

are implicitly optimizing and

• Observation 2: ML depends on and while depends on
• When we keep the master policy and the termination conditions

fixed and optimize , we are implicitly optimizing

πH

πL

{πo}

πH

π

{βo}

{πo}

πH

π

{βo}

π

{βo}

πL

{πo}

π

πL

{βo}

{πo}

Do we need two critics?

• Proposition: When state-value functions are used as

critics, one critic can be expressed in terms of the other, and hence only one critic is necessary

23

Experiments setup

• Want to see
• DAC vs. existing gradient based option learnings
• DAC vs. hierarchy-free methods
• DAC can use any policy optimization (AC, SAC, NAC,

PPO, …).

• Here focus on DAC + PPO.

24

Results

• Single task
• DAC + A2C similar to OC
• DAC + PPO similar to PPO

25

Results

• Transfer Learning

CartPole = (balance, balance_sparse) Reacher = (easy, hard) Cheetah = (run, backward) Fish = (upright, downleft) Walker1 = (squat, stand) Walker2 = (walk, backward)

26

Recap

• Problem: An end to end, policy based method to learn options

and policy over them

• Solution: Option Critic
• Limitation: Can’t use other policy optimization algorithms off-

the-shelf

• Solution: Reformulate the SMDP of the option framework

as two augmented MDPs (Double Actor Critic)

27

Thank you