Deep RBF Value Functions for Continuous Control Kavosh Asadi - - PowerPoint PPT Presentation

Deep RBF Value Functions for Continuous Control

Kavosh Asadi Ronald Parr George Konidaris Michael Littman

1

The RL Problem

2

agent

a

s, r

3

“The state-space complexity for Go has been estimated at 10174, which is more than the total number of atoms in the universe.”

4

Background

Gt := rt + γrt+1 + γ2rt+2 + ... =

∞

X

i=0

γirt+i

Qπ(s, a) := E[Gt|st = s, at = a, π]

• return
• Q function
• ptimal Q function

Q∗(s, a) := max

π

Qπ(s, a)

hS, A, R, T, γi

• MDP
• policy

π : S → Pr(A)

5

learn from interaction.

s

θ

θ

. . .

ˆ Q(s, a1; θ)

ˆ Q(s, a|A|; θ)

• ptimal action selection:

6

ˆ Q(s, a; θ) ≈

assume a perfect is learned

ˆ Q

arg max

a∈A

ˆ Q(s, a; θ)

[Mnih et al., 2015]

for learning itself:

ˆ Q

θ θ + α

• r + γ max

a0

ˆ Q(s0, a0; θ) ˆ Q(s, a; θ)

• rθ ˆ

Q(s, a; θ)

[Watkins, 1989]

Q∗(s, a) := max

π

Qπ(s, a)

s

Maximization: Approach #1

7

learn a policy network

w w + αrw ˆ Q(s, π(s; w); θ)

π(s; w)

no need to solve , however, exploration is hard

max

a

ˆ Q(s, a; θ)

[Silver et al., 2014]

gradually improve the policy, by increasing :

ˆ Q(s, π(s; w); θ)

[Kakade, Langford 2002 & Matheron et al., 2019]

seems to me like an over-reaction! avoid it using policy gradient methods (super popular recently)

s

a

θ

start with an initial

8

repeat:

ˆ Q(s, a; θ)

max

a∈A

ˆ Q(s, a; θ)

a ∈ A

a a + µra ˆ Q(s, a; θ)

Maximization: Approach #2

Maximization: Approach #3

9

restrict the function class so that max

a

ˆ Q(s, a; θ)

input-convex neural networks [Amos et al., 2017]

| max

a

Q∗(s, a) − max

a

ˆ Q(s, a; θ)|

is easy

s

a

ˆ Q(s, a; θ)

θa

θs

θa

θa

θs

θs

+

θa ≥ 0

two properties better co-exist.

Our Solution: Deep RBF Q Functions

10

• offers both desired properties at the same time
• accurate and efficient action maximization
• universal function approximation.

softmax

… …

dot product

hidden layers

s

a

a1:N

v1:N

k·k

RBF layer

b Q

(RBF: radial-basis function)

Deep RBF Q Functions (1):

11

a

a1

v1

ˆ Q(a)

ˆ Q(a) :=

N

X

i=1

g(a − ai)vi

g(a − ai) := e−β||a−ai||

ˆ Qβ(a) := PN

i=1 e−β||a−ai||vi

PN

i=1 e−β||a−ai||

ˆ Qβ(a) :=

N

X

i=1

e−β||a−ai||vi

“normalized” (stateless)

g

||a − ai||

β = 1

g

||a − ai||

β = 10

softmax

… …

dot product

hidden layers

s

a

a1:N

v1:N

k·k

RBF layer

b Q

12

Deep RBF Q Functions (2):

ˆ Qβ(s, a; θ) := PN

i=1 e−β||a−ai(s;θ)|| vi(s; θ)

PN

i=1 e−β||a−ai(s;θ)||

θ

ˆ Qβ ≈ Q∗

ˆ Qβ(a) := PN

i=1 e−β||a−ai||vi

PN

i=1 e−β||a−ai||

(extension to states)

13

max

a∈A

ˆ Qβ(s, a; θ) = max

i∈[1,N]

ˆ Qβ(s, ai; θ)

0 ≤ max

a∈A

ˆ Qβ(s, a; θ) − max

i∈[1,N]

ˆ Qβ(s, ai; θ) ≤ O(e−β)

A = R

A = Rd

Deep RBF Q Functions (3):

ˆ Qβ

(maximization theory)

14

yes! are deep RBF Q functions capable of universal function approximation? [Hornik et al., 1989] no need to trade expressiveness

Deep RBF Q Functions (4):

∀Q∗ ∀ ≥ 0 ∃✓ |Q∗(s, a) − ˆ Qβ(s, a; ✓)| ≤ ✏

(UFA)

the two desired properties do co-exist.

RBF-DQN

15

• therwise, similar to DQN: use target network, experience replay,
• greedy policy, & RMSProp like DQN.

✏

θ θ + α

• r + γ max

a0

ˆ Qβ(s0, a0; θ) ˆ Qβ(s, a; θ

• r ˆ

Qβ(s, a; θ)

≈ max

i2[1,N]

ˆ Qβ(s0, a0

i; θ)

use a deep RBF Q function, and update parameters:

Evaluations

16

9 standard continuous-action RL problems from Open AI Gym: Classic Control, Box2D, Mujoco.

Comparisons: (1)

17

learn Q using standard neural nets, perform action-maximization using gradient ascent input-convex neural network [Amos et al., 2017] RBF-DQN wire-fitting excluded due to its poor performance [Baird & Klopf, 1994] (more domains in the paper). (with value-function-based baselines)

18

DDPG: Deep Deterministic Policy Gradient [Lilicrap et al., 2015] TD3: Twin Delayed Deep Deterministic policy gradient [Fujimoto et al., 2015].

Comparisons: (2)

RBF-DQN (with policy-gradient baselines)

Conclusion

19

Future Work

deep RBF Q functions, great for continuous control

• facilitate efficient action-maximization
• capable of UFA
• scale, and do well relative to deep RL baselines
• better exploration (uncertainty-based, optimistic initialization, …)
• other RBFs
• better Q training (Double-Q, Distributional Q, …)
• other (real-world) applications: robotics, dialog, education & health care.

20

Thanks

Deep RBF Value Functions for Continuous Control

Kavosh Asadi, Ronald Parr, George Konidaris, Michael Littman

paper: Arxiv.org/abs/2002.01883 code: Github.com/kavosh8/RBFDQN