SLIDE 1
DualDICE
Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
Ofir Nachum,* Yinlam Chow,* Bo Dai, Lihong Li
Google Research
*Equal contribution
DualDICE Behavior-Agnostic Estimation of Discounted Stationary - - PowerPoint PPT Presentation
DualDICE Behavior-Agnostic Estimation of Discounted Stationary - - PowerPoint PPT Presentation
DualDICE Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections Ofir Nachum ,* Yinlam Chow,* Bo Dai, Lihong Li Google Research *Equal contribution Reinforcement Learning Reinforcement Learning A policy acts on an
SLIDE 2
SLIDE 3
Reinforcement Learning
- A policy acts on an environment.
SLIDE 4
Reinforcement Learning
- A policy acts on an environment.
s0 Initial state distribution β
SLIDE 5
Reinforcement Learning
- A policy acts on an environment.
s0 Initial state distribution β 𝛒(-|s0) a0
SLIDE 6
Reinforcement Learning
- A policy acts on an environment.
s0 Initial state distribution β 𝛒(-|s0) a0 R(-|s0, a0) r0
SLIDE 7
Reinforcement Learning
- A policy acts on an environment.
s0 Initial state distribution β 𝛒(-|s0) T(-|s0, a0) a0 s1 R(-|s0, a0) r0
SLIDE 8
Reinforcement Learning
- A policy acts on an environment.
s0 Initial state distribution β 𝛒(-|s0) T(-|s0, a0) a0 s1 R(-|s0, a0) r0 𝛒(-|s1) T(-|s1, a1) a1 R(-|s1, a1) r1
SLIDE 9
Reinforcement Learning
- A policy acts on an environment.
s0 Initial state distribution β 𝛒(-|s0) T(-|s0, a0) a0 s1 R(-|s0, a0) r0 𝛒(-|s1) T(-|s1, a1) a1 s2 R(-|s1, a1) r1 𝛒(-|s2) T(-|s2, a2) a2 R(-|s2, a2) r2
SLIDE 10
Reinforcement Learning
- A policy acts on an environment.
s0 Initial state distribution β 𝛒(-|s0) T(-|s0, a0) a0 s1 R(-|s0, a0) r0 𝛒(-|s1) T(-|s1, a1) a1 s2 R(-|s1, a1) r1 𝛒(-|s2) T(-|s2, a2) a2 R(-|s2, a2) r2
- Question: What is the value (average reward) of the policy?
SLIDE 11
Off-policy Policy Estimation
SLIDE 12
Off-policy Policy Estimation
- Want to estimate average discounted per-step reward of policy,
SLIDE 13
Off-policy Policy Estimation
- Want to estimate average discounted per-step reward of policy,
- Only have access to finite experience dataset
where transitions are from some unknown distribution
s, a, r, s’ s, a, r, s’ s, a, r, s’ s, a, r, s’ s, a, r, s’
. . .
SLIDE 14
Off-policy Policy Estimation
- Want to estimate average discounted per-step reward of policy,
- Only have access to finite experience dataset
where transitions are from some unknown distribution
- Don’t even know the behavior policy!
s, a, r, s’ s, a, r, s’ s, a, r, s’ s, a, r, s’ s, a, r, s’
. . .
SLIDE 15
Reduction of OPE to Density Ratio Estimation
SLIDE 16
Reduction of OPE to Density Ratio Estimation
- Can write where dπ is discounted on-policy distribution
SLIDE 17
Reduction of OPE to Density Ratio Estimation
- Can write where dπ is discounted on-policy distribution
- Using importance weighting trick, we have,
SLIDE 18
Reduction of OPE to Density Ratio Estimation
- Can write where dπ is discounted on-policy distribution
- Using importance weighting trick, we have,
- Given finite dataset, this corresponds to weighted average,
SLIDE 19
Reduction of OPE to Density Ratio Estimation
- Can write where dπ is discounted on-policy distribution
- Using importance weighting trick, we have,
- Given finite dataset, this corresponds to weighted average,
- Problem reduces to estimating weights (density ratios)
SLIDE 20
Reduction of OPE to Density Ratio Estimation
- Can write where dπ is discounted on-policy distribution
- Using importance weighting trick, we have,
- Given finite dataset, this corresponds to weighted average,
- Problem reduces to estimating weights (density ratios)
SLIDE 21
Reduction of OPE to Density Ratio Estimation
- Can write where dπ is discounted on-policy distribution
- Using importance weighting trick, we have,
- Given finite dataset, this corresponds to weighted average,
- Problem reduces to estimating weights (density ratios)
- Difficult because we don’t have access to environment and we don’t have explicit
knowledge of dD(s,a), only samples.
SLIDE 22
- Define zero-reward Bellman operator as
The DualDICE Objective
SLIDE 23
- Define zero-reward Bellman operator as
The DualDICE Objective
SLIDE 24
- Define zero-reward Bellman operator as
The DualDICE Objective
minimize squared Bellman error
SLIDE 25
- Define zero-reward Bellman operator as
The DualDICE Objective
s0 s1 s2 s3 s4
minimize squared Bellman error
SLIDE 26
- Define zero-reward Bellman operator as
The DualDICE Objective
maximize initial “nu-values”
s0 s1 s2 s3 s4
minimize squared Bellman error
SLIDE 27
- Define zero-reward Bellman operator as
The DualDICE Objective
maximize initial “nu-values”
s0 s1 s2 s3 s4
minimize squared Bellman error
SLIDE 28
- Define zero-reward Bellman operator as
The DualDICE Objective
maximize initial “nu-values”
s0 s1 s2 s3 s4
minimize squared Bellman error
SLIDE 29
- Define zero-reward Bellman operator as
- Nice: Objective is based on expectations from dD, β, and π, which we have access to.
The DualDICE Objective
maximize initial “nu-values”
s0 s1 s2 s3 s4
minimize squared Bellman error
SLIDE 30
- Define zero-reward Bellman operator as
- Nice: Objective is based on expectations from dD, β, and π, which we have access to.
- Extension 1: Can remove appearance of Bellman operator from both objective and
solution by application of Fenchel conjugate!
The DualDICE Objective
maximize initial “nu-values”
s0 s1 s2 s3 s4
minimize squared Bellman error
SLIDE 31
- Define zero-reward Bellman operator as
- Nice: Objective is based on expectations from dD, β, and π, which we have access to.
- Extension 1: Can remove appearance of Bellman operator from both objective and
solution by application of Fenchel conjugate!
- Extension 2: Can generalize this result to any convex function (not just square)!
The DualDICE Objective
maximize initial “nu-values”
s0 s1 s2 s3 s4
minimize squared Bellman error
SLIDE 32
DualDICE Results
- DualDICE accuracy during training compared to existing methods.
SLIDE 33
DualDICE Results
- DualDICE accuracy during training compared to existing methods.