SLIDE 1
Per-Decision Option Discounting
Anna Harutyunyan, Peter Vrancx, Philippe Hamel, Ann Nowe, Doina Precup
Motivation: Agents that reason over long temporal horizons
Per-Decision Option Discounting Anna Harutyunyan, Peter Vrancx, - - PowerPoint PPT Presentation
Per-Decision Option Discounting Anna Harutyunyan, Peter Vrancx, - - PowerPoint PPT Presentation
Per-Decision Option Discounting Anna Harutyunyan, Peter Vrancx, Philippe Hamel, Ann Nowe, Doina Precup Motivation: Agents that reason over long temporal horizons Motivation: Agents that reason over long temporal horizons Horizon depends on
SLIDE 2
SLIDE 3
Horizon depends on discount γ Motivation: Agents that reason over long temporal horizons
SLIDE 4
Motivation: Agents that reason over long temporal horizons Horizon depends on discount γ
SLIDE 5
Horizon depends on discount γ Larger grid requires a larger γ Motivation: Agents that reason over long temporal horizons
SLIDE 6
Horizon depends on discount γ Larger grid requires a larger γ Large γ-s are inefficient in practice :( Motivation: Agents that reason over long temporal horizons
SLIDE 7
Horizon depends on discount γ Larger grid requires a larger γ Temporal abstraction? Motivation: Agents that reason over long temporal horizons
SLIDE 8
Motivation: Agents that reason over long temporal horizons Horizon depends on discount γ Larger grid requires a larger γ Temporal abstraction? Options still tied to γ!
SLIDE 9
Motivation: Agents that reason over long temporal horizons Horizon depends on discount γ Larger grid requires a larger γ Temporal abstraction? Options still tied to γ! Contribution: Generalize the options framework to let it extend the agent’s horizon.
SLIDE 10
The Options Framework
Reward model: Transition model:
SLIDE 11
The Options Framework
Reward model: Transition model:
SLIDE 12
Options with Time Dilation
Reward model: Transition model:
(1) decouple (2) per-decision
SLIDE 13
Options with Time Dilation
Reward model: Transition model:
(1) decouple (2) per-decision
SLIDE 14
Options with Time Dilation
Reward model: Transition model:
γp controls how much we care about option duration (pseudo-primitive when γp=1) (1) decouple (2) per-decision
SLIDE 15
Options with Time Dilation
Reward model: Transition model: Key intuition: Insulate option time from global time
γp controls how much we care about option duration (pseudo-primitive when γp=1) (1) decouple (2) per-decision
SLIDE 16
Primitive Timestep Invariance
Ours Classical
SLIDE 17
Bias-Variance Tradeoff
Analytical variance bound Empirical error (Four Rooms)
SLIDE 18
Bias-Variance Tradeoff
Analytical variance bound Empirical error (Four Rooms) Larger γp can induce less variance!
SLIDE 19
Bias-Variance Tradeoff
Analytical variance bound Empirical error (Four Rooms)
Thanks! More at poster #114 :)
Larger γp can induce less variance!