SLIDE 1
NeurIPS 2019
Hindsight Credit Assignment
Anna Harutyunyan, Will Dabney, Thomas Mesnard, Mo Azar, Bilal Piot, Nicolas Heess, Hado van Hasselt, Greg Wayne, Satinder Singh, Doina Precup, Remi Munos
Hindsight Credit Assignment Anna Harutyunyan , Will Dabney, Thomas - - PowerPoint PPT Presentation
Hindsight Credit Assignment Anna Harutyunyan , Will Dabney, Thomas - - PowerPoint PPT Presentation
Hindsight Credit Assignment Anna Harutyunyan , Will Dabney, Thomas Mesnard, Mo Azar, Bilal Piot, Nicolas Heess, Hado van Hasselt, Greg Wayne, Satinder Singh, Doina Precup, Remi Munos NeurIPS 2019 How did past actions infmuence future
SLIDE 2
SLIDE 3
RL relies on MDP structure, and takes time as main proxy for credit relevance
SLIDE 4
RL relies on MDP structure, and takes time as main proxy for credit relevance
SLIDE 5
RL relies on MDP structure, and takes time as main proxy for credit relevance
Time
SLIDE 6
RL relies on MDP structure, and takes time as main proxy for credit relevance
Time
Credit
SLIDE 7
RL relies on MDP structure, and takes time as main proxy for credit relevance
Time
Credit
Meeting at 4pm
SLIDE 8
RL relies on MDP structure, and takes time as main proxy for credit relevance
Time
Credit
Lunch
SLIDE 9
RL relies on MDP structure, and takes time as main proxy for credit relevance
Time
Credit
Umbrella
SLIDE 10
RL relies on MDP structure, and takes time as main proxy for credit relevance
Time
Credit Credit
SLIDE 11
Instead of only relying
- n MDP assumptions,
let’s learn credit relevance explicitly!
SLIDE 12
x
SLIDE 13
future
- utcome
past action x
SLIDE 14
future
- utcome
past action x
SLIDE 15
How did past actions infmuence future outcomes?
future
- utcome
past action x
SLIDE 16
future
- utcome
past action
How did past actions infmuence future outcomes?
x y
SLIDE 17
future
- utcome
past action
How did past actions infmuence future outcomes?
x y
SLIDE 18
future
- utcome
past action
How did past actions infmuence future outcomes?
x y
SLIDE 19
future
- utcome
past action
How did past actions infmuence future outcomes?
x y
SLIDE 20
future
- utcome
past action
How did past actions infmuence future outcomes?
x y z
SLIDE 21
future
- utcome
past action
How did past actions infmuence future outcomes?
x y z
SLIDE 22
future
- utcome
past action
How did past actions infmuence future outcomes?
x y z
SLIDE 23
future
- utcome
past action
How did past actions infmuence future outcomes?
x y z State
SLIDE 24
future
- utcome
past action
How did past actions infmuence future outcomes?
x y z State Return
SLIDE 25
Hindsight Credit Assignment
How relevant was a to get to a state Xk?
SLIDE 26
Hindsight Credit Assignment
How relevant was a to get to a state Xk? How relevant was a to achieve the return Z?
SLIDE 27
Hindsight Credit Assignment
How relevant was a to get to a state Xk? How relevant was a to achieve the return Z?
SLIDE 28
Hindsight Credit Assignment
How relevant was a to get to a state Xk? How relevant was a to achieve the return Z?
HCA Algorithms: Learn the hindsight distribution P, and use it to better estimate value functions or policy gradients
SLIDE 29
Experiments
SLIDE 30
Experiments
SLIDE 31
Thank you for your attention!
Poster #204 :)