SLIDE 1
Motivation
Problem: standard Q-Learning/SARSA “propagates rewards” only one state back per time step – n-step bootstrapping is one way to address this problem – we will see other ways in subsequent slide decks.
N-step bootstrapping Robert Platt Northeastern University Id love - - PowerPoint PPT Presentation
N-step bootstrapping Robert Platt Northeastern University Id love - - PowerPoint PPT Presentation
N-step bootstrapping Robert Platt Northeastern University Id love to use my experiences more efficiently... Motivation Left: path taken by agent in grid world. Gets zero reward everywhere except for goal state where it gets positive
SLIDE 2
SLIDE 3
TD and MC are two extremes of a continuum
SLIDE 4
TD and MC are two extremes of a continuum
What are these?
SLIDE 5
TD and MC are two extremes of a continuum
SLIDE 6
TD and MC are two extremes of a continuum
Update equation:
SLIDE 7
TD and MC are two extremes of a continuum
Is called the target of the update Update equation:
SLIDE 8
TD and MC are two extremes of a continuum
What’s the target for this one? Update equation:
SLIDE 9
TD and MC are two extremes of a continuum
Complete update equation: What’s the target for this one?
SLIDE 10
TD and MC are two extremes of a continuum
Complete update equation: What’s the target for this one?
SLIDE 11
TD and MC are two extremes of a continuum
Complete update equation: What’s the target for this one?
SLIDE 12
TD and MC are two extremes of a continuum
Complete update equation: What’s the target for this one?
Notice that you can’t do this update until time step t+3 – TD update happens on next time step – MC update happens at end of episode – n-step TD update happens on time step n
SLIDE 13
How well does this work?
This comparison is for: – a 19 state random walk policy – n-step TD policy evaluation
SLIDE 14
n-step TD algorithm
SLIDE 15
n-step SARSA
Same idea as in n-step TD – how is this backup diagram different from that of n-step TD? – why is it different?
SLIDE 16
n-step SARSA
Same idea as in n-step TD – how is this backup diagram different from that of n-step TD? – why is it different? Why does the backup start with a dot rather than a circle?
SLIDE 17