Better Transfer Learning with Inferred Successor Maps Tamas Madarasz - - PowerPoint PPT Presentation

▶
better transfer learning with inferred successor maps
SMART_READER_LITE
LIVE PREVIEW

Better Transfer Learning with Inferred Successor Maps Tamas Madarasz - - PowerPoint PPT Presentation

May 23, 2023 104 likes •470 views

Better Transfer Learning with Inferred Successor Maps Tamas Madarasz 1,2 , Tim Behrens 1,2 arXiv:1906.07663 Spotlight NeurIPS 2019 1: University of Oxford 2: UCL The successor representation (SR) Dayan, 1993 Neural Computation The successor

slide-1
SLIDE 1

Better Transfer Learning with Inferred Successor Maps

Tamas Madarasz1,2, Tim Behrens1,2 arXiv:1906.07663 Spotlight NeurIPS 2019

1: University of Oxford 2: UCL

slide-2
SLIDE 2
slide-3
SLIDE 3
slide-4
SLIDE 4
slide-5
SLIDE 5
slide-6
SLIDE 6
slide-7
SLIDE 7
slide-8
SLIDE 8
slide-9
SLIDE 9
slide-10
SLIDE 10
slide-11
SLIDE 11
slide-12
SLIDE 12

The successor representation (SR)

Dayan, 1993 Neural Computation

slide-13
SLIDE 13

The successor representation (SR)

Dayan, 1993 Neural Computation reward function

slide-14
SLIDE 14
slide-15
SLIDE 15
slide-16
SLIDE 16
slide-17
SLIDE 17
slide-18
SLIDE 18
slide-19
SLIDE 19
slide-20
SLIDE 20

Wilson et al. 2007, ICML Lazaric and Ghamazadev 2010, ICML Finn et al. 2017, ICML

Main approach

  • Cluster tasks and try to map current task to the cluster such

that SR is easiest to adapt

  • Use the SR’s flexibility to approximate the optimal value

function

slide-21
SLIDE 21

Generative model over reward functions

slide-22
SLIDE 22

Generative model over reward functions

Dirichlet Process mixture model of kernel- smoothed rewards

slide-23
SLIDE 23

Generative model over reward functions

Dirichlet Process mixture model of kernel- smoothed rewards

slide-24
SLIDE 24

Generative model over reward functions

Dirichlet Process mixture model of kernel- smoothed rewards

slide-25
SLIDE 25

Bayesian Successor Representation (BSR)

M: Successor Representation CR: Convolved reward map

slide-26
SLIDE 26

Bayesian Successor Representation (BSR)

slide-27
SLIDE 27

Bayesian Successor Representation (BSR)

slide-28
SLIDE 28

Bayesian Successor Representation (BSR)

slide-29
SLIDE 29

Bayesian Successor Representation (BSR)

slide-30
SLIDE 30

Bayesian Successor Representation (BSR)

slide-31
SLIDE 31

Results

Barreto et al. 2017 NeurIPS

slide-32
SLIDE 32

Multi-task exploration bonus by offsetting the reward belief vector w

w UCB inspired constant offset w Offset using CR maps, acting as priors for rewards

Auer 2002 JMLR

slide-33
SLIDE 33

Results

slide-34
SLIDE 34

Results

slide-35
SLIDE 35

Hippocampus

Results

Boccara et al. 2019 Science Jezek et al. 2019 Nature Grieves et al. 2016 Elife Blum and Abbot 1996 Levy et al. 2005 Stachenfeld et al. 2017

slide-36
SLIDE 36

Thank you!

arXiv:1906.07663 Transfer and Multi-task learning Poster#52

10:45 AM - 12:45 PM