Michael J. Frank Laboratory for Neural Computation and Cognition - - PowerPoint PPT Presentation

Clustering and generalization of abstract structures in reinforcement learning 


Michael J. Frank

Laboratory for Neural Computation and Cognition Brown University

Reinforcement learning in neural nets and AI

Mnih et al, 2015, Nature

Kansky et al, 2017 See also Witty et al 2018 arXiv Offset Paddle Breakout Breakout trained on Asynchronous Advantage Actor-Critic (A3C)

• 3

But nets show failure to transfer learned knowledge

• Limited WM capacity (curse of dimensionality)
• Multi-tasking (shared representations enhance

learning & generalization)

• Robustness to task contingencies (OpAL vs RL)
• (catastrophic) interference in episodic memory
• Unsupervised (hebbian) vs supervised
• Motor learning – hierarchical structure
• In defense of “small” problems:
• need to understand key elements
• link to neural data / experiments
• Theory of Everything before Theory of Anything!
• 4

What can we learn from limitations of models and humans? Trade-offs

Why does motor learning develop so slowly in humans?

• Standard story: infants born early due to large head, small birth canal
• ‘Fourth trimester’

Why does motor learning develop so slowly in humans?

• Standard story: infants born early due to large head, small birth canal
• ‘Fourth trimester’
• But 3 month old infants are still pretty incompetent (from babycenter.com):

Why does motor learning develop so slowly in humans?

• Standard story: infants born early due to large head, small birth canal
• ‘Fourth trimester’
• But 3 month old infants are still pretty incompetent (from babycenter.com):

‘you no longer need to support his head. When he’s on his stomach he can lift his head and chest. He can open and close his hands..’

Why does motor learning develop so slowly in humans?

• Standard story: infants born early due to large head, small birth canal
• ‘Fourth trimester’
• But 3 month old infants are still pretty incompetent (from babycenter.com):

‘you no longer need to support his head. When he’s on his stomach he can lift his head and chest. He can open and close his hands..’

• Hypothesis: human brain is wired to discover latent generalizable structure,

which is initially inefficient – see Werchan et al 2016!

Why does motor learning develop so slowly in humans?

• Standard story: infants born early due to large head, small birth canal
• ‘Fourth trimester’
• But 3 month old infants are still pretty incompetent (from babycenter.com):

‘you no longer need to support his head. When he’s on his stomach he can lift his head and chest. He can open and close his hands..’

• Hypothesis: human brain is wired to discover latent generalizable structure,

which is initially inefficient

Why does motor learning develop so slowly in humans?

• Standard story: infants born early due to large head, small birth canal
• ‘Fourth trimester’
• But 3 month old infants are still pretty incompetent (from babycenter.com):

‘you no longer need to support his head. When he’s on his stomach he can lift his head and chest. He can open and close his hands..’

• Hypothesis: human brain is wired to discover latent generalizable structure,

which is initially inefficient

Driving rules UK Driving rules Montreal and…

Humans learn contextualized
 rule structures

A key structure: Task-sets (TS)

stimuli actions Cue 1

Task-sets (TS)

S1 S2 S3 A1 A2 A3 C1

Task-sets (TS)

Si1 Ai1 C1 Si2 Ai2 C2 Si3 Ai3 C3 Si4 Ai4 C4 Si5 Ai5 C5 Si6 Ai6 C6

Abstracting Task-set rules

C1 Si1 Ai1 C2 C3 C4 Si2 Ai2 C5 C6

TS1 TS2

Latent task-set space

Collins & Frank 2013

Popularity Prior on Task-set rules

C1 Si1 Ai1 C2 C3 C4 Si2 Ai2 C5 C6

TS1 TS2

C7

?

Collins & Frank 2013

C2

CRP Prior on TS in a new context: P0(TS = TSj |Cnew) = N(TSj|C*) / [α + Σi N(TSi| C*)] P0(TS = new|Cnew) = α / [α + Σi N(TSi| C*)]

Ability to create new Task-set rules

C1 Si1 Ai1 C2 C3 C4 Si2 Ai2 C5 C6

TS1 TS2

Si Ai C7

TSnew

Latent task-set space: Unknown size

Collins & Frank 2013

Linking algorithmic model and neural network model

Both models are approximations of the same process: TS space building

Collins & Frank, Psych Review, 2013

DA BG TSi BG Ai

CTS-model Neural Network-model

Clustering vs partitioning task space in frontostriatal circuits via RL

Old TS New TS generalization & transfer

RL

Collins & Frank 2013; 2016; Frank & Badre, 2012

Clustering vs partitioning task space in frontostriatal circuits via RL

C-PFC sparseness

Fitted clustering

prior Old TS New TS generalization & transfer

RL

Collins & Frank 2013; 2016; Frank & Badre, 2012

Model mimicry: C-TS and hierarchical neural net are approximations of same structure building process

Vector reward prediction errors:
 “actor-specific” computations

• DA signals are tailored to computations of underlying FC-BG circuit
• “Mixture of Experts” (Frank & Badre 2012; fMRI: Badre & Frank 2012; Collins & Frank 2013…)
• Vector RPEs

“Mixture of Experts”

MIXTURE

Hierarchical task

Flat task

S1 S2 C0 A1 A2 C1 A1 A2 C2 A3 A4 C3 C4

Initial Phase

Appending to latent task structures: beyond the identity mapping..

S1 S2 S3 S4 C0 A1 A2 A1 A4 C1 A1 A2 A1 A4 C2 A3 A4 A3 A2 C3 C4

Initial Phase Transfer Phase 1

Appending to latent task structures: extrapolating beyond the identity mapping

C0 C1 C2 TS1 TS2 A1 A2 A3 A4 ? ? 1/4 1/4 1/2 Trial# per input pattern Proportion Correct C0, C1 C2

*

Subjects (N=34)

Initial phase

Trial# per input pattern Proportion Correct C0, C1 C2

*

Subjects (N=34)

Phase 2

2 4 6 8 0.2 0.4 0.6 0.8 1 H

Proportion Correct C0, C1 C2 Model Trial# per input pattern

Trial# per input pattern Proportion Correct C0, C1 C2

*

Subjects (N=34)

Initial phase

2 4 6 8 0.2 0.4 0.6 0.8 1 H

Proportion Correct C0, C1 C2 Model

2 4 6 8 0.2 0.4 0.6 0.8 1 init

Proportion Correct C0, C1 C2 Model Trial# per input pattern Proportion Correct C0, C1 C2

*

Subjects (N=34)

Phase 2

Can subjects generalize learned rules to new contexts?

C0 C1 C2 TS1 TS2 C4 TS3 C3 C3

S1 S2 S3 S4 C0 TS1 TS1 C1 TS1 TS1 C2 TS2 TS2 C3 TS old C4 TS new

Initial Phase Transfer Phase 2 Transfer Phase 1

Can subjects generalize learned rules to new contexts?

C0 C1 C2 TS1 TS2 A1 A2 A3 A4 C3 C3 C4 TS3 A1 A4

Subjects (N = 34) Trial# per input pattern Proportion Correct *

2 4 6 8 0.2 0.4 0.6 0.8 1 CV

C3: TS old C4: TS new Model Trial# per input pattern Proportion Correct

C0 C1 C2 TS1 TS2 A1 A2 A3 A4 ? ? Correct Correct Correct Correct

Prediction error:

PE = reward - expectation PE PE Structure learning PE

Prediction error (PE) in EEG signal

Collins & Frank (2016), Cognition Time from FB

~ β0 + βPE + βStr trial number For each subject: βPE(electrodes, time) βStr(electrodes, time)

Prediction error (PE) in EEG signal
 Structure PE in EEG signal

PE effect Time from feedback (ms) average βPE

Collins & Frank (2016) Cognition

EEG(trial) ~ β0 + βPE PE(trial) + βStr StructurePE(trial) ** * ns Unique effect of Structure learning PE ** * ROI1 ROI2 ROI1 ROI2

Structure PE signal predicts transfer.

2 4 6 8 0.2 0.4 0.6 0.8 1 P(Correct) Iteration # 2 4 6 8 0.2 0.4 0.6 0.8 1 P(Correct) Iteration #

C3-TS old C4-TS new Unique effect of Structure learning PE

TS1 TS2

• ther

0.1 0.2 0.3 0.4 0.5 0.6 % Choices

New context Prior

TS1 action TS2 action

• ther

action

Collins & Frank, Cognition, accepted

ROI1+2

It affords transfer It depends on clustering priors It informs neural representations of reward predictions Structure learning

** * ns

Unique effect of Structure learning PE

** *

PE effect

Trial# per input pattern Proportion Correct * Trial# per input pattern Proportion Correct C0, C1 C2 *

Neural model & EEG: TS switch effects

Neural model & EEG: TS switch effects

S1 S2 S3 S4 C0 TS1 TS1 C1 TS1 TS1 C2 TS2 TS2 C3 TS old C4 TS new

Initial Phase Transfer Phase 2 Transfer Phase 1

Structure learning affords transfer

• f new information within learned

clusters No early clustering benefit

• early structure learning is

costly Structure learning affords transfer of known rules to new contexts – with popularity clustering prior Neural signatures of

hierarchical prediction errors predict structure learning/ transfer:

Badre & Frank 2012; Collins et al 2014, 2016

New Context: Old TS New TS

*

N = 33

Significant whole group positive transfer

Do we build structure a priori?

Werchan et al, 2016, JNeurosci

Share: Physical Movements (mappings from sounds to notes) Share: Chord progression, rhythm, etc (desired sound/ song)

• 38

• 39

Piccolo Need compositionality: reuse flute mappings to play a song usually played on guitar

Cluster Rewards Transitions

C

Rewards What do you want to do? Transitions How can you do it?

• 40

vs vs Clustering goals and transitions jointly will not allow generalization of each independent of the other

Cluster Rewards Transitions Rewards Transitions

ClusterR ClusterT

C C

• 41

Joint Clustering Independent Clustering Nick Franklin

𝑺 𝝔 𝑺 𝝔 𝝔 𝑺 𝝔

Joint Clustering Independent Clustering Policy

contexts clusters functions policies

Policy Policy Policy

C C C Franklin & Frank, 2018 PLOS Computational Biology C C C

a4 a2 a1 a8 a7 a6 a5 a3

𝑏 = { } 𝐵 = { } 𝜚:𝑏 → 𝐵 Reduced Transition Cardinal Movements Actions

N S E W

Rewards 𝑆:𝑦 → {0, + 1} What to do? Rewards: How to do it? Transition: 𝑦 ∈ { < 𝑦𝑗, 𝑦𝑘 > :𝑗 = 1,…, 6; 𝑘 = 1, …, 6} Model-based RL: What-to-do vs How-to-do-it?

• 43

Mapping () Reward ()

C1 C2 C3 C4

Agent Performance

• Independent clustering: Fewer steps to

solve task domain

• Independent clustering: better estimates
• f and (lower KL-Divergence)

Franklin & Frank, 2018 PLOS Computational Biology

• Joint clustering: Fewer steps to solve

task domain

• Joint clustering: better estimates of

and (lower KL-Divergence)

Function Estimates Agent Performance

𝜚(𝑏, 𝐵) 𝑆(𝑦)

CC CC CC CC

• 45

Transitions

C Franklin & Frank, 2018 PLOS Computational Biology

• 46

𝜚(𝑏, 𝐵) 𝑆(𝑦)

C C C C C C C C

Meta Model

• Meta-agent uses RL processes to learn to

select independent/joint clustering as actor

• Choose Indep./Joint for a single trial
• Normative: Better than the worst of Joint/

Independent (i.e. minimax)

• Why is doing better than the worst a good

idea?

Meta- agent Joint actor Ind. actor 𝑥 1 − 𝑥

Mapping () Reward ()

C1 C2 C3 C4

Franklin & Frank, 2018 PLOS Computational Biology

Generalizing to generalize

• New room = new context
• Different transitions in each room
• Exploration costs: potentially non-linear
• Generalize benefit
• Joint: ~2x faster than flat
• Indep: ~2.5x faster than joint
• Advantage grows exponentially with size of grid-world
• Advantage grows linearly with more rooms

“Diabolical Rooms”

• 47

2. Theory

Back to Start Back to Start End

• 48

C C C C

𝜚(𝑏, 𝐵) 𝑆(𝑦) 𝜚(𝑏, 𝐵) 𝑆(𝑦)

C C C C C C C C

?

Generalization via Bayesian Inference

Posterior probability context is in a cluster Likelihood

• f data given a

context is in a cluster Prior probability a context is in a cluster

Cluster C C C C C C Cluster

task specific task general

• 49

2. Theory

How good of a guess about cluster assignment is the prior? Formally, how good of an estimator is the prior of the generative process, regardless of task specifics? ∝ ×

Joint vs Independent Clustering

Joint Clustering

• Consider transitions when generalizing goals conditional
• More complex
• Bias/Variance better asymptotically

Independent clustering

• Generalize goals independent of transitions marginal
• Less complex
• Bias/Variance robust to noise

C C C Rewards Transitions Rewards Transitions Transitions Rewards C C C Transitions

Joint Clustering Independent Clustering

A

• 51

a1 a4 a2 a8 a7 a6 a5 a3

1 B 1

a1 a4 a2 a8 a7 a6 a5 a3

2 A 2 C

Only consider prior: Don’t need statistics of specific transitions, rewards; replace with arbitrary symbol

A

• 52

1 B 1 2 A 2 C

R-Sequence T-Sequence

1 ?

Update CRP prior in each context and ask how well it can guess the next reward? * CRP with independent clustering * CRP with joint clustering

G Sequence: AAAAAAAAAABBBBBBBBBB T Sequence: 11111222222222211111 G Sequence: AAAAAAAAAABBBBBBBBBB T Sequence: 11111122222222221111 G Sequence: AAAAAAAAAABBBBBBBBBB T Sequence: 11111112222222222111 Hold the structure of each sequence constant Increase mutual information () by progressively switching pairs Independent Clustering better with low mutual information

• 53

2. Theory

Independent vs Joint (bits)

• Previous analysis assumed T fully observable and

noiseless

• Relax assumption?
• 54

2. Theory

G Sequence: AAAAAAAAAAAAAAAAAAAABCD T Sequence: 11111111111111111111234 G Sequence: AAAAAAAAAAAAAAAAAAAABCD T Sequence: 11111111111111111115234 G Sequence: AAAAAAAAAAAAAAAAAAAABCD T Sequence: 11111111111111111156234 Hold mutual information constant T perfectly predicts G (HGT=0) Increase chance of misidentifying T i.e. noise Even w/ T perfectly predicts G, independent clustering is useful when T observations are noisy

• 55

2. Theory

Independent vs Joint (bits)

G Sequence: AAAAAAAAAAAAAAAAAAAABCDBCD T Sequence: 11111111111111111111234342 G Sequence: AAAAAAAAAAAAAAAAAAAABCDBCD T Sequence: 11111111111111111115234342 G Sequence: AAAAAAAAAAAAAAAAAAAABCDBCD T Sequence: 11111111111111111156234342 Reduce Larger compositionality benefit Reduce by adding to sequence Hold mutual information constant T perfectly predicts G Increase chance of misidentifying T i.e. noise Even w/ T perfectly predicts G, independent clustering is useful when T observations are noisy

• 56

2. Theory

𝐼(𝐻 𝑈 ) ≈ 0.2 bits Independent vs Joint (bits)

1. Normative behavior depends on task-domain 2. Meaningful differences 3. Independent simpler statistical model

a) Joint better asymptotically b) Independent better in noisy and/or independent environments

• 57

2. Theory

• 1. Background
• 2. Theory
• 3. Human Behavior
• 58

N S E W N S E W N S E W

f s a ; l k j d

B A C

Transitions:

relationship between keypresses and cardinal movements

Goal Values

3. Human Behavior 3. Human Behavior

N S E W

f s a ; l k j d

B A C

Transitions:

relationship between keypresses and cardinal movements

Goal Values Where do you want to go? How do you get there?

Do people cluster Goals and Transitions together (jointly) ? Do people cluster Goals and Transitions independently?

3. Human Behavior 3. Human Behavior

• 61

3. Human Behavior 3. Human Behavior

A B C Goal Values

B A C

• 62

3. Human Behavior

f s a ; l k j d

Transitions

3. Human Behavior

B A C

A B C Goal Values

• 63

3. Human Behavior

Transitions

f s a ; l k j d

3. Human Behavior

A B C Goal Values

• 64

3. Human Behavior

Transitions

f s a ; l k j d

B A C

3. Human Behavior

B A C

A B C Goal Values

• 65

3. Human Behavior

Transitions

f s a ; l k j d

3. Human Behavior

A B C Goal Values

• 66

3. Human Behavior

Transitions

B A C

f s a ; l k j d

3. Human Behavior

A B C Goal Values

• 67

3. Human Behavior

Transitions

f s a ; l k j d

?

3. Human Behavior

A B C

• 68

3. Human Behavior Joint Clustering

• Sensitive to transitions

B A C B A C

Independent Clustering

• Insensitive to transitions

B A C

Goal Popularity Goal Popularity

3. Human Behavior

1 4 2 3

Test Contexts

• 69

Model and Human Behavior: Ambiguous Structure

• When training structure is ambiguous,

generalization is a combination of indep & joint: Meta-generalization

Follow-up:

• 1. Can we encourage independent clustering?
• 2. Can we encourage joint clustering?
• 70

• 71

Test Contexts

Model and Human Behavior: Independent Structure

Test Contexts A B C

1 4 2 3

D

Model and Human Behavior: Independent Structure

• 73

Model and Human Behavior: Joint Structure

Following-up:

• 1. Can we encourage independent clustering? Yes!
• 2. Can we encourage joint clustering? Yes!
• 3. Meta Model?
• 74

Summary

• Learning involves decision making at multiple levels: choose task structure,

choose action, etc.

• Hierarchical PFC-BG interactions support structure learning
• Slower to acquire correct action contingencies: have to learn world structure (what

is C, what is S?) simultaneous with actions

• But structure learning affords generalization:
• Expanding a rule in a known context-cluster
• Transferring a rule to a new context
• Role in development?
• Joint & Independent Clustering are dissociable; Normative strategy reflects

task structure

LNCC Anne Collins (Berkeley)

Thanks!

Nick Franklin Denise Werchan Dima Amso Jim Cavanagh Nick Franklin (Harvard)