[PPT] - Universality and Individuality in Recurrent Neural Networks Niru PowerPoint Presentation

SLIDE 1

Universality and Individuality in Recurrent Neural Networks

Niru Maheswaranathan, Alex Williams, Matthew D Golub, Surya Ganguli, David Sussillo Google Brain & Stanford University NeurIPS 2019 arxiv:1907.08549 Poster #179

SLIDE 2

Artificial neural networks in neuroscience

Advantages:

Can train ANNs to accomplish tasks analogous to those studied in animals.
Can inspect/probe/dissect artificial networks very easily.
Can easily initiate a huge number of in silico studies

...

arxiv:1907.08549 Poster #179

SLIDE 3 Dots

n

Dots

ff

Dots

ff

Rotate Sort by irrelevant colour Choice Motion 6 1 Choice Colour Choice Colour Choice 1 Choice 2 Choice 1 Choice 2 Choice 1 Choice 2 Strong Weak Strong To choice 1 To choice 2 Motion Strong Weak Strong To choice 1 To choice 2 Colour

a b c

Attractor Line attractor Colour context Motion context Choice axis Motion context Dots

n

Dots

ff

Dots

ff

Rotate Sort by irrelevant colour Choice Motion 1.5 1 Choice Colour Choice Colour Choice 1 Choice 2 Choice 1 Choice 2 Choice 1 Choice 2 Strong Weak Strong To choice 1 To choice 2 Motion Strong Weak Strong To choice 1 To choice 2 Colour a b c

Mante & Sussillo et al. Nature 2013

Yamins & DiCarlo, 2014 Carnevale et al, NEURON, 2015 Rajan, Harvey, Tank, Neuron, 2016

grid-like band-like border a b c

Cueva & Wei, ICLR, 2018

d

Normalized activity

e

Orhan & Ma, Nature Neuroscience, 2019

target move onset

Sussillo et al., Nature Neuroscience, 2015

Go RT Go Dly Go Anti RT Anti Dly Anti DM 1 DM 2 Ctx DM 1 Ctx DM 2 MultSen DM Dly DM 1 Dly DM 2 Ctx Dly DM 2 MultSen Dly DM DMS DNMS DMC DNMC 0.5 –0.5 Performance change after lesioning Normalized task variance 1 1 2 3 4 5 6 Clusters Units 7 9 10 11 12 8 1 2 3 4 5 6 Clusters tSNE Task Unit 145 Unit 145 Go a c b d e Time (s) 1.5 0.0 DNMC Go 0.00 0.05 0.5 1.0 0.0 Task variance Activitity (a.u.) 7 9 10 11 12 8 Ctx Dly DM 1 Go RT Go Dly Go Anti RT Anti Dly Anti DM 1 DM 2 Ctx DM 1 Ctx DM 2 MultSen DM Dly DM 1 Dly DM 2 Ctx Dly DM 2 MultSen Dly DM DMS DNMS DMC DNMC Ctx Dly DM 1

Yang, et al., Nature Neuroscience 2019

Artificial & biological neural networks

Zipser & Andersen, Science, 1988 Maheswaranathan et al, 2018

SLIDE 4 Dots

n

Dots

ff

Dots

ff

Rotate Sort by irrelevant colour Choice Motion 6 1 Choice Colour Choice Colour Choice 1 Choice 2 Choice 1 Choice 2 Choice 1 Choice 2 Strong Weak Strong To choice 1 To choice 2 Motion Strong Weak Strong To choice 1 To choice 2 Colour

a b c

Attractor Line attractor Colour context Motion context Choice axis Motion context Dots

n

Dots

ff

Dots

ff

Rotate Sort by irrelevant colour Choice Motion 1.5 1 Choice Colour Choice Colour Choice 1 Choice 2 Choice 1 Choice 2 Choice 1 Choice 2 Strong Weak Strong To choice 1 To choice 2 Motion Strong Weak Strong To choice 1 To choice 2 Colour a b c

Mante & Sussillo et al. Nature 2013

Yamins & DiCarlo, 2014 Carnevale et al, NEURON, 2015 Rajan, Harvey, Tank, Neuron, 2016

grid-like band-like border a b c

Cueva & Wei, ICLR, 2018

d

Normalized activity

e

Orhan & Ma, Nature Neuroscience, 2019

target move onset

Sussillo et al., Nature Neuroscience, 2015

Go RT Go Dly Go Anti RT Anti Dly Anti DM 1 DM 2 Ctx DM 1 Ctx DM 2 MultSen DM Dly DM 1 Dly DM 2 Ctx Dly DM 2 MultSen Dly DM DMS DNMS DMC DNMC 0.5 –0.5 Performance change after lesioning Normalized task variance 1 1 2 3 4 5 6 Clusters Units 7 9 10 11 12 8 1 2 3 4 5 6 Clusters tSNE Task Unit 145 Unit 145 Go a c b d e Time (s) 1.5 0.0 DNMC Go 0.00 0.05 0.5 1.0 0.0 Task variance Activitity (a.u.) 7 9 10 11 12 8 Ctx Dly DM 1 Go RT Go Dly Go Anti RT Anti Dly Anti DM 1 DM 2 Ctx DM 1 Ctx DM 2 MultSen DM Dly DM 1 Dly DM 2 Ctx Dly DM 2 MultSen Dly DM DMS DNMS DMC DNMC Ctx Dly DM 1

Yang, et al., Nature Neuroscience 2019

Artificial & biological neural networks

Zipser & Andersen, Science, 1988

Networks have surprisingly similar representations…

Maheswaranathan et al, 2018

SLIDE 5

Artificial & biological neural networks

Biological neuron Artificial neuron …but are composed of drastically different elements!

arxiv:1907.08549 Poster #179

SLIDE 6

When trained to perform the same task, why should we expect artificial and biological networks to be similar, given the drastic differences in underlying mechanism?

Central question

arxiv:1907.08549 Poster #179

SLIDE 7

This work: an empirical approach

Similarity measures

Canonical correlation analysis (CCA) Centered kernel alignment (CKA)

Tasks

Decision making Pattern generation Working memory

Network mechanisms

RNN architectures (e.g. LSTMs, GRUs, …) Nonlinearities (e.g. ReLU, tanh) … arxiv:1907.08549 Poster #179

SLIDE 8

Evidence of both universality and individuality

arxiv:1907.08549 Poster #179

SLIDE 9

Evidence of both universality and individuality

Pattern generation task arxiv:1907.08549 Poster #179

SLIDE 10

Evidence of both universality and individuality

Pattern generation task arxiv:1907.08549 Poster #179 Analyzing trained networks

PC #1 PC #2 PC #3

SLIDE 11

Evidence of both universality and individuality

Pattern generation task arxiv:1907.08549 Poster #179 Analyzing trained networks

PC #1 PC #2 PC #3

SLIDE 12

Evidence of both universality and individuality

Network representations show individuality Pattern generation task arxiv:1907.08549 Poster #179 Analyzing trained networks

PC #1 PC #2 PC #3

SLIDE 13

Evidence of both universality and individuality

Network representations show individuality Pattern generation task arxiv:1907.08549 Poster #179 but aspects of the computation are universal

Target Linear prediction

Analyzing trained networks

PC #1 PC #2 PC #3

SLIDE 14

Universality and Individuality in Recurrent Neural Networks Niru - - PowerPoint PPT Presentation

Universality and Individuality in Recurrent Neural Networks

Niru Maheswaranathan, Alex Williams, Matthew D Golub, Surya Ganguli, David Sussillo Google Brain & Stanford University NeurIPS 2019 arxiv:1907.08549 Poster #179

Artificial neural networks in neuroscience

Advantages:

arxiv:1907.08549 Poster #179

Artificial & biological neural networks

Artificial & biological neural networks

Networks have surprisingly similar representations…

Artificial & biological neural networks

Biological neuron Artificial neuron …but are composed of drastically different elements!

arxiv:1907.08549 Poster #179

When trained to perform the same task, why should we expect artificial and biological networks to be similar, given the drastic differences in underlying mechanism?

Central question

arxiv:1907.08549 Poster #179

This work: an empirical approach

Similarity measures

Canonical correlation analysis (CCA) Centered kernel alignment (CKA)

Tasks

Decision making Pattern generation Working memory

Network mechanisms

RNN architectures (e.g. LSTMs, GRUs, …) Nonlinearities (e.g. ReLU, tanh) … arxiv:1907.08549 Poster #179

Evidence of both universality and individuality

arxiv:1907.08549 Poster #179

Evidence of both universality and individuality

Pattern generation task arxiv:1907.08549 Poster #179

Evidence of both universality and individuality

Pattern generation task arxiv:1907.08549 Poster #179 Analyzing trained networks

Evidence of both universality and individuality

Pattern generation task arxiv:1907.08549 Poster #179 Analyzing trained networks

Evidence of both universality and individuality

Network representations show individuality Pattern generation task arxiv:1907.08549 Poster #179 Analyzing trained networks

Evidence of both universality and individuality

Network representations show individuality Pattern generation task arxiv:1907.08549 Poster #179 but aspects of the computation are universal

Analyzing trained networks

Learn more at Poster #179

arxiv:1907.08549 Poster #179