Cognitive computational neuroscience of vision Nikolaus - - PowerPoint PPT Presentation

Artificial intelligence Computational neuroscience Cognitive science Nikolaus Kriegeskorte

Department of Psychology, Department of Neuroscience Zuckerman Mind Brain Behavior Institute Affiliated member, Electrical Engineering, Columbia University

Cognitive computational neuroscience

• f vision

Artificial intelligence Computational neuroscience Cognitive science

Kriegeskorte & Douglas 2018

neural network models

Cognitive computational neuroscience

A common language for expressing theories about brain information processing

How can we test neural network models with brain-activity data?

?

activity patterns experimental stimuli

...

... ...

brain model

Predicting representational spaces

activity pattern

stimuli responses

activity profile activity pattern

stimuli responses

activity profile Diedrichsen & Kriegeskorte 2017, Kriegeskorte & Diedrichsen 2019

Predicting representational spaces

activity pattern

stimuli responses

activity profile

Diedrichsen & Kriegeskorte 2017, Kriegeskorte & Diedrichsen 2019

Predicting representational spaces

encoding model representational similarity analysis Diedrichsen & Kriegeskorte 2017, Kriegeskorte & Diedrichsen 2019

distance matrix

response 1

(e.g. neuron, voxel)

stimulus 1

activity pattern

stimuli responses

activity profile

weights model features stimulus 3 activity response 3 activity

L2 weight penalty

Predicting representational spaces

model activity-profiles distribution encoding model pattern component model representational similarity analysis model representational distances model each response separately model stimulus-by-stimulus matrix

• f summary statistics

Diedrichsen & Kriegeskorte 2017, Kriegeskorte & Diedrichsen 2019

distance matrix second-moment matrix

response 1

(e.g. neuron, voxel)

stimulus 1

weights model features response 3 activity

Core commonality: All three test hypotheses about the second moment of the activity profiles.

stimulus 3 activity stimulus 3 activity

L2 weight penalty

1 spatially organized neuronal population code

(neuronal locations and activity profiles ) L U

2 activity-profiles distribution

(activity profiles

• r all moments

U

• f activity-profiles distribution)

3 representational geometry

(2 moment

• f activity profiles or

nd

G representational distance matrix ) D

4 total encoded information

(downstream neuron can perform arbitrary linear or nonlinear readout from all neurons)

5 linear neuronal readout

(downstream neuron can perform linear readout from all neurons)

6 restricted-input linear readout

(downstream neuron can perform linear readout from a limited number of neurons)

7 local linear readout

(downstream neuron can perform linear or radial-basis readout from neurons in a restricted spatial neighborhood)

The onion of brain representations

information potentially used by researchers information potentially extracted by single readout neurons

encoded information

explicit implicit

researcher information

focused comprehensive

Kriegeskorte & Diedrichsen 2019

1 spatially organized neuronal population code

(neuronal locations and activity profiles ) L U

2 activity-profiles distribution

(activity profiles

• r all moments

U

• f activity-profiles distribution)

3 representational geometry

(2 moment

• f activity profiles or

nd

G representational distance matrix ) D

4 total encoded information

(downstream neuron can perform arbitrary linear or nonlinear readout from all neurons)

5 linear neuronal readout

(downstream neuron can perform linear readout from all neurons)

6 restricted-input linear readout

(downstream neuron can perform linear readout from a limited number of neurons)

7 local linear readout

(downstream neuron can perform linear or radial-basis readout from neurons in a restricted spatial neighborhood)

The onion of brain representations

information potentially used by researchers information potentially extracted by single readout neurons

encoded information

explicit implicit

researcher information

focused comprehensive

Kriegeskorte & Diedrichsen 2019

?

stimuli stimuli

000000000000

stimuli stimuli

000000000000

activity patterns experimental stimuli

...

... ...

brain model

representational dissimilarity matrix (RDM)

dissimilarity (e.g. crossvalidated Mahalanobis distance estimator)

Representational similarity analysis

!

Kriegeskorte et al. 2008

Representational feature weighting with non-negative least-squares

f1 w2 f2 f2 fk w1 f1 wkfk . . . . . . model RDM weighted-model RDM

Representational feature weighting with non-negative least-squares

wk weight given to model feature k fk(i) model feature k for stimulus i di,j distance between stimuli i,j w is the weight vector [w1 w2 ... wk] that minimizes the sum of squared errors

𝐱 = arg min

𝐱∈𝐒+𝒐 ෍ 𝑗≠𝑘

𝑒𝑗,𝑘

2 − መ

𝑒𝑗,𝑘

2 2

መ 𝑒𝑗,𝑘

2 = ෍ 𝑙=1 𝑜

[𝑥𝑙𝑔

𝑙 𝑗 − 𝑥𝑙𝑔 𝑙 𝑘 ]2

= ෍

𝑙=1 𝑜

𝑥𝑙

2 ∙ [𝑔 𝑙 𝑗 − 𝑔 𝑙 𝑘 ]2

w1

2  feature-1 RDM

+w2

2 feature-2 RDM

+wk

2 feature-k RDM

= weighted-model RDM

... = arg min

𝐱∈𝐒+𝒐 ෍ 𝑗≠𝑘

𝑒2 − ෍

𝑙=1 𝑜

𝑥𝑙2 ∙ RDM𝑙

𝑗,𝑘 2

The squared distance RDM

• f weighted model features

equals a weighted sum

• f single-feature RDMs.

model feature k weight k stimuli i, j predicted distance

convolutional fully connected weighted combination of layers and SVM discriminants

highest accuracy any model can achieve

• ther subjects’ average

as model

accuracy above 0 p < 0.05, Bonf. corr. (stimulus bootstrap) SE (stimulus bootstrap)

Khaligh-Razavi & Kriegeskorte 2014, Nili et al. 2014 (RSA Toolbox), Storrs et al. (in prep.)

model comparisons (stimulus bootstrap, p < 0.05, Bonferroni corrected for all pairwise comparisons)

Deep convolutional networks predict IT representational geometry

accuracy

• f human IT

dissimilarity matrix prediction

[group-average of Spearman’s  ] 0.7 0.6 0.5 0.4 0.3 0.2 0.1 accuracy below noise ceiling p < 0.05, Bonf. corr. (stimulus bootstrap)

noise ceiling

performance range of computer-vision features

Do recurrent neural networks provide better models of vision?

Courtney Spoerer

Recurrent networks can recycle their limited computational resources over time.

Kriegeskorte & Golan 2019

This might boost the performance of a physically finite model or brain.

Layer 1 lateral connectivity is consistent with primate V1 connectivity

RCNN, layer 1, lateral connectivity templates

(first 5 principal components)

Spoerer et al. pp2019

Spoerer et al. pp2019

recurrent convolutional

accuracy computational cost

[floating-point operations ×1011]

Recurrent models can trade off speed of computation for accuracy

feedforward models

Spoerer et al. pp2019

recurrent convolutional

accuracy computational cost

[floating-point operations ×1011]

Recurrent models can trade off speed of computation for accuracy

feedforward models

RCNN reaction times tend to be slower for images humans are uncertain about

correlation between human certainty and RCNN reaction time [Spearman]

Tim Kietzmann

Can recurrent neural network models capture the representational dynamics in the human ventral stream?

Fitting model representational dynamics with deep representational distance learning

Task: find an image-computable network to model the first 300ms

• f representational dynamics of the ventral stream.

McClure & Kriegeskorte 2016

magnetoencephalography functional magnetic resonance imaging

Recurrent networks significantly outperform ramping feedforward models in predicting ventral-stream representations (MEG and fMRI).

feedforward recurrent

Recurrent models better explain representations and their dynamics

How can we build neural network models

• f mind and brain?

big models Divergent: Exploring the space of computational models with world data

• Training
• different sets of stimuli
• different tasks
• Units
• stochasticity
• context-modulation
• Architecture
• skipping connections
• recurrent connections

Convergent: Constraining models with brain and behavioral data

• inferential model selection (model

parameters learned for a task)

• reweighting of units
• linear remixing of units
• deep learning of model parameters

from brain-activity data

big world data big behavioral data big brain data