SLIDE 1
Intuition behind our method Perceptual Categorical
fMRI-EVC MEG-early fMRI-ITC MEG-Late DNN-L1 DNN-L3 DNN-L2 DNN-L5 DNN-L4
visual cortices responses Agustin Lage Castellanos 1,2 and Federico - - PowerPoint PPT Presentation
visual cortices responses Agustin Lage Castellanos 1,2 and Federico - - PowerPoint PPT Presentation
RDM mixtures for predicting visual cortices responses Agustin Lage Castellanos 1,2 and Federico De Martino 2 1-Cuban Neuroscience Center, 2-Maastricht University Algonauts Challenge 2019 Intuition behind our method fMRI-ITC Categorical
SLIDE 2
SLIDE 3
Combining RDMs to improve predictions
Predicted RDM Perceptual Categorical DNN
= 𝑥1 + 𝑥2 + 𝑥3
SLIDE 4
Perceptual RDMs
SLIDE 5
Perceptual RDMs
Only uses image information
Extract Edges and Smooth
Perceptual-RDM Pixel Overlap
SLIDE 6
Categorical RDMs
SLIDE 7
Categorical Structure of the 92 image set
Objects-Scenes animals Human Fruits-vegetables Faces Hands Monkey faces Animal Faces
SLIDE 8
Within category RDM based on fMRI/MEG data similarity
92 x 92 8 x 8
mean Between image fMRI/MEG similarity Between category fMRI/MEG similarity
fMRI-ITC
SLIDE 9
Training a GNB classifier as predicting category
GNB Class Labels
Last fully connected layer (defines category membership) Leave one out CV on the 92 image training set
SLIDE 10
Classification of the 78 test set images 𝑋
𝐻𝑂𝐶
Predicted Labels x Predicted as Human Faces Predicted as Animal Faces in the 78 set
Objects-Scenes Animal Faces animals Human Fruits-vegetables Faces Hands
SLIDE 11
Assigning distances between new test images based on categorical RDM and predicted labels
Test Set Image 1 Test Set Image 2
human face animal face Assigned distance 0.37
SLIDE 12
Predicted categorical RDM for the 78 images test data
Same distance for all the images within the same category
SLIDE 13
Mixing perceptual and categorical components
Large impact on fMRI-ITC and MEG-Late.
𝑆 = 1 − 𝑥2 𝑆𝑞𝑓𝑠 + 𝑥2𝑆𝑑𝑏𝑢
Training data: 92 image set
SLIDE 14
Results Test set: Perceptual + Categorical RDMs
SLIDE 15
DNN based RDMs
SLIDE 16
RDM based on DNN features at one layer
117 𝑦 117 mean 0.12
corr
DNN
1 64 1 64 2 63 2 63 Vgg L-1
SLIDE 17
Model Improvement including DNN Based RDMs
𝑆 = 1 − 𝑥3 𝑆(𝑞𝑓𝑠+𝑑𝑏𝑢) + 𝑥3𝑆𝑒𝑜𝑜
Improvement of 𝑆2 (explained variance) in EVC for the 92 image set
𝑥3 𝑥3 𝑥3
SLIDE 18
Results Test set including DNNs
SLIDE 19
Conclusions
- A mixture of perceptual and categorical RDMs
made the largest contribution to the prediction accuracy in fMRI-ITC/MEG-Late.
- VGG was the DNN that produced the largest
improvement on the model performance.
- However, it is necessary to evaluate the