Neural network representations and visual processing in brains - - PowerPoint PPT Presentation
Neural network representations and visual processing in brains Katja Seeliger [ www.ccnlab.net | Marcel van Gerven ] Neural network representations and visual processing in brains Computational Cognitive Neuroscience Lab Develop computational
Katja Seeliger [ www.ccnlab.net | Marcel van Gerven ]
Neural network representations and visual processing in brains
Computational Cognitive Neuroscience Lab
Neural network representations and visual processing in brains
www.ccnlab.net
Develop computational models of cognitive processes (sensory, attention, control, language …) Connectionist modeling with artificial neural networks Validation with neural / behavioural data
Representation of visual processing
Neural network representations and visual processing in brains
www.ccnlab.net
Natural world Neural representation
Biological visual system basics: Striate cortex
Sherrington (1906) Hubel, D. H., & Wiesel, T. N. (1968).
Neural network representations and visual processing in brains "The receptive field of an individual sensory neuron is the particular region
neuron." (Sherrington, 1906) Specific firing for: Location, stimulus orientation, colour, right / left eye (…?)
Basics: Learning of feature detectors
Colin Blakemore and Grahame F Cooper (1970). “Development of the brain depends on the visual environment”. In: Nature 228, pp. 477 –478.
Neural network representations and visual processing in brains
Feature detectors in the early visual cortex don’t exist a-priori, but are learned from exposure to the visual environment.
—> There must be a learning algorithm.
Neural network representations and visual processing in brains Visual system regions (simplified)
Basics: Hierarchy, local regions
Basics: Two streams-hypothesis
Image source: https://en.wikipedia.org/wiki/Two-streams_hypothesis
Dorsal stream "Where? How?" Localization and motion perception Ventral stream "What?" Object recognition Neural network representations and visual processing in brains
Hand-crafted (and shallow) representations
Neural network representations and visual processing in brains Gabor edge detectors Object categories Fourier powers
Encoding in visual alphabet ( feature space )
Great shallow representations can get you quite far…
Shinji Nishimoto et al. (2011). “Reconstructing visual experiences from brain activity evoked by natural movies”. In: Current Biology 21.19.
Neural network representations and visual processing in brains
Feature space: Gabor filter pyramid + motion energy
Method: Voxel-wise encoding models
Neural network representations and visual processing in brains
Naselaris, Thomas, et al. "Encoding and decoding in fMRI." Neuroimage 56.2 (2011): 400-410.
natural stimuli brain activity
Method: Voxel-wise encoding models
Neural network representations and visual processing in brains
Naselaris, Thomas, et al. "Encoding and decoding in fMRI." Neuroimage 56.2 (2011): 400-410.
encode predict
Method: Voxel-wise encoding models
Neural network representations and visual processing in brains
Naselaris, Thomas, et al. "Encoding and decoding in fMRI." Neuroimage 56.2 (2011): 400-410.
Based on the chosen encoding / feature space: … we predict the activity for each voxel using regularized linear regression. Correlation[ predicted , measured ]
Model performance
Deep representations
Neural network representations and visual processing in brains Visual system regions (simplified) Deep convolutional neural network
Deep representations: AlexNet first layer
Neural network representations and visual processing in brains
Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems. 2012.
Specific for: Location, stimulus
Deep representations: AlexNet convolutional hierarchy
Neural network representations and visual processing in brains
Güçlü, U., & van Gerven, M. A. (2015). Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. The Journal of Neuroscience, 35(27), 10005-10014.
Deep representations across ventral stream
Güçlü, U., & van Gerven, M. A. (2015). Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. The Journal of Neuroscience, 35(27), 10005-10014.
Neural network representations and visual processing in brains
Relating fMRI responses to static images and VGG-S layers (via predictive voxel-wise linear models (encoding models) )
Low and high deep convolutional layers in time
based Encoding and Decoding of Visual Object Recognition in Space and Time (bioRxiv , and in press)
Neural network representations and visual processing in brains
Relating MEG responses to static images and VGG-S layers in time (via predictive voxel-wise linear models (encoding models) )
Assignment of VGG-S layers to anatomy
Neural network representations and visual processing in brains
based Encoding and Decoding of Visual Object Recognition in Space and Time (bioRxiv , and in press)
Deep representations across dorsal stream
U Güçlü, MAJ van Gerven (2015): Increasingly complex representations of natural movies across the dorsal stream are shared between subjects. NeuroImage
Neural network representations and visual processing in brains
Relating fMRI responses to movies and action recognition DNN > spatiotemporal receptive fields
Felleman, D. J., & Van Essen, D. C. (1991). Distributed hierarchical processing in the primate cerebral cortex. Cerebral cortex, 1(1), 1-47.
Neural network representations and visual processing in brains
Macaque visual system connections and hierarchy by van Essen (1991)
Visual cortex == 8 representations from AlexNet?
Presentation Title Neural network representations and visual processing in brains
Questions?