Spikes and Computation in Sensory Processing Simon Thorpe CerCo ( Brain and Cognition Research Center ) & SpikeNet Technology SARL, Toulouse, France
Simon Thorpe • 1974 - 77Physiology and Psychology ( Oxford ) • 1977 - 81 Doctorate with Edmund Rolls ( Oxford ) • Recording neurons in orbitofrontal cortex, striatum, parietal cortex, lateral hypothalamus, amygdala, hippocampus, supplementary motor area ( !! ) • 1982 - 83Post - doc with Max Cynader ( Halifax, Canada ) • Attempting to test Hebb's hypothesis in kitten visual cortex • 1983 - 93Paris with Michel Imbert • Testing Hebb's Hypothesis ( with Y ves Frégnac & Elie Bienenstock ) • Dynamics of responses in monkey V1 ( with Simona Celebrini & Y ves T rotter ) • 1993 - nowCERCO Toulouse • W ork on ultra - rapid scene processing • 1999 - now SpikeNet • Spike - based image processing • Bioinspired vision
Overview Part 2 Part 1 • Ultrarapid visual categorization • Biologically inspired learning • Spike - Time Dependent Plasticity • Biological vs Computer Vision ( STDP ) • Temporal Constraints • Applications in Vision • Applications in Audition • Coding with Spikes • Development of Neural Selectivity • Convolutional Neural Networks in 1999 • Memories that can last a lifetime • SpikeNet • Grandmother Cells • The current state of the art in • Neocortical Dark Matter Convolutional Neural Networks • Supervision & GoogleNet
Ultra - Rapid Visual Processing Behavioural Reaction Times Event Related Potentials Difference Targets Distractors Scene Processing in 150 ms
Ultra - Rapid Visual Processing • Saccades towards faces • Latency 100ms! Crouzet, ¡Kirchner ¡& ¡Thorpe, ¡2010
Ultra - Rapid Visual Processing RSVP at 10 fps
Activating Memories • “That Dalmation picture that I saw in Psych 101” Some Questions • How does the brain recognize visual objects and scenes? • Can a machine be built that can do the same thing?
Biological and Computer Vision Common Problems • Identification and Categorising objects and events in complex dynamically changing natural scenes • As fast as possible • As reliably as possible • Using the most energy e ffi cient hardware possible • Using the smallest size and weight footprint Common Solutions? • David Marr ( 1982 ) “Vision : A computational investigation into the Human Representation and Processing of Visual Information” • Recent years - is there convergence?
Hardware Constraints Brain Electronics • 1 KHz • Nvidia GeForce GTX Titan • 86 billion processors • 1 - 2 m/s conduction velocity • 20 watts Questions Are we going to be able to implement brain • 4.5 TeraFlops! style computing with conventional computing? • 2668 cores • 7.1 billion transistors How many teraflops does the brain need? • 288 Gbytes/sec Memory bus How much memory bandwidth? • $ 999 Response It depends if we can understand how the brain computes
Classic Neural Computing • To simulate the visual system • 4 billion neurons • 10000 connections each • Update at 1 kHz • 40 Petaflops
What’s missing? • Real brains use spikes • Simulation • 16.7 million neurons • 21 billion synapses • The Brain • 86 billion neurons • 16 billion in the cortex • 100 trillion synapses • Spikes • Firing rate • 0 - 200 Hz • Spontaneous activity • 1 - 2 Hz?
Classes of Brain Simulation • Connectionist Simulators ( non - spiking ) • Backpropagation, ART, Kohonen Maps, Time Delay Networks, • Computational Neuroscience Simulators • Up to 30 000 compartments • Lots of channel kinetics etc. • Spike - Based Simulators • Software • Hardware
The Human Brain Project • European Flagship Proposal • Three simulation approaches • Blue Brain • Henry Markram ( Lausanne, Switzerland ) • Analog Chips • Karl - Heinz Meyer ( Heidelburg, Germany ) • SpiNNaker • Steve Furber ( Manchester, UK )
SpiNNaker Project • A simulation system for Spiking Neural Networks • Prof. Steve Furber, Computer Science, University of Manchester • 18 ARM968 cores • each with 64 Kbytes of Data and 32 Kbytes of Instructions • 128 MBytes of shared memory • 48 chips on a board A billion neurons • 18 boards in a 19” frame in real time
IBM T rueNorth Chip SCIENCE • What can be computed?
Temporal Constraints – Early 1980s • Jerry Feldman’s 100 - step limit • High level decisions in about 0.5 seconds • Interval between spikes around 5 ms • No more than 100 ( massively parallel ) computational steps • Development of connectionist and PDP modelling • Surely, more detail can help
Temporal Contraints - 1989 Face selectivity at 100 ms T IT T V4 T V2 Food selectivity at 150 ms T V1 T LGN T Retina Therefore Argument • Mainly feedforward ( ?! ) • Roughly 10 layers • One spike per neuron ( ?! ) • 10 ms per layer • Firing rates 0 - 100 Hz
Temporal Constraints 1988 - 1989 Inferotemporal Cortex :Face selectivity at 100 ms Perrett, Caan & Rolls, 1982 See also : Jerry Feldman & John Tsotsos Leonard Uhr ( 1987 )
Ultra - Rapid Visual Processing • Feedforward processing • Only a few milliseconds per processing step • One spike per neuron • V ery sparse coding • Processing without context based help How can you code with just one spike per neuron?
Sensory Coding with Spikes • Adrian ( 1920s ) • Hubel & Wiesel ( 1962 ) • Orientation selectivity in striate cortec • First recordings from sensory fibres
Sensory Coding with Spikes • Wurtz & Goldberg ( 1972, 1976 ) Raster Display Post Stimulus Time Histogram Assumption : Firing rate is enough to describe the response
The Classic View 2 0,213 • Spikes don't really matter 4 0,432 1 0,112 • Neurons send floating point numbers 2 0,238 3 0,309 • The floating point numbers are transformed into spikes trains using a Poisson process • God plays dice with spike generation 0,375
Temporal Coding Option • Spikes do really matter • The temporal patterning of spikes across neurons is critical for computation • Synchrony • Repeating patterns • etc • The apparent noise in spiking is unexplained variation
Simon Thorpe's V ersion • Ordering of spikes is critical • The most activated neurons fire first • Temporal coding is used even for stimuli that are not temporally structured • Computation theoretically possible even when each neuron emits one spike Threshold Medium Strong Weak Stimulus Stimulus Stimulus Time
Sensory Coding with Spikes • Adrian ( 1920s ) • First recordings from sensory fibres Higher ¡peak ¡ Higher ¡maintained ¡ firing firing High ¡intensity Low ¡intensity Shorter ¡ Latency!
Coding in the Optic Nerve 1,000,000 fibres
Coding in the Optic Nerve Intensity n n n n n n n n n n n n n n n n
Coding in the Optic Nerve Intensity A mini retina 32 x 32 pixels n n n n n n n n n n n n n n n n
Coding with Spike Ordering Example • A toy retina Less than 1 % of cells need to fire for recognition!
Early Studies • Face identification directly from the output of oriented filters
Early Studies • Virtually all the faces correctly identified • V ery robust to low contrast • V ery robust to noise
SpikeNet Technology • Created in 1999 • Simon Thorpe, Rufin V anRullen & Arnaud Delorme • Currently 12 employees
SpikeNet - Invariance • Luminance • Contrast • Blurring • Noise
SpikeNet - Invariance • Size • Perspective • Rotation
SpikeNet - Invariance • 3D Rotation • Identity • What is the current state of the art?
Biological vs Computer Hardware Computer Brain • Nvidia GeForce GTX Titan • 4.5 TeraFlops! • 86 billion neurons • 2668 cores • 16 billion in the cortex • 7.1 billion transistors • 4 billion in the visual system • 288 Gbytes/sec Memory bus • 1 KHz • 200 watts • 1 - 2 m/s conduction velocity • $ 999 • 20 watts Is that enough to reproduce human performance?
The ImageNet Challenge • 10,000,000 training images • 10,000+ labels • Systems tested on new images, with 1000 possible labels • ECCV 2012 Firenze • The state of the art was beaten by a “simple” feedforward convolutional neural network trained with Back - Propagation
SuperVision 253440 186624 64896 64896 43264 Output Layer Fully Connected Layers Convolutional Layers • 650,000 “neurons” • 60,000,000 parameters • 630,000,000 “synapses”
SuperVision Animals
And then…. • Geo ff Hinton and his two • Y ann LeCun, a pioneer of feed - students launched a start - up forward convolutional networks ( DNNresearch ) since the end of the 1980s • bought by Google… • Hired by Facebook…
Supervision vs Primate Vision Convergent Evolution!
Coding at higher levels ? ? ? ?
Basic Methodology • Neural Activity • Human performance • Computer models • Parallel Multielectrode Recordings • Crowd sourced data • 57 parameters • V4 and IT • Amazon Mechanical Turk • 1 - 3 layers • Activity ( 70 - 170 ms ) • 104 subjects
Recommend
More recommend
