Putting vision into context: Influence of behaviour and context on - - PowerPoint PPT Presentation

Sonja Hofer

Sensory Systems Module PhD course 26/10/2018

Putting vision into context:

Influence of behaviour and context on sensory processing

Primary visual cortex V1

Visual system

Lateral geniculate nucleus

The same stimulus can be perceived differently depending on the context

The same stimulus can be perceived differently depending on the context

The same stimulus can be perceived differently depending on the context

Effect of context on perception:

Effect of context on perception:

Integration of sensory and contextual ‘top-down’ signals

Visual cortex

Visual information Contextual influences

Eye

Attention Expectations Actions Behavioural relevance Associations Knowledge

Top-down cortical inputs

Eye

dLGN

V1

Higher visual areas Association areas

Pulvinar

Cortex Thalamus

Superior colliculus

Neuromodulation Higher-order thalamic inputs

Integration of sensory and contextual ‘top-down’ signals

Outline

• Neuronal signals related to attention and reward

expectation

• Behavioural relevance & Learning
• Motor signals in sensory cortex
• Predictive coding

Modulation of sensory responses by attention

Spatial attention (Top-down)

V1 V2 V4 Buffalo et al 2009

Modulation of sensory responses by attention

Object-based attention

Curve-tracing task Roelfsema et al 1998

Modulation of sensory responses by reward expectation

Attention or reward expectation?

Adapted curve-tracing task Stănişor et al, 2013

Relative reward value

Normalized response

V1

Modulation of sensory responses by reward expectation

Neuronal signals in V1 related to reward timing (or motor signals?)

Flash to left or right eye: Shuler & Bear, 2006 Shubykin et al, 2013

Changes of sensory responses during learning How do responses to visual stimuli change as they become behaviourally relevant to an animal?

Reward zone

Approach

Approach corridor Grating corridors Vertical: rewarded

(drop of soya milk)

Angled (40°): non-rewarded

Changes of sensory responses during learning

Visual discrimination task in virtual reality

Adil Khan

Trained mouse performing the task

Head-fixed mouse on a cylinder, running through a virtual corridor (only half of virtual reality visible)

Implantation of a chronic cranial window:

Access to the cortex for chronic recordings

Holtmaat et al., 2009

Two-photon calcium imaging of GCaMP calcium indicators

GCaMP6-expressing neurons in visual cortex (V1)

Trained mouse performing the task Neurons in visual cortex expressing GCaMP6 Eye position

In vivo two-photon calcium imaging during the discrimination task Speed 2.5x

Example cell response to grating corridors:

100 50 100 50

Trials Trials Trials rewarded grating (vertical) Trials non-rewarded grating (angled)

• 2

2 4

• 2

2 4 Time (s) Time (s) Vertical grating Angled grating

Average response (ΔF/F)

Grating onset

Average response

10 s 50% ΔF/F

2 1

ΔF/F

Neuronal responses to task-relevant stimuli

Cell 1

Vertical grating Angled grating

Day 1 Day 2 Day 5 Day 6

1 ΔF/F

Cell 2 Cell 3 Cell 4

0.2 ΔF/F 0.5 ΔF/F 1 ΔF/F

Neuronal responses to task-relevant stimuli

Neuronal population performance Neuronal population discrimination

2 3 4 5 6 7 8

Session

Time (s)

• 1
• 0.5

0.5 1 0.1 0.2 0.3 0.4 0.5 0.6 1 1 2 3 4 5 6 7 8 9

• 1

1 2 3 4 5

Mouse M2 Session

Session

Time (s)

• 1
• 0.5

0.5 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 1 2 3 4 5 6 7 8 9

Time (s)

1 2 3 4 5 6 7 8 9 10

• 1

1 2 3

Mouse M5 Session

• 1
• 0.5

0.5 1 0.2 0.4 0.6 0.8 1 1.2 1.4 1 2 3 4 5 6 7 8 9 10

Session

1 2 3 4 5 6 7 8 1 2 3

Mouse M7

Behavioural discrimination (d’)

Session

Behavioural performance

Relationship between behavioural and neuronal performance

Poort, Khan et al., Neuron 2015

• 1
• 0.5

0.5 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Time (s)

Population selectivity

• 1.0 to -0.1 (N=19 sessions)
• 0.1 to 0.8 (N=8)

0.8 to 1.7 (N=8) 1.7 to 2.7 (N=26) 2.7 to 3.6 (N=17) 3.6 to 4.5 (N=15)

Behavioural performance (d-prime)

Population performance

The visual cortex gets better at distinguishing the two task- relevant stimuli, tightly correlated with behavioural performance

Poort, Khan et al., Neuron 2015

Learning may increase the salience of task-relevant visual information to better inform behavioural decisions

Neuronal changes with learning

Learning

Run up Odour1

Irrelevant Grating (vert or ang)

Odour2

Irrelevant Grating (vert or ang)

OLFACTORY BLOCK VISUAL BLOCK

Run up

Vertical grating Angled grating

Switching between visual and olfactory discrimination task

Trained mouse switching between visual and olfactory task

Mice switch between a visual and an olfactory task (the same visual stimuli are shown but ignored)

0.5 1

0.2 0.4 0.6 0.8 1.0

Population selectivity

Olfactory blocks (N=11) Visual blocks (N=11 sessions)

Time (s)

• 0.5

Population discriminability

Neurons in V1 are more selective when visual stimuli are relevant

Poort, Khan et al., Neuron 2015

Visual stimulus relevant Visual stimulus irrelevant

Switching between visual and olfactory discrimination task

Modulation of sensory responses by task demands

Task-dependent changes in auditory cortex receptive fields

Fritz et al, 2003

STRF: spectrotemporal response field Average change in response field passive listening vs during task

Sensory response properties are not fixed but reflect behavioural demands!

Motor signals in sensory areas

Electrophysiological recordings in primary visual cortex in head-fixed, running mice

Niell & Styker, 2010

Visual responses in V1 are increased during locomotion

Motor signals in sensory areas

Circuit-mechanisms of locomotion-related signals in visual cortex?

Lee at al., 2014

MLR: mesencephalic locomotor region

Motor signals in sensory areas

Circuit-mechanisms of locomotion-related signals in visual cortex?

Pakan at al., 2016

Complex networks!! -> Modelling

Del Molino at al., 2017 Fu at al., 2014

Motor signals in sensory areas

Just gain control? No!

Erisken at al., 2014 Activity in visual cortex excitatory cells: modulated in the dark and carry detailed running speed information Saleem at al., 2013

Motor signals in sensory areas

Anterior cingulate cortex (+ secondary motor cortex)?

Origin of motor signals?

Leinweber at al., 2017

Motor signals in sensory areas

Origin of motor signals?

V1

100 µm

LGN

axons

Pulvinar

axons Eye

LGN

Lateral geniculate nucleus

V1

Primary visual cortex

Higher visual areas Association areas

Pulvinar

(Lateral posterior nucleus LP)

Cortex Thalamus

Superior colliculus

Thalamus?

1 mm

AM PM

V1

AL LM

Intrinsic signal imaging to determine position of visual areas

LP axons in V1

100 μm

Expression of calcium indicator in pulvinar or LGN

10 μm

1 2

20 s 2 ΔF/F

1 2

Two-photon imaging of thalamic projections in V1

Lateral posterior nucleus LP

Imaging activity of thalamic projections in V1

In vivo two-photon calcium imaging of thalamic axons and boutons in layer 1 of V1

15 µm

Speed 5x

• Trained to run through

virtual corridor

• Running uncoupled from

visual flow

Imaging activity of thalamic projections in V1

Visuo-motor ‘task’

Visuo-motor signals in thalamic boutons in V1

5 ΔF/F 10 s 20 cm/s

Visual Flow Speed (VF) ΔF/F

VF dLGN dLGN

30 1 30 1 20

LP

1

Visual flow speed (cm/sec) Response strength dLGN

30 1 30 1 30 1

Running speed (cm/sec) LP LP ΔF/F RS

10 s

Running Speed (RS)

20 cm/s 5 ΔF/F

Proportion of highly informative boutons (%) Running speed Visual flow

10 15 5

dLGN Pulvinar

Roth, Dahmen, Muir et al., Nature Neurosciene 2016

Motor signals in sensory areas

Motor signals seem to dominate neuronal activity across the cortical surface

Musall at al., bioRxiv 2018

Widefield calcium imaging of cortical activity during a very simple spatial discrimination task

Motor signals in sensory areas

Motor signals as efference copy?

Visual cortex

Visual information Contextual influences

Eye

Attention Expectations Actions Behavioural relevance Associations Knowledge

Integration of sensory and contextual ‘top-down’ signals

Comparison of predicted and actual input

Visual cortex

Visual information

Eye

Predictions

Signal: discrepancies

The importance of predictions for sensory perception

Visual discrepancy

During eye or head movements: Motor system Visual information

Eye

Information about own body’s movement Difference calculator Motor command Prediction

Visual feedback Efference copy Predicted visual feedback

The importance of predictions for sensory perception

Predictive coding framework

Kant, Helmholtz,…Friston, Clark, Mumford

Predictive coding framework

Keller et al., 2012

Experimental evidence for predictive coding in cortical circuits A subset of neurons in V1 shows strong mismatch (prediction error) responses Mismatch responses are dependent on experience

• f visuo-motor coupling

Attinger et al., 2017

Predictive coding framework

Potential circuit for mismatch computation in visual cortex

Mismatch response in V1 is weaker when ACC is silenced Attinger et al., 2017

ACC

Muscimol in Anterior Cingulate Cortex (ACC)

Leinweber et al., 2017

Predictive coding framework

Potential circuit for mismatch computation in visual cortex

Optogenetic manipulation of SOM neurons alters mismatch response

(consistent with the model but no proof)

Attinger et al., 2017 Somatostatin (SOM) neurons are most strongly driven by visual flow

ACC SOM

Predictive coding framework

Spatial prediction and prediction error signals in visual cortex

Some V1 neurons become selective to spatial location

Fiser et al., 2016

Some V1 start firing in expectation

• f visual stimuli

Predictive coding framework

Spatial prediction and prediction error signals in visual cortex

Strong response in V1 when an expected visual stimulus is omitted

Fiser et al., 2016

Summary

• Sensory processing is highly dynamic, allowing animals to flexibly access

and process sensory information according to their current perceptual and behavioural demands.

• Still unclear to what degree top-down predictions influence or dominate

sensory representations

• Recently, the focus has shifted away from the neocortex towards

subcortical structures such as the superior colliculus, thalamus, cerebellum and the basal ganglia as sources for contextual modulation.

• The sources of different contextual signals are mostly still unknown and

we are only starting to determine the circuit mechanisms of how some of these signals are integrated with sensory information

• “Sensory” cortical areas are strongly influenced by context and

behaviour

Further reading

Kahn A, Hofer SB. Contextual signals in visual cortex, Current Opinion in Neurobiology, 2018, 52: 131 Gilbert CD, Li W. Top down influences on visual processing, Nature Reviews Neuroscience, 2013 Maunsell JHR. Neuronal Mechanisms of Visual Attention, Annu. Rev. Vis. Sci, 2015 Keller GB, Mrsic-Flogel TD. Predictive Processing: A Canonical Cortical Computation, Neuron, 2018