Exploring Neural Mechanisms for Prediction Keith L. Downing The - - PowerPoint PPT Presentation

Primitives Procedural -vs- Declarative Predictive Networks

Exploring Neural Mechanisms for Prediction

Keith L. Downing

The Norwegian University of Science and Technology (NTNU) Trondheim, Norway keithd@idi.ntnu.no

February 19, 2011

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Philosophy: What is Prediction? Neuroscience: Synaptic Modification

Prediction: Essential for Action and Cognition

i of the Vortex: From Neurons to Self. Llinas, 2001 You only need a brain if you move The faster and more intricate the moves, the more you need to predict their outcomes, since sensory processing is slow. On Intelligence. Hawkins, 2004 Intelligence and understanding started as a memory system that fed predictions into the sensory stream. These predictions are the essence of understanding. To know something means that you can make predictions about it... We can now see where Alan Turing went wrong. Prediction, not behavior, is the proof of intelligence.

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Philosophy: What is Prediction? Neuroscience: Synaptic Modification

Cognitive Incrementalism

Llinas (pg. 35) ...that which we call thinking is the evolutionary internalization of movement.. Mindware (pg. 135), Andy Clark, 2001 This is the idea that you do indeed get full-blown human cognition by gradually adding bells and whistles to basic (embodied and embedded) strategies of relating to the present at hand.

Is the predictive machinery evolved for motion also used for cognition? Could it be the basis of common sense? Is it the key to Artificial General Intelligence (AGI)?

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Philosophy: What is Prediction? Neuroscience: Synaptic Modification

Predict

To declare or indicate in advance To foretell on the basis of observation Declare To make known formally, officially, or explicitly Indicate To point out or point to To be a sign, symptom or index of ...Webster’s Online Dictionary

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Philosophy: What is Prediction? Neuroscience: Synaptic Modification

Declarative Prediction

Recognition Associative Learning Prediction Time

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Philosophy: What is Prediction? Neuroscience: Synaptic Modification

Procedural Prediction

To an observer, the agent’s actions indicate knowledge of a future world state.

Observer

Time

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Philosophy: What is Prediction? Neuroscience: Synaptic Modification

Eye Tracking Simulations

Kettner, Mahamud, Leung, Sitkoff, Houk, Peterson and Barto (1997)

This tracking behavior is considered predictive because visual signals are processed by the smooth pursuit system with considerable delays (≈ 100 ms)...One would expect tracking to lag by similar delays if the eye were controlled exclusively by a simple negative feedback system based on visual input...(pg. 2115) Procedurally Predictive To an observer, it may appear that the controller has an explicit representation of the ball’s future location, but actually it just knows how to move the eye to point there.

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Philosophy: What is Prediction? Neuroscience: Synaptic Modification

Spike-Timing Dependent Plasticity (STDP)

Markram et. al. (1997), Bi et. al. (1998)

∆W ∆T = Tpost - Tpre LTP LTD ∆T*

• ∆T*

∆T* = 10 - 50 ms

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Philosophy: What is Prediction? Neuroscience: Synaptic Modification

Bi-Modal Thresholding

Artola, Brocher and Singer (1990)

LTD LTP P S P+S Stimulation Intensity Synaptic Strength Change P- prediction only S - sensory input only P+S - sensory input and prediction N Prediction Sensory Input Dendrites Soma

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Philosophy: What is Prediction? Neuroscience: Synaptic Modification

Chemistry of Eligibility Traces

Houk, Adams and Barto (1998)

Glutamate Neurotransmitter cAMP Receptors Dopamine Ca ++CAM Ca ++ Kinase A DARPP-32 2nd Messenger Phosphatase 2B Depolarization CAM PK-II PO4 Phosphatase 2A High for ≈ 100 ms after depolarization Disinhibition Promote Inhibit LTP / LTD Autophosphorylation

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Generic Procedural Prediction Network

Competitive Context-Detecting Layer Sensory Inputs Internally-Generated Activation Patterns Actions Inhibit Excite Salient Event Detection Feedback Signals

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

The Cerebellum

Parallel Fibers Granular Cells Mossy Fibers Climbing Fibers Purkinje Cells Inferior Olive Inhibition of deep cerebellar neurons To Cerebral Cortex To Spinal Cord Sensory + Cortical Inputs Somatosensory (touch, pain, body position) + Cortical Inputs Golgi Cells Efference Copy Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Basal Ganglia: Anatomy

Putamen Caudate Nucleus VL Nucleus

• f Thalamus

STN Substantia Nigra Globus Palllidus (GP) Endopedunclar Nucleus (EP) Striatum Cortex Inhibit Excite

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Basal Ganglia: Function

Houk, Davis and Beiser (1998). Models of Info Proc in the BG

Neocortex

Striatum STN Thalamus GP EP SNr

Actors Critic

Hyperdirect Pathway Direct Pathway Direct Pathway Indirect Pathway Inhibit Excite Striosome Matriosome Midbrain & Brainstem SNc Dopamine Primary reinforcement from the limbic system

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Mixed-Temporal Context Coding

et et+2 et+1 et-2 et+1 et+1

Granular Cells Mossy Fibers Golgi Cells Parallel Fibers

et+3 et+1 Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Predictive Learning in Procedural Networks

Time

P1 P3 P2 Context Action Salient Event Detected Salient Event Occurs Feedback 100 ms Sensory & Internal Patterns Salient Context Detected Feedback

At time T, and given situation at T −δ1 compute the best action for T +δ2

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Eligibility Traces in the Cerebellum

LTD: Those PF-PC synapses active ≈ 100 ms before error detection (≈ when error occured) are ⇓ most.

Time

C1 A3 Error Detected Error Occurs 100 ms C1 C2 A2 A1 C8 A2 C4 A1 Eligibility Trace

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Eligibility Traces in the Basal Ganglia

LTP: Corticostriatal and striatal-pallidal synapses that are active 100 ms before treward (≈ tsalient event) are ⇑ most.

Time

C3 A3 Internal Reward Signal Salient Event 100 ms Eligibility Trace Substantia Nigra C3 A3 Salient Event Internal Reward Signal C2 A2

Trial 1 Trial 2

Learn link to reward Learn link to reward

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Reinforcement Learning in the Basal Ganglia

R R R R Z Z Y Z Y X Z Y Y X X X Time (msec) Learning Trial SN SN SN SN

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Regressive Procedural Prediction

Displaying predictive behavior at progressively earlier times.

Time

Trial 1 Trial 2 Trial k Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Procedural Prediction in CB and BG

1

Neural architectures and dynamics that adapt (over evolutionary and lifetime) timescales to inherent delays in sensory processing and motor activation.

2

Implicit predictive knowledge in context-action circuitry:

Choose appropriate actions for time T +δ2 based on state

• f the world at T −δ1.

Learn to indicate future situations well before they occur.

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Generic Declarative Prediction Network

A C B W X Y Z Stimulus A Stimulus C Stimulus B T1 T4 T3 T2

Level K+1 Level K

Monitoring High Activity Low Activity Learning

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Cortical Columns

Hawkins (2004), On Intelligence

Predictions Sensory Inputs Layers 2, 3 Layer 4 Layers 5,6 Layer 1

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Hippocampus: Anatomy

Entorhinal Cortex (EC) Dentate gyrus (DG) CA3 CA1 Subiculum EC Sub DG CA3 CA1 CA2 6-layered 3-layered

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Recurrent Topology

3 1 4 5 6 2 Dentate Gyrus

1 - 5% Recurrence CA3

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

First Input and Context Formation

Wallenstein, Eichenbaum and Hasselmo(1998)

1 2 5 4 3 6

Dentate Gyrus Random Bursting Learning High Activity Low Activity 10-20% active at any time

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Monitoring by Context Neurons

1 2 5 4 3 6

Dentate Gyrus High Activity Low Activity Monitoring Learning

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Second Input and Linkage to Context

1 2 5 4 3 6

Dentate Gyrus Monitoring High Activity Low Activity Learning

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Using the New Prediction

1 2 5 4 3 6

Dentate Gyrus High Activity Low Activity 1-3 predicts 2-6 Early in Theta Cycle Late in Theta Cycle Contexts are Self-Sustaining

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Thalamacortical System: Anatomy

Nucleus Reticularis

Excite Inhibit Core Cell Matrix Cell N.R. Cell

Cortex

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Incremental Sensory Processing via GDPN

Rodriguez, Whitson and Granger(2004)

Layers 2, 3 Layer 4 Layer 6 Layer 1 Layer 5 1 2 3 1 2 3 Stimulus 1

1 2 3

Matrix ? 1 ? Excite Inhibit Inactive Active Learning Monitoring Stimulus 2 Core N.R.

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Asymmetric Place Cells

Neural Dynamics of Predictive Coding. Mehta, 2001

Place Cell Firing Rate Space P* Time T*

P*

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

STDP → Asymmetry → Predictive Coding

X A C D E F B CA1 CA3 Time T* A F X A C D E F B CA1 CA3 STDP

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Place Cell Linkage → Prediction

A B C D E F G

A C D E F B G CA3 Place Cells

• Random recurrence (2-5%)
• No topology

STDP can produce these connections 50 ms STDP Window Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Theta Phase Precession → Prediction Chunks

Prediction, sequences and the hippocampus. Lisman & Redish, 2009

A C D E F B G

Chunking

Theta Wave (6-10 Hz) A G E F D B C 50 ms STDP Window C D E F G H I G E F Gamma Waves (40-100 Hz) ride atop the theta waves, with each gamma peak stimulating the next place cell in the sequence. Sensory Input A B C Look- ahead

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Generality of Place Cells and Chunking

A C D E B P1 => P2 => P3 => P4 => P5 Places Words "Neurons that fire together wire" Concepts

• 1. Integrate Inputs
• 2. Apply activation function
• 3. Propagate outputs
• 4. Apply learning rule
• 5. Normalize weights

Place cells and theta phase precession useful for linking any declarative sequences (episodes).

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Episodes to Concepts

Rhythms of the Brain. Buzsaki, 2006. Place cells are unidirectional in corridors but

• mnidirectional in open arenas.

They generalize after exploration and path crossings. Similarly, a set of overlapping episodes can generalize to a concept. Concept = essence of episodes, minus (a lot of detailed) spatiotemporal context. What is the neural basis for this type of generalization?

Place-Cell Remapping (Place representation within hippocampal networks is modified by LTP. Dragoi, Harris & Buzsaki, 2003) Biased synaptic potentiation and/or stability

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Place Cell Generalization

A B C D E F G

T1 T2 C1 D B C2

T1 T2

C B D C1 = C2 After more exploration and re-mapping Neuron C may or may not be neuron C1 or C2

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Episodes to Concepts: Possible Neural Mechanisms

Disjuncts Favored Biased Re-mapping

C1 D B C2 C

C2 C1

B D

Weakened before Tk Formed at Tk Formed at T1 Formed at T2

1

Broader (disjunctive) context more stable, since it is active more often.

2

B-D cue overlap makes C1 most likely target for D if some D → C2 synapses weaken.

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Innate Hippocampal Predictive Mechanisms

New experiences enhance coordinated neural activity in the hippocampus. Cheng & Frank, Neuron, 2008. Some phase precession seen in novel environments. How? Grid Cells Microstructure of a spatial map in the entorhinal cortex. Hafting et. al., Nature, 2005. Hippocampus-independent phase precession in entorhinal grid cells. Hafting et. al., Nature, 2008.

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Grid Cells

Medial Entorhinal Cortex (MEC) Layers II & III Mainly layer II shows theta precession, but II and III could support theta-based prediction in CA3.

A B A A A A A A A A A A A A A A B B B B B B B B B A B B B A Place fields for neurons A & B B

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Proposed Grid-Cell Circuitry

McNaughton et. al., 2006

B A C Velocity Y

X

Z A A A A A A B B B

X

C C C Z Y

X

Z Y

X

Z Y

Q

Q A & Q Groups have same frequency and orientation. Vary only in phase. Rotation Space MEC Prediction Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

MEC Grid Cells ⇒ CA3 Place Cells

B A C Y

X

Z K

J

L

MEC

Ventral Dorsal

Grid Spatial Frequency Decreases

CA3

P5 P4 P6 P7 P1 P2 P3

K (> 4) intersecting grids yield sparse place fields General-purpose predictive mechanism for navigation (and more?) in MEC, possibly configured during development.

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Comparative Anatomy for Prediction

Cerebellum & BG Hippocampus NeoCortex

Action Context Detector

EC CA3 DG CA1 Subiculum Sensory Input Region Associative Region

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Comparative Anatomy for Prediction

1

CB & BG have dedicated tracts without much overlap. Context → action.

2

HC & Cortical areas have massive interconnections and recurrence that support:

Spreading activation for recall, information integration, concept formation and declarative prediction. Maintenance of activity patterns for conscious attention

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Prediction & Cognitive Incrementalism

Prediction is essential for sensorimotor behavior. But a lot of that is done using procedural predictive mechanisms. Cognition may build on this, since procedural areas support basic aspects of cognition: timing, focus of attention, etc. But high-level cognition requires declarative prediction, handled by evolutionarily newer brain regions. The hippocampus could be pivotal for cognitive incrementalism: Essential for both navigation (a sensorimotor act with varying cognitive demands) and declarative memory formation. Theta phase precession useful in concept formation and general sequence learning.

Keith L. Downing Exploring Neural Mechanisms for Prediction

Primitives Procedural -vs- Declarative Predictive Networks Procedural Networks Declarative Networks

Related Articles

Downing (2009).Predictive models in the brain. Connection Science, 21(1), pp. 39-74. Downing (2007). Neuroscientific implications for situated and embodied AI. Connection Science, 19(1), pp.75-104. Downing (2005). The predictive basis of situated and embodied AI. GECCO Proceedings, pp. 43-50. All available at www.idi.ntnu.no/(tilde)keithd

Keith L. Downing Exploring Neural Mechanisms for Prediction