Invertebrate Learning Computational Models of Neural Systems - - PowerPoint PPT Presentation

Invertebrate Learning

Computational Models of Neural Systems

Lecture 3.3

David S. Touretzky October, 2007

10/10/07 Computational Models of Neural Systems 2

Eric Kandel Nobel Prize in Physiology or Medicine, 2000

10/10/07 Computational Models of Neural Systems 3

Aplysia Californica

10/10/07 Computational Models of Neural Systems 4

Aplysia

10/10/07 Computational Models of Neural Systems 5

Mantle, Siphon and Gill

siphon and gill withdrawal

10/10/07 Computational Models of Neural Systems 6

Abdominal Ganglion

10/10/07 Computational Models of Neural Systems 7

Review of Learning Terms

• Non-associative learning

– habituation: response to a repeated stimulus gradually decreases – dishabituation: response restored to a more normal level – sensitization: elevated response to a stimulus

• Associative learning

– classical conditioning

train: CS + UCS → UCR test: CS → CR

– instrumental (operant) conditioning

behavior → reinforcement

10/10/07 Computational Models of Neural Systems 8

Habituation in Aplysia

• Tactile stimulus to siphon causes brief withdrawal of gill + siphon
• With repeated exposure, withdrawal response is greatly reduced.
• Effect can last from minutes to weeks, depending on the

stimulus protocol.

• Short term mechanism: decreased transmitter release at

synapse from sensory to motor neuron, due to decreased Ca2+ influx, due to inactivation of presynaptic calcium channels.

• Long term mechanism: decrease in number and size of active

zones in synapses.

10/10/07 Computational Models of Neural Systems 9

Sensitization

• Animal initially responds weakly to weakly aversive/neutral CS.
• Noxious stimulus (strong shock to the neck or tail) enhances

defensive responses to subsequent weak or neutral stimuli.

• Densive responses include siphon- and gill-withdrawal reflexes,

inking, and walking.

• Dishabituation shown to be a special case of sensitization.
• Effects last for minutes to weeks.
• Cause: increase in sensory neuron transmitter release onto the

motor neurons.

• Mechanism: presynaptic facilitiation by facilitory interneuron.

10/10/07 Computational Models of Neural Systems 10

Inking Behavior

10/10/07 Computational Models of Neural Systems 11

Presynaptic Facilitation Mechanism

• Tail stimulation activates a group
• f facilitator interneurons.
• Transmitter released by them (may be

serotonin or a related amine) activates an adenylate cyclase in the presymaptic terminals and causes elevation of free cAMP.

• Free cAMP activates a cAMP-dependent protein kinase.
• The protein kinase closes a particular type of K+ channel in the presynpatic

terminal.

• Reduction in K+ current leads to a broadening of action potentials, allowing

more Ca2+ to enter the terminal.

• Increase Ca2+ leads to a higher probability of transmitter release onto the

motor neuron.

10/10/07 Computational Models of Neural Systems 12

Classical Conditioning

• CS1: weak stimulus to the siphon, produces feeble response.
• UCS: strong shock to the tail, produces strong response.
• Training: CS1, then UCS. Repeat for 15 trials.
• Conditioning result:

CS1 → strong withdrawal response CS2 (weak stimulus to mantle) → feeble response

10/10/07 Computational Models of Neural Systems 13

Classical Conditioning

• Training: CS1 and UCS, mixed with CS2 and no UCS.
• Stimulus specificity result:

CS1 → strong response CS2 → less strong response

• Mechanism: presynaptic facilitation is increase is a synapse that

has recently been activated. Elevated presynaptic Ca2+ enhances effect of serotonin.

• Generalization result: strength of CS2 response depends on

how similar CS2 is to CS1.

– Suggests shared representations.

10/10/07 Computational Models of Neural Systems 14

Stimulus Specificity and Generalization

shared representations

10/10/07 Computational Models of Neural Systems 15

Cellular Mechanisms of Conditioning

10/10/07 Computational Models of Neural Systems 16

Classical Conditioning Phenomena

• Basic result: CS1 + UCS → UCR, then CS1 → CR

– specificity: CS2 → small CR (if similar to CS1) or no CR – extinction: CS (repeated presentations with no UCS) → no response – recovery: strong stimulus reinstates CS → CR

(Dashed line shows normal slow decay of learning in the absence of extinction trials.)

10/10/07 Computational Models of Neural Systems 17

Classical Conditioning Phenomena

• Second order conditioning:

– First: CS1 + UCS → UCR – Second: CS2 + CS1 → CR (CS1 may extinguish) – Test: CS2 → CR

10/10/07 Computational Models of Neural Systems 18

Classical Conditioning Phenomena

• Blocking:

– First: CS1 + UCS → UCR – Second: (CS1,CS2) + UCS → UCR – Test: CS2 → little or no CR

• Preconditioning:

– First: UCS → UCR – Second: CS + UCS → UCR – Test: CS → reduced CR

10/10/07 Computational Models of Neural Systems 19

A Cell-Biological Alphabet

• Hawkins and Kandel (1984):

– Habituation, sensitization, and classical conditioning may form the basic

“alphabet” from which higher-order learning behavios are synthesized.

– Showed how a variety of classical conditioning effects could be

implemented in known Aplysia circuitry.

– Didn't actually do any simulations.

• Gluck and Thompson (1987):

– Simulated the Hawkins and Kandel model. – Found that blocking didn't work. – Proposed modifications that could produce blocking.

10/10/07 Computational Models of Neural Systems 20

Extinction

• After training, presenting CS alone, with no UCS, eventually

reduces the CR back to baseline level.

• Explanation: this is the result of habituation (decreased

transmitter release due to decreased presynaptic Ca2+.)

• Note that habitution involves a presynaptic mechanism that is

independent of the sensitization/conditioning mechanism.

10/10/07 Computational Models of Neural Systems 21

Recovery From Extinction

• Trained animals will spontaneously recover from extinction.
• Explanation: after a rest period, habituation wears off, while the

separate changes caused by conditioning are still in effect.

• Disinhibition: a strong stimulus can undo the effects of

habituation.

• Explanation: disinhibition is a special case of sensitization; the

effect counters that of habituation.

10/10/07 Computational Models of Neural Systems 22

Second Order Conditioning

• First: siphon stimulus + tail shock → gill withdrawal

Second: mantle stimulus + siphon stimulus → gill withdrawl Test: mantle stimulus → gill withdrawal ?

• To explain second order conditioning, Hawkins & Kandel

introduced 3 new features to their model:

– Facilitory interneurons can be excited by the CS, not just the UCS – Facilitory interneurons also excite sensory-to-facilitory neuron synapses – Facilitory interneurons also excite motor neurons, either directly or

indirectly

10/10/07 Computational Models of Neural Systems 23

Extended Model

10/10/07 Computational Models of Neural Systems 24

Model of Second Order Conditioning

• 1. Stimulate siphon (CS1) and then shock tail (UCS).
• 2. Activity-dependent presynaptic facilitation occurs:
• at the siphon-to-motor neuron synapse (S-R learning)
• at the siphon-to-facilitory neuron synapse (S-S learning)
• 3. Now, siphon-to-facilitory neuron synapse is strong enough to fire the facilitory

neuron.

• 4. Stimulate mantle (CS2) followed by siphon (CS1).
• 5. Siphon sensory neuron fires the facilitory interneuron, and also the motor

neuron, producing a CR.

• 6. Activity-dependent facilitation occurs at mantle-to-motor neuron synapse.
• 7. Now, mantle-to-motor neuron synapse will cause a CR.
• 8. Note: no way to get 3rd order conditioning. We've run out of fac. stages.

10/10/07 Computational Models of Neural Systems 25

Rescorla-Wagner Learning Rule

Predicted UCS strength: V = ∑

i

V i Xi V i is the response strength associated with stimulus i. Xi is 1 if the stimulus is present, else 0. Learning rule: V i = −V ⋅i Xi  is the actual strength of the stimulus.  is the learning rate. i is the salience of stimulus i.

10/10/07 Computational Models of Neural Systems 26

How Rescorla-Wagner Produces Blocking

• Train on CS1 + UCS
• V1 becomes roughly equal to λ.
• Train on (CS1, CS2) + UCS.
• Since ( λ–V ) is close to zero, ∆V2 is also close to zero.
• No learning occurs for CS2.

10/10/07 Computational Models of Neural Systems 27

How Hawkins-Kandel Produce Blocking

• Train on CS1 + UCS
• UCS fires facilitory interneuron, causing facilitation:

– of the CS1-to-motor neuron synapse – of the CS1-to-facilitory interneuron synapse

• After training, CS1 can fire the facilitory interneuron, which goes

into a refractory state.

– When UCS arrives, it can't fire the facilitory inteneuron

10/10/07 Computational Models of Neural Systems 28

How Hawkins-Kandel Produce Blocking

• Now train on (CS1,CS2) + UCS.
• Since CS1 fires the facilitory

interneuron at the same time as CS2 is firing, the timing is not right for presynaptic facilitation to help CS2.

• When the UCS arrives, it can't do

anything to help CS2 because the facilitory neuron is already in a refractory state.

• Result: no learning for CS2.

10/10/07 Computational Models of Neural Systems 29

Variations on Blocking

• Blocking can be observed even if CS1+UCS trials are

intermixed with (CS1,CS2)+UCS trials.

• Hawkins and Kandel explanation: extra CS1-UCS pairings

mean that the CS1 synapse will be facilitated more quickly than the CS2 synapse.

• Once CS1-CS2 synapses are together strong enough to fire the

facilitory neuron, learning stops on the (CS1,CS2) trials, but continues on the CS1 trials (because CS1 alone isn't yet able to fire the facilitory interneuron.)

• CS2 then undergoes extinction.
• CS1 extinguishes on (CS1,CS2) trials but is increased on the

pure CS1 trials, until it asymptotes.

10/10/07 Computational Models of Neural Systems 30

Pre-Exposure

• Pre-exposure to the UCS alone, or training on CS1+UCS pairs

with intermittent unpaired UCS's, reduces learning.

• Rescorla-Wagner explanation: a “background” CS is credited for

the UCS, and therefore acts to block CS1.

• Hawkins-Kandel: this would require continuous excitation of the

facilitator neurons so the UCS couldn't fire them. Might result from a “generalized anxiety” state.

• Alternative: extra UCS shocks habituate the UCS pathway,

which makes the UCS less effective on CS1+UCS trials.

• UCS also causes sensitization, so CS2 would show elevated

effect due to sensitization while CS1 shows reduced learning (but still more than CS2) due to habituation.

10/10/07 Computational Models of Neural Systems 31

Hawkins-Kandel Conclusions

• Conditioning of the gill- and siphon-withdrawal effect in Aplysia

shows stimulus specificity, extinction, recovery, and the effects

• f contingency.
• Second-order conditioning, blocking, and US pre-exposure have

not been demonstrated in Aplysia, but have been shown in Limax.

• The Hawkins-Kandel model does not account for learned

behavioral inhibition, only learned excitation.

– Learmed inhibition: pair CS1+UCS, then (CS1,CS2) with no UCS;

animal learns that CS2 has negative association with UCS.

– Not known if Aplysia withdrawl reflex can show learned inhibition.

10/10/07 Computational Models of Neural Systems 32

Hawkins-Kandel Summary

• The Hawkins-Kandel model depends on synaptic depression for

negative learning. No inhibitory synapses anywhere.

• The “alphabet” proposal is purely speculative.
• What can Aplyisa learn?

– Maybe only the things it is pre-wired to learn.

• Vertebrates aren't constrained this way.

10/10/07 Computational Models of Neural Systems 33

Gluck and Thompson (1987)

• Built a computational instantiation of the Hawkins-Kandel model.
• Includes temporal structure of trials; parameters for learning and

decay rates.

• Doesn't include any biophysics: no channels or transmitters.
• Yields new insight into the Hawkins-Kandel model:

– Their blocking mechanism doesn't work! – But it can be fixed.

• This is what modeling is supposed to do.

10/10/07 Computational Models of Neural Systems 34

Basic Formulation

Activation level of a unit: At Spike function: St = 1 with probability At

• therwise

Synaptic strength: V t Synaptic effect: Pt = 1 with probability St⋅V t

• therwise

At St Pt

10/10/07 Computational Models of Neural Systems 35

Eligibility Trace

T t = 1 when a spike occurs, i.e., Pt is 1. T t1 = T t − ⋅T t = 1− ⋅Tt t is the epsp decay rate. Eligibility trace: t = Tt⋅ [1−T t] Note: t is zero when a spike occurs. It rises quickly, falls slowly, due to exponential decay of T(t). T(t) Φ(t)

10/10/07 Computational Models of Neural Systems 36

Learning Rule

Pairing-specific enhancement of sensitization: V cst = 1[1−V cst] with probability t

• therwise

Learning at optimal ISI.

10/10/07 Computational Models of Neural Systems 37

Stimulus Timing

Simultaneous CS + UCS produces almost no learning.

10/10/07 Computational Models of Neural Systems 38

Stimulus Timing

Long ITI produces reduced learning.

10/10/07 Computational Models of Neural Systems 39

Further Equations

Habituation: every time a CS synapse passes a pulse to a motor neuron, its strength decreases: V cst = −2V cst if Pcst is 1

• therwise

Motor neuron firing:  Amnt = 1[1−Amnt] if Pcst is 1 or Pust is 1 −2 Amnt

• therwise

1 is activation growth rate parameter 2 is activation decay rate parameter

10/10/07 Computational Models of Neural Systems 40

Facilitory Interneuron

• Activity level is AFI(t)
• Refractory variable RFI is set to

1 when AFI exceeds a fixed threshold, then decays slowly back to 0.  AFIt = 1[1−AFIt] with probablility 1−RFI if Pcst or Pust is 1 −2 AFIt

• therwise

10/10/07 Computational Models of Neural Systems 41

Second Order Conditioning

10/10/07 Computational Models of Neural Systems 42

Results So Far

• Classical conditioning
• Sensitivity to ISI (inter-stimulus interval)
• Differential conditioning (stimulus specificity)
• Adding a facilitator interneuron gives second-order conditioning,

as in Hawkins-Kandel model.

10/10/07 Computational Models of Neural Systems 43

Blocking

• Facilitory interneuron needs to go through a refractory stage.
• Because refractory period is longer than the ISI, we must have

a direct connection from UCS sensory neuron to the motor neuron so that we can still get the UCR.

• Problem: this formulation fails to produce blocking.

10/10/07 Computational Models of Neural Systems 44

Why Asymptotic Blocking Fails

• If interneuron firing to UCS is completely eliminated by

refractory state, there will be no activity-dependent presynaptic facilitation.

• The CS1 connections therefore habituate downward.
• Eventually, CS1 can no longer fire the facilitory interneuron.
• So, can't have complete inibition of the FI; let it fire enough to
• ffset habituation. But this allows learning of CS2.
• Result: can only get mild, preasymptotic blocking effect.

10/10/07 Computational Models of Neural Systems 45

Why Asymptotic Blocking Fails

• Root cause: Rescorla-Wagner use an associative learning rule

for extinction/habituation: these occur only when there is no

• UCS. With no UCS, λ = 0, giving:
• Hawkins-Kandel use a non-associative learning rule, so they get

habituation on every trial, even when they don't want it: V j = − j 2∑

i

V i V cst = −2V cst if CS present

10/10/07 Computational Models of Neural Systems 46

Failure of Blocking

1 = 0.4 ; CS2 blocked only in first 2-3 trials Dashed line shows result without CS1 pretraining.

10/10/07 Computational Models of Neural Systems 47

How to Get Blocking

• Single interneuron model of blocking: switch from a negatively

accelerated learning curve to one that is S-shaped: positively accelerated at the beginning, negatively at the end. V j = 1[V j⋅1−V j] V j V j1−V j

1 negative accel. S-shaped Trials V(t)

10/10/07 Computational Models of Neural Systems 48

Successful Blocking

• If interneuron is only slightly active during UCS, this can block

learning of CS2 while maintaining learning of CS1 (by countering habituation).

10/10/07 Computational Models of Neural Systems 49

Multi-Interneuron Blocking

• There are other interneurons in Aplysia that could contribute to

maintenance of learned associations.

• This includes four interneurons with input from sensory

neurons, and two inhibitory interneurons.

• Suppose there is a second facilitory interneuron that:

– does not have a refractory period – sensitizes CS synapses in proportion to their current assoc. strength

V cst = 1[V cst] with probability t

• therwise

10/10/07 Computational Models of Neural Systems 50

Multi-Interneuron Blocking

• This circuit generates a more extreme case of blocking, with

stronger learning of the CS1 → CR association.

10/10/07 Computational Models of Neural Systems 51

Conclusion

• Modeling demonstrated that the Hawkins-Kandel “cell-biological

alphabet” idea could be made to work.

• But it revealed a problem with blocking. Two solutions:

– S-shaped instead of asymptotic learning curve for activity-dependent

presynaptic facilitation

– Second facilitory neuron for help in maintaining learned associations

• The modeling has also generated suggestions for things that

physiologists should look for in Aplysia circuitry.

• These are the things that modeling should do.

10/10/07 Computational Models of Neural Systems 52

Matlab Simulation

10/10/07 Computational Models of Neural Systems 53

Matlab Simulation

10/10/07 Computational Models of Neural Systems 54

Matlab Simulation

10/10/07 Computational Models of Neural Systems 55

Glanzman: It's Less Simple Than You Think

• There is a Hebbian component to Aplysia conditioning, in

addition to activity-dependent presynpatic facilitation.

• There are polysynaptic pathways. Present models only treat

monosynaptic pathways.

• There are post-synaptic

modification mechanisms in addition to presynaptic ones.

➢ Aplysia can also exhibit operant

conditioning.

Figure from Eric Kandel's Nobel lecture.