CSE/NEURO 528 Lecture 13: Reinforcement Learning & Course - - PDF document
CSE/NEURO 528 Lecture 13: Reinforcement Learning & Course Review (Chapter 9) 1 Animation: Tom Creed, SJU Early Results: Pavlov and his Dog F Classical (Pavlovian) conditioning experiments F Training: Bell Food F After: Bell
1 Animation: Tom Creed, SJU
(Chapter 9)
2
F Classical (Pavlovian)
conditioning experiments
F Training: Bell Food F After: Bell Salivate F Conditioned stimulus
(bell) predicts future reward (food)
Image: Wikimedia Commons; Animation: Tom Creed, SJU
3
F How do we predict rewards delivered some time after a
stimulus is presented?
F Given: Many trials, each of length T time steps F Time within a trial: 0 t T with stimulus u(t) and reward
r(t) at each time step t (Note: r(t) can be zero for some t)
F We would like a neuron whose output v(t) predicts the
expected total future reward starting from time t
trials t T
4
F Use a set of synaptic weights w(t) and predict
based on all past stimuli u(t):
F Learn weights w() that minimize error:
t
2
t T
Yes, BUT future rewards are not yet available! (Can we minimize this using gradient descent and delta rule?)
) (t v
) (t u ) 1 ( t u
) ( u ) ( w ) (t w ) (T w
(Linear filter!)
5
F Key Idea: Rewrite error function to get rid of future terms: F Temporal Difference (TD) Learning:
2 2 1 2
t T t T
Expected future reward Prediction
Minimize this using gradient descent!
6
Stimulus at t = 100 and reward at t = 200
Prediction error for each time step (over many trials)
Image Source: Dayan & Abbott textbook
7
Dopaminergic cells in Ventral Tegmental Area (VTA) Before Training After Training
Reward Prediction error δ? No error
)] 1 ( ) ( [ t v t v )] ( ) 1 ( ) ( [ t v t v t r
Image Source: Dayan & Abbott textbook
8
Dopaminergic cells in VTA after Training
Negative error
Reward expected but not delivered
Image Source: Dayan & Abbott textbook
9
Agent Environment State ut Reward rt Action at
10
Agent Environment State ut Action at Reward rt Learn a state-to-action mapping or “policy”: which maximizes the expected total future reward:
trials t T
11
States = locations A, B, or C Actions= L (go left)
If the rat chooses L or R at random (random “policy”), what is the expected reward (or “value”) v for each state?
Image Source: Dayan & Abbott textbook
12
For random policy: Can learn value of states using TD learning:
75 . 1 ) ( 2 1 ) ( 2 1 ) ( 1 2 1 2 2 1 ) ( 5 . 2 5 2 1 2 1 ) ( C v B v A v C v B v
Let value of state u v(u) = weight w(u)
(Location, action) new location i.e., (u,a) u’
13
1.75 2.5 1 Once I know the values, I can pick the action that leads to the higher valued state!
(For all three, = 0.5)
Image Source: Dayan & Abbott textbook
14
Values act as surrogate immediate rewards Locally
to globally optimal policy for “Markov” environments (Related to Dynamic Programming)
15
F
Two separate components: Actor (selects action and maintains policy) and Critic (maintains value of each state)
Value of state u = v(u) = w(u)
3. Repeat 1 and 2
)) ; ' ( )]( ( ) ' ( ) ( [ ) ( ) (
' ' '
u a P u v u v u r u Q u Q
aa a a
b b a
u Q u Q u a P )) ( exp( )) ( exp( ) ; ( Probabilistically select an action a at state u
(same as TD rule) For all actions a’:
16
Probability of going Left at each location
Image Source: Dayan & Abbott textbook
17
Cortex Striatum GPi/SNr Thalamus STN GPe SNc
DA
State Estimate Hidden Layer TD error Action Value Actor Critic
(See Supplementary Materials for references)
18
Example: Google’s AlphaGo beats human champion in Go, Autonomous Helicopter Flight (learned from human demonstrations)
(Videos and papers at: http://heli.stanford.edu/)
19
20
What is the neural code?
What is the nature of the code? Representing the spiking output: single cells vs populations rates vs spike times vs intervals What features of the stimulus does the neural system represent?
21
Encoding: building functional models of neurons/neural systems and predicting the spiking output given the stimulus Decoding: what can we say about the stimulus given what we
22
23
Biophysical Models of Neurons
24
Ohm’s law: and Kirchhoff’s law
Capacitive current Ionic currents Externally applied current
25
V
m e L m
If V > Vthreshold Spike Then reset: V = Vreset Integrate-and- Fire Model
26
Output Decay Input Feedback
27
w
T T
w
T
w
Hebb rule performs principal component analysis (PCA)
28
] ; | [ G p v u ] ; | [ G p u v
Causes v Data u Generative model Recognition model (data likelihood) (posterior) Unsupervised learning = learning the hidden causes of input data G = (v, v) Causes of clustered data “Causes”
images
Use EM algorithm for learning
29
Droning lecture Mathematical derivations Lack of sleep
30
)) ( (
m k k jk j ij m i
u w g W g v
2 ,
m i i m m i jk ij
m k
m j
Goal: Find W and w that minimize errors:
Backpropagation for Multilayered Networks
jk m j m j jk jk jk jk ij ij ij
w x x E w w E w w W E W W Gradient descent learning rules:
Desired output (Delta rule) (Chain rule)
31
rewards (TD learning):
state using TD learning
based on value of next state (using the TD error)
(http://employees.csbsju.edu/tcreed/pb/pdoganim.html)
2.5 1
32
synapses
advantages?
channels and their density
such as efficient coding and Bayesian inference?
environment and engage in purposeful behavior?
33
et al., MIT Press, 1997
Oxford University Press, 1999
al., MIT Press, 2002
2007
Sutton and A. Barto, MIT Press, 1998
34
mins/group (with questions)
Introduction, Background, Methods, Results, Conclusion
laptop (Windows machine) OR
email to both Adrienne, Rich, and Raj before midnight)
35
Have a great weekend!