At Attent ntio ion The The proble lem For very long sentences, - - PowerPoint PPT Presentation
At Attent ntio ion The The proble lem For very long sentences, the score for machine translation really goes down after 30-40 words. With attention Performance degradation Bahdanau et al 2014. Neural machine translation by jointly
translation really goes down after 30-40 words.
Bahdanau et al 2014. Neural machine translation by jointly learning to align and translate.
With attention Performance degradation
2
[Christopher Olah] Understanding LSTMs
Hidden state input Previous hidden state
3
Hidden state Parameters to be learned
4
Hidden state Same parameters for each time step = generalization! Output
5
[Christopher Olah] Understanding LSTMs
Hidden state is the same
6
7
I mo moved to Germany any … so I speak German an fluently
8
I mo moved to Germany any … so I speak German an fluently
ATTENTION: Which hidden states are more important to predict my output?
9
Context
I mo moved to Germany any … so I speak German an fluently
αt,t+1 αt+1,t+1 α1,t+1
10
the information
input:
– Previous decoder hidden state – Previous output – Attention
D D D
Context
αt,t+1 αt+1,t+1
11
is important to translate the work in position
Soft ft attention: All attention masks alpha sum up to 1
α1,t+1 t + 1 + 1 ct+1 =
t+1
X
k=1
αk,t+1ak
12
NN
a1 dt
Hidden state of the encoder Previous state of the decoder
f1,t+1 α1,t+1 = expf1,t+1 Pt+1
k=1 expfk,t+1
13
BIRD
15
to just run our classification network on the patch?
16
18
Xu et al 2015. Show attention and tell: neural image caption generation with visual attention.
Encoder: a classification CNN (VGGNet, AlexNet). This computes a feature maps over the image.
Decoder: an attention-based RNN
– In each time step, the decoder computes an attention map over the entire image, effectively deciding which regions to focus on. – It receives a context vector, which is the weighted average of the conv net features.
19
Encoder Decoder
Image from: https://blog.heuritech.com/2016/01/20/attention-mechanism/
20
LSTM only sees the image once!
A girl is throwing a frisbee in the park
21
A girl is throwing a frisbee in the park
22
A girl is throwing a frisbee in the park
23
A girl is throwing a frisbee in the park
24
y": Output of encoder are the image features which still retain spatial information (no FC layer!) Z": Output of attention model h": Hidden state of LSTM
25
How does the attention model look like?
26
Image: https://blog.heuritech.com/2016/01/20/attention-mechanism/
27
Any past hidden state Visual features Output attention
28
29
Still related to the spatial location of the image
!" = tanh (
)*+ + (
30
and 1
31
importance
32
probabilities
33
si
attent ntion: deterministic process that can be backproped
ard at attent ntion: stochastic process, gradient is estimated through Monte Carlo sampling.
be incorporated into the optimization more easily
34
35
Soft Hard
36
Image: Stanford CS231n lecture
Att Attention module Final context
37
Image: Stanford CS231n lecture
image cropping!
gradient descent, what alternative could we use to train this function? Reinforcement Learning
38
Xu et al 2015. Show attention and tell: neural image caption generation with visual attention.
Translation,” EMNLP 2015
2015
2015
Spatial Attention for Visual Question Answering”, arXiv 2015
2015
Tracker with Spatial-Temporal Attention Mechanism“. ICCV 2017
39
41
(question) needs to be understood, the image is needed to create the answer.
42
(question) needs to be understood, the image is needed to create the answer.
43
n the he cont ntext of the other?
nditioni ning ng: the computation carried out by a model is conditioned or modulated by information extracted from an auxiliary input.
(see p. 39)
44
45
Image: https://distill.pub/2018/feature-wise-transformations/
46
Image: https://distill.pub/2018/feature-wise-transformations/
47
Image: https://distill.pub/2018/feature-wise-transformations/
48
Image: https://distill.pub/2018/feature-wise-transformations/
à expressed as an image (same dimensionality)
49
50
Wait for the GAN intro in a few weeks!
à expressed as an image (same dimensionality)
measurements (low-dimensional)
51
52
Image: https://distill.pub/2018/feature-wise-transformations/
Think about the similarities with concatenation
conditioning
53
Image: https://distill.pub/2018/feature-wise-transformations/
– Long-Short Term Memory units
related between e.g. the two sources
and effective à number of parameters to be learned scales linearly with the number of features of the NN
54
55
Conditional Affine Transformation
56
Image: https://distill.pub/2018/feature-wise-transformations/
Information coming from e.g. the other source
57
Image: https://distill.pub/2018/feature-wise-transformations/
Strub et al. ECCV 2018.
Dumoulin et al ICLR 2017
van den Oord et al. NIPS 2016
58
59
60
Image: https://distill.pub/2018/feature-wise-transformations/
ations ns contain the most useful information
ature map aps contain the most useful information
61
Image: https://distill.pub/2018/feature-wise-transformations/
62