SLIDE 1
Attention, Transformer and BERT
- Prof. Kuan-Ting Lai
2020/6/16
Attention is All You Need!
- A. Waswani et al., NIPS, 2017
Google Brain & University of Toronto
2
Attention, Transformer and BERT Prof. Kuan-Ting Lai 2020/6/16 - - PowerPoint PPT Presentation
Attention, Transformer and BERT Prof. Kuan-Ting Lai 2020/6/16 - - PowerPoint PPT Presentation
Attention, Transformer and BERT Prof. Kuan-Ting Lai 2020/6/16 Attention is All You Need! A. Waswani et al., NIPS , 2017 Google Brain & University of Toronto 2 Attention Visual attention and textual attention
SLIDE 2
SLIDE 3
Attention
- Visual attention and textual
attention
3
https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html
SLIDE 4
Seq2seq model
- Language translation
4
SLIDE 5
Attention = Vector of Importance Weights
5
SLIDE 6
Transformer
- http://jalammar.github.io/illustrated-transformer/
6
SLIDE 7
Encoder and Decoder
7
SLIDE 8
8
SLIDE 9
Structure of the Encoder and Decoder
- Self-attention
- Encoder-decoder attention
9
SLIDE 10
10
SLIDE 11
Tensor2Tensor Notebook
- https://colab.research.google.co
m/github/tensorflow/tensor2ten sor/blob/master/tensor2tensor/ notebooks/hello_t2t.ipynb
11
SLIDE 12
Self-attention (query, key, value)
12
https://www.youtube.com/watch?v=ugWDIIOHtPA&t=1089s
SLIDE 13
Self-attention
13
SLIDE 14
14
SLIDE 15
Calculating 𝑐2
15
SLIDE 16
Matrix Mutiplication
16
SLIDE 17
17
SLIDE 18
Adding Residual Connections
18
SLIDE 19
Layer Normalization
19
SLIDE 20
20
SLIDE 21
References
- 1. https://lilianweng.github.io/lil-log/2018/06/24/attention-
attention.html
- 2. http://jalammar.github.io/illustrated-transformer/
- 3. Hong-Yi Lee, Transformer, 2019
https://www.youtube.com/watch?v=ugWDIIOHtPA
21