IN5550: Neural Methods in Natural Language Processing IN5550 - - PowerPoint PPT Presentation

IN5550: Neural Methods in Natural Language Processing – IN5550 – Neural Methods in Natural Language Processing Transformers

Jeremy Barnes

University of Oslo

March 31, 2020

Attention - tl;dr

Pay attention to a weighted combination of input states to generate the right output state

2

Attention

RNNs + attention work great, but are inefficient ◮ impossible to make computation parallel ◮ leads to long train times and smaller models ◮ To enjoy the benefits of deep learning, our models need to be truly deep!

3

Desiderta for a new kind of model

• 1. Reduce the total computational complexity per layer
• 2. Increase the amount of computation that can be parallelized
• 3. Ensure that the model can efficiently learn long range dependencies.

4

Self-attention

John Lennon, 1967: love is all u need Vaswani et al., 2017:

5

Self-attention

Main principle: instead of a target paying attention to different parts of the source, make the source pay attention to itself.

6

Self-attention

I am not so happy . I am not so happy . Values (V) Keys (K)

◮ By making parts of a sentence pay attention to other parts of itself, we get better representations ◮ This can be an RNN replacement ◮ Where an RNN carries long-term information down a chain, self-attention acts more like a tree

7

Transformer

Remember for attention: K = key vector V = value vector

8

Transformer

For Transformers, we will add another: Q = query vector K = key vector V = value vector We will see what the difference is in a minute

9

Transformer

10

Transformer attention

11

Scaled dot product attention

Remember, we have two ways of doing attention

• 1. easy, fast way: dot product attention
• 2. parameterized : 1-layer feed-forward network to determine attention

weights But can’t we get the benefits of the 2nd without the extra parameters? hypothesis: With large values, dot product leads to large values, and the gradient becomes small (squashing with sigmoid/tanh) solution: Scaled dot-product attention Make sure the dot product doesn’t get too big

12

Scaled dot product attention

The important bit: The maths: Attention(Q, K, V ) = softmax(QKT √dk )V

13

Transformer

Attention(Q, K, V ) = softmax(QKT √dk )V What’s happening at a token level: ◮ Obtain three representations of the input, Q, K and V - query, key and value ◮ Obtain a set of relevance strengths: QKT . For words i and j, Qi · Kj represents the strength of the association - exactly like in seq2seq attention. ◮ Scale it (stabler gradients, boring maths) and softmax for αs. ◮ Unlike seq2seq, use different ‘value’ vectors to weight. In a sense, this is exactly like seq2seq attention, except: a) non-recurrent representations, b) same source/target, c) different value vectors

14

Intuition behind Query, Key, and Value vectors

15

Multi-head attention

16

Adding heads

Revolutionary idea: if representations learn so much from attention, why not learn many attentions Multi-headed attention is many self-attentions (Simplified) transformer:

17

Point-wise feed-forward layers

18

Point-wise feed-forward layers

◮ Add 2-layer feed-forward layers after attention layers ◮ Same across all positions, but different for layers ◮ Again a trade-off to increase model complexity while keeping computation costs down

19

Position embeddings

20

Position embeddings

But wait, now we lost our sequence information :( ◮ Use an encoding that gives this information ◮ Mix of sine and cosine functions

◮ How would this work? ◮ Why do they need both?

In the end, learning positional embeddings is often better ◮ But, it has a very large disadvantage ◮ No way to represent sequences longer than those seen in training ◮ You have to chop your data off at an arbitrary length

21

Depth

To get the benefits of deep learning, we need depth. ◮ Let’s make it deep: ◮ encoder: 6 layers ◮ decoder : 6 layers

22

Transformer

23

Transformer

◮ Can be complicated to train ◮ Has its own ADAM setup (learning rate is proportional to step−0.5) ◮ dropout added just before residual ◮ label smoothing ◮ during decoding add length penalties ◮ checkpoint averaging

24

Transformer

◮ Have a look at The Annotated Transformer ◮ http://nlp.seas.harvard.edu/2018/04/03/attention.html

25

Evolution of Transformer-based Models

26

Evolution of Transformer-based Models

27

Evolution of Transformer-based Models

28

Carbon footprint

*from Strubel et al. (2019) Energy and policy considerations for deep learning in NLP. 29

What can we to avoid wasting resources?

30

Sharing is caring

To avoid retraining lots of models ◮ We can share the trained models ◮ Nordic Language Processing Laboratory (NLPL) is a good example ◮ But it’s important to get things right

◮ METADATA!!! ◮ same format for all models

31

Reduce model size?

What if we can reduce the size of these giant models? ◮ Often, overparameterized transformer models lead to better performance, even with less data ◮ Lottery-ticket hypothesis: for large enough models, there is a small chance that random initialization will lead to a submodel that already has good weights for the task ◮ But interestingly, you can often remove a large number of the parameters for only a small decrease in performance

32

Reduce model size?

33

Model distillation

This is an example.

Pretrained T eacher Model

This is an example.

Pretrained T eacher Model Student Model Model Model 34

Head pruning

Pretrained T eacher Model

Performance = 93.7 Performance = 93.7 Performance = 93.0

But how can we NLPers contribute to sustainability?

◮ When possible use pre-trained models ◮ If you train a strong model, similarly make it available to the community. ◮ Try to reduce the amount of hyperparameter tuning we do (for example, by working with models that are more robust to hyperparameters)

36