Neural Machine Translation Spring 2020 2020-03-12 Adapted from - - PowerPoint PPT Presentation

Neural Machine Translation

Spring 2020

2020-03-12

CMPT 825: Natural Language Processing

SFU NatLangLab

Adapted from slides from Danqi Chen, Karthik Narasimhan, and Jetic Gu. (with some content from slides from Abigail See, Graham Neubig)

Course Logistics

• Project proposal is due today
• What problem are you addressing? Why is it interesting?
• What specific aspects will your project be on?
• Re-implement paper? Compare different methods?
• What data do you plan to use?
• What is your method?
• How do you plan to evaluate? What metrics?

• Statistical MT
• Word-based
• Phrase-based
• Syntactic

Last time

Neural Machine Translation

• A single neural network is used to translate from source

to target

• Architecture: Encoder-Decoder
• Two main components:
• Encoder: Convert source sentence (input) into a vector/

matrix

• Decoder: Convert encoding into a sentence in target

language (output)

Sequence to Sequence learning (Seq2seq)

• Encode entire input sequence into a single vector (using an RNN)
• Decode one word at a time (again, using an RNN!)
• Beam search for better inference
• Learning is not trivial! (vanishing/exploding gradients)

(Sutskever et al., 2014)

Encoder

xt ht−1 xt+1 ht xt+2 ht+1 ht+2 xt+3 ht+3

h This cat is cute Sentence: This cat is cute

word embedding

Encoder

x1 h0 xt+1 h1 xt+2 ht+1 ht+2 xt+3 ht+3

h Sentence: This cat is cute

word embedding

This cat is cute

x1 h0 x2 h1 x3 h2 h3 x4 h4

Encoder

xt+2 ht+2 xt+3 ht+3

h Sentence: This cat is cute

word embedding

This cat is cute

Encoder

x1 h0 x2 h1 x3 h2 h3 x4 h4

(encoded representation)

Sentence: This cat is cute

word embedding

henc

This cat is cute

<s> ce chat est mianon

Decoder

x′

1

x′

2

z1 x′

3

z2

ce

• z3
• x′

4

z4

• chat

mignon est

x′

5

z5

• <e>

mignon

word embedding

henc

<s> ce chat est mianon

Decoder

y1

henc

x′

2

z1 x′

3

z2

ce

• z3
• x′

4

z4

• chat

mignon est

x′

5

z5

• <e>

mignon

word embedding

<s> ce chat est mianon

Decoder

y1 y2 z1 x′

3

z2

ce

• z3
• x′

4

z4

• chat

mignon est

x′

5

z5

• <e>

mignon

word embedding

henc

Decoder

y1 y2 z1 y3 z2

ce

• z3
• y4

z4

• <s> ce chat est mianon

chat mignon

• A conditioned language model

est

y5 z5

• <e>

word embedding

henc

Seq2seq training

• Similar to training a language model!
• Minimize cross-entropy loss:
• Back-propagate gradients through both decoder and encoder
• Need a really big corpus

T

∑

t=1

− log P(yt|y1, . . . , yt−1, x1, . . . , xn)

English: Machine translation is cool!

36M sentence pairs

Russian: Машинный перевод - это крутo!

Seq2seq training

(slide credit: Abigail See)

Remember masking

1 1 1 1 1 1 1 1 1 1 1 1 1 1

Use masking to help compute loss for batched sequences

Scheduled Sampling

(figure credit: Bengio et al, 2015)

Possible decay schedules (probability using true y decays over time)

How seq2seq changed the MT landscape

MT Progress

(source: Rico Sennrich)

(Wu et al., 2016)

NMT vs SMT

Pros

• Better performance
• Fluency
• Longer context
• Single NN optimized end-to-end
• Less engineering
• Works out of the box for many

language pairs

Cons

• Requires more data and

compute

• Less interpretable
• Hard to debug
• Uncontrollable
• Heavily dependent on data -

could lead to unwanted biases

• More parameters

Seq2Seq for more than NMT

Task/Application Input Output Machine Translation French English Summarization Document Short Summary Dialogue Utterance Response Parsing Sentence Parse tree (as sequence) Question Answering Context + Question Answer

Cross-Modal Seq2Seq

Task/Application Input Output Speech Recognition Speech Signal Transcript Image Captioning Image Text Video Captioning Video Text

Issues with vanilla seq2seq

• A single encoding vector,

, needs to capture all the information about source sentence

• Longer sequences can lead to vanishing gradients
• Overfitting

henc

Bottleneck

Issues with vanilla seq2seq

• A single encoding vector,

, needs to capture all the information about source sentence

• Longer sequences can lead to vanishing gradients
• Overfitting

henc

Bottleneck

Remember alignments?

Attention

• The neural MT equivalent of alignment models
• Key idea: At each time step during decoding, focus on a

particular part of source sentence

• This depends on the decoder’s current hidden state (i.e.

notion of what you are trying to decode)

• Usually implemented as a probability distribution over the

hidden states of the encoder ( )

henc

i

Seq2seq with attention

(slide credit: Abigail See)

Seq2seq with attention

(slide credit: Abigail See)

Seq2seq with attention

(slide credit: Abigail See)

Seq2seq with attention

(slide credit: Abigail See)

Can also use as input for next time step

̂ y1

Seq2seq with attention

(slide credit: Abigail See)

Computing attention

• Encoder hidden states:
• Decoder hidden state at time :
• First, get attention scores for this time step (we will see what is soon!):

• Obtain the attention distribution using softmax:

• Compute weighted sum of encoder hidden states:

• Finally, concatenate with decoder state and pass on to output layer:

henc

1 , . . . , henc n

t hdec

t

g et = [g(henc

1 , hdec t

), . . . , g(henc

n , hdec t

)] αt = softmax (et) ∈ ℝn at =

n

∑

i=1

αt

ihenc i

∈ ℝh [at; hdec

t

] ∈ ℝ2h

Types of attention

• Assume encoder hidden states

and decoder hidden state

• 1. Dot-product attention (assumes equal dimensions for and :

• 2. Multiplicative attention:


, where is a weight matrix

• 3. Additive attention:



 where are weight matrices and is a weight vector

h1, h2, . . . , hn z

a b ei = g(hi, z) = zThi ∈ ℝ g(hi, z) = zTWhi ∈ ℝ W g(hi, z) = vT tanh (W1hi + W2z) ∈ ℝ W1, W2 v

Issues with vanilla seq2seq

• A single encoding vector,

, needs to capture all the information about source sentence

• Longer sequences can lead to vanishing gradients
• Overfitting

henc

Bottleneck

Dropout

• Form of regularization for RNNs (and any NN in general)
• Idea: “Handicap” NN by removing hidden units

stochastically

• set each hidden unit in a layer to 0 with probability

during training ( usually works well)

• scale outputs by
• hidden units forced to learn more general patterns
• Test time: Use all activations (no need to rescale)

p p = 0.5 1/(1 − p)

Handling large vocabularies

• Softmax can be expensive for large vocabularies
• English vocabulary size: 10K to 100K

P(yi) = exp(wi ⋅ h + bi) ∑|V|

j=1 exp(wj ⋅ h + bj)

Expensive to compute

Approximate Softmax

• Negative Sampling
• Structured softmax
• Embedding prediction

Negative Sampling

• Softmax is expensive when vocabulary size is large

(figure credit: Graham Neubig)

Negative Sampling

• Sample just a subset of the vocabulary for negative
• Saw simple negative sampling in word2vec (Mikolov 2013)

(figure credit: Graham Neubig)

Other ways to sample: Importance Sampling (Bengio and Senecal 2003) Noise Contrastive Estimation (Mnih & Teh 2012)

Hierarchical softmax

(Morin and Bengio 2005) (figure credit: Quora)

Class based softmax

• Two-layer: cluster words into classes, predict class and

then predict word.

(Gooding 2001, Mikolov et al 2011)

• Clusters can be based on frequency, random, or word

contexts.

(figure credit: Graham Neubig)

Embedding prediction

• Directly predict embeddings of outputs

themselves

• What loss to use? (Kumar and Tsvetkov 2019)
• L2? Cosine?
• Von-Mises Fisher distribution loss, make

embeddings close on the unit ball

(slide credit: Graham Neubig)

Generation

How can we use our model (decoder) to generate sentences?

• Sampling: Try to generate a random sentence

according the the probability distribution

• Argmax: Try to generate the best sentence,

the sentence with the highest probability

Decoding Strategies

• Ancestral sampling
• Greedy decoding
• Exhaustive search
• Beam search

Ancestral Sampling

• Randomly sample words one by one
• Provides diverse output (high variance)

(figure credit: Luong, Cho, and Manning)

Greedy decoding

• Compute argmax at every step of decoder to

generate word

• What’s wrong?

Exhaustive search?

• Find
• Requires computing all possible sequences
• complexity!
• Too expensive

arg max

y1,...,yT

P(y1, . . . , yT|x1, . . . , xn) O(VT)

Recall: Beam search (a middle ground)

• Key idea: At every step, keep track of the k most

probable partial translations (hypotheses)

• Score of each hypothesis = log probability
• Not guaranteed to be optimal
• More efficient than exhaustive search

j

∑

t=1

log P(yt|y1, . . . , yt−1, x1, . . . , xn)

Beam decoding

(slide credit: Abigail See)

Beam decoding

(slide credit: Abigail See)

Beam decoding

(slide credit: Abigail See)

Backtrack

(slide credit: Abigail See)

Beam decoding

• Different hypotheses may produce

(end) token at different time steps

• When a hypothesis produces

, stop expanding it and place it aside

• Continue beam search until:
• All hypotheses produce

OR

• Hit max decoding limit T
• Select top hypotheses using the normalized likelihood score
• Otherwise shorter hypotheses have higher scores

⟨eos⟩ ⟨eos⟩ k ⟨eos⟩ 1 T

T

∑

t=1

log P(yt|y1, . . . , yt−1, x1, . . . , xn)

Under translation

• Under translation
• Beam search
• Ensemble
• Coverage models
• Crucial information are left untranslated; premature generation

<EOS>

• Out-of-Vocabulary (OOV) Problem; Rare word problem
• Frequent word are translated correctly, rare words are not
• In any corpus, word frequencies are exponentially unbalanced
• Under translation
• Crucial information are left untranslated; premature generation

Under-trans<EOS>

! RNN <EOS> nobody RNN expects expects RNN the the RNN spanish spanish RNN inquisition inquisition RNN ! <SOS> RNN nobody <EOS>

'inquisition': 0.49 <EOS>: 0.51

<EOS>

• Out-of-Vocabulary (OOV) Problem; Rare word problem
• Frequent word are translated correctly, rare words are not
• In any corpus, word frequencies are exponentially unbalanced
• Under translation
• Crucial information are left untranslated; premature generation

<EOS>

Under-trans<EOS>

! RNN <EOS> nobody RNN expects expects RNN the the RNN spanish spanish RNN inquisition inquisition RNN ! <SOS> RNN nobody <EOS>

'inquisition': 0.49 <EOS>: 0.51

Ensemble

• Similar to voting mechanism, but with probabilities
• multiple models of different parameters (usually from different checkpoints
• f the same training instance)
• use the output with the highest probability across all models

C

• n

c e p t

P0 XXXXXXX P2 Ensemble

Ensemble

D e m

• P0

XXXXXXX P2 Ensemble

!

RNN RNN RNN <SOS> prof loves prof loves cheese RNN cheese <EOS>

! ! !

RNN

Ensemble

D e m

• P0

XXXXXXX P2 Ensemble RNN RNN RNN prof loves cheese RNN <EOS> RNN RNN RNN RNN prof loves cheese RNN <EOS> RNN RNN RNN RNN prof loves cheese RNN <EOS> RNN

seq2seq_E049.pt

Ensemble

D e m

• P0

XXXXXXX P2 Ensemble RNN cp2 cp2 prof loves cheese cp2 <EOS> cp2 RNN cp3 cp3 prof loves pizza cp3 <EOS> cp3 RNN cp1 cp1 prof loves kebab cp1 <EOS> cp1

seq2seq_E049.pt seq2seq_E048.pt seq2seq_E047.pt

Ensemble

D e m

• P0

XXXXXXX P2 Ensemble RNN cp2 cp2 prof loves cheese cp2 <EOS> cp2 RNN cp3 cp3 prof loves pizza cp3 <EOS> cp3 RNN cp1 cp1 prof loves kebab cp1 <EOS> cp1

seq2seq_E049.pt seq2seq_E048.pt seq2seq_E047.pt

'kebab': 0.9 'cheese': 0.05 'pizza': 0.05 …… 'kebab': 0.3 'cheese': 0.55 'pizza': 0.05 …… 'kebab': 0.3 'cheese': 0.05 'pizza': 0.55 ……

'kebab': 0.3 'cheese': 0.55 'pizza': 0.05 …… 'kebab': 0.3 'cheese': 0.05 'pizza': 0.55 ……

Ensemble

D e m

• P0

XXXXXXX P2 Ensemble RNN cp2 cp2 prof loves cheese cp2 <EOS> cp2 RNN cp3 cp3 prof loves pizza cp3 <EOS> cp3 RNN cp1 cp1 prof loves kebab cp1 <EOS> cp1

seq2seq_E049.pt seq2seq_E048.pt seq2seq_E047.pt

'kebab': 0.9 'cheese': 0.05 'pizza': 0.05 …… kebab kebab

Ensemble

D e m

• P0

XXXXXXX P2 Ensemble RNN cp2 cp2 prof loves cheese cp2 <EOS> cp2 RNN cp3 cp3 prof loves pizza cp3 <EOS> cp3 RNN cp1 cp1 prof loves kebab cp1 <EOS> cp1

seq2seq_E049.pt seq2seq_E048.pt seq2seq_E047.pt

kebab kebab

Existing challenges with NMT

• Out-of-vocabulary words
• Low-resource languages
• Long-term context
• Common sense knowledge (e.g. hot dog, paper jam)
• Fairness and bias
• Uninterpretable

Existing challenges with NMT

• Out-of-vocabulary words
• Low-resource languages
• Long-term context
• Common sense knowledge (e.g. hot dog, paper jam)
• Fairness and bias
• Uninterpretable

Out-of-vocabulary words

• Out-of-Vocabulary (OOV) Problem; Rare word problem
• Frequent word are translated correctly, rare words are not
• In any corpus, word frequencies are exponentially unbalanced

OOV

• 1. https://en.wikipedia.org/wiki/Zipf%27s_law

Zipf’s Law

Occurrence Count 1 10 100 1000 10000 100000 1000000 Frequency Rank 1 25000 50000

• rare word are exponentially less frequent than frequent words
• e.g. in HW4, 45%|65% (src|tgt) of the unique words occur once

Out-of-vocabulary words

• Out-of-Vocabulary (OOV) Problem; Rare word problem
• Copy Mechanisms
• Subword / character models
• Out-of-Vocabulary (OOV) Problem; Rare word problem

Copy Mechanism

C

• n

c e p t

P0 XXXXXXX P1 Copy

!

RNN RNN RNN <SOS> prof loves prof loves cheese RNN cheese <EOS>

! ! !

RNN

• 1. CL2015015 [Luong et al.] Addressing the Rare Word Problem in Neural Machine Translation

Copy Mechanism

C

• n

c e p t

P0 XXXXXXX P1 Copy

!

RNN RNN RNN <SOS> prof loves prof loves cheese RNN cheese <EOS>

! ! !

RNN <UNK> prof liebt kebab <UNK> αt,1 = 0.1 αt,2 = 0.2 αt,3 = 0.7 kebab

• 1. CL2015015 [Luong et al.] Addressing the Rare Word Problem in Neural Machine Translation

Copy Mechanism

C

• n

c e p t

P0 XXXXXXX P1 Copy

!

RNN RNN RNN <SOS> prof loves prof loves cheese RNN cheese <EOS>

! ! !

RNN <UNK> prof liebt kebab <UNK> αt,1 = 0.1 αt,2 = 0.2 αt,3 = 0.7 kebab

• 1. CL2015015 [Luong et al.] Addressing the Rare Word Problem in Neural Machine Translation

Copy Mechanism

C

• n

c e p t

P0 XXXXXXX P1 Copy

!

RNN RNN RNN <SOS> prof loves prof loves cheese

! !

RNN prof liebt kebab <UNK> αt,1 = 0.1 αt,2 = 0.2 αt,3 = 0.7 kebab

• Sees <UNK> at step
• looks at attention weight
• replace <UNK> with the source word

t αt fargmaxiαt,i

• 1. CL2015015 [Luong et al.] Addressing the Rare Word Problem in Neural Machine Translation

*Pointer-Generator Copy Mechanism

C

• n

c e p t

P0 XXXXXXX P1 Copy

• 1. CL2016032 [Gulcehre et al.] Pointing the Unknown Words

RNN <UNK>

!

pg

!

RNN RNN RNN <SOS> prof loves prof loves cheese RNN cheese <EOS>

! ! !

RNN <UNK> prof liebt kebab <UNK> αt,1 = 0.1 αt,2 = 0.2 αt,3 = 0.7 kebab

• binary classifier switch
• use decoder output
• use copy mechanism

*Pointer-Generator Copy Mechanism

C

• n

c e p t

P0 XXXXXXX P1 Copy

• 1. CL2016032 [Gulcehre et al.] Pointing the Unknown Words

RNN <UNK>

!

pg

• Sees <UNK> at step , or
• looks at attention weight
• replace <UNK> with the source word

t pg([hdec

t

; contextt]) ≤ 0.5 αt fargmaxiαt,i

*Pointer-Based Dictionary Fusion

C

• n

c e p t

P0 XXXXXXX P1 Copy

• 1. CL2019331 [Gū et al.] Pointer-based Fusion of Bilingual Lexicons into Neural Machine Translation

RNN <UNK>

!

pg

• Sees <UNK> at step , or
• looks at attention weight
• replace <UNK> with translation of the source word

t pg([hdec

t

; contextt]) ≤ 0.5 αt dict(fargmaxiαt,i)

Out-of-vocabulary words

• Out-of-Vocabulary (OOV) Problem; Rare word problem
• Copy Mechanisms
• Subword / character models
• Out-of-Vocabulary (OOV) Problem; Rare word problem

OOV

Subword / character models

• Character based seq2seq models

Subword / character models

• Character based seq2seq models
• Byte pair encoding (BPE)

Subword / character models

• Character based seq2seq models
• Byte pair encoding (BPE)

Subword / character models

• Character based seq2seq models
• Byte pair encoding (BPE)

Subword / character models

• Character based seq2seq models
• Byte pair encoding (BPE)
• Subword units help model morphology

Existing challenges with NMT

• Out-of-vocabulary words
• Low-resource languages
• Long-term context
• Common sense knowledge (e.g. hot dog, paper jam)
• Fairness and bias
• Uninterpretable

Massively multilingual MT

(Arivazhagan et al., 2019)

• Train a single neural network on 103 languages paired with

English (remember Interlingua?)

• Massive improvements on low-resource languages

Existing challenges with NMT

• Out-of-vocabulary words
• Low-resource languages
• Long-term context
• Common sense knowledge (e.g. hot dog, paper jam)
• Fairness and bias
• Uninterpretable

Next time

• Contextualized embeddings
• ELMO, BERT, and friends
• Transformers
• Multi-headed self-attention