Generating Alignments using Target Foresight in Attention-Based - - PowerPoint PPT Presentation

▶
generating alignments using target foresight in attention
SMART_READER_LITE
LIVE PREVIEW

Generating Alignments using Target Foresight in Attention-Based - - PowerPoint PPT Presentation

Oct 05, 2022 16 likes •402 views

Generating Alignments using Target Foresight in Attention-Based Neural Machine Translation Jan-Thorsten Peter, Arne Nix, Hermann Ney peter@cs.rwth-aachen.de Mai 29, 2017 EAMT 2017, Prag Human Language Technology and Pattern Recognition

slide-1
SLIDE 1

Generating Alignments using Target Foresight in Attention-Based Neural Machine Translation

Jan-Thorsten Peter, Arne Nix, Hermann Ney

peter@cs.rwth-aachen.de Mai 29, 2017 EAMT 2017, Prag Human Language Technology and Pattern Recognition Computer Science Department RWTH Aachen University

J.-T. Peter, A.Nix, H. Ney:Target Foresight 1/23 29.05.2017

slide-2
SLIDE 2

Outline

Motivation Neural Machine Translation Target Foresight Guided Alignment Training Target Foresight with Guided Alignment Training Conclusion

J.-T. Peter, A.Nix, H. Ney:Target Foresight 2/23 29.05.2017

slide-3
SLIDE 3

Motivation

◮ Alignment use to be important for SMT ◮ Neural Machine Translation (NMT) uses attention ◮ There are still application for alignments: ⊲ Guided alignment training [Chen & Matusov+ 16] ⊲ Transread 1 ⊲ Linguee 2 ◮ Using the attention as alignment produces bad results ◮ Can we use NMT to create alignments?

1https://transread.limsi.fr 2http://www.linguee.com J.-T. Peter, A.Nix, H. Ney:Target Foresight 3/23 29.05.2017

slide-4
SLIDE 4

Related Work

  • D. Bahdanau, K. Cho, Y. Bengio [Bahdanau & Cho+ 15]:

Neural machine translation by jointly learning to align and translate. ICLR, May 2015. ◮ Introducing an attention mechanism to neural machine translation

  • W. Chen, E. Matusov, S. Khadivi, J.-T. Peter [Chen & Matusov+ 16]:

Guided alignment training for topic-aware neural machine

  • translation. AMTA, October 2016.

◮ Introduces guided alignment training

  • Z. Tu, Z. Lu, Y. Liu, X. Liu, H. Li [Tu & Lu+ 16]:

Modeling coverage for neural machine translation. ACL, August 2016. ◮ Analysing attention of neural machine translation using SAER

J.-T. Peter, A.Nix, H. Ney:Target Foresight 4/23 29.05.2017

slide-5
SLIDE 5

Outline

Motivation Neural Machine Translation Target Foresight Guided Alignment Training Target Foresight with Guided Alignment Training Conclusion

J.-T. Peter, A.Nix, H. Ney:Target Foresight 5/23 29.05.2017

slide-6
SLIDE 6

Attention Based NMT

◮ Bidirectional RNN encodes source sentence f J

1 into −

→ h J

1 and ←

− h J

1

◮ hj := [− → h T

j ; ←

− h T

j ]T

J.-T. Peter, A.Nix, H. Ney:Target Foresight 6/23 29.05.2017

slide-7
SLIDE 7

Attention Based NMT

◮ Energies computed through MLP: ˜ αij = vT

a tanh(Wasi−1 + Uahj)

Wa ∈ Rn×n, Ua ∈ Rn×2n, va ∈ Rn: weight parameters

J.-T. Peter, A.Nix, H. Ney:Target Foresight 6/23 29.05.2017

slide-8
SLIDE 8

Attention Based NMT

◮ Attention weights normalized with softmax: αij =

exp(˜ αij) J

k=1 exp(˜

αik)

J.-T. Peter, A.Nix, H. Ney:Target Foresight 6/23 29.05.2017

slide-9
SLIDE 9

Attention Based NMT

◮ Context vector as weighted sum: ci = J

j=1 αijhj

J.-T. Peter, A.Nix, H. Ney:Target Foresight 6/23 29.05.2017

slide-10
SLIDE 10

Attention Based NMT

◮ Neural network output: p(ei|ei−1

1

, f J

1 ) = gout(ei−1, si−1, ci)

gout: output function

J.-T. Peter, A.Nix, H. Ney:Target Foresight 6/23 29.05.2017

slide-11
SLIDE 11

Attention Based NMT

◮ Hidden decoder state: si = gdec(ei, ci; si−1) gdec: gated recurrent unit

J.-T. Peter, A.Nix, H. Ney:Target Foresight 6/23 29.05.2017

slide-12
SLIDE 12

GIZA++ vs. NMT Alignment

GIZA++ NMT ◮ GIZA++ creates a clean alignment ◮ Noise NMT alignment

J.-T. Peter, A.Nix, H. Ney:Target Foresight 7/23 29.05.2017

slide-13
SLIDE 13

Alignment Error Rate

◮ Alignment Evaluation: AER(S, P ; A) = 1 − |A ∩ S| + |A ∩ P | |A| + |S| [Och & Ney 03] SAER(MS, MP; MA) = 1 − |MA ⊙ MS| + |MA ⊙ MP| |MA| + |MS| [Tu & Lu+ 16]

Europarl De-En Alignment Test Model AER% SAER % GIZA++ 21.0 26.8 Attention-Based 38.1 63.6

◮ Attention is converted into hard alignment in both directions ◮ Merged using Och’s refined method [Och & Ney 03].

J.-T. Peter, A.Nix, H. Ney:Target Foresight 8/23 29.05.2017

slide-14
SLIDE 14

Alignment Error Rate

◮ Alignment Evaluation: AER(S, P ; A) = 1 − |A ∩ S| + |A ∩ P | |A| + |S| [Och & Ney 03] SAER(MS, MP; MA) = 1 − |MA ⊙ MS| + |MA ⊙ MP| |MA| + |MS| [Tu & Lu+ 16]

Europarl De-En Alignment Test Model AER% SAER % GIZA++ 21.0 26.8 Attention-Based 38.1 63.6

◮ Attention is converted into hard alignment in both directions ◮ Merged using Och’s refined method [Och & Ney 03].

J.-T. Peter, A.Nix, H. Ney:Target Foresight 8/23 29.05.2017

slide-15
SLIDE 15

Outline

Motivation Neural Machine Translation Target Foresight Guided Alignment Training Target Foresight with Guided Alignment Training Conclusion

J.-T. Peter, A.Nix, H. Ney:Target Foresight 9/23 29.05.2017

slide-16
SLIDE 16

Target Foresight

◮ Idea: Use knowledge of the target sentence eI

1 to improve the attention

˜ αij = vT

a tanh(Wasi−1 + Uahj + Va˜

ei) Va ∈ Rn×p

J.-T. Peter, A.Nix, H. Ney:Target Foresight 10/23 29.05.2017

slide-17
SLIDE 17

Raw Target Foresight

◮ Target word encoded in source embedding and attention weights

J.-T. Peter, A.Nix, H. Ney:Target Foresight 11/23 29.05.2017

slide-18
SLIDE 18

Target Foresight with Noise

NMT

die Kommission hat diesen Appell vernommen . </S> </S> . call this heeded Commission the

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Target Foresight with Noise

die Kommission hat diesen Appell vernommen . </S> </S> . call this heeded commission the

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

◮ Adding noise on attention does not help

J.-T. Peter, A.Nix, H. Ney:Target Foresight 12/23 29.05.2017

slide-19
SLIDE 19

Freeze Encoder and Decoder

NMT Target Foresight Frozen ◮ Train baseline system ◮ Freeze encoder and decoder weights ◮ Continue training with target foresight

J.-T. Peter, A.Nix, H. Ney:Target Foresight 13/23 29.05.2017

slide-20
SLIDE 20

Freeze Encoder and Decoder

NMT Target Foresight Frozen Alignment Test Model AER % SAER % GIZA++ 21.0 26.8 Attention-Based 38.1 63.6 + Target foresight with frozen en-/decoder 33.9 55.6

J.-T. Peter, A.Nix, H. Ney:Target Foresight 13/23 29.05.2017

slide-21
SLIDE 21

Outline

Motivation Neural Machine Translation Target Foresight Guided Alignment Training Target Foresight with Guided Alignment Training Conclusion

J.-T. Peter, A.Nix, H. Ney:Target Foresight 14/23 29.05.2017

slide-22
SLIDE 22

Guided Alignment Training

◮ Idea: Introducing target alignment A as a second objective [Chen & Matusov+ 16] ◮ Cross-Entropy cost Lalign between the attention weights α and target alignment A Lalign(A, α) := − 1 N

  • n

In

  • i=1

Jn

  • j=1

An,ij log αn,ij ◮ Optimize w.r.t. L(A, α, eI

1, f J 1 ) := λCE · LCE + λalign · Lalign

⊲ LCE: standard decoder cost function (cross-entropy) ⊲ λalign, λCE: weights determined through experiments

J.-T. Peter, A.Nix, H. Ney:Target Foresight 15/23 29.05.2017

slide-23
SLIDE 23

Guided Alignment Training

IWSLT De-En Test Alignment Test Model BLEU% AER% SAER% Attention-Based 29.3 41.8 66.3 + GA 30.3 35.4 44.2 ◮ Improves translation by 1.0 BLEU on IWSLT2013 Test ◮ Great improvement in AER and SAER and Alignment Test ◮ Trained an all IWSLT 2013 data

J.-T. Peter, A.Nix, H. Ney:Target Foresight 16/23 29.05.2017

slide-24
SLIDE 24

Outline

Motivation Neural Machine Translation Target Foresight Guided Alignment Training Target Foresight with Guided Alignment Training Conclusion

J.-T. Peter, A.Nix, H. Ney:Target Foresight 17/23 29.05.2017

slide-25
SLIDE 25

GIZA++ vs. Target Foresight with Guided Alignment

GIZA++ TF + GA ◮ Target foresight creates correct alignment

J.-T. Peter, A.Nix, H. Ney:Target Foresight 18/23 29.05.2017

slide-26
SLIDE 26

Results

Alignment Test Model AER % SAER % fast_align 27.9 33.0 GIZA++ 21.0 26.8 BerkeleyAligner 20.5 26.4 Attention-Based 38.1 63.6 + Guided alignment 29.8 38.0 + Target foresight with frozen en-/decoder 33.9 55.6 + Target foresight with guided alignment 19.0 34.9 + converted to hard alignment 19.0 24.6 ◮ Trained on Europal data ◮ Target foresight improves AER by 2.0% compared to GIZA++ ◮ SAER is biased towards hard alignments

J.-T. Peter, A.Nix, H. Ney:Target Foresight 19/23 29.05.2017

slide-27
SLIDE 27

Results

Alignment Test Model AER % SAER % fast_align 27.9 33.0 GIZA++ 21.0 26.8 BerkeleyAligner 20.5 26.4 Attention-Based 38.1 63.6 + Guided alignment 29.8 38.0 + Target foresight with frozen en-/decoder 33.9 55.6 + Target foresight with guided alignment 19.0 34.9 + converted to hard alignment 19.0 24.6 ◮ Trained on Europal data ◮ Target foresight improves AER by 2.0% compared to GIZA++ ◮ SAER is biased towards hard alignments

J.-T. Peter, A.Nix, H. Ney:Target Foresight 19/23 29.05.2017

slide-28
SLIDE 28

Retrain Guided Alignment

◮ Use improved alignment for guided alignment training ◮ Test data: IWSLT 2013 ◮ Train data: Europarl corpus Test Alignment Test Model BLEU AER % SAER % Attention-Based 16.0 38.1 63.6 + GA using GIZA++ 18.4 29.8 38.0 + GA using target-foresight alignments 18.8 28.5 36.7

J.-T. Peter, A.Nix, H. Ney:Target Foresight 20/23 29.05.2017

slide-29
SLIDE 29

Outline

Motivation Neural Machine Translation Target Foresight Guided Alignment Training Target Foresight with Guided Alignment Training Conclusion

J.-T. Peter, A.Nix, H. Ney:Target Foresight 21/23 29.05.2017

slide-30
SLIDE 30

Conclusion

◮ Improvement of AER by 2.0% compared to GIZA++ ◮ Can easily be used to align unseen data ◮ Aligned data can again be used for guided alignment training ◮ Neural networks will cheat if it is possible ◮ Guided alignment keeps it from cheating

J.-T. Peter, A.Nix, H. Ney:Target Foresight 22/23 29.05.2017

slide-31
SLIDE 31

Thank you for your attention! Jan-Thorsten Peter, Arne Nix, Hermann Ney

peter@cs.rwth-aachen.de

J.-T. Peter, A.Nix, H. Ney:Target Foresight 23/23 29.05.2017

slide-32
SLIDE 32

Experiment Setup

System configuration: ◮ 30000 most frequent words as source and target vocabulary ◮ Out-of-vocabulary words are mapped to unknown tokens ◮ Bi-directional encoder with 1000 GRU nodes each ◮ GRU based decoder with 1000 nodes ◮ Alignment computation also has an internal dimension of 1000 Training: ◮ 250000 iterations ◮ Evaluation after each 10000 iterations on corresponding dev set

J.-T. Peter, A.Nix, H. Ney:Target Foresight 24/23 29.05.2017

slide-33
SLIDE 33

Europarl De-En

German English train (full data) Sentences 1.2M Running Words 32M 34M Vocabulary 305K 100K align-test Sentences 504 Running Words 9.9K 10.3K Vocabulary 2.8K 2.4K OOVs with full vocabulary 6 (0.1%) 1 (0.0%) OOVs with 30K shortlist (Rate) 276 (2.8%) 50 (0.5%)

J.-T. Peter, A.Nix, H. Ney:Target Foresight 25/23 29.05.2017

slide-34
SLIDE 34

Analyzing Attention-based Alignments

◮ How good is the alignment quality of attention-based NMT? ◮ How can we evaluate attention-based alignments? ◮ How important are attention-based alignments for translation?

J.-T. Peter, A.Nix, H. Ney:Target Foresight 26/23 29.05.2017

slide-35
SLIDE 35

Analysing Attention-based Alignments (Europarl)

◮ Compare BLEU to AER, SAER on model with increasing noise on alignment parameters

0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 Noise (Standard Deviation) 10 20 30 40 50 60 70 Accuracy in %

Europarl Baseline BLEU 100 - AER Pearson:0.995 100 - SAER Pearson:0.998

J.-T. Peter, A.Nix, H. Ney:Target Foresight 27/23 29.05.2017

slide-36
SLIDE 36

Analysing Attention-based Alignments (Europarl)

◮ Attention parameters are robust to noise up to a certain degree ◮ Alignment quality correlates with translation quality for all evaluation methods

0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 Noise (Standard Deviation) 10 20 30 40 50 60 70 Accuracy in %

Europarl Baseline BLEU 100 - AER Pearson:0.995 100 - SAER Pearson:0.998

J.-T. Peter, A.Nix, H. Ney:Target Foresight 27/23 29.05.2017

slide-37
SLIDE 37

Analysing Attention-based Alignments (IWSLT2013)

◮ Attention parameters are robust to noise up to a certain degree ◮ Alignment quality correlates with translation quality for all evaluation methods

0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 Noise (Standard Deviation) 10 20 30 40 50 60 70 Accuracy in %

IWSLT Baseline BLEU 100 - AER Pearson:0.992 100 - SAER Pearson:0.996

J.-T. Peter, A.Nix, H. Ney:Target Foresight 27/23 29.05.2017

slide-38
SLIDE 38
  • D. Bahdanau, K. Cho, Y. Bengio.

Neural machine translation by jointly learning to align and translate. In ICLR, May 2015.

  • W. Chen, E. Matusov, S. Khadivi, J.-T. Peter.

Guided alignment training for topic-aware neural machine translation. Austion, Texas, October 2016. Association for Machine Translation in the Americas.

  • F. J. Och, H. Ney.

A systematic comparison of various statistical alignment models. Computational Linguistics, Vol. 29, No. 1, pp. 19–51, 2003.

  • Z. Tu, Z. Lu, Y. Liu, X. Liu, H. Li.

Modeling coverage for neural machine translation. In 54th Annual Meeting of the Association for Computational Linguistics, August 2016.

J.-T. Peter, A.Nix, H. Ney:Target Foresight 28/23 29.05.2017