neural machine translation breaking the performance
play

Neural Machine Translation: Breaking the Performance Plateau Rico - PowerPoint PPT Presentation

Mar 24, 2023 •230 likes •452 views

Neural Machine Translation: Breaking the Performance Plateau Rico Sennrich Institute for Language, Cognition and Computation University of Edinburgh July 4 2016 Rico Sennrich Neural Machine Translation 1 / 15 Is Machine Translation Getting

  1. Neural Machine Translation: Breaking the Performance Plateau Rico Sennrich Institute for Language, Cognition and Computation University of Edinburgh July 4 2016 Rico Sennrich Neural Machine Translation 1 / 15

  2. Is Machine Translation Getting Better Over Time? [Graham et al., 2014] B LEU on newstest2007 (EN → DE) 30 23 . 6 20 14 . 6 10 0 2007 best system current system (2014) Rico Sennrich Neural Machine Translation 1 / 15

  3. Edinburgh’s WMT Results Over the Years B LEU on newstest2013 (EN → DE) 30 24 . 7 22 . 1 22 21 . 5 20 . 9 20 . 8 20 . 3 20 . 2 20 19 . 4 10 0 2013 2014 2015 2016 phrase-based SMT syntax-based SMT neural MT Rico Sennrich Neural Machine Translation 2 / 15

  4. Edinburgh’s WMT Results Over the Years B LEU on newstest2013 (EN → DE) 30 24 . 7 22 . 1 22 21 . 5 20 . 9 20 . 8 20 . 3 20 . 2 20 19 . 4 10 0 2013 2014 2015 2016 phrase-based SMT syntax-based SMT neural MT Rico Sennrich Neural Machine Translation 2 / 15

  5. Edinburgh’s WMT Results Over the Years B LEU on newstest2013 (EN → DE) 30 24 . 7 22 . 1 22 21 . 5 20 . 9 20 . 8 20 . 3 20 . 2 20 19 . 4 10 0 2013 2014 2015 2016 phrase-based SMT syntax-based SMT neural MT Rico Sennrich Neural Machine Translation 2 / 15

  6. Neural Machine Translation [Bahdanau et al., 2015] Kyunghyun Cho http://devblogs.nvidia.com/parallelforall/introduction-neural-machine-translation-gpus-part-3/ Rico Sennrich Neural Machine Translation 3 / 15

  7. Why Neural Machine Translation? qualitative differences main strength of neural MT: improved grammaticality [Neubig et al., 2015] phrase-based SMT strong independence assumptions log-linear combination of many “weak” features neural MT output conditioned on full source text and target history end-to-end trained model Rico Sennrich Neural Machine Translation 4 / 15

  8. Example (WMT16 EN → DE) source But he wants an international reporter to be there to write about it. reference Aber er will , dass ein internationaler Reporter anwesend ist , um dort zu schreiben . PBSMT Aber er will einen internationalen Reporter zu sein , darüber zu schreiben . SBSMT Aber er will einen internationalen Reporter , um dort zu sein , über sie zu schreiben . neural MT Aber er will , dass ein internationaler Reporter da ist , um darüber zu schreiben . Rico Sennrich Neural Machine Translation 5 / 15

  9. Recent Advances in Neural MT some problems: networks have fixed vocabulary → poor translation of rare/unknown words models are trained on parallel data; how do we use monolingual data? recent solutions: subword models allow translation of rare/unknown words [Sennrich et al., 2016b] train on back-translated monolingual data [Sennrich et al., 2016a] Rico Sennrich Neural Machine Translation 6 / 15

  10. Problem with Word-level Models they charge a carry-on bag fee . sie erheben eine Hand|gepäck|gebühr . Neural MT architectures have small and fixed vocabulary translation is an open-vocabulary problem productive word formation (example: compounding) names (may require transliteration) Rico Sennrich Neural Machine Translation 7 / 15

  11. Why Subword Models? transparent translations many translations are semantically/phonologically transparent → translation via subword units possible morphologically complex words (e.g. compounds): solar system (English) Sonnen|system (German) Nap|rendszer (Hungarian) named entities: Barack Obama (English; German) ➪àðàê ❰áàìà (Russian) バラク ・ オバマ (ba-ra-ku o-ba-ma) (Japanese) cognates and loanwords: claustrophobia (English) Klaustrophobie (German) ✃ëàóñòðîôîáèÿ (Russian) Rico Sennrich Neural Machine Translation 8 / 15

  12. Examples system sentence source health research institutes reference Gesundheitsforschungsinstitute word-level Forschungsinstitute character bigrams Fo|rs|ch|un|gs|in|st|it|ut|io|ne|n joint BPE Gesundheits|forsch|ungsin|stitute source rakfisk reference ðàêôèñêà (rakfiska) word-level rakfisk → UNK → rakfisk character bigrams ra|kf|is|k → ðà⑤êô⑤èñ⑤ê (ra|kf|is|k) joint BPE rak|f|isk → ðàê⑤ô⑤èñêà (rak|f|iska) Rico Sennrich Neural Machine Translation 9 / 15

  13. Monolingual Training Data why monolingual data for phrase-based SMT? relax independence assumptions ✓ more training data ✓ more appropriate training data (domain adaptation) ✓ why monolingual data for neural MT? relax independence assumptions ✗ more training data ✓ more appropriate training data (domain adaptation) ✓ Rico Sennrich Neural Machine Translation 10 / 15

  14. Monolingual Data in NMT solutions previous work: combine NMT with separately trained LM [Gülçehre et al., 2015] our idea: decoder is already a language model → train encoder-decoder with added monolingual data monolingual training instances how do we get approximation of source context? dummy source context (moderately effective) automatically back-translate monolingual data into source language Rico Sennrich Neural Machine Translation 11 / 15

  15. Results: WMT 15 English → German system B LEU syntax-based 24.4 Neural MT baseline 22.0 +subwords 22.8 +back-translated data 25.7 +ensemble of 4 26.5 Rico Sennrich Neural Machine Translation 12 / 15

  16. WMT16 Results (B LEU ) uedin-nmt 34.2 metamind 32.3 uedin-nmt 26.0 NYU-UMontreal 30.8 uedin-nmt 31.4 amu-uedin 25.3 cambridge 30.6 jhu-pbmt 30.4 jhu-pbmt 24.0 uedin-syntax 30.6 PJATK 28.3 LIMSI 23.6 KIT/LIMSI 29.1 cu-mergedtrees 13.3 AFRL-MITLL 23.5 KIT 29.0 CS → EN NYU-UMontreal 23.1 uedin-pbmt 28.4 AFRL-MITLL-verb-annot 20.9 jhu-syntax 26.6 EN → RU uedin-pbmt 35.2 EN → DE uedin-nmt 33.9 uedin-syntax 33.6 amu-uedin 29.1 uedin-nmt 38.6 jhu-pbmt 32.2 NRC 29.1 uedin-pbmt 35.1 LIMSI 31.0 uedin-nmt 28.0 jhu-pbmt 34.5 RO → EN AFRL-MITLL 27.6 uedin-syntax 34.4 AFRL-MITLL-contrast 27.0 KIT 33.9 RU → EN QT21-HimL-SysComb 28.9 jhu-syntax 31.0 uedin-nmt 28.1 DE → EN RWTH-SYSCOMB 27.1 uedin-pbmt 26.8 uedin-nmt 25.8 uedin-lmu-hiero 25.9 NYU-UMontreal 23.6 KIT 25.8 jhu-pbmt 23.6 lmu-cuni 24.3 cu-chimera 21.0 LIMSI 23.9 uedin-cu-syntax 20.9 jhu-pbmt 23.5 cu-tamchyna 20.8 usfd-rescoring 23.1 cu-TectoMT 14.7 EN → RO cu-mergedtrees 8.2 EN → CS Rico Sennrich Neural Machine Translation 13 / 15

  17. WMT16 Results (B LEU ) uedin-nmt 34.2 metamind 32.3 uedin-nmt 26.0 NYU-UMontreal 30.8 uedin-nmt 31.4 amu-uedin 25.3 cambridge 30.6 jhu-pbmt 30.4 jhu-pbmt 24.0 uedin-syntax 30.6 PJATK 28.3 LIMSI 23.6 KIT/LIMSI 29.1 cu-mergedtrees 13.3 AFRL-MITLL 23.5 KIT 29.0 CS → EN NYU-UMontreal 23.1 uedin-pbmt 28.4 AFRL-MITLL-verb-annot 20.9 jhu-syntax 26.6 EN → RU uedin-pbmt 35.2 EN → DE uedin-nmt 33.9 uedin-syntax 33.6 amu-uedin 29.1 uedin-nmt 38.6 jhu-pbmt 32.2 NRC 29.1 uedin-pbmt 35.1 LIMSI 31.0 uedin-nmt 28.0 jhu-pbmt 34.5 RO → EN AFRL-MITLL 27.6 uedin-syntax 34.4 AFRL-MITLL-contrast 27.0 KIT 33.9 RU → EN QT21-HimL-SysComb 28.9 jhu-syntax 31.0 uedin-nmt 28.1 DE → EN RWTH-SYSCOMB 27.1 uedin-pbmt 26.8 Edinburgh NMT uedin-nmt 25.8 uedin-lmu-hiero 25.9 NYU-UMontreal 23.6 KIT 25.8 jhu-pbmt 23.6 lmu-cuni 24.3 cu-chimera 21.0 LIMSI 23.9 uedin-cu-syntax 20.9 jhu-pbmt 23.5 cu-tamchyna 20.8 usfd-rescoring 23.1 cu-TectoMT 14.7 EN → RO cu-mergedtrees 8.2 EN → CS Rico Sennrich Neural Machine Translation 13 / 15

  18. WMT16 Results (B LEU ) uedin-nmt 34.2 metamind 32.3 uedin-nmt 26.0 NYU-UMontreal 30.8 uedin-nmt 31.4 amu-uedin 25.3 cambridge 30.6 jhu-pbmt 30.4 jhu-pbmt 24.0 uedin-syntax 30.6 PJATK 28.3 LIMSI 23.6 KIT/LIMSI 29.1 cu-mergedtrees 13.3 AFRL-MITLL 23.5 KIT 29.0 CS → EN NYU-UMontreal 23.1 uedin-pbmt 28.4 AFRL-MITLL-verb-annot 20.9 jhu-syntax 26.6 EN → RU uedin-pbmt 35.2 EN → DE uedin-nmt 33.9 uedin-syntax 33.6 amu-uedin 29.1 uedin-nmt 38.6 jhu-pbmt 32.2 NRC 29.1 uedin-pbmt 35.1 LIMSI 31.0 uedin-nmt 28.0 jhu-pbmt 34.5 RO → EN AFRL-MITLL 27.6 uedin-syntax 34.4 AFRL-MITLL-contrast 27.0 KIT 33.9 RU → EN QT21-HimL-SysComb 28.9 jhu-syntax 31.0 uedin-nmt 28.1 DE → EN RWTH-SYSCOMB 27.1 uedin-pbmt 26.8 Edinburgh NMT uedin-nmt 25.8 uedin-lmu-hiero 25.9 NYU-UMontreal 23.6 KIT 25.8 jhu-pbmt 23.6 lmu-cuni 24.3 System cu-chimera 21.0 LIMSI 23.9 uedin-cu-syntax 20.9 jhu-pbmt 23.5 Combination with cu-tamchyna 20.8 usfd-rescoring 23.1 Edinburgh NMT cu-TectoMT 14.7 EN → RO cu-mergedtrees 8.2 EN → CS Rico Sennrich Neural Machine Translation 13 / 15

  19. Neural MT and Phrase-based SMT Neural MT Phrase-based SMT translation quality ✓ model size ✓ training time ✓ model interpretability ✓ decoding efficiency ✓ ✓ ✓ ✓ toolkits (for simplicity) (for maturity) special hardware requirement GPU lots of RAM Rico Sennrich Neural Machine Translation 14 / 15

  20. Conclusions and Outlook conclusions neural MT is SOTA on many tasks subword models and back-translated data contributed to success future predictions performance lead over phrase-based SMT will increase industry adoption will happen, but beware: some hard things are suddenly easy (incremental training) some easy things are suddenly hard (manual changes to model) exciting research opportunities relax independence assumptions: document-level translation, multimodal input, ... share parts of network between tasks: universal translation models, multi-task models, ... Rico Sennrich Neural Machine Translation 15 / 15

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Neural Machine Translation Gongbo Tang 8 October 2018 Outline Neural Machine Translation 1

Neural Machine Translation Gongbo Tang 8 October 2018 Outline Neural Machine Translation 1

Neural Machine Translation Gongbo Tang 8 October 2018 Outline Neural Machine Translation 1 Advances and Challenges 2 Gongbo Tang Neural Machine Translation 2/52 Neural Machine Translation Figure Recurrent neural network based NMT

906 views • 73 slides
Neural Machine Translation Philipp Koehn 6 October 2020 Philipp Koehn Machine Translation:

Neural Machine Translation Philipp Koehn 6 October 2020 Philipp Koehn Machine Translation:

Neural Machine Translation Philipp Koehn 6 October 2020 Philipp Koehn Machine Translation: Neural Machine Translation 6 October 2020 Language Models 1 Modeling variants feed-forward neural network recurrent neural network long

485 views • 36 slides
Convolutional over Recurrent Encoder for Neural Machine Translation Praveen Dakwale and Christof

Convolutional over Recurrent Encoder for Neural Machine Translation Praveen Dakwale and Christof

Annual Conference of the European Association for Machine Translation 2017 Convolutional over Recurrent Encoder for Neural Machine Translation Praveen Dakwale and Christof Monz Neural Machine Translation End to end neural network with RNN

354 views • 20 slides
Machine Translation 12: (Non-neural) Statistical Machine Translation Rico Sennrich University of

Machine Translation 12: (Non-neural) Statistical Machine Translation Rico Sennrich University of

Machine Translation 12: (Non-neural) Statistical Machine Translation Rico Sennrich University of Edinburgh R. Sennrich MT 2018 12 1 / 27 Todays Lecture So far, main focus of lecture was on: neural machine translation research

740 views • 46 slides
Adaptive Multi-pass Decoder for Neural Machine Translation EMNLP 2018

Adaptive Multi-pass Decoder for Neural Machine Translation EMNLP 2018

Adaptive Multi-pass Decoder for Neural Machine Translation EMNLP 2018 http://aclweb.org/anthology/D18-1048 Neural Machine Translation(NMT) The encoder-decoder are widely used in neural machine translation the encoder transforms the source

121 views • 11 slides
Semi-supervised Learning for Neural Machine Translation Yong Cheng joint work with Wei Xu,

Semi-supervised Learning for Neural Machine Translation Yong Cheng joint work with Wei Xu,

Semi-supervised Learning for Neural Machine Translation Yong Cheng joint work with Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, Yang Liu 1 Machine Translation Automated translation using computer software 2 Machine Translation Rule-based

650 views • 26 slides
Measuring Immediate Adaptation Performance for Neural Machine Translation Patrick Simianer , Joern

Measuring Immediate Adaptation Performance for Neural Machine Translation Patrick Simianer , Joern

Measuring Immediate Adaptation Performance for Neural Machine Translation Patrick Simianer , Joern Wuebker, John DeNero Lilt NAACL 19 Outline Motivation & Approach 1 2 Evaluation Conclusion 3 2 / 20 Motivation Online adaptation is

765 views • 52 slides
What can Statistical Machine Translation teach Neural Machine Translation about Structured

What can Statistical Machine Translation teach Neural Machine Translation about Structured

What can Statistical Machine Translation teach Neural Machine Translation about Structured Prediction? Graham Neubig @ ICLR Workshop on Deep Reinforcement Learning Meets Structured Prediction 5/6/2019 Types of Prediction Types of

1.73k views • 127 slides
Googles Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation

Googles Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation

Googles Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation Author: Melvin Johnson , Mike Schuster , Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Vigas, Martin Wattenberg, Greg

344 views • 16 slides
Introduction to Neural Machine Translation Gongbo Tang 16 September 2019 Outline Why Neural

Introduction to Neural Machine Translation Gongbo Tang 16 September 2019 Outline Why Neural

Introduction to Neural Machine Translation Gongbo Tang 16 September 2019 Outline Why Neural Machine Translation ? 1 Introduction to Neural Networks 2 Neural Language Models 3 Gongbo Tang Introduction to Neural Machine Translation 2/38

686 views • 50 slides
BREAKTHROUGHS IN NEURAL MACHINE TRANSLATION Olof Mogren Chalmers University of Technology

BREAKTHROUGHS IN NEURAL MACHINE TRANSLATION Olof Mogren Chalmers University of Technology

BREAKTHROUGHS IN NEURAL MACHINE TRANSLATION Olof Mogren Chalmers University of Technology 2016-09-29 COMING SEMINARS T oday: Olof Mogren Neural Machine Translation October 6: John Wiedenhoeft Fast Bayesian inference in Hidden Markov

607 views • 22 slides
Neural Machine Translation: directions for improvement CMSC 470 Marine Carpuat How can we

Neural Machine Translation: directions for improvement CMSC 470 Marine Carpuat How can we

Neural Machine Translation: directions for improvement CMSC 470 Marine Carpuat How can we improve on state-of-the-art machine translation approaches? Model Training Data Objective Algorithm Addressing domain mismatch

490 views • 36 slides
Sparse and Constrained Attention for Neural Machine Translation Chaitanya Malaviya 1 ,

Sparse and Constrained Attention for Neural Machine Translation Chaitanya Malaviya 1 ,

Sparse and Constrained Attention for Neural Machine Translation Chaitanya Malaviya 1 , Pedro Ferreira 2 , Andr F.T. Martins 2,3 1 Carnegie Mellon University, 2 Instituto Superior Tcnico, 3 Unbabel 1 Adequacy in Neural Machine

574 views • 22 slides
An Introduction to Neural Machine Translation Prof. John D. Kelleher @johndkelleher ADAPT Centre

An Introduction to Neural Machine Translation Prof. John D. Kelleher @johndkelleher ADAPT Centre

An Introduction to Neural Machine Translation Prof. John D. Kelleher @johndkelleher ADAPT Centre for Digital Content Technology Dublin Institute of Technology, Ireland June 25, 2018 1 / 57 Outline The Neural Machine Translation Revolution

610 views • 57 slides
Neural machine translation with less supervision CMSC 470 Marine Carpuat Neural MT only helps

Neural machine translation with less supervision CMSC 470 Marine Carpuat Neural MT only helps

Neural machine translation with less supervision CMSC 470 Marine Carpuat Neural MT only helps in high-resource settings Ongoing research Learn from other sources of supervision than pairs (E,F) Monolingual text Multiple languages

479 views • 29 slides
Neural Machine Translation Decoding Philipp Koehn 8 October 2020 Philipp Koehn Machine

Neural Machine Translation Decoding Philipp Koehn 8 October 2020 Philipp Koehn Machine

Neural Machine Translation Decoding Philipp Koehn 8 October 2020 Philipp Koehn Machine Translation: Neural Machine Translation Decoding 8 October 2020 Inference 1 Given a trained model ... we now want to translate test sentences We

682 views • 39 slides
Translation , 1 2,1 2 3,1 3 0,1 1,1

Translation , 1 2,1 2 3,1 3 0,1 1,1

0 1 2 3 4 5 6 7 3,3 3 3 2,3 2 3 1,3 3 0,3 3 3,2 3 2 2,2 2 2 1,2 2 0,2 2 3,1 3 2,1 2 1,2

344 views • 16 slides
Empirical Methods in Natural Language Processing Lecture 14 Machine translation (I): Introduction

Empirical Methods in Natural Language Processing Lecture 14 Machine translation (I): Introduction

Empirical Methods in Natural Language Processing Lecture 14 Machine translation (I): Introduction Philipp Koehn 21 February 2008 Philipp Koehn EMNLP Lecture 14 21 February 2008 1 Machine translation Task: make sense of foreign text like

511 views • 28 slides
Machine Translation 2: Statistical MT: Phrase-Based and Neural Ond rej Bojar

Machine Translation 2: Statistical MT: Phrase-Based and Neural Ond rej Bojar

Machine Translation 2: Statistical MT: Phrase-Based and Neural Ond rej Bojar bojar@ufal.mff.cuni.cz Institute of Formal and Applied Linguistics Faculty of Mathematics and Physics Charles University, Prague December 2017 MT2: PBMT, NMT

1.14k views • 72 slides
2D Geometric Transformations Question : How do we represent a geometric object in the plane?

2D Geometric Transformations Question : How do we represent a geometric object in the plane?

2D Geometric Transformations (Chapter 5 in FVD) 1 2D Geometric Transformations Question : How do we represent a geometric object in the plane? Answer : For now, assume that objects consist of points and lines. A point is represented

353 views • 13 slides
Surprise Language Evaluation: Rapid-Response Cross-Language IR Maryland: Douglas W. Oard, Marine

Surprise Language Evaluation: Rapid-Response Cross-Language IR Maryland: Douglas W. Oard, Marine

Surprise Language Evaluation: Rapid-Response Cross-Language IR Maryland: Douglas W. Oard, Marine Carpuat, Petra Galuscakova, Joseph Barrow, Suraj Nair, Xing Niu, Han-Chin Shing, Weijia Xu, Elena Zotkina Columbia: Kathleen McKeown, Smaranda

485 views • 25 slides
A first step towards interactivity and language tools convergence Arnaud Vi 1 Luis Villarejo

A first step towards interactivity and language tools convergence Arnaud Vi 1 Luis Villarejo

A first step towards interactivity and language tools convergence Arnaud Vi 1 Luis Villarejo Muoz 2 Mireia Farrs Cabeceran 2 Jimmy ORegan 3 1 Informations Systems Engineering Grenoble INP - Ensimag arnaud.vie@ensimag.fr 2 Learning

760 views • 31 slides
Machine Translation Prof. Sameer Singh CS 295: STATISTICAL NLP WINTER 2017 February 28, 2017

Machine Translation Prof. Sameer Singh CS 295: STATISTICAL NLP WINTER 2017 February 28, 2017

Machine Translation Prof. Sameer Singh CS 295: STATISTICAL NLP WINTER 2017 February 28, 2017 Based on slides from Jason Eisenstein, Chris Dyer, Alan Ritter, Yejin Choi, and everyone else they copied from. Upcoming Paper summaries:

995 views • 46 slides
Experiments in Term Translation Mihael Arcan DERI, NUI Galway Supervised by Dr. Paul Buitelaar

Experiments in Term Translation Mihael Arcan DERI, NUI Galway Supervised by Dr. Paul Buitelaar

Experiments in Term Translation Mihael Arcan DERI, NUI Galway Supervised by Dr. Paul Buitelaar Monnet is supported by the European Union under Grant No. 248458 Motivation Generation of multilingual ontologies most of the

1.14k views • 53 slides

More recommend