Semi-supervised Learning for Neural Machine Translation Yong Cheng - PowerPoint PPT Presentation

Semi-supervised Learning for Neural Machine Translation Yong Cheng joint work with Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, Yang Liu 1

Machine Translation Automated translation using computer software 2

Machine Translation Rule-based Machine Translation 1970s Example-based Machine Translation 1984 Statical Machine Translation (SMT) 1993 Neural Machine Translation � NMT � 2014 Trends: learning to translate from DATA 3

Machine Translation Parallel corpora are usually limited in & & quantity quality coverage Monolingual Corpora Parallel Corpora 4

Monolingual Corpora Used in SMT and NMT N-gram language model in SMT Koehn et al., [2007] Monolingual corpora as decipherment Ravi and Knight [2011] Integrate a neural language model into NMT. Gulccehre et al. [2015] Additional pseudo parallel corpus. Sennrich et al. [2016] 5

Supervised Training Parallel Corpus Objective 6

Unsupervised Training Monolingual Corpus 7

cc Our Approach — Autoencoders x bushi yu shalong juxing le huitan 8

cc Our Approach — Autoencoders ! θ ) P ( y | x ; x bushi yu shalong juxing le huitan 9

cc Our Approach — Autoencoders y latent Bush held a talk with sharon ! θ ) P ( y | x ; x bushi yu shalong juxing le huitan 10

cc Our Approach — Autoencoders ! θ ) P ( x | y ; y latent Bush held a talk with sharon ! θ ) P ( y | x ; x bushi yu shalong juxing le huitan 11

cc Our Approach — Autoencoders ′ x bushi yu shalong juxing le huitan ! θ ) P ( x | y ; y latent Bush held a talk with sharon ! θ ) P ( y | x ; x bushi yu shalong juxing le huitan 12

cc Our Approach — Autoencoders source autoencoder target autoencoder 13

Unsupervised Training (Autoencoders) Monolingual Corpus target autoencoder 14

Semi-supervised Training Training Objective 15

Translation Results Compared with Moses (SMT) and RNNSearch (NMT) 16

Translation Results Compared with Sennrich et al. [2015a] 21

Example Translation of Monolingual Corpus 22

Conclusion Monolingual corpora is an important resource for neural machine translation. We have proposed a semi-supervised approach to training bidirectional neural machine translation models for exploiting monolingual corpora. As our method is sensitive to the OOVs present in monolingual corpora, we plan to integrate Jean et al. (2015)’s technique on using very large vocabulary into our approach. 23

Thank You ! 24

Effect of Sample Size ZH-EN EN-ZH 25

Effect of OOV ratio ZH-EN EN-ZH 26

Semi-supervised Learning for Neural Machine Translation Yong Cheng - PowerPoint PPT Presentation

Semi-supervised Learning for Neural Machine Translation Yong Cheng joint work with Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, Yang Liu 1 Machine Translation Automated translation using computer software 2 Machine Translation Rule-based

Semi-Supervised Learning Maria-Florina Balcan 03/30/2015 Readings: Semi-Supervised Learning.

Unsupervised and Semi-supervised Learning of Structure Graham Neubig Site

Unsupervised and Semi-supervised Learning of Structure Graham Neubig Site

Semi-Supervised Learning and Text Analysis Machine Learning 10-701 November 29, 2005 Tom M.

NEURAL RENDERING MODEL (NRM): JOINT GENERATION AND PREDICTION FOR SEMI-SUPERVISED LEARNING Tan

Neural Machine Translation Gongbo Tang 8 October 2018 Outline Neural Machine Translation 1

Fundamentals of Machine Learning for Neural Machine Translation Dr. John D. Kelleher ADAPT

What can Statistical Machine Translation teach Neural Machine Translation about Structured

IN5550: Neural Methods in Natural Language Processing Lecture 2 Supervised Machine Learning:

IN5550: Neural Methods in Natural Language Processing Lecture 2 Supervised Machine Learning:

Neural Machine Translation Philipp Koehn 6 October 2020 Philipp Koehn Machine Translation:

Convolutional over Recurrent Encoder for Neural Machine Translation Praveen Dakwale and Christof

Machine Translation 12: (Non-neural) Statistical Machine Translation Rico Sennrich University of

Adaptive Multi-pass Decoder for Neural Machine Translation EMNLP 2018

Tangent-Normal Adversarial Regularization for Semi-Supervised Learning Bing Yu , Jingfeng Wu

CPSC 340: Machine Learning and Data Mining Non-Parametric Models Summer 2020 Course Map

4CSLL5 Advanced Computational Linguistics Introduction Phrase Based Machine Trans Martin

Syntax-Directed Translation for Top-Down Parsing 1 Midterm next week during class online

Leveling for Non-Volatile Main Memories Haikun Liu , Yuanyuan Ye, Xiaofei Liao, Hai Jin, Yu Zhang,

The HIT-LTRC Machine Translation System for IWSLT 2012 Xiaoning Zhu, Yiming Cui, Conghui Zhu,

Neural Machine Translation II Refinements Philipp Koehn 17 October 2017 Philipp Koehn Machine

23 Advanced Topics 5: Multi-lingual Models Up until now, we have assumed that in the case of

Evaluating MT Quality Evaluation of Why do we want to do it? Translation Quality - Want to

ANSI TAG 37 Committee F43 Language Services and Products Interagency Language Roundtable