SLIDE 1
Semi-supervised Learning for Neural Machine Translation
Yong Cheng
joint work with Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, Yang Liu
1
Machine Translation
2
Semi-supervised Learning for Neural Machine Translation Yong Cheng - - PowerPoint PPT Presentation
Semi-supervised Learning for Neural Machine Translation Yong Cheng - - PowerPoint PPT Presentation
Semi-supervised Learning for Neural Machine Translation Yong Cheng joint work with Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, Yang Liu 1 Machine Translation Automated translation using computer software 2 Machine Translation Rule-based
SLIDE 2
SLIDE 3
Machine Translation
3
Rule-based Machine Translation 1970s Example-based Machine Translation 1984 Statical Machine Translation (SMT) 1993 Neural Machine Translation NMT 2014 Trends: learning to translate from DATA
SLIDE 4
Machine Translation
4
Parallel Corpora Monolingual Corpora Parallel corpora are usually limited in
quantity quality coverage
& &
SLIDE 5
Monolingual Corpora Used in SMT and NMT
N-gram language model in SMT Koehn et al., [2007] Monolingual corpora as decipherment Ravi and Knight [2011] Integrate a neural language model into NMT. Gulccehre et al. [2015] Additional pseudo parallel corpus. Sennrich et al. [2016]
5
SLIDE 6
Supervised Training
Parallel Corpus Objective
6
SLIDE 7
Unsupervised Training
Monolingual Corpus
7
SLIDE 8
cc
Our Approach — Autoencoders
8
bushi yu shalong juxing le huitan
x
SLIDE 9
cc
Our Approach — Autoencoders
9
bushi yu shalong juxing le huitan
x
P(y | x; ! θ )
SLIDE 10
cc
Our Approach — Autoencoders
10
bushi yu shalong juxing le huitan
x
Bush held a talk with sharon
y
P(y | x; ! θ )
latent
SLIDE 11
cc
Our Approach — Autoencoders
11
bushi yu shalong juxing le huitan
x
Bush held a talk with sharon
y
P(y | x; ! θ ) P(x | y; ! θ )
latent
SLIDE 12
cc
Our Approach — Autoencoders
12
bushi yu shalong juxing le huitan
x
Bush held a talk with sharon
′ x
bushi yu shalong juxing le huitan
y
P(y | x; ! θ ) P(x | y; ! θ )
latent
SLIDE 13
cc
Our Approach — Autoencoders
13
source autoencoder target autoencoder
SLIDE 14
Unsupervised Training (Autoencoders)
Monolingual Corpus
14
target autoencoder
SLIDE 15
Semi-supervised Training
15
Training Objective
SLIDE 16
Translation Results
Compared with Moses (SMT) and RNNSearch (NMT)
16
SLIDE 17
Translation Results
Compared with Moses (SMT) and RNNSearch (NMT)
17
SLIDE 18
Translation Results
Compared with Moses (SMT) and RNNSearch (NMT)
18
SLIDE 19
Translation Results
Compared with Moses (SMT) and RNNSearch (NMT)
19
SLIDE 20
Translation Results
Compared with Moses (SMT) and RNNSearch (NMT)
20
SLIDE 21
Translation Results
Compared with Sennrich et al. [2015a]
21
SLIDE 22
Example Translation of Monolingual Corpus
22
SLIDE 23
Conclusion
Monolingual corpora is an important resource for neural machine translation. We have proposed a semi-supervised approach to training bidirectional neural machine translation models for exploiting monolingual corpora. As our method is sensitive to the OOVs present in monolingual corpora, we plan to integrate Jean et
- al. (2015)’s technique on using very large
vocabulary into our approach.
23
SLIDE 24
Thank You !
24
SLIDE 25
Effect of Sample Size
ZH-EN EN-ZH
25
SLIDE 26
Effect of OOV ratio
ZH-EN EN-ZH
26