Cross linguality and machine translation without bilingual data ith - - PowerPoint PPT Presentation
Cross linguality and machine translation without bilingual data ith t bili l d t Enek Eneko Agirr girre @ @eagirre i Joint work with Mikel Artetxe Gorka Labaka Joint work with: Mikel Artetxe, Gorka Labaka IXA NLP group University
IXA NLP group – University of the Basque Country (UPV/EHU) IXA NLP group – University of the Basque Country (UPV/EHU)
Cr Cross
lingual wo word rd repr presen esentations: tions: W d W d b ddi ddi k f N l L P i
embeddi ddings ngs key ey for Natural Language Processing
pped embeddings embeddings represent languages in a single space
bilingual dictionaries dictionaries
bilingual dictionaries dictionaries
Exciting citing re results in dictionary induction, transfer learning, crosslingual applications, interlingual semantic representations 2
Cr Cross
lingual wo word rd repr presen esentations: tions: W d W d b ddi ddi k f N l L P i
embeddi ddings ngs key ey for Natural Language Processing
pped embeddings embeddings represent languages in a single space
bilingual dictionaries dictionaries
bilingual dictionaries dictionaries
Exciting citing re results in dictionary induction, transfer learning, crosslingual applications, interlingual semantic representations Our focus: ex extend mappings mappings to to an any pair pair of
languages
very fe few bilingual bilingual re resources
wide de adop adoption tion of NLP tools 3
Cr Cross
lingual wo word rd repr presen esentations: tions: W d W d b ddi ddi k f N l L P i
embeddi ddings ngs key ey for Natural Language Processing
pped embeddings embeddings represent languages in a single space
bilingual dictionaries dictionaries
bilingual dictionaries dictionaries
Exciting citing re results in dictionary induction, transfer learning, crosslingual applications, interlingual semantic representations Our focus: ex extend mappings mappings to to an any pair pair of
languages
very fe few bilingual bilingual re resources
wide de adop adoption tion of NLP tools In particular: no no bilingual bilingual re resources at at all all
Unsuper ervised vised embedding mappings 4 p g pp g
Unsupervised ised neural machine translation
Ar Arab abic ic monolin nolingual al co corpora Chin Chinese ese monolin nolingu gual al co corpora
Ar Arabic abic embed embeddings ings Chinese Chinese embed embeddings ings
5
Ar Arab abic ic monolin nolingual al co corpora Chin Chinese ese monolin nolingu gual al co corpora
Ar Arabic abic embed embeddings ings Chinese Chinese embed embeddings ings Bilin ilingual embed embeddings ings
6
Ar Arab abic ic monolin nolingual al co corpora Chin Chinese ese monolin nolingu gual al co corpora
Ar Arabic abic embed embeddings ings Chinese Chinese embed embeddings ings Bilin ilingual embed embeddings ings
7
Ar Arab abic ic monolin nolingual al co corpora Chin Chinese ese monolin nolingu gual al co corpora
Ar Arabic abic embed embeddings ings Chinese Chinese embed embeddings ings Bilin ilingual embed embeddings ings
8
lf l f ll d ( )
10
lf l f ll d ( )
11
12
S i S i Seman emantic space space
‐ Words ‐ Meaningful distances
Donostia Bilbo Maule
Baiona
g ‐ Meaningful relations ‐ 300 dimensions ‐ Neural networks / linear algebra
Gasteiz Iruñea
from co‐occurrence counts
Apple House Pear
Geogr Geographi phical space space
C Banana Calendar
‐ Cities ‐ Meaningful distances ‐ Meaningful relations
Moo Bark Dog cat Cow
34
Meaningful relations ‐ 2 dimensions ‐ Cartographers from 3D world
Meow
43
Ba Basque que En English glish
44
Seed Seed dictionar dictionary
Ba Basque que En English glish
45
Seed Seed dictionar dictionary
Ba Basque que En English glish
Txakur Sagar Dog Apple
46
⋮ Egutegi ⋮ Calendar
Seed Seed dictionar dictionary
Ba Basque que En English glish
Txakur Sagar Dog Apple
47
⋮ Egutegi ⋮ Calendar
Seed Seed dictionar dictionary
Ba Basque que En English glish
Txakur Sagar Dog Apple
48
⋮ Egutegi ⋮ Calendar
Seed Seed dictionar dictionary
Ba Basque que En English glish
Txakur Sagar Dog Apple
49
⋮ Egutegi ⋮ Calendar
Seed Seed dictionar dictionary
Ba Basque que En English glish
Txakur Sagar Dog Apple
50
⋮ Egutegi ⋮ Calendar
Mikolov et al (2013b) Ba Basque que En English glish Mikolov et al. (2013b)
Txakur Sagar Dog Apple
51
⋮ Egutegi ⋮ Calendar
Mikolov et al (2013b) Ba Basque que En English glish Mikolov et al. (2013b)
Txakur Sagar Dog Apple
52
⋮ Egutegi ⋮ Calendar
Mikolov et al (2013b) Ba Basque que En English glish Mikolov et al. (2013b)
Txakur Sagar Dog Apple
53
⋮ Egutegi ⋮ Calendar
Mikolov et al (2013b) Ba Basque que En English glish Mikolov et al. (2013b)
Txakur Sagar Dog Apple
54
⋮ Egutegi ⋮ Calendar
( ) P (opt.) Pre‐process 1. (opt.) Whitening h l h l 2. 2. Or Orthogona
mapping 3. (opt.) Re‐weighting ( ) 4. (opt.) De‐whitening
55
S0 (opt.) Pre‐processing: length normalization, mean centering
Two sequences of (optional) linear transformations: S0 (opt.) Pre‐processing: length normalization, mean centering
Two sequences of (optional) linear transformations: S0 (opt.) Pre‐processing: length normalization, mean centering S1 (opt ) Whitening : turn covariance S1 (opt.) Whitening : turn covariance matrices into the identity matrix
58
Two sequences of (optional) linear transformations: S0 (opt.) Pre‐processing: length normalization, mean centering S1 (opt ) Whitening : turn covariance S1 (opt.) Whitening : turn covariance matrices into the identity matrix S2 S2 Ort Orthog
mapping: map into a S2 S2 Ort Orthog
mapping: map into a shared space (Procrustes)
Two sequences of (optional) linear transformations: S0 (opt.) Pre‐processing: length normalization, mean centering S1 (opt ) Whitening : turn covariance S1 (opt.) Whitening : turn covariance matrices into the identity matrix S2 S2 Ort Orthog
mapping: map into a S2 S2 Ort Orthog
mapping: map into a shared space (Procrustes) S3 (opt ) Re eight each component S3 (opt.) Re‐weight each component according to its cross‐correlation
Two sequences of (optional) linear transformations: S0 (opt.) Pre‐processing: length normalization, mean centering S1 (opt ) Whitening : turn covariance S1 (opt.) Whitening : turn covariance matrices into the identity matrix S2 S2 Ort Orthog
mapping: map into a S2 S2 Ort Orthog
mapping: map into a shared space (Procrustes) S3 (opt ) Re eight each component S3 (opt.) Re‐weight each component according to its cross‐correlation S4 ( t ) D hit i t i i l S4 (opt.) De‐whitening: restore original variance in every direction
61
Two sequences of (optional) linear transformations: S0 (opt.) Pre‐processing: length normalization, mean centering S1 (opt ) Whitening : turn covariance S1 (opt.) Whitening : turn covariance matrices into the identity matrix S2 S2 Ort Orthog
mapping: map into a S2 S2 Ort Orthog
mapping: map into a shared space (Procrustes) S3 (opt ) Re eight each component S3 (opt.) Re‐weight each component according to its cross‐correlation S4 ( t ) D hit i t i i l S4 (opt.) De‐whitening: restore original variance in every direction S5 ( ) Di i li d i k h fi l S5 (opt) Dimensionality reduction: keep the first n components only
62
S0 S0 (l) (l) S0 (m) S1 S 1 S2 S 2 S3 S4 (sr (src) S4 (tr (trg) S5 OLS Mikolov et al. (2013) x x src trg trg Shigeto et al. (2015) x x trg src src CCA Faruqui and Dyer (2014) x x x x x q y ( ) Orth. Xing et al. (2015) x x Artetxe et al. (2016) x x x h l ( ) Zhang et al. (2016) x Smith et al. (2017) x x x
70
S0 S0 (l) (l) S0 (m) S1 S 1 S2 S 2 S3 S4 (sr (src) S4 (tr (trg) S5 OLS Mikolov et al. (2013) x x src trg trg Shigeto et al. (2015) x x trg src src CCA Faruqui and Dyer (2014) x x x x x q y ( ) Orth. Xing et al. (2015) x x Artetxe et al. (2016) x x x h l ( ) Zhang et al. (2016) x Smith et al. (2017) x x x Our method (AAAI18) x x x x trg src trg x
71
Dataset by Dinu et al. (2015) extended to German, Finnish, Spanish
72
Dataset by Dinu et al. (2015) extended to German, Finnish, Spanish ⇒ Monolingual embeddings (CBOW + negative sampling)
73
Dataset by Dinu et al. (2015) extended to German, Finnish, Spanish ⇒ Monolingual embeddings (CBOW + negative sampling) ⇒ S d di ti 5 000 d i ⇒ Seed dictionary: 5,000 word pairs
74
Dataset by Dinu et al. (2015) extended to German, Finnish, Spanish ⇒ Monolingual embeddings (CBOW + negative sampling) ⇒ S d di ti 5 000 i ⇒ Seed dictionary: 5,000 pairs ⇒ Test dictionary: 1,500 pairs (Nearest neighbor, P@1)
75
Dataset by Dinu et al. (2015) extended to German, Finnish, Spanish ⇒ Monolingual embeddings (CBOW + negative sampling) ⇒ S d di ti 5 000 i
M t M th d EN EN IT IT EN EN DE DE EN EN FI FI EN EN ES ES
⇒ Seed dictionary: 5,000 pairs ⇒ Test dictionary: 1,500 pairs (Nearest neighbor, P@1)
Meth thod EN EN‐IT IT EN EN‐DE DE EN EN‐FI FI EN EN‐ES ES
76
Dataset by Dinu et al. (2015) extended to German, Finnish, Spanish ⇒ Monolingual embeddings (CBOW + negative sampling) ⇒ S d di ti 5 000 i
M t M th d EN EN IT IT EN EN DE DE EN EN FI FI EN EN ES ES
⇒ Seed dictionary: 5,000 pairs ⇒ Test dictionary: 1,500 pairs (Nearest neighbor, P@1)
Meth thod EN EN‐IT IT EN EN‐DE DE EN EN‐FI FI EN EN‐ES ES Mikolov et al. (2013) 34.93† 35.00† 25.91† 27.73† Faruqui and Dyer (2014) 38.40* 37.13* 27.60* 26.80* Shigeto et al. (2015) 41.53† 43.07† 31.04† 33.73† Dinu et al. (2015) 37.7 38.93* 29.14* 30.40* Lazaridou et al. (2015) 40.2 ‐ ‐ ‐ ( ) Xing et al. (2015) 36.87† 41.27† 28.23† 31.20† Artetxe et al. (2016) 39.27 41.87* 30.62* 31.40* Zhang et al (2016) 36 73† 40 80† 28 16† 31 07† Zhang et al. (2016) 36.73 40.80 28.16 31.07 Smith et al. (2017) 43.1 43.33† 29.42† 35.13†
78
Dataset by Dinu et al. (2015) extended to German, Finnish, Spanish ⇒ Monolingual embeddings (CBOW + negative sampling) ⇒ S d di ti 5 000 i ⇒ Seed dictionary: 5,000 pairs ⇒ Test dictionary: 1,500 pairs (Nearest neighbor, P@1)
M t M th d EN EN IT IT EN EN DE DE EN EN FI FI EN EN ES ES Meth thod EN EN‐IT IT EN EN‐DE DE EN EN‐FI FI EN EN‐ES ES Mikolov et al. (2013) 34.93† 35.00† 25.91† 27.73† Faruqui and Dyer (2014) 38.40* 37.13* 27.60* 26.80* Shigeto et al. (2015) 41.53† 43.07† 31.04† 33.73† Dinu et al. (2015) 37.7 38.93* 29.14* 30.40* Lazaridou et al. (2015) 40.2 ‐ ‐ ‐ ( ) Xing et al. (2015) 36.87† 41.27† 28.23† 31.20† Ar Artetx txe et al.
2016) 39.27 39.27 41.87 41.87* 30.62 30.62* 31.40 31.40* Zhang et al (2016) 36 73† 40 80† 28 16† 31 07† Zhang et al. (2016) 36.73 40.80 28.16 31.07 Smith et al. (2017) 43.1 43.33† 29.42† 35.13† Our method (AAAI18) 45.27 45.27 44.13 44.13 32.94 32.94 36.60 36.60
79
80
81
82
83
84
85
86
lf l f ll d ( )
87
88
89
Previous work
bilingual signal for training
91
for training
Previous work
bilingual signal for training
‐ parallel corpora ‐ comparable corpora 94
for training
comparable corpora ‐ (big) dictionaries
Previous work
bilingual signal for training
‐ parallel corpora ‐ comparable corpora 95
for training
comparable corpora ‐ (big) dictionaries
Our work Previous work
bilingual signal for training
‐ 25 word dictionary ‐ numerals (1, 2, 3…) ‐ parallel corpora ‐ comparable corpora 99
for training
numerals (1, 2, 3…) ‐ nothing comparable corpora ‐ (big) dictionaries
100
Monolingual embeddings
101
Monolingual embeddings Dictionary
102
Monolingual embeddings Dictionary
103
Monolingual embeddings Mapping Dictionary
104
Monolingual embeddings Mapping Dictionary
105
Monolingual embeddings Mapping Dictionary Dictionary
106
Monolingual embeddings Mapping Dictionary Dictionary
bet better! er! 107
Monolingual embeddings Mapping Dictionary Dictionary
bet better! er! 108
Monolingual embeddings Mapping Dictionary Dictionary
bet better! er!
M i Mapping
109
Monolingual embeddings Mapping Dictionary Dictionary
bet better! er!
M i Mapping
110
Monolingual embeddings Mapping Dictionary Dictionary
bet better! er!
M i Di ti Mapping Dictionary
111
Monolingual embeddings Mapping Dictionary Dictionary
bet better! er!
M i Di ti
eve even bet better! er!
Mapping Dictionary
eve even bet better! er! 112
Monolingual embeddings Mapping Dictionary Dictionary
bet better! er!
M i Di ti
eve even bet better! er!
Mapping Dictionary
eve even bet better! er! 113
Monolingual embeddings Mapping Dictionary Dictionary
bet better! er!
M i Di ti
eve even bet better! er!
Mapping Dictionary
eve even bet better! er! 114
Mapping
Monolingual embeddings Mapping Dictionary Dictionary
bet better! er!
M i Di ti
eve even bet better! er!
Mapping Dictionary
eve even bet better! er! 115
Mapping
Monolingual embeddings Mapping Dictionary Dictionary
bet better! er!
M i Di ti
eve even bet better! er!
Mapping Dictionary
eve even bet better! er! 116
Mapping Dictionary
Monolingual embeddings Mapping Dictionary Dictionary
bet better! er!
M i Di ti
eve even bet better! er!
Mapping Dictionary
eve even bet better! er! 117
Mapping Dictionary
eve even bet better! er!
Monolingual embeddings Mapping Dictionary Dictionary
118
Monolingual embeddings Mapping Dictionary Dictionary
proposed self‐learning method Too good to be true?
120
121
122
123
124
125
wo word rd tr tran ansla slation ion induction duction
138
Implicit objective: 139
Implicit objective: 140
Implicit objective: 141
Implicit objective: 142
Implicit objective: 143
Implicit objective: 144
Implicit objective: 145
Implicit objective: 146
Implicit objective: 147
Implicit objective: 148
Implicit objective: 149
Implicit objective: 150
Implicit objective: 151
Implicit objective: 152
Implicit objective: 153
Implicit objective: 154
Implicit objective: 155
Implicit objective: 156
Implicit objective:
Independent from seed dictionary!
157
Implicit objective: 158
Implicit objective:
Independent from seed dictionary! So why do we need a seed dictionary? Avoid poor local optima!
159
Avoid poor local optima!
Implicit objective: 160
161
162
Initial dictionary:
p g g y
would have analoguous similarity histograms (isometry hyp.)
163
Initial dictionary:
p g g y
would have analoguous similarity histograms (isometry hyp.)
164
Initial dictionary:
p g g y
would have analoguous similarity histograms (isometry hyp.)
165
Initial dictionary:
p g g y
would have analoguous similarity histograms (isometry hyp.)
166
Initial dictionary:
p g g y
would have analoguous similarity histograms (isometry hyp.) It works, but very weak: Accuracy 0.52%
167
Initial dictionary:
p g g y
would have analoguous similarity histograms (isometry hyp.) It works, but very weak: Accuracy 0.52% For self‐learning to work we had to add: g
2 Frequency‐based vocabulary cut‐off
nearest‐neighbour use CSLS (Lample et al. 2018) nearest neighbour use CSLS (Lample et al. 2018)
168
Super upervisi ision Method ethod EN‐IT IT EN EN‐DE DE EN EN‐FI FI EN EN‐ES ES
171
⇒ Monolingual embeddings (CBOW + negative sampling) ⇒ Monolingual embeddings (CBOW + negative sampling)
Super upervisi ision Method ethod EN‐IT IT EN EN‐DE DE EN EN‐FI FI EN EN‐ES ES
172
⇒ Monolingual embeddings (CBOW + negative sampling) ⇒ Monolingual embeddings (CBOW + negative sampling) ⇒ Seed dictionary: 5,000 word pairs / 25 word pairs / none
Super upervisi ision Method ethod EN‐IT IT EN EN‐DE DE EN EN‐FI FI EN EN‐ES ES 5k dict. 25 dict. None
173
⇒ Monolingual embeddings (CBOW + negative sampling) ⇒ Monolingual embeddings (CBOW + negative sampling) ⇒ Seed dictionary: 5,000 word pairs / 25 word pairs / none ⇒ Test dictionary: 1,500 word pairs
Super upervisi ision Method ethod EN‐IT IT EN EN‐DE DE EN EN‐FI FI EN EN‐ES ES 5k dict. 25 dict. None
174
⇒ Monolingual embeddings (CBOW + negative sampling) ⇒ Monolingual embeddings (CBOW + negative sampling) ⇒ Seed dictionary: 5,000 word pairs / 25 word pairs / none ⇒ Test dictionary: 1,500 word pairs
Super upervisi ision Method ethod EN‐IT IT EN EN‐DE DE EN EN‐FI FI EN EN‐ES ES Mikolov et al. (2013) 34.93† 35.00† 25.91† 27.73† A l (2016) 39 27 41 87* 30 62* 31 40* 5k dict. Artetxe et al. (2016) 39.27 41.87* 30.62* 31.40* Smith et al. (2017) 43.1 43.33† 29.42† 35.13† Our method (AAAI18) 45.27 45.27 44.13 44.13 32.94 32.94 36.60 36.60 25 dict. None
176
⇒ Monolingual embeddings (CBOW + negative sampling) ⇒ Monolingual embeddings (CBOW + negative sampling) ⇒ Seed dictionary: 5,000 word pairs / 25 word pairs / none ⇒ Test dictionary: 1,500 word pairs
Super upervisi ision Method ethod EN‐IT IT EN EN‐DE DE EN EN‐FI FI EN EN‐ES ES Mikolov et al. (2013) 34.93† 35.00† 25.91† 27.73† A l (2016) 39 39 27 27 41 41 87 87* 30 30 62 62* 31 40* 5k dict. Artetxe et al. (2016) 39 39.27 27 41 41.87 87* 30 30.62 62* 31.40* 25 dict. Our method (ACL17) None
177
⇒ Monolingual embeddings (CBOW + negative sampling) ⇒ Monolingual embeddings (CBOW + negative sampling) ⇒ Seed dictionary: 5,000 word pairs / 25 word pairs / none ⇒ Test dictionary: 1,500 word pairs
Super upervisi ision Method ethod EN‐IT IT EN EN‐DE DE EN EN‐FI FI EN EN‐ES ES Mikolov et al. (2013) 34.93† 35.00† 25.91† 27.73† A l (2016) 39 39 27 27 41 41 87 87* 30 30 62 62* 31 40* 5k dict. Artetxe et al. (2016) 39 39.27 27 41 41.87 87* 30 30.62 62* 31.40* 25 dict. Our method (ACL17) 37.27 39.60 28.16 ‐ None
178
⇒ Monolingual embeddings (CBOW + negative sampling) ⇒ Monolingual embeddings (CBOW + negative sampling) ⇒ Seed dictionary: 5,000 word pairs / 25 word pairs / none ⇒ Test dictionary: 1,500 word pairs
Super upervisi ision Method ethod EN‐IT IT EN EN‐DE DE EN EN‐FI FI EN EN‐ES ES 5k dict. 25 dict. None Zhang et al. (2017) Conneau et al. (2018)
180
( )
⇒ Monolingual embeddings (CBOW + negative sampling) ⇒ Monolingual embeddings (CBOW + negative sampling) ⇒ Seed dictionary: 5,000 word pairs / 25 word pairs / none ⇒ Test dictionary: 1,500 word pairs
Super upervisi ision Method ethod EN‐IT IT EN EN‐DE DE EN EN‐FI FI EN EN‐ES ES
5k dict. 25 dict. None Zhang et al. (2017) 0.00 0.00 0.01 0.01 Conneau et al. (2018) 13.55 42.15 0.38 21.23
181
( )
⇒ Monolingual embeddings (CBOW + negative sampling) ⇒ Monolingual embeddings (CBOW + negative sampling) ⇒ Seed dictionary: 5,000 word pairs / 25 word pairs / none ⇒ Test dictionary: 1,500 word pairs
Super upervisi ision Method ethod EN‐IT IT EN EN‐DE DE EN EN‐FI FI EN EN‐ES ES Mikolov et al. (2013) 34.93† 35.00† 25.91† 27.73† A l (2016) 39 27 41 87* 30 62* 31 40* 5k dict. Artetxe et al. (2016) 39.27 41.87* 30.62* 31.40* Smith et al. (2017) 43.1 43.33† 29.42† 35.13† Our method (AAAI18) 45.27 44.13 32.94 32.94 36.60 25 dict. Our method (ACL17) 37.27 39.60 28.16 ‐ None Zhang et al. (2017) 0.00 0.00 0.01 0.01 Conneau et al. (2018) 13.55 42.15 0.38 21.23
182
( ) Our method (ACL18) 48.13 48.13 48.19 48.19 32.63 37.33 37.33
⇒ Monolingual embeddings (CBOW + negative sampling) ⇒ Monolingual embeddings (CBOW + negative sampling) ⇒ Seed dictionary: 5,000 word pairs / 25 word pairs / none ⇒ Test dictionary: 1,500 word pairs
Super upervisi ision Method ethod EN‐IT IT EN EN‐DE DE EN EN‐FI FI EN EN‐ES ES Mikolov et al. (2013) 34.93† 35.00† 25.91† 27.73† A l (2016) 39 27 41 87* 30 62* 31 40* 5k dict. Artetxe et al. (2016) 39.27 41.87* 30.62* 31.40* Smith et al. (2017) 43.1 43.33† 29.42† 35.13† Our method (AAAI18) 45.27 44.13 32.94 32.94 36.60 25 dict. Our method (ACL17) 37.27 39.60 28.16 ‐ None Zhang et al. (2017) 0.00 0.00 0.01 0.01 Conneau et al. (2018) 13.55 42.15 0.38 21.23
183
( ) Our method (ACL18) 48.13 48.13 48.19 48.19 32.63 37.33 37.33
184
185
High quality dictionaries: Manual analysis shows that real accuracy > 60% High frequency words up to 80%
186
High quality dictionaries: Manual analysis shows that real accuracy > 60% High frequency words up to 80%
Full repr produci
lity (i (incl ncludi uding ng da datase sets): ts):
https://github.com/artetxem/vecmap 187
High quality dictionaries: Manual analysis shows that real accuracy > 60% High frequency words up to 80%
Full repr produci
lity (i (incl ncludi uding ng da datase sets): ts):
https://github.com/artetxem/vecmap
190
Generalizing zing and and Im Impr proving Bi Bilingual ngual Wo Word Embeddi Embedding Ma Mappings gs wi with a Mu Multi‐ and and Im Impr proving Bi Bilingual ngual Wo Word Embeddi Embedding Ma Mappings gs wi with a Mu Multi St Step ep Fr Fram amework
inear Trans nsform rmatio ions
Learning ng bil bilingual ngual wor
eddings with ith (alm almost) no bil ilingual ingual da
2017.
A robust s t self‐ learnin learning meth thod for fully ully unsup upervise vised cross ss‐lingual ngual mappi appings ngs of d b ddi ddi I ACL 2018 wo word em embeddi ddings
191
lf l f ll d ( )
192
193
194
encoder der based on Recurrent Neural Nets return all hidden states, encoding input x1…xn
1 n
195
encoder der based on Recurrent Neural Nets return all hidden states, encoding input x1…xn
1 n
decoder der based on Recurrent Neural Nets ‐ based on hidden states and last word in translation y based on hidden states and last word in translation yi-1 ‐ plus an atte attent ntion mechanism ‐ classifier guesses next word y classifier guesses next word yi
196
encoder der based on Recurrent Neural Nets return all hidden states, encoding input x1…xn
1 n
decoder der based on Recurrent Neural Nets ‐ based on hidden states and last word in translation y based on hidden states and last word in translation yi-1 ‐ plus an atte attent ntion mechanism ‐ classifier guesses next word y classifier guesses next word yi End End‐to to‐end end tr trainin aining
197
198
Source: Wu et al. 2016 (~ 30 authors – Also known as Google NMT)
Encoder for L1 L2 decoder n softmax
. . . . . .
ttention
…
L1 embeddings
…
at
199
can re represent wo word rds in in tw two languag languages in in the the same same embeddings embeddings space space without bilingual dictionaries… … what can we do? 200
can re represent wo word rds in in tw two languag languages in in the the same same embeddings embeddings space space without bilingual dictionaries… … what can we do?
archit chitectur ecture of
the NM NMT sy system:
Shared encoder for the two languages (E)
201
202
203
204
archit chitectur ecture of
the NM NMT sy system:
Shared encoder for the two languages (E)
training aining re regime, mi mixing ng mi mini ni‐ba batches hes:
i i d i i i L1 i h l ( 1)
205
p , , , p
Training aining 206
Training aining
Une fusillade a eu lieu à l’aéroport international p de Los Angeles.
207
Training aining
Supervised Supervised
Une fusillade a eu lieu à l’aéroport international p de Los Angeles.
208
Training aining
Supervised Supervised
Une fusillade a eu lieu à l’aéroport international There was a shooting in Los Angeles International Airport. p de Los Angeles.
209
Training aining
Supervised Supervised
Une fusillade a eu lieu à l’aéroport international There was a shooting in Los Angeles International Airport. p de Los Angeles.
210
Training aining
Supervised Supervised
Une fusillade a eu lieu à l’aéroport international There was a shooting in Los Angeles International Airport. p de Los Angeles.
211
Training aining
Supervised Supervised
Une fusillade a eu lieu à l’aéroport international p de Los Angeles.
212
Training aining
Supervised Supervised Autoencoder
Une fusillade a eu lieu à l’aéroport international de Los Angeles. Une fusillade a eu lieu à l’aéroport international p de Los Angeles.
213
Training aining
Supervised Supervised Autoencoder
Une fusillade a eu lieu à l’aéroport international de Los Angeles. Une fusillade a eu lieu à l’aéroport international p de Los Angeles.
214
Training aining
Supervised Supervised Denoising Autoencoder
Une fusillade a eu lieu à l’aéroport international de Los Angeles. Une lieu fusillade a eu à l’aéroport de Los p international Angeles.
215
Training aining
Supervised Supervised Denoising Autoencoder
Une fusillade a eu lieu à l’aéroport international de Los Angeles. Une lieu fusillade a eu à l’aéroport de Los p international Angeles.
216
Training aining
Supervised Supervised Denoising Autoencoder
There a shooting was in Airport Los Angeles International. There was a shooting in Los Angeles International Airport.
217
Training aining
Supervised Supervised Denoising Autoencoder
There a shooting was in Airport Los Angeles International. There was a shooting in Los Angeles International Airport.
218
Training aining
Supervised Supervised Denoising Backtranslation
Une fusillade a eu lieu à l’aéroport international p de Los Angeles.
219
Training aining
Supervised Supervised Denoising Backtranslation
Une fusillade a eu lieu à l’aéroport international p de Los Angeles.
220
Training aining
Supervised Supervised Denoising Backtranslation
Une fusillade a eu lieu à l’aéroport international A shooting has had place in airport international of Los Angeles. p de Los Angeles.
221
Training aining
Supervised Supervised Denoising Backtranslation
Une lieu fusillade a eu à l’aéroport de Los A shooting has had place in airport international of Los Angeles. p international Angeles.
222
Training aining
Supervised Supervised Denoising Backtranslation
Une fusillade a eu lieu à l’aéroport international de Los Angeles. Une lieu fusillade a eu à l’aéroport de Los A shooting has had place in airport international of Los Angeles. p international Angeles.
223
archit chitectur ecture of
the NM NMT sy system:
Shared encoder for the two languages (E)
training aining re regime, mi mixing ng mi mini ni‐ba batches hes:
i i d i i i L1 i h l ( 1)
224
p , , , p
Only WMT released data (test and monolingual corpora)
FR FR‐EN EN EN EN‐FR FR DE DE‐EN EN EN EN‐DE DE
y g p
Unsupervised NMT
226
Only WMT released data (test and monolingual corpora)
FR FR‐EN EN EN EN‐FR FR DE DE‐EN EN EN EN‐DE DE
y g p
Unsupervised NMT Baseline (emb. nearest neighbor) 9.98 6.25 7.07 4.39
227
Only WMT released data (test and monolingual corpora)
FR FR‐EN EN EN EN‐FR FR DE DE‐EN EN EN EN‐DE DE
y g p
Unsupervised NMT Baseline (emb. nearest neighbor) 9.98 6.25 7.07 4.39 Proposed (denoising) 7.28 5.33 3.64 2.40
228
Only WMT released data (test and monolingual corpora)
FR FR‐EN EN EN EN‐FR FR DE DE‐EN EN EN EN‐DE DE
y g p
Unsupervised NMT Baseline (emb. nearest neighbor) 9.98 6.25 7.07 4.39 Proposed (denoising) 7.28 5.33 3.64 2.40 Proposed (+backtranslation) 15 15 56 56 15 15 13 13 10 10 21 21 6 5 6 55 Proposed (+backtranslation) 15 15.56 56 15 15.13 13 10 10.21 21 6.55 55
230
Only WMT released data (test and monolingual corpora)
FR FR‐EN EN EN EN‐FR FR DE DE‐EN EN EN EN‐DE DE
y g p
Unsupervised NMT Baseline (emb. nearest neighbor) 9.98 6.25 7.07 4.39 Proposed (denoising) 7.28 5.33 3.64 2.40 Proposed (+backtranslation) 15 56 15 13 10 21 6 55 Proposed (+backtranslation) 15.56 15.13 10.21 6.55 Semi‐supervised NMT Proposed (full) + 10k parallel 18.57 18.57 17.34 17.34 11.47 11.47 7.86 7.86 Proposed (full) + 100k parallel 21.81 21.81 21.74 21.74 15.24 15.24 10.95 10.95
233
Only WMT released data (test and monolingual corpora)
FR FR‐EN EN EN EN‐FR FR DE DE‐EN EN EN EN‐DE DE
y g p
Unsupervised NMT Baseline (emb. nearest neighbor) 9.98 6.25 7.07 4.39 Proposed (denoising) 7.28 5.33 3.64 2.40 Proposed (+backtranslation) 15 15 56 56 15 15 13 13 10 10 21 21 6 5 6 55 Proposed (+backtranslation) 15 15.56 56 15 15.13 13 10 10.21 21 6.55 55 Lample et al. 2018* (Same conference!) 14.31 15.06 ‐ ‐
*Experimental conditions are not exactly the same
237
Only WMT released data (test and monolingual corpora)
FR FR‐EN EN EN EN‐FR FR DE DE‐EN EN EN EN‐DE DE
y g p
Unsupervised NMT Baseline (emb. nearest neighbor) 9.98 6.25 7.07 4.39 Proposed (denoising) 7.28 5.33 3.64 2.40 Proposed (+backtranslation) 15 15 56 56 15 15 13 13 10 10 21 21 6 5 6 55 Proposed (+backtranslation) 15 15.56 56 15 15.13 13 10 10.21 21 6.55 55 Lample et al. 2018 14.31 15.06 ‐ ‐ Lample et al. 2018b
Lample et al. 2018b (EMNLP)
No embedding mappings
238
Only WMT released data (test and monolingual corpora)
FR FR‐EN EN EN EN‐FR FR DE DE‐EN EN EN EN‐DE DE
y g p
Unsupervised NMT Baseline (emb. nearest neighbor) 9.98 6.25 7.07 4.39 Proposed (denoising) 7.28 5.33 3.64 2.40 Proposed (+backtranslation) 15 56 15 13 10 21 6 55 Proposed (+backtranslation) 15.56 15.13 10.21 6.55 Lample et al. 2018 14.31 15.06 ‐ ‐ Lample et al. 2018b 24.2 24.2 25.1 25.1 ‐ ‐
Lample et al. 2018b (EMNLP)
No embedding mappings
239
Only WMT released data (test and monolingual corpora)
FR FR‐EN EN EN EN‐FR FR DE DE‐EN EN EN EN‐DE DE
y g p
Unsupervised NMT Baseline (emb. nearest neighbor) 9.98 6.25 7.07 4.39 Proposed (denoising) 7.28 5.33 3.64 2.40 Proposed (+backtranslation) 15 56 15 13 10 21 6 55 Proposed (+backtranslation) 15.56 15.13 10.21 6.55 Lample et al. 2018 14.31 15.06 ‐ ‐ Lample et al. 2018b 24.2 25.1 ‐ ‐ L l d C 2019 33 33 3 33 33 4 Lample and Conneau 2019 33 33.3 33 33.4 ‐ ‐
Lample and Conneau 2019 (Arxiv)
larger batch sizes (+6 Bleu!) 240
241
l b ( ) Artetxe et al. 2018b (EMNLP)
242 p g
Only WMT released data (test and monolingual corpora). WMT14 and WMT16
FR FR‐EN EN EN EN‐FR FR DE DE‐EN EN EN EN‐DE DE DE DE‐EN EN EN EN‐DE DE
y g p
Unsupervised NMT Artetxe et al. 2018 15.56 15.13 10.21 6.55 Lample and C. 2019 33.3 33.3 33.4 33.4 34.3 34.3 26.2 26.2 Unsupervised Unsupervised PBMT
l b ( ) Artetxe et al. 2018b (EMNLP)
243 p g
Only WMT released data (test and monolingual corpora). WMT14 and WMT16
FR FR‐EN EN EN EN‐FR FR DE DE‐EN EN EN EN‐DE DE DE DE‐EN EN EN EN‐DE DE
y g p
Unsupervised NMT Artetxe et al. 2018 15.56 15.13 10.21 6.55 Lample and C. 2019 33.3 33.3 33.4 33.4 34.3 34.3 26.2 26.2 Unsupervised Artetxe et al 2018b 25 87 26 22 17 43 14 08 23 05 18 23 Unsupervised Artetxe et al. 2018b 25.87 26.22 17.43 14.08 23.05 18.23 PBMT
l b ( ) Artetxe et al. 2018b (EMNLP)
244 p g
Only WMT released data (test and monolingual corpora). WMT14 and WMT16
FR FR‐EN EN EN EN‐FR FR DE DE‐EN EN EN EN‐DE DE DE DE‐EN EN EN EN‐DE DE
y g p
Unsupervised NMT Artetxe et al. 2018 15.56 15.13 10.21 6.55 Lample and C. 2019 33.3 33.3 33.4 33.4 34.3 34.3 26.2 26.2 Unsupervised Artetxe et al 2018b 25 87 26 22 17 43 14 08 23 23 05 05 18 18 23 23 Unsupervised Artetxe et al. 2018b 25.87 26.22 17.43 14.08 23 23.05 05 18 18.23 23 PBMT Lample et al. 2018b 27.16 27.16 28.11 28.11 22.68 17.77
l b ( ) Artetxe et al. 2018b (EMNLP)
245 p g
Only WMT released data (test and monolingual corpora). WMT14 and WMT16
FR FR‐EN EN EN EN‐FR FR DE DE‐EN EN EN EN‐DE DE DE DE‐EN EN EN EN‐DE DE
y g p
Unsupervised NMT Artetxe et al. 2018 15.56 15.13 10.21 6.55 Lample and C. 2019 33.3 33.4 34.3 26.2 Unsupervised Artetxe et al 2018b 25 87 26 22 17 43 14 08 23 05 18 23 Unsupervised Artetxe et al. 2018b 25.87 26.22 17.43 14.08 23.05 18.23 PBMT Lample et al. 2018b 27.16 28.11 22.68 17.77 NMT + Lample et al. 2018b 27.7 27.6 25.2 20.2 PBMT A t t t l SUBM 33 33 5 36 36 2 27 27 0 22 22 5 34 34 4 26 26 9 PBMT Artetxe et al. SUBM 33 33.5 36 36.2 27 27.0 22 22.5 34 34.4 26 26.9
Combina inatio ions ns improve further UMT is at the level MT was at 2014 246
247
248
249
251
252
The main Machine Translation competition (WMT) has now an unsuper unsupervised ised sub subtrack ck
p y
https://github.com/artetxem/undreamt https://github.com/artetxem/monoses (soon) 260
Unsu Unsuper ervi vised sed Neur Neural al Ma Machine Tr
p
, , g Uns Unsuper upervised ed St Statis istic tical Ma Machine Tr
261
Word embeddi embeddings ngs key ey for Natural Language Processing
languages in in co commo mmon space space
languages in in co commo mmon space space
very fe few re resources
monolingual ngual re resources
monolingual ngual re resources
262
Word embeddi embeddings ngs key ey for Natural Language Processing
languages in in co commo mmon space space
languages in in co commo mmon space space
very fe few re resources
monolingual ngual re resources
monolingual ngual re resources
Cross
ngual unsuper unsupervised ed mappi mappings ngs enabled breakthroughs in Bili l di ti i d ti
263
Word embeddi embeddings ngs key ey for Natural Language Processing
languages in in co commo mmon space space
languages in in co commo mmon space space
very fe few re resources
monolingual ngual re resources
monolingual ngual re resources
Cross
ngual unsuper unsupervised ed mappi mappings ngs enabled breakthroughs in Bili l di ti i d ti
U l d i it i f i f
ancy
MT in in lo low re resource lan languages and and dom domains ins l f f h l d l d
trans ansformi
ng the NLP NLP lan andsc scap ape
ltilingual al to tool
264
Univer ersal sal sen senten ence ce re representat tations
265