On the Limitations of Unsupervised Bilingual Dictionary Induction - - PowerPoint PPT Presentation

On the Limitations of Unsupervised Bilingual Dictionary Induction

Sebastian Ruder Ivan Vulić Anders Søgaard

2

Background:
 Unsupervised MT

• Recently: Unsupervised neural machine translation

(Artetxe et al., ICLR 2018; Lample et al., ICLR 2018)

2

Background:
 Unsupervised MT

• Recently: Unsupervised neural machine translation

(Artetxe et al., ICLR 2018; Lample et al., ICLR 2018)

2

Background:
 Unsupervised MT

• Recently: Unsupervised neural machine translation

(Artetxe et al., ICLR 2018; Lample et al., ICLR 2018)

2

Background:
 Unsupervised MT

• Key component:

Initialization via unsupervised cross-lingual alignment of word embedding spaces

3

Background:
 Cross-lingual word embeddings

• Cross-lingual word embeddings enable cross-lingual

transfer

3

Background:
 Cross-lingual word embeddings

• Cross-lingual word embeddings enable cross-lingual

transfer

• Most common approach: Project one word embedding

space into another by learning a transformation matrix 
 between source embeddings and their translations

3

Background:
 Cross-lingual word embeddings

W xi yi n

• Cross-lingual word embeddings enable cross-lingual

transfer

• Most common approach: Project one word embedding

space into another by learning a transformation matrix 
 between source embeddings and their translations (Mikolov et al., 2013)

3

Background:
 Cross-lingual word embeddings

n

∑

i=1

∥Wxi − yi∥2 W xi yi n

• Cross-lingual word embeddings enable cross-lingual

transfer

• Most common approach: Project one word embedding

space into another by learning a transformation matrix 
 between source embeddings and their translations (Mikolov et al., 2013)

• More recently: Use an adversarial setup to learn an

unsupervised mapping

3

Background:
 Cross-lingual word embeddings

n

∑

i=1

∥Wxi − yi∥2 W xi yi n

• Cross-lingual word embeddings enable cross-lingual

transfer

• Most common approach: Project one word embedding

space into another by learning a transformation matrix 
 between source embeddings and their translations (Mikolov et al., 2013)

• More recently: Use an adversarial setup to learn an

unsupervised mapping

• Assumption: Word embedding spaces are approximately

isomorphic, i.e. same number of vertices, connected the same way.

3

Background:
 Cross-lingual word embeddings

n

∑

i=1

∥Wxi − yi∥2 W xi yi n

4

How similar are embeddings across languages?

• Nearest neighbour (NN) graphs of top 10 most frequent

words in English and German are not isomorphic.

4

How similar are embeddings across languages?

• Nearest neighbour (NN) graphs of top 10 most frequent

words in English and German are not isomorphic.

• NN graphs of top 10 most frequent English words and their

translations into German

4

How similar are embeddings across languages?

English German

• Nearest neighbour (NN) graphs of top 10 most frequent

words in English and German are not isomorphic.

• NN graphs of top 10 most frequent English words and their

translations into German

4

How similar are embeddings across languages?

English German

• Not isomorphic

5

How similar are embeddings across languages?

• NN graphs of top 10 most frequent English nouns and their

translations

5

How similar are embeddings across languages?

English German

• NN graphs of top 10 most frequent English nouns and their

translations

5

How similar are embeddings across languages?

English German

• Not isomorphic

• NN graphs of top 10 most frequent English nouns and their

translations

5

How similar are embeddings across languages?

English German

• Not isomorphic

Word embeddings are not approximately isomorphic across languages.

6

How do we quantify similarity?

• Need a metric to measure how similar two NN graphs

and of different languages are

6

How do we quantify similarity?

G1 G2

• Need a metric to measure how similar two NN graphs

and of different languages are

• Propose eigenvector similarity

6

How do we quantify similarity?

G1 G2

• Need a metric to measure how similar two NN graphs

and of different languages are

• Propose eigenvector similarity
• : adjacency matrices of

6

How do we quantify similarity?

G1 G2 A1, A2 G1, G2

• Need a metric to measure how similar two NN graphs

and of different languages are

• Propose eigenvector similarity
• : adjacency matrices of
• : degree matrices of

6

How do we quantify similarity?

G1 G2 A1, A2 G1, G2 G1, G2 D1, D2

• Need a metric to measure how similar two NN graphs

and of different languages are

• Propose eigenvector similarity
• : adjacency matrices of
• : degree matrices of
• : Laplacians of

6

How do we quantify similarity?

G1 G2 A1, A2 G1, G2 G1, G2 D1, D2 L1 = D1 − A1, L2 = D2 − A2 G1, G2

• Need a metric to measure how similar two NN graphs

and of different languages are

• Propose eigenvector similarity
• : adjacency matrices of
• : degree matrices of
• : Laplacians of
• : eigenvalues (spectra) of

6

How do we quantify similarity?

G1 G2 A1, A2 G1, G2 G1, G2 D1, D2 L1 = D1 − A1, L2 = D2 − A2 G1, G2 λ1, λ2 L1, L2

• Need a metric to measure how similar two NN graphs

and of different languages are

• Propose eigenvector similarity
• : adjacency matrices of
• : degree matrices of
• : Laplacians of
• : eigenvalues (spectra) of

6

How do we quantify similarity?

G1 G2 A1, A2 G1, G2 G1, G2 D1, D2 L1 = D1 − A1, L2 = D2 − A2 G1, G2 λ1, λ2 L1, L2 Δ =

k

∑

i=1

(λ1i − λ2i)2 k = min

j {

∑k

i=1 λji

∑n

i=1 λji

> 0.9}

• Metric: where

7

How do we quantify similarity?

Δ =

k

∑

i=1

(λ1i − λ2i)2 k = min

j {

∑k

i=1 λji

∑n

i=1 λji

> 0.9}

• Metric: where

7

How do we quantify similarity?

• Quantifies how much two NN graphs are isospectral, i.e.

they have the same spectrum (same sets of eigenvalues). Δ =

k

∑

i=1

(λ1i − λ2i)2 k = min

j {

∑k

i=1 λji

∑n

i=1 λji

> 0.9}

• Metric: where

7

How do we quantify similarity?

• Quantifies how much two NN graphs are isospectral, i.e.

they have the same spectrum (same sets of eigenvalues).

• Isomorphic isospectral, but isospectral isomorphic

→ ↛ Δ =

k

∑

i=1

(λ1i − λ2i)2 k = min

j {

∑k

i=1 λji

∑n

i=1 λji

> 0.9}

• Metric: where

7

How do we quantify similarity?

• Quantifies how much two NN graphs are isospectral, i.e.

they have the same spectrum (same sets of eigenvalues).

• Isomorphic isospectral, but isospectral isomorphic
• →

↛ Δ : G1, G2 → [0,∞) Δ =

k

∑

i=1

(λ1i − λ2i)2 k = min

j {

∑k

i=1 λji

∑n

i=1 λji

> 0.9}

• Metric: where

7

How do we quantify similarity?

• Quantifies how much two NN graphs are isospectral, i.e.

they have the same spectrum (same sets of eigenvalues).

• Isomorphic isospectral, but isospectral isomorphic
• : are isospectral (very similar)

→ ↛ Δ : G1, G2 → [0,∞) Δ = 0 G1, G2 Δ =

k

∑

i=1

(λ1i − λ2i)2 k = min

j {

∑k

i=1 λji

∑n

i=1 λji

> 0.9}

• Metric: where

7

How do we quantify similarity?

• Quantifies how much two NN graphs are isospectral, i.e.

they have the same spectrum (same sets of eigenvalues).

• Isomorphic isospectral, but isospectral isomorphic
• : are isospectral (very similar)
• : become less similar

→ ↛ Δ : G1, G2 → [0,∞) Δ = 0 G1, G2 Δ → ∞ G1, G2 Δ =

k

∑

i=1

(λ1i − λ2i)2 k = min

j {

∑k

i=1 λji

∑n

i=1 λji

> 0.9}

• Metric: where

8

Unsupervised cross-lingual learning assumptions

• Besides isomorphism, several other implicit assumptions

8

Unsupervised cross-lingual learning assumptions

• Besides isomorphism, several other implicit assumptions
• May or may not scale to low-resource languages

8

Unsupervised cross-lingual learning assumptions

• Besides isomorphism, several other implicit assumptions
• May or may not scale to low-resource languages

8

Unsupervised cross-lingual learning assumptions

Conneau et al. (2018) This work

• Besides isomorphism, several other implicit assumptions
• May or may not scale to low-resource languages

8

Unsupervised cross-lingual learning assumptions

Conneau et al. (2018) This work Languages Dependent-marking, fusional and isolating Agglutinative, many cases

• Besides isomorphism, several other implicit assumptions
• May or may not scale to low-resource languages

8

Unsupervised cross-lingual learning assumptions

Conneau et al. (2018) This work Languages Dependent-marking, fusional and isolating Agglutinative, many cases Corpora Comparable (Wikipedia) Different domains

• Besides isomorphism, several other implicit assumptions
• May or may not scale to low-resource languages

8

Unsupervised cross-lingual learning assumptions

Conneau et al. (2018) This work Languages Dependent-marking, fusional and isolating Agglutinative, many cases Corpora Comparable (Wikipedia) Different domains Algorithms/ hyperparameters Same Different

9

Conneau et al. (2018)

• 1. Monolingual word embeddings:


Learn monolingual vector spaces and .

9

Conneau et al. (2018)

X Y

• 1. Monolingual word embeddings:


Learn monolingual vector spaces and .

• 2. Adversarial mapping:


Learn a translation matrix . Train discriminator to discriminate samples from and .

9

Conneau et al. (2018)

X Y W WX Y

• 3. Refinement (Procrustes analysis):


Build bilingual dictionary of frequent words using . Learn a new based on frequent word pairs.

10

Conneau et al. (2018)

W W

• 3. Refinement (Procrustes analysis):


Build bilingual dictionary of frequent words using . Learn a new based on frequent word pairs.

• 4. Cross-domain similarity local scaling (CSLS):


Use similarity measure that increases similarity of isolated word vectors, decreases similarity of vectors in dense areas.

10

Conneau et al. (2018)

W W

11

A simple weakly supervised method

• Extract identically spelled words in both languages

11

A simple weakly supervised method

• Extract identically spelled words in both languages
• Use these as bilingual seed words

11

A simple weakly supervised method

• Extract identically spelled words in both languages
• Use these as bilingual seed words
• Run refinement step of Conneau et al. (2018)

11

A simple weakly supervised method

12

Experiments:
 Bilingual dictionary induction

12

Experiments:
 Bilingual dictionary induction

• Given a list of source language words, find the closest

target language word in the cross-lingual embedding space

12

Experiments:
 Bilingual dictionary induction

• Given a list of source language words, find the closest

target language word in the cross-lingual embedding space

• Compare against a gold standard dictionary

12

Experiments:
 Bilingual dictionary induction

• Given a list of source language words, find the closest

target language word in the cross-lingual embedding space

• Compare against a gold standard dictionary
• Metric: Precision at 1 (P@1)

12

Experiments:
 Bilingual dictionary induction

• Given a list of source language words, find the closest

target language word in the cross-lingual embedding space

• Compare against a gold standard dictionary
• Metric: Precision at 1 (P@1)
• Use fastText monolingual embeddings

12

Experiments:
 Bilingual dictionary induction

• Given a list of source language words, find the closest

target language word in the cross-lingual embedding space

• Compare against a gold standard dictionary
• Metric: Precision at 1 (P@1)
• Use fastText monolingual embeddings

Conneau et al. (2018) This work Languages
 (English to) French, German, Chinese, Russian, Spanish Estonian (ET), Finnish (FI), Greek (EL), Hungarian (HU), Polish (PL), Turkish

13

Impact of language similarity

P@1 22.5 45 67.5 90 EN-ES EN-ET EN-FI EN-EL EN-HU EN-PL EN-TR ET-FI

Unsupervised (Adversarial) Weakly supervised (Identical strings)

13

Impact of language similarity

P@1 22.5 45 67.5 90 EN-ES EN-ET EN-FI EN-EL EN-HU EN-PL EN-TR ET-FI

Unsupervised (Adversarial) Weakly supervised (Identical strings)

13

Impact of language similarity

• Unsupervised approaches are challenged by languages that

are not isolating and not dependent marking

P@1 22.5 45 67.5 90 EN-ES EN-ET EN-FI EN-EL EN-HU EN-PL EN-TR ET-FI

Unsupervised (Adversarial) Weakly supervised (Identical strings)

13

Impact of language similarity

• Unsupervised approaches are challenged by languages that

are not isolating and not dependent marking

• Naive supervision leads to competitive performance on

similar language pairs and better results for dissimilar pairs

P@1 22.5 45 67.5 90 EN-ES EN-ET EN-FI EN-EL EN-HU EN-PL EN-TR ET-FI

Unsupervised (Adversarial) Weakly supervised (Identical strings)

14

Impact of language similarity

Eigenvector similarity 2 4 6 8 BDI performance 22.5 45 67.5 90

14

Impact of language similarity

Eigenvector similarity 2 4 6 8 BDI performance 22.5 45 67.5 90

14

Impact of language similarity

• Eigenvector similarity strongly correlates with BDI

performance

Eigenvector similarity 2 4 6 8 BDI performance 22.5 45 67.5 90

(ρ ∼ 0.89)

15

Impact of domain differences

15

Impact of domain differences

• Source and target embeddings induced on 3 corpora:


EuroParl (EP), Wikipedia (Wiki), Medical (EMEA)

15

Impact of domain differences

English-Spanish

P@1

17.5 35 52.5 70

Domain similarity

0.2 0.4 0.6 0.8 EP-EP EP-Wiki EP-EMEA Wiki-EP Wiki-Wiki Wiki-EMEA EMEA-EP EMEA-Wiki EMEA-EMEA

Domain similarity Unsupervised Weakly supervised

• Source and target embeddings induced on 3 corpora:


EuroParl (EP), Wikipedia (Wiki), Medical (EMEA)

15

Impact of domain differences

English-Spanish

P@1

17.5 35 52.5 70

Domain similarity

0.2 0.4 0.6 0.8 EP-EP EP-Wiki EP-EMEA Wiki-EP Wiki-Wiki Wiki-EMEA EMEA-EP EMEA-Wiki EMEA-EMEA

Domain similarity Unsupervised Weakly supervised

• Source and target embeddings induced on 3 corpora:


EuroParl (EP), Wikipedia (Wiki), Medical (EMEA)

15

Impact of domain differences

English-Spanish

P@1

17.5 35 52.5 70

Domain similarity

0.2 0.4 0.6 0.8 EP-EP EP-Wiki EP-EMEA Wiki-EP Wiki-Wiki Wiki-EMEA EMEA-EP EMEA-Wiki EMEA-EMEA

Domain similarity Unsupervised Weakly supervised

• Source and target embeddings induced on 3 corpora:


EuroParl (EP), Wikipedia (Wiki), Medical (EMEA)

15

Impact of domain differences

English-Spanish

P@1

17.5 35 52.5 70

Domain similarity

0.2 0.4 0.6 0.8 EP-EP EP-Wiki EP-EMEA Wiki-EP Wiki-Wiki Wiki-EMEA EMEA-EP EMEA-Wiki EMEA-EMEA

Domain similarity Unsupervised Weakly supervised

• Source and target embeddings induced on 3 corpora:


EuroParl (EP), Wikipedia (Wiki), Medical (EMEA)

• Unsupervised approaches break down when domains are

dissimilar

15

Impact of domain differences

English-Spanish

P@1

17.5 35 52.5 70

Domain similarity

0.2 0.4 0.6 0.8 EP-EP EP-Wiki EP-EMEA Wiki-EP Wiki-Wiki Wiki-EMEA EMEA-EP EMEA-Wiki EMEA-EMEA

Domain similarity Unsupervised Weakly supervised

• Source and target embeddings induced on 3 corpora:


EuroParl (EP), Wikipedia (Wiki), Medical (EMEA)

• Unsupervised approaches break down when domains are

dissimilar

16

Impact of domain differences

English-Finnish

P@1

7.5 15 22.5 30

Domain similarity

0.2 0.4 0.6 0.8 EP-EP EP-Wiki EP-EMEA Wiki-EP Wiki-Wiki Wiki-EMEA EMEA-EP EMEA-Wiki EMEA-EMEA

Domain similarity Unsupervised Weakly supervised

16

Impact of domain differences

English-Finnish

P@1

7.5 15 22.5 30

Domain similarity

0.2 0.4 0.6 0.8 EP-EP EP-Wiki EP-EMEA Wiki-EP Wiki-Wiki Wiki-EMEA EMEA-EP EMEA-Wiki EMEA-EMEA

Domain similarity Unsupervised Weakly supervised

16

Impact of domain differences

English-Finnish

P@1

7.5 15 22.5 30

Domain similarity

0.2 0.4 0.6 0.8 EP-EP EP-Wiki EP-EMEA Wiki-EP Wiki-Wiki Wiki-EMEA EMEA-EP EMEA-Wiki EMEA-EMEA

Domain similarity Unsupervised Weakly supervised

• Domain differences may exacerbate difficulties of

generalising across dissimilar languages

17

Impact of domain differences

English-Hungarian

P@1

7.5 15 22.5 30

Domain similarity

0.2 0.4 0.6 0.8 EP-EP EP-Wiki EP-EMEA Wiki-EP Wiki-Wiki Wiki-EMEA EMEA-EP EMEA-Wiki EMEA-EMEA

Domain similarity Unsupervised Weakly supervised

17

Impact of domain differences

English-Hungarian

P@1

7.5 15 22.5 30

Domain similarity

0.2 0.4 0.6 0.8 EP-EP EP-Wiki EP-EMEA Wiki-EP Wiki-Wiki Wiki-EMEA EMEA-EP EMEA-Wiki EMEA-EMEA

Domain similarity Unsupervised Weakly supervised

17

Impact of domain differences

English-Hungarian

P@1

7.5 15 22.5 30

Domain similarity

0.2 0.4 0.6 0.8 EP-EP EP-Wiki EP-EMEA Wiki-EP Wiki-Wiki Wiki-EMEA EMEA-EP EMEA-Wiki EMEA-EMEA

Domain similarity Unsupervised Weakly supervised

• Weak supervision helps to bridge domain differences, but

performance still deteriorates

18

Impact of hyper-parameters

18

Impact of hyper-parameters

• Settings: English with skipgram, win=2, ngrams=3-6

18

Impact of hyper-parameters

• Settings: English with skipgram, win=2, ngrams=3-6
• Vary hyper-parameters of Spanish embeddings

18

Impact of hyper-parameters

P@1 22.5 45 67.5 90 == win=10 ngrams=2-7 win=10, ngrams=2-7

English-Spanish (skipgram) English-Spanish (cbow)

• Settings: English with skipgram, win=2, ngrams=3-6
• Vary hyper-parameters of Spanish embeddings

≠ ≠ ≠

18

Impact of hyper-parameters

P@1 22.5 45 67.5 90 == win=10 ngrams=2-7 win=10, ngrams=2-7

English-Spanish (skipgram) English-Spanish (cbow)

• Settings: English with skipgram, win=2, ngrams=3-6
• Vary hyper-parameters of Spanish embeddings

≠ ≠ ≠

18

Impact of hyper-parameters

P@1 22.5 45 67.5 90 == win=10 ngrams=2-7 win=10, ngrams=2-7

English-Spanish (skipgram) English-Spanish (cbow)

• Settings: English with skipgram, win=2, ngrams=3-6
• Vary hyper-parameters of Spanish embeddings

≠ ≠ ≠

19

Impact of hyper-parameters

P@1 22.5 45 67.5 90 == win=10 ngrams=2-7 win=10, ngrams=2-7

English-Spanish (skipgram) English-Spanish (cbow)

• Different algorithms introduce embedding spaces with

wildly different structures.

≠ ≠ ≠

20

Impact of dimensionality

P@1 22.5 45 67.5 90 EN-ES EN-ET EN-FI EN-EL EN-HU EN-PL EN-TR

300-dimensional embeddings 40-dimensional embeddings

20

Impact of dimensionality

P@1 22.5 45 67.5 90 EN-ES EN-ET EN-FI EN-EL EN-HU EN-PL EN-TR

300-dimensional embeddings 40-dimensional embeddings

• Worse performance overall, but better performance for

dissimilar language pairs (Estonian, Finnish, Greek).

20

Impact of dimensionality

P@1 22.5 45 67.5 90 EN-ES EN-ET EN-FI EN-EL EN-HU EN-PL EN-TR

300-dimensional embeddings 40-dimensional embeddings

• Worse performance overall, but better performance for

dissimilar language pairs (Estonian, Finnish, Greek).

• Monolingual word embeddings may overfit to rare

peculiarities of languages.

20

Impact of dimensionality

P@1 22.5 45 67.5 90 EN-ES EN-ET EN-FI EN-EL EN-HU EN-PL EN-TR

300-dimensional embeddings 40-dimensional embeddings

21

Impact of evaluation procedure

21

Impact of evaluation procedure

• Part-of-speech:


Performance on verbs is lowest across the board.

21

Impact of evaluation procedure

• Part-of-speech:


Performance on verbs is lowest across the board.

• Frequency:


Sensitivity to frequency for Hungarian, but less so for Spanish.

21

Impact of evaluation procedure

• Part-of-speech:


Performance on verbs is lowest across the board.

• Frequency:


Sensitivity to frequency for Hungarian, but less so for Spanish.

• Homographs:


Lower precision due to loan words/proper names. High precision for free with weak supervision.

22

Takeaways

• Word embedding spaces are not approximately

isomorphic across languages.

22

Takeaways

• Word embedding spaces are not approximately

isomorphic across languages.

• We can use eigenvector similarity to characterise the

relatedness of two monolingual vector spaces.

22

Takeaways

• Word embedding spaces are not approximately

isomorphic across languages.

• We can use eigenvector similarity to characterise the

relatedness of two monolingual vector spaces.

• Eigenvector similarity strongly correlates with

unsupervised bilingual dictionary induction performance.

22

Takeaways

• Word embedding spaces are not approximately

isomorphic across languages.

• We can use eigenvector similarity to characterise the

relatedness of two monolingual vector spaces.

• Eigenvector similarity strongly correlates with

unsupervised bilingual dictionary induction performance.

• Limitations of unsupervised bilingual dictionary induction:

22

Takeaways

• Word embedding spaces are not approximately

isomorphic across languages.

• We can use eigenvector similarity to characterise the

relatedness of two monolingual vector spaces.

• Eigenvector similarity strongly correlates with

unsupervised bilingual dictionary induction performance.

• Limitations of unsupervised bilingual dictionary induction:
• Morphologically rich languages.

22

Takeaways

• Word embedding spaces are not approximately

isomorphic across languages.

• We can use eigenvector similarity to characterise the

relatedness of two monolingual vector spaces.

• Eigenvector similarity strongly correlates with

unsupervised bilingual dictionary induction performance.

• Limitations of unsupervised bilingual dictionary induction:
• Morphologically rich languages.
• Corpora from different domains.

22

Takeaways

• Word embedding spaces are not approximately

isomorphic across languages.

• We can use eigenvector similarity to characterise the

relatedness of two monolingual vector spaces.

• Eigenvector similarity strongly correlates with

unsupervised bilingual dictionary induction performance.

• Limitations of unsupervised bilingual dictionary induction:
• Morphologically rich languages.
• Corpora from different domains.
• Different word embedding algorithms.

22

Takeaways