ACL2018 Incorporating Latent Meanings of Morphological Compositions - - PowerPoint PPT Presentation

β–Ά
acl2018 incorporating latent meanings of morphological
SMART_READER_LITE
LIVE PREVIEW

ACL2018 Incorporating Latent Meanings of Morphological Compositions - - PowerPoint PPT Presentation

Jun 02, 2023 6 likes β€’309 views

ACL2018 Incorporating Latent Meanings of Morphological Compositions to Enhance Word Embeddings Yang Xu, Jiawei Liu, Wei Yang, and Liusheng Huang School of Computer Science and Technology, University of Science and Technology of China,

slide-1
SLIDE 1

July 17th 2018

Yang Xu, Jiawei Liu, Wei Yang, and Liusheng Huang

School of Computer Science and Technology, University of Science and Technology of China, Hefei, 230027, China

Incorporating Latent Meanings of Morphological Compositions to Enhance Word Embeddings ACL2018

slide-2
SLIDE 2

OUTLINE

Introduction

01

Latent Meaning Models

02

Experimental Setup

03

Experimental Results

04

Conclusion

05

2

ACL2018

slide-3
SLIDE 3

01 Introduction

3

ACL2018

slide-4
SLIDE 4

Word-level Word Embedding

01

Neural Network-Based

02

Matrix Factorization-Based ( Spectral Methods ) word-word co-occurrence matrix e.g., GloVe (Pennington et al.)

w(t-2) w(t-1) w(t+1) w(t+2) w(t) SUM INPUT PROJECTION OUTPUT C B O W w(t-2) w(t-1) w(t+1) w(t+2) w(t) SUM INPUT PROJECTION OUTPUT Skip-gram

e.g., CBOW, Skip-gram (Mikolov et al.)

4

ACL2018

slide-5
SLIDE 5

Morphology-based Word Embedding

Training Model

Morpheme Embeddings Word Embeddings Prefix Root Suffix Word

β†’ π‘—π‘œβˆ’ β†’ 𝑑𝑠𝑓𝑒 β†’ π‘—π‘π‘šπ‘“ β†’ incredible

Generated Word Vectors Morpheme Embeddings Prefix Root Suffix Generated Word Generative Model

01 02

5

ACL2018

slide-6
SLIDE 6

Our Original Intention

6

ACL2018

Word-level models: InputWords; Output Word Embeddings Morphology-based models: Input Words + Morphemes Output Word Embeddings + Morpheme Embeddings Our Latent Meaning Models: InputWords + Latent Meanings of Morphemes Output Word Embeddings ( no by-product, e.g., morpheme embedding) PURPOSE: to not only encode morphological properties into words, but also enhance the semantic similarities among word embeddings

slide-7
SLIDE 7

Explicit Models & Our Models

it is an incredible unbelievable thing it is that

  • in

cred ible un believ able not believe able capable not believe able capable

Prefix Latent Meaning

in un in, not not

Root Latent Meaning

believ cred believe believe

Suffix Latent Meaning

able ible able, capale able, capale sentence i : sentence j : Explicit models directly use morphemes

  • Our models

employ the latent meanings

  • f morphemes

Corpus Lookup table

in

*Note: The lookup table can be derived from morphological lexicons. 7

ACL2018

slide-8
SLIDE 8

02 Latent Meaning Models

8

ACL2018

slide-9
SLIDE 9

CBOW with Negative Sampling

9

ti-2 ti-1 ti+1 ti+2 ti

SUM INPUT PROJECTION OUTPUT

(Context Words) (Target Word) Sequence of tokens

𝑀 = 1 π‘œ

π‘œ

βˆ‘

𝑗=1

logπ‘ž(𝑒𝑗|π·π‘π‘œπ‘’π‘“π‘¦π‘’(𝑒𝑗))

Objective Function: Negative Sampling: ACL2018

slide-10
SLIDE 10

Three Specific Models

10

ACL2018

01

LMM-A

(Latent Meaning Model-Average)

02

LMM-S

(Latent Meaning Model-Similarity)

03

LMM-M

(Latent Meaning Model-Max)

slide-11
SLIDE 11

Word Map

Prefix Latent Meaning

in un in, not not

Root Latent Meaning

believ cred believe believe

Suffix Latent Meaning

able ible able, capale able, capale

Lookup table incredible in cred ible unbelievable un believ able Word Prefix Root Suffix incredible in not believe able capable unbelievable not believe able capable

Word Map

*Note: The derivational morphemes, not the inflectional morphemes, are mainly concerned 11

ACL2018

#rows = |vocabulary|

slide-12
SLIDE 12

Latent Meaning Model-Average (LMM-A)

A paradigm of LMM-A

not Latent Meaning Prefix Root Suffix it is incredible thing an SUM in capable believe able An item of the Word Map incredible not in believe able capable Word Prefix Root Suffix 1/ 5 1/ 5 1/ 5 1/ 5 1/ 5

12

Sequence of tokens The latent meanings of ’s morphemes have equal contributions to The modified embedding of : is utilized for training

: a set of latent meanings of ’s morphemes : the length of

ACL2018

𝒖𝒋 π‘«π’‘π’π’–π’‡π’šπ’–(𝒖𝒋)

slide-13
SLIDE 13

Latent Meaning Model-Similarity (LMM-S)

not Latent Meaning Prefix Root Suffix it is incredible thing an SUM in capable believe able An item of the Word Map incredible not in believe able capable Word Prefix Root Suffix

? in ? not ? believe ? capable ? able

A paradigm of LMM-S

13

The latent meanings of ’s morphemes are assigned with different weights: The modified embedding of :

: a set of latent meanings of ’s morphemes

𝝏<π’–π’Œ, 𝒙> = cos(π’˜π’–π’Œ, π’˜π’™) βˆ‘π’šβˆˆπ‘΅π’Œ cos(π’˜π’–π’Œ, π’˜π’š) , 𝒙 ∈ π‘΅π’Œ Sequence of tokens ACL2018

𝒖𝒋 π‘«π’‘π’π’–π’‡π’šπ’–(𝒖𝒋)

slide-14
SLIDE 14

Latent Meaning Model-Max (LMM-M)

not Latent Meaning Prefix Root Suffix it is incredible thing an SUM in capable believe able An item of the Word Map incredible not in believe able capable Word Prefix Root Suffix

? not ? believe ? able

A paradigm of LMM-M

14

Keep the latent meanings that have maximum similarities to : The modified embedding of : π‘π‘˜

𝑛𝑏𝑦 = {π‘„π‘˜ 𝑛𝑏𝑦, π‘†π‘˜ 𝑛𝑏𝑦, π‘‡π‘˜ 𝑛𝑏𝑦}

π‘„π‘˜

𝑛𝑏𝑦 = 𝑏𝑠𝑕max π‘₯ 𝑑𝑝𝑑(π‘€π‘’π‘˜, 𝑀π‘₯), π‘₯ ∈ π‘„π‘˜

π‘†π‘˜

𝑛𝑏𝑦 = 𝑏𝑠𝑕max π‘₯ 𝑑𝑝𝑑(π‘€π‘’π‘˜, 𝑀π‘₯), π‘₯ ∈ π‘†π‘˜

π‘‡π‘˜

𝑛𝑏𝑦 = 𝑏𝑠𝑕max π‘₯ 𝑑𝑝𝑑(π‘€π‘’π‘˜, 𝑀π‘₯), π‘₯ ∈ π‘‡π‘˜

Sequence of tokens ACL2018

𝒖𝒋 π‘«π’‘π’π’–π’‡π’šπ’–(𝒖𝒋)

slide-15
SLIDE 15

Update Rules for LMMs

15

New Objective Function (After modifying the input layer of CBOW): ^ 𝑀 = 1 π‘œ

π‘œ

βˆ‘

𝑗=1

logπ‘ž(𝑀𝑒𝑗| βˆ‘

π‘’π‘˜βˆˆπ·π‘π‘œπ‘’π‘“π‘¦π‘’(𝑒𝑗)

^ π‘€π‘’π‘˜) All parameters introduced by our models can be directly derived using the word map and word embeddings Update not just but the embeddings of the latent meanings with the same weights as they are assigned in the forward propagation period ACL2018

slide-16
SLIDE 16

03 Experimental Setup

16

ACL2018

slide-17
SLIDE 17

Corpus & Word Map

17

Corpus Word Map

  • News corpus of 2009 (2013 ACL

Eighth Workshop)

  • Size: 1.7GB
  • ~500 million tokens
  • ~600,000 words
  • Digits & punctuation marks are

filtered

  • Morpheme segmentation using

Morefessor (Creutz & Lagus, 2007)

  • Assign latent meanings
  • Lookup table

β–Ί derived from the resources provided by Michigan State University* β–Ί 90 prefixes, 382 roots, 67 suffixes *Resources web link: https://msu.edu/~defores1/gre/roots/gre_rts_afx1.htm

ACL2018

slide-18
SLIDE 18

Baselines & Parameter Settings

18

Baselines:

Word-level models: CBOW, Skip-gram, GloVe Explicitly Morpheme-related Model (EMM)

Morphemes Prefix Root Suffix it is incredible thing an SUM in ible cred

A paradigm of EMM Super-parameter Settings:

Equal settings to all models Vector Dimension: 200 Context window size: 5 #Negative_Samples: 20

ACL2018

slide-19
SLIDE 19

Evaluation Benchmarks (1/2)

19

Word Similarity: Syntactic Analogy:

β€œa b as c ? (d) ” e.g., Queen King as Woman (Man) Microsoft Research Syntactic Analogies dataset (8000 items)

Name #Pairs Name #Pairs Name #Pairs RG-65 65 Rare-Word 2034 Men-3k 3000 Wordsim-353 353 SCWS 2003 WS-353-Related 252 Dataset

Gold Standard Datasets Widely-used Datasets

ACL2018

slide-20
SLIDE 20

20

Evaluation Benchmarks (2/2)

Text Classification: 20 Newsgroups dataset (19000 documents of 20 different topics) 4 text classification tasks, each involves 10 topics Training/Validation/Test subsets (6:2:2) Feature vector: average word embedding of words in each document L2-regularized logistic regression classifier ACL2018

slide-21
SLIDE 21

04 Experimental Results

21

ACL2018

slide-22
SLIDE 22

The Results on Word Similarity

22

(Given different models) Spearman’s rank correlation (%) on different datasets

CBOW Skip-gram GloVe EMM LMM-A LMM-S LMM-M Wordsim-353 58.77 61.94 49.40 60.01 62.05 63.13 61.54 Rare-Word 40.58 36.42 33.40 40.83 43.12 42.14 40.51 RG-65 56.50 62.81 59.92 60.85 62.51 62.49 63.07 SCWS 63.13 60.20 47.98 60.28 61.86 61.71 63.02 Men-3k 68.07 66.30 60.56 66.76 66.26 68.36 64.65 WS-353-Related 49.72 57.05 47.46 54.48 56.14 58.47 55.19

ACL2018

slide-23
SLIDE 23

The Results on Syntactic Analogy

23

Syntactic analogy performance (%)

CBOW Skip-gram GloVe EMM LMM-A LMM-S LMM-M Syntactic Analogy 13.46 13.14 13.94 17.34 20.38 17.59 18.30

ACL2018

Question: β€œa b as c (d) ” Answer:

slide-24
SLIDE 24

The Results on Text Classification

24

Average text classification accuracy across the 4 tasks (%)

CBOW Skip-gram GloVe EMM LMM-A LMM-S LMM-M Text Classification 78.26 79.40 77.01 80.00 80.67 80.59 81.28

ACL2018

slide-25
SLIDE 25

The Impact of Corpus Size

25

Results on Wordsim-353 task with different corpus size ACL2018

slide-26
SLIDE 26

The Impact of Context Window Size

26

Results on Wordsim-353 task with different context window size ACL2018

slide-27
SLIDE 27

Word Embedding Visualization

27

Visualization of word embeddings based on PCA

β˜’ latent meanings of morphemes

ACL2018

slide-28
SLIDE 28

05 Conclusions

28

ACL2018

slide-29
SLIDE 29

Conclusions

  • Employ latent meanings of morphemes rather than the internal

compositions themselves to train word embeddings

  • By modifying the input layer and update rules of CBOW, we

proposed three latent meaning models (LMM-A, LMM-S, LMM-M)

  • The comprehensive quality of word embedings are enhanced by

incorporating latent meanings of morphemes

  • In the future, we intend to evaluate our models for some

morpheme-rich languages like Russian, German, etc.

29

ACL2018

slide-30
SLIDE 30

Questions?

Thank you!

ACL2018