Lecture 7: Word Embeddings Kai-Wei Chang CS @ University of - - PowerPoint PPT Presentation

Lecture 7: Word Embeddings

Kai-Wei Chang CS @ University of Virginia kw@kwchang.net Couse webpage: http://kwchang.net/teaching/NLP16

1 6501 Natural Language Processing

This lecture

v Learning word vectors (Cont.) v Representation learning in NLP

2 6501 Natural Language Processing

Recap: Latent Semantic Analysis

v Data representation

vEncode single-relational data in a matrix

v Co-occurrence (e.g., from a general corpus) v Synonyms (e.g., from a thesaurus)

v Factorization

vApply SVD to the matrix to find latent components

v Measuring degree of relation

vCosine of latent vectors

Recap: Mapping to Latent Space via SVD v SVD generalizes the original data

v Uncovers relationships not explicit in the thesaurus v Term vectors projected to 𝑙-dim latent space

v Word similarity: cosine of two column vectors in 𝚻𝐖$

𝑫

𝐕

𝐖'

≈

𝑒×𝑜 𝑒×𝑙 𝑙×𝑙 𝑙×𝑜

𝚻

Low rank approximation

v Frobenius norm. C is a 𝑛×𝑜 matrix

||𝐷||/ = 1 1 |𝑑34|5

6 478 9 378

v Rank of a matrix.

vHow many vectors in the matrix are independent to each other

6501 Natural Language Processing 5

Low rank approximation

v Low rank approximation problem: min

= ||𝐷 − 𝑌||/ 𝑡. 𝑢. 𝑠𝑏𝑜𝑙 𝑌 = 𝑙

v If I can only use k independent vectors to describe the points in the space, what are the best choices?

6501 Natural Language Processing 6

Essentially, we minimize the “reconstruction loss” under a low rank constraint

Low rank approximation

v Low rank approximation problem: min

= ||𝐷 − 𝑌||/ 𝑡. 𝑢. 𝑠𝑏𝑜𝑙 𝑌 = 𝑙

v If I can only use k independent vectors to describe the points in the space, what are the best choices?

6501 Natural Language Processing 7

Essentially, we minimize the “reconstruction loss” under a low rank constraint

Low rank approximation

v Assume rank of 𝐷 is r v SVD: 𝐷 = 𝑉Σ𝑊', Σ = diag(𝜏

8,𝜏5 … 𝜏P, 0,0,0, …0)

v Zero-out the r − 𝑙 trailing values Σ′ = diag(𝜏

8, 𝜏5 …𝜏U, 0,0,0,… 0)

v 𝐷V = UΣV𝑊' is the best k-rank approximation: CV = 𝑏𝑠𝑕 min

= ||𝐷 − 𝑌||/ 𝑡.𝑢. 𝑠𝑏𝑜𝑙 𝑌 = 𝑙

6501 Natural Language Processing 8

Σ = 𝜏8 ⋱ 𝑠 non-zeros

Word2Vec

v LSA: a compact representation of co-

• ccurrence matrix

v Word2Vec:Predict surrounding words (skip-gram) vSimilar to using co-occurrence counts Levy&Goldberg

(2014), Pennington et al. (2014)

v Easy to incorporate new words

• r sentences

6501 Natural Language Processing 9

Word2Vec

v Similar to language model, but predicting next word is not the goal.

v Idea: words that are semantically similar often

• ccur near each other in text

v Embeddings that are good at predicting neighboring words are also good at representing similarity

6501 Natural Language Processing 10

Skip-gram v.s Continuous bag-of-words v What differences?

6501 Natural Language Processing 11

Skip-gram v.s Continuous bag-of-words

6501 Natural Language Processing 12

Objective of Word2Vec (Skip-gram)

v Maximize the log likelihood of context word 𝑥\]9, 𝑥\]9^8, …, 𝑥\]8 , 𝑥\^8, 𝑥\^5, …, 𝑥\^9 given word 𝑥\ v m is usually 5~10

6501 Natural Language Processing 13

Objective of Word2Vec (Skip-gram)

v How to model log 𝑄(𝑥\^4|𝑥\)? 𝑞 𝑥\^4 𝑥\ =

cde (fghij ⋅ lgh) ∑ cde (fgn ⋅ lgh)

gn

vsoftmax function Again!

v Every word has 2 vectors

v𝑤p : when 𝑥 is the center word v𝑣p: when 𝑥 is the outside word (context word)

6501 Natural Language Processing 14

How to update?

𝑞 𝑥\^4 𝑥\ =

cde (fghij ⋅ lgh) ∑ cde (fgn ⋅ lgh)

gn

v How to minimize 𝐾(𝜄)

vGradient descent! vHow to compute the gradient?

6501 Natural Language Processing 15

Recap: Calculus

6501 Natural Language Processing 16

v Gradient:

𝒚' = 𝑦8 𝑦5 𝑦z , ∇𝜚 𝒚 = 𝜖𝜚(𝒚) 𝜖𝑦8 𝜖𝜚(𝒚) 𝜖𝑦5 𝜖𝜚(𝒚) 𝜖𝑦z

v 𝜚 𝒚 = 𝒃 ⋅ 𝒚 (or represented as 𝒃'𝒚)

∇𝜚 𝒚 = 𝒃

Recap: Calculus

6501 Natural Language Processing 17

v If 𝑧 = 𝑔 𝑣 and 𝑣 = 𝑕 𝑦 (i.e,. 𝑧 = 𝑔(𝑕 𝑦 )

ƒ„ ƒ… = ƒ†(f) ƒf ƒ‡(…) ƒ…

(

ƒ„ ƒf ƒf ƒ… )

• 1. 𝑧 = 𝑦ˆ + 6 z
• 2. y = ln

(𝑦5 + 5)

• 3. y = exp(x• + 3𝑦 + 2)

Other useful formulation

v 𝑧 = exp 𝑦 dy dx = exp x v y = log x dy dx = 1 x

6501 Natural Language Processing 18

When I say log (in this course), usually I mean ln

6501 Natural Language Processing 19

Example

v Assume vocabulary set is 𝑋. We have one center word 𝑑, and one context word 𝑝. v What is the conditional probability 𝑞 𝑝 𝑑 𝑞 𝑝 𝑑 = exp (𝑣• ⋅ 𝑤–) ∑ exp (𝑣pn ⋅ 𝑤–)

pV

v What is the gradient of the log likelihood w.r.t 𝑤–?

𝜖 log 𝑞 𝑝 𝑑 𝜖𝑤– = 𝑣• − 𝐹p∼™ 𝑥 𝑑 [𝑣p]

6501 Natural Language Processing 20

Gradient Descent

min

p 𝐾(𝑥)

Update w: 𝑥 ← 𝑥 − 𝜃∇𝐾(𝑥)

6501 Natural Language Processing 21

Local minimum v.s. global minimum

6501 Natural Language Processing 22

Stochastic gradient descent

v Let 𝐾 𝑥 = 8

6 ∑

𝐾4(𝑥)

6 478

v Gradient descent update rule:

𝑥 ← 𝑥 −

ž 6 ∑

𝛼𝐾4 𝑥

6 478

v Stochastic gradient descent:

v Approximate 8

6∑

𝛼𝐾4 𝑥

6 478

by the gradient at a single example 𝛼𝐾3 𝑥 (why?) v At each step:

6501 Natural Language Processing 23

Randomly pick an example 𝑗 𝑥 ← 𝑥 − 𝜃𝛼𝐾3 𝑥

Negative sampling

v With a large vocabulary set, stochastic gradient descent is still not enough (why?)

𝜖 log𝑞 𝑝 𝑑 𝜖𝑤– = 𝑣• − 𝐹p∼™ 𝑥 𝑑 [𝑣p]

vLet’s approximate it again!

vOnly sample a few words that do not appear in the context vEssentially, put more weights on positive samples

6501 Natural Language Processing 24

More about Word2Vec – relation to LSA v LSA factorizes a matrix of co-occurrence counts v (Levy and Goldberg 2014) proves that skip-gram model implicitly factorizes a (shifted) PMI matrix! v PMI(w,c) =log

¡(„|…) ¡(„) = log ¡(…,„) ™(…)¡(„)

= log # 𝑥, 𝑑 ⋅ |𝐸| #(𝑥)#(𝑑)

6501 Natural Language Processing 25

All problem solved?

6501 Natural Language Processing 26

Continuous Semantic Representations

sunny rainy windy cloudy car wheel cab sad joy emotion feeling

6501 Natural Language Processing 27

Semantics Needs More Than Similarity

Tomorrow will be rainy. Tomorrow will be sunny.

𝑡𝑗𝑛𝑗𝑚𝑏𝑠(rainy, sunny)? 𝑏𝑜𝑢𝑝𝑜𝑧𝑛(rainy, sunny)?

6501 Natural Language Processing 28

Polarity Inducing LSA [Yih, Zweig, Platt 2012]

v Data representation

vEncode two opposite relations in a matrix using “polarity”

v Synonyms & antonyms (e.g., from a thesaurus)

v Factorization

vApply SVD to the matrix to find latent components

v Measuring degree of relation

vCosine of latent vectors

joy gladden sorrow sadden goodwill Group 1: “joyfulness” 1 1

• 1
• 1

Group 2: “sad”

• 1
• 1

1 1

Group 3: “affection”

1

Encode Synonyms & Antonyms in Matrix

v Joyfulness: joy, gladden; sorrow, sadden v Sad: sorrow, sadden; joy, gladden Inducing polarity Cosine Score: + 𝑇𝑧𝑜𝑝𝑜𝑧𝑛𝑡 Target word: row- vector

joy gladden sorrow sadden goodwill Group 1: “joyfulness” 1 1

• 1
• 1

Group 2: “sad”

• 1
• 1

1 1

Group 3: “affection”

1

Encode Synonyms & Antonyms in Matrix

v Joyfulness: joy, gladden; sorrow, sadden v Sad: sorrow, sadden; joy, gladden Inducing polarity Cosine Score: − 𝐵𝑜𝑢𝑝𝑜𝑧𝑛𝑡 Target word: row- vector

Continuous representations for entities

6501 Natural Language Processing 32

? Michelle Obama Democratic Party George W Bush Laura Bush Republic Party

Continuous representations for entities

6501 Natural Language Processing 33

• Useful resources for NLP applications
• Semantic Parsing & Question Answering
• Information Extraction