Word Embeddings - Word2Vec Fall 2020 2020-09-30 Adapted from slides - - PowerPoint PPT Presentation

Adapted from slides from Dan Jurafsky, Chris Manning, Danqi Chen and Karthik Narasimhan

Word Embeddings - Word2Vec

Fall 2020 2020-09-30 CMPT 825: Natural Language Processing

SFU NatLangLab

1

Announcements

• Homework 1 due today
• Both parts are due
• Programming component has 2 grace days, but something must be turned in by tonight
• Single person groups - highly encouraged to team up with each other
• Video Lectures
• Summary of logisBc regression (opBonal)
• Word vectors (required) - covers PPMI
• Word vectors TF-IDF (required, not yet posted) - covers TF-IDF
• Word vectors Summary (opBonal, not yet posted)
• Using SVD to get dense word vectors, and connecBons to word2vec
• TA video summarizing key points about word vectors

2

Representing words by their context

Distributional hypothesis: words that occur in similar contexts tend to have similar meanings

J.R.Firth 1957

• “You shall know a word by the company it keeps”
• One of the most successful ideas of modern statistical

NLP!

These context words will represent banking.

3

Word Vectors

4

word-word (term-context) co-occurrence matrix

• Represent words by their context
• One-hot vectors

hotel = [0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0] motel = [0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0] matrix

|V| × |V|

context

(other words in the span around the target word)

term

sugar, a sliced lemon, a tablespoonful of apricot jam, a pinch each of, their enjoyment. Cautiously she sampled her first pineapple and another fruit whose taste she likened well suited to programming on the digital computer. In finding the optimal R-stage policy from for the purpose of gathering data and information necessary for the study authorized in the

Sparse vs dense vectors

5

Vectors we get from word-word (term-context) co-occurrence matrix are

• long (length |V|= 20,000 to 50,000)
• sparse (most elements are zero)

True for both one-hot, U-idf and PPMI vectors AlternaBve: we want to represent words as

• short (50-300 dimensional)
• dense (real-valued) vectors
• The focus of this lecture
• The basis of all the modern NLP systems

Dense vectors

employees =               0.286 0.792 −0.177 −0.107 10.109 −0.542 0.349 0.271 0.487              

6

short + dense

Why dense vectors?

• Short vectors are easier to use as features in ML systems
• Dense vectors may generalize better than storing explicit counts
• They do better at capturing synonymy
• co-occurs with “car”,

co-occurs with “automobile”

w1 w2

• Different methods for getting dense vectors:
• Singular value decomposition (SVD)
• word2vec and friends: “learn” the vectors!

7

Word2vec and friends

8

Download pretrained word embeddings

Word2vec (Mikolov et al.) https://code.google.com/archive/p/word2vec/ Fasttext http://www.fasttext.cc/ Glove (Pennington, Socher, Manning) http://nlp.stanford.edu/projects/glove/

9

Word2Vec

• Popular embedding method
• Very fast to train
• Idea: predict rather than count

(Mikolov et al, 2013): Distributed Representations of Words and Phrases and their Compositionality

10

Word2Vec

• Instead of counting how often each word occurs near “apricot”
• Train a classifier on a binary prediction task:
• Is likely to show up near “apricot”?
• We don’t actually care about this task
• But we’ll take the learned classifier weights as the word embeddings

w w

11

(Bengio et al, 2003): A Neural Probabilistic Language Model

Word2Vec

Insight: use running text as implicitly supervised training data!

• A word near apricot
• Act as gold “correct answer” to the question “Is word w likely to show up

near apricot?”

• No need for hand-labeled supervision
• The idea comes from neural language modeling
• Bengio et al (2003)
• Collobert et al (2011)

s

12

Word2vec

• Input: a large text corpora, V, d
• Output:
• : a pre-defined vocabulary
• : dimension of word vectors (e.g. 300)
• Text corpora:
• Wikipedia + Gigaword 5: 6B
• Twitter: 27B
• Common Crawl: 840B

V d

vcat =     −0.224 0.130 −0.290 0.276    

vdog =     −0.124 0.430 −0.200 0.329    

vthe =     0.234 0.266 0.239 −0.199    

vlanguage =     0.290 −0.441 0.762 0.982    

f : V → Rd

13

Learn by training classifiers to predict words and take learned weights as word vectors.

f

Word2vec

Continuous Bag of Words (CBOW)

Skip-grams

14

Predict center word from context words Predict context words from center word

Skip-gram

• The idea: we want to use words to predict their context words
• Context: a fixed window of size 2

15

Skip-gram: Basic Setup

Let's represent words as vectors of some length (say 300), randomly iniBalized. So we start with 300 * V random parameters Over the enBre training set, we’d like to adjust those word vectors such that we

• Predict the probability distribuBon for how likely a word is to be a context word

16

P(c|t)

Skip-gram

17

Training sentence: ... lemon, a tablespoon of apricot jam a pinch ... c1 c2 t c3 c4

Training data: input/output pairs centering on apricot

assume a +/- 2 word window

Training data Goal

Given a tuple = target, context (apricot, jam) (apricot, aardvark) Return probability that is a real context word:

(t, c) c

P(c|t)

Skip-gram: objective function

• For each position

, predict context words within context size , given center word :

t = 1,2,…T m wj L(θ) =

T

Y

t=1

Y

mjm,j6=0

P(wt+j | wt; θ)

all the parameters to be optimized

• The objective function

is the (average) negative log likelihood:

J(θ)

J(θ) = − 1 T log L(θ) = − 1 T

T

X

t=1

X

mjm,j6=0

log P(wt+j | wt; θ)

18

How to define ?

P(wt+j ∣ wt; θ)

• We have two sets of vectors for each word in the vocabulary

ui ∈ Rd

: embedding for target word i

: embedding for context word i’

Q: Why two sets of vectors?

vi0 ∈ Rd

• Use inner product to measure how likely word i

appears with context word i’, the larger the better

“softmax” we learned last time!

ui · vi0

P(wt+j | wt) = exp(uwt · vwt+j) P

k∈V exp(uwt · vk)

“softmax” we learned last time!

are all the parameters in this model!

θ = {{uk}, {vk}}

19

Normalized over entire vocabulary

How to train the model

Calculating all the gradients together!

Q: How many parameters are in total?

) = − 1 T

T

X

t=1

X

mjm,j6=0

log P(wt+j | wt; θ)

J(θ) = −

rθJ(θ) =?

We can apply stochastic gradient descent (SGD)! Let’s walk through the math..

θ(t+1) = θ(t) ηrθJ(θ)

θ = {{uk}, {vk}}

20

Warm-up

f(x) = exp(x)

d f dx =

f(x) = log(x)

d f dx =

1 x

a

f(x) = f1(f2(x))

exp(x)

d f1(z) dz d f2(x) dx

z = f2(x)

d f dx =

chain rule:

f(x) = x · a

∂f ∂x =

∂f ∂x = [ ∂f ∂x1 , ∂f ∂x2 , . . . , ∂f ∂xn ]

21

Computing the gradients

Consider one pair of target/context words (t, c): y = − log ✓ exp(ut · vc) P

k∈V exp(ut · vk)

◆

Make sure you know how to do this!

22

Putting it together

• Input: text corpus, context size

, embedding size ,

m d V

• Initialize

randomly

ui, vi

• Walk through the training corpus and collect training data (t, c):
• Update
• Update

ut ← ut − η ∂y ∂ut

vk ← vk − η ∂y ∂vk , ∀k ∈ V

Any issues?

23

Skip-gram with negative sampling (SGNS)

Problem: every time you get one pair of (t, c), you need to update with all the words in the vocabulary! It is very computationally expensive.

vk

Negative sampling: instead of considering all the words in V, let’s randomly sample K (5-20) negative examples. softmax: NS: y = − log ✓ exp(ut · vc) P

k∈V exp(ut · vk)

◆

Q: Why is it faster?

24

Skip-gram with negative sampling (SGNS)

σ(x) = 1 1 + exp(−x)

Same as training a logistic regression for binary classification!

P(D = 1 | t, c) = σ(ut · vc)

25

Skip-gram with negative sampling (SGNS)

• 1. Treat the target word and a neighboring context word as posiBve examples.
• 2. Randomly sample other words in the lexicon to get negaBve samples
• 3. Use logisBc regression to train a classifier to disBnguish those two cases
• 4. Use the weights as the embeddings

26

Skip-gram with negative sampling

27

Training sentence: ... lemon, a tablespoon of apricot jam a pinch ... c1 c2 t c3 c4

Training data: input/output pairs centering on apricot

assume a +/- 2 word window

Training data Goal

Given a tuple = target, context (apricot, jam) (apricot, aardvark) Return probability that is a real context word:

(t, c) c

P( + |t, c) P( − |t, c) = 1 − P( + |t, c)

SGNS: Basic Setup

Let's represent words as vectors of some length (say 300), randomly iniBalized. So we start with 300 * V random parameters Over the enBre training set, we’d like to adjust those word vectors such that we

• Maximize the similarity of the target word, context word pairs (t,c) drawn

from the posiBve data

• Minimize the similarity of the (t,c) pairs drawn from the negaBve data.

28

1 . k . n . V 1.2…….j………V 1 . . . d

W C

• 1. .. … d

increase similarity( apricot , jam) wj . ck

jam apricot aardvark

decrease similarity( apricot , aardvark) wj . cn

“…apricot jam…”

neighbor word random noise word

29

+ sample

• sample

SGNS: Learning the classifier

IteraBve process. We’ll start with 0 or random weights Then adjust the word weights to

• make the posiBve pairs more likely
• and the negaBve pairs less likely
• ver the enBre training set

30

SGNS Training

positive examples + t c apricot tablespoon apricot of apricot preserves apricot or

• For each posiBve example, we'll create negaBve examples.
• Using noise words
• Any random word that isn't

K t

Training sentence:

... lemon, a tablespoon of apricot jam a pinch ... c1 c2 t c3 c4

31

Choosing noise words

Could pick according to their unigram frequency More common to chose them according to

α= ¾ works well because it gives rare noise words slightly higher probability

To show this, imagine two events and :

w P(w) Pα(w)

P

α(w) =

count(w)α P

w count(w)α

p(a) = 0.99 p(b) = 0.01

P

α(a) =

.99.75 .99.75 +.01.75 = .97 P

α(b) =

.01.75 .99.75 +.01.75 = .03

32

SGNS: objective function

We want to maximize… Maximize the + label for the pairs from the posiBve training data, and the – label for the pairs sample from the negaBve data.

X

(t,c)∈+

logP(+|t, c) + X

(t,c)∈−

logP(−|t, c)

33

Focusing on one target word t:

L(θ) = logP(+|t,c)+

k

X

i=1

logP(−|t,ni) = logσ(c·t)+

k

X

i=1

logσ(−ni ·t) = log 1 1+e−c·t +

k

X

i=1

log 1 1+eni·t

34

Skip-gram with negative sampling (SGNS)

σ(x) = 1 1 + exp(−x)

Same as training a logistic regression for binary classification!

P(D = 1 | t, c) = σ(ut · vc)

35

(see also http://jalammar.github.io/illustrated-word2vec/)

Continuous Bag of Words (CBOW)

L(θ) =

T

Y

t=1

P (wt | {wt+j}, m  j  m, j 6= 0)

¯ vt = 1 2m X

mjm,j6=0

vt+j

36

GloVe: Global Vectors

(Pennington et al, 2014): GloVe: Global Vectors for Word Representation

• Let’s take the global co-occurrence statistics: Xi,j
• Training faster
• Scalable to very large corpora

37

GloVe: Global Vectors

(Pennington et al, 2014): GloVe: Global Vectors for Word Representation

38

FastText: Sub-Word Embeddings

(Bojanowski et al, 2017): Enriching Word Vectors with Subword Information

• More to come! Contextualized word embeddings
• Similar as Skip-gram, but break words into n-grams with n = 3 to 6

where: 3-grams: <wh, whe, her, ere, re> 4-grams: <whe, wher, here, ere> 5-grams: <wher, where, here> 6-grams: <where, where>

• Replace by

ui · vj

X

g∈n-grams(wi)

ug · vj

39

Trained word embeddings available

• word2vec: https://code.google.com/archive/p/word2vec/
• GloVe: https://nlp.stanford.edu/projects/glove/
• FastText: https://fasttext.cc/

Differ in algorithms, text corpora, dimensions, cased/uncased…

40

Evaluating Word Embeddings

41

Extrinsic evaluation

• Let’s plug these word embeddings into

a real NLP system and see whether this improves performance

• Could take a long time but still the

most important evaluation metric

I

( 0.31 −0.28)( 0.01 −0.91) ( 1.87 0.03) ( −3.17 −0.18) ( 1.23 1.59)

don’t like this movie

ML model

👏

Extrinsic vs intrinsic evaluation

Intrinsic evaluation

• Evaluate on a specific/intermediate subtask
• Fast to compute
• Not clear if it really helps the downstream task

42

Intrinsic evaluation

Word similarity

Example dataset: wordsim-353 353 pairs of words with human judgement

http://www.cs.technion.ac.il/~gabr/resources/data/wordsim353/

Cosine similarity:

Metric: Spearman rank correlation

43

Intrinsic evaluation

Word Similarity

44

• Spearman rank

correlation of word vector similarities with different human judgements

• n different

datasets

• All vectors are 300-

dimensional

Intrinsic evaluation

Word analogy

man: woman king: ?

≈

arg max

i

(cos(ui, ub − ua + uc))

semantic

Chicago:Illinois Philadelphia: ?

≈

bad:worst cool: ?

≈

syntactic

http://download.tensorflow.org/data/questions-words.txt

More examples at

45

Analogy: Embeddings capture relational meaning!

46

vector(‘king’) - vector(‘man’) + vector(‘woman’) ≈ vector(‘queen’) vector(‘Paris’) - vector(‘France’) + vector(‘Italy’) ≈ vector(‘Rome’)

(slide credit: Stanford CS124, Dan Jurafsky)

Applications of Word Embeddings

47

Embeddings can help study word history!

48

Train embeddings on old books to study changes in word meaning

Will Hamilton

~30 million books, 1850-1990, Google Books data

Project 300 dimensions down into 2

(slide credit: Stanford CS124, Dan Jurafsky)

Bias in word embeddings

49

woman daughter her she girl man he him son boy

nurse carpenter

(slide credit: Stanford CS124, Dan Jurafsky)

Embeddings reflect cultural bias Biases change over time

Using word embeddings to study bias

From the historical embeddings for each decade

• 1. Compute historical biases of each word:
• Gender bias: how much closer a word is to "woman" synonyms than "man"

synonyms.

• Chinese bias: how much closer a word is to Chinese names (Wang, Yang,

Chen) than Anglo names

• 2. Correlate with occupaBonal data from historical census
• 3. Look at how all these change over Bme

50

Word embeddings quantify 100 years of gender and ethnic stereotypes, Garg et al, 2018

(slide credit: Stanford CS124, Dan Jurafsky)

Embedding bias correlates with actual occupation data

51

(slide credit: Stanford CS124, Dan Jurafsky)

Embeddings reflects gender bias in occupations across time (1910-1990)

52

(slide credit: Stanford CS124, Dan Jurafsky)

Embeddings reflect ethnic stereotypes over time

53

• Princeton trilogy experiments
• Aitudes toward ethnic groups (1933, 1951, 1969)

scores for adjecBves

• industrious, supers--ous, na-onalis-c, etc
• Cosine of Chinese name embeddings with those

adjecBve embeddings correlates with human raBngs.

avail- dif- their social ear- gender patterns con- fully day. atti-

Cultural bias over time

54

(slide credit: Stanford CS124, Dan Jurafsky)

1910 Irresponsible Envious Barbaric Aggressive Transparent Monstrous Hateful Cruel Greedy Bizarre

Cultural bias over time

55

(slide credit: Stanford CS124, Dan Jurafsky)

1910 1950 1990 Irresponsible Disorganized Inhibited Envious Outrageous Passive Barbaric Pompous Dissolute Aggressive Unstable Haughty Transparent Effeminate Complacent Monstrous Unprincipled Forceful Hateful Venomous Fixed Cruel Disobedient Active Greedy Predatory Sensitive Bizarre Boisterous Hearty

Beyond Simple Word Embeddings

56

Sense Embeddings

57

(slide credit: Stanford CS224N, Chris Manning)