Natural Language Processing with Deep Learning Word Embeddings - - PowerPoint PPT Presentation

Institute of Computational Perception

Natural Language Processing with Deep Learning Word Embeddings

Navid Rekab-Saz

navid.rekabsaz@jku.at

Agenda

• Introduction
• Count-based word representation
• Prediction-based word embedding

Agenda

• Introduction
• Count-based word representation
• Prediction-based word embedding

4

Distributional Representation § An entity is represented with a vector of 𝑒 dimensions § Distributed Representations

• Each dimension (units) is a feature of the entity
• Units in a layer are not mutually exclusive
• Two units can be ‘‘active’’ at the same time

𝑦! 𝑦" 𝑦# … 𝑦$

𝒚

𝑒

5

Word Embedding Model 𝑤1 𝑤2 𝑤𝑂 𝑒

Word Embedding Model

When vector representations are dense, they are often called embedding e.g. word embedding

Word embeddings projected to a two-dimensional space

7

Word Embeddings – Nearest neighbors

frog

frogs toad litoria leptodactylidae rana

book

books foreword author published preface

asthma

bronchitis allergy allergies arthritis diabetes

https://nlp.stanford.edu/projects/glove/

8

Word Embeddings – Linear substructures § Analogy task:

• man to woman is like king to ? (queen)

𝒚!"#$% − 𝒚#$% + 𝒚&'%( = 𝒚∗ 𝒚∗ ≈ 𝒚*+,-.

https://nlp.stanford.edu/projects/glove/

Agenda

• Introduction
• Count-based word representation
• Prediction-based word embedding

10

Intuition for Computational Semantics

“You shall know a word by the company it keeps!”

• J. R. Firth, A synopsis of

linguistic theory 1930–1955 (1957)

11

Nida[1975]

Tesgüino

d r i n k f e r m e n t e d b

• t

t l e

• ut of corn

s a c r e d beverage Mexico alcoholic

12

Ale

d r i n k b a r g r a i n medieval pale b

• t

t l e brew fermentation alcoholic

13

Tesgüino ←→ Ale

Algorithmic intuition:

Two words are related when they have common context words

14

Word-Document Matrix – recap

§ 𝔼 is a set of documents (plays of Shakespeare) 𝔼 = [𝑒1, 𝑒2, … , 𝑒𝑁] § 𝕎 is the set of words (vocabularies) in dictionary 𝕎 = [𝑤1, 𝑤2, … , 𝑤𝑂] § Words as rows and documents as columns § Values: term count tc!,# § Matrix size 𝑂×𝑁 𝑒1 𝑒2 𝑒3 𝑒4

As You Like It Twelfth Night Julius Caesar Henry V battle 1 1 8 15 soldier 2 2 12 36 fool 37 58 1 5 clown 6 117 ... ... … ... ...

15

Cosine § Cosine is the normalized dot product of two vectors

• Its result is between -1 and +1

cos(𝒚, 𝒛) = 𝒚 𝒚 / . 𝒛 𝒛 / = ∑0 𝑦0𝑧0 ∑0 𝑦0

/

∑0 𝑧0

/

§ 𝒚 = 1 1 0 𝒛 = 4 5 6 cos(𝒚, 𝒛) = 1 ∗ 4 + 1 ∗ 5 + 0 ∗ 6 1$ + 1$ + 0$ 4$ + 5$ + 6$ = 9 ~12.4

16

𝑒1 𝑒2 𝑒3 𝑒4

As You Like It Twelfth Night Julius Caesar Henry V battle 1 1 8 15 soldier 2 2 12 36 fool 37 58 1 5 clown 6 117 ... ... … ... ...

Word-Document Matrix § Similarity between two words: similarity soldier, clown = cos 𝒚1"23',4, 𝒚52"!%

17

Context § Context can be

• Document
• Paragraph, tweet
• Window of (2-10) context words on each side of the word

§ Word-Context matrix

• Every word as a unit (dimension)

ℂ = [𝑑1, 𝑑2, … , 𝑑𝑀]

• Matrix size: 𝑂×𝑀
• Usually ℂ = 𝕎 and therefore 𝑀 = 𝑂

18

Word-Context Matrix

sugar, a sliced lemon, a tablespoonful of apricot preserve or jam, a pinch each of, their enjoyment. Cautiously she sampled her first pineapple and another fruit whose taste she likened well suited to programming on the digital computer. In finding the optimal R-stage policy from for the purpose of gathering data and information necessary for the study authorized in the

𝑑1 𝑑2 𝑑3 𝑑4 𝑑5 𝑑6

aardvark computer data pinch result sugar

𝑤1 apricot

1 1

𝑤2 pineapple

1 1

𝑤3 digital

2 1 1

𝑤4 information

1 6 4

§ Window context of 7 words

19

𝑑1 𝑑2 𝑑3 𝑑4 𝑑5 𝑑6

aardvark computer data pinch result sugar

𝑤1 apricot

1 1

𝑤2 pineapple

1 1

𝑤3 digital

2 1 1

𝑤4 information

1 6 4

Word-to-Word Relations § First-order co-occurrence relation

• Each cell of the word-context matrix
• Words that appear in the proximity of each other
• Like drink to beer, and drink and wine

§ Second-order similarity relation

• Cosine similarity between the representation vectors
• Words that appear in similar contexts
• Like beer to wine, tesgüino to ale, and frog to toad

20

Point Mutual Information § Problem with raw counting methods

• Biased towards high frequent words (“and”, “the”) although they

don’t contain much of information

§ Point Mutual Information (PMI)

• Rooted in information theory
• A better measure for the first-order relation in word-context

matrix in order to reflect the informativeness of co-occurrences

• Joint probability of two events (random variables) divided by

their marginal probabilities

PMI 𝑌, 𝑍 = log 𝑞(X, Y) 𝑞 X 𝑞(Y)

21

Point Mutual Information PMI 𝑤, 𝑑 = log 𝑞(𝑤, 𝑑) 𝑞 𝑤 𝑞(𝑑)

𝑞 𝑤, 𝑑 = # 𝑤, 𝑑 𝑇 𝑞 𝑤 = ∑%&'

(

# 𝑤, 𝑑% 𝑇 𝑞 𝑑 = ∑)&'

*

# 𝑤), 𝑑 𝑇 𝑇 = H

)&' *

H

%&' (

# 𝑤), 𝑑%

§ Positive Point Mutual Information (PPMI) PPMI 𝑢, 𝑑 = max(PMI, 0)

22

Point Mutual Information

𝑄 𝑤 = information, 𝑑 = data = Q 6 19 = .32 𝑄 𝑤 = information = Q 11 19 = .58 𝑄 𝑑 = data = Q 7 19 = .37 PPMI 𝑤 = information, 𝑑 = data = max(0, log .32 .58 ∗ .37) = .39 𝑑1 𝑑2 𝑑3 𝑑4 𝑑5 𝑑6

aardvark computer data pinch result sugar

𝑤1 apricot

1 1

𝑤2 pineapple

1 1

𝑤3 digital

2 1 1

𝑤4 information

1 6 4

24

Singular Value Decomposition - Recap § An 𝑂×𝑀 matrix 𝒀 can be factorized to three matrices:

𝒀 = 𝑽𝜯𝑾𝐔

§ 𝑽 (left singular vectors) is an 𝑂×𝑀 unitary matrix § 𝜯 is an 𝑀×𝑀 diagonal matrix, diagonal entries

• are eigenvalues,
• show the importance of corresponding 𝑀 dimensions in 𝒀
• are all positive and sorted from large to small values

§ 𝑾𝐔 (right singular vectors) is an 𝑀×𝑀 unitary matrix

* The definition of SVD is simplified. Refer to https://en.wikipedia.org/wiki/Singular_value_decomposition for the exact definition

25

Singular Value Decomposition – Recap

𝑂×𝑀 𝑂×𝑀 𝑀×𝑀 𝑀×𝑀 =

• riginal matrix

𝒀 eigenvalues 𝜯 right singular vectors 𝑾𝐔 left singular vectors 𝑽

26

Applying SVD to Word-Context Matrix

𝑂×𝑀 words contexts = 𝑂×𝑀 𝑀×𝑀 𝑀×𝑀

word vectors 𝑽 eigenvalues 𝜯 context vectors 𝑾𝐔 (sparse) word-context matrix 𝒀

§ Step 1: create a sparse PPMI matrix of the size 𝑂✕𝑀, § Apply SVD

27

Applying SVD to Term-Context Matrix § Step 2: keep only top 𝑒 eigenvalues in 𝜯 and set the rest to zero § Truncate the 𝑽 and 𝑾𝐔 matrices based on the changes in 𝜯, called N 𝑽 and N 𝑾𝐔 respectively

28

×𝑀 𝑀×𝑀 𝑀×𝑀

truncated word vectors

: 𝑽

truncated eigenvalues % 𝜯 truncated context word vectors

: 𝑾𝐔

Applying SVD to Term-Context Matrix

𝑒 𝑒 𝑂 𝑒 𝑒 𝑀

§ N 𝑽 matrix is the dense low-dimensional word vectors

Agenda

• Introduction
• Count-based word representation
• Prediction-based word embedding

30

Word Embedding with Neural Networks § Design a neural network architecture! § Loop over training data (𝑤, 𝑑) for some epochs

• Pass the word 𝑤 as input and execute forward passing
• Calculate the probability of observing context word 𝑑 at output:

𝑞(𝑑|𝑤)

• Optimize the network to maximize this likelihood probability

Recipe for creating (dense) word embedding with neural networks Details come next!

31

Training Data 𝒠

Window of size 2

http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/

32

https://web.stanford.edu/~jurafsky/slp3/

Train sample: (Tesgüino, drink)

Linear activation

Encoder embedding Decoder embedding Input Layer (One-hot encoder) Output Layer (softmax)

Neural embeddings – architecture

1×𝑂 1×𝑒 1×𝑂

𝑭𝑂×𝑒 𝑽%

𝑒×𝑂

𝑞(𝐞𝐬𝐣𝐨𝐥|𝐔𝐟𝐭𝐡ü𝐣𝐨𝐩)

Forward pass Backpropagation

33

Embedding vector Ale Tesgüino

34

Embedding vector Ale Tesgüino

35

Embedding vector Decoding vector Ale Tesgüino

36

drink Ale Tesgüino Embedding vector Decoding vector

37

drink Ale Tesgüino Embedding vector Decoding vector

38

drink

• Train sample: (Tesgüino, drink)
• Update vectors to maximize 𝑞(drink|Tesgüino)

Ale Tesgüino Embedding vector Decoding vector

39

Neural embeddings – prediction § Since hidden layer is linear, output vector is:

𝒜 = 𝒇)𝑽*

§ Probability distribution of output words using softmax: 𝑞 𝑑 𝑤 = softmax(𝒜)? = exp 𝒇@𝒗?

A

∑ ̃

?∈𝕎 exp 𝒇@𝒗 ̃ ? A

§ In this example: 𝑞 drink Tesgüino = exp 𝒇A,1(ü'%"𝒗34'%&

A

∑ ̃

?∈𝕎 exp 𝒇A,1(ü'%"𝒗 ̃ ? A

Denominator (normalization) is expensive!

40

Neural embeddings – loss § Loss function with NLL over all training samples 𝒠

ℒ = −𝔽 j,k ~𝒠 log 𝑞 𝑑 𝑤 ℒ = −𝔽 j,k ~𝒠 log exp 𝒇j𝒗k

m

∑ ̃

k∈𝕎 exp 𝒇j𝒗 ̃ k m

ℒ = −𝔽 j,k ~𝒠 𝒇j𝒗k

m − log ? ̃ k∈𝕎

exp 𝒇j𝒗 ̃

k m

41

Neural embeddings – word embedding § word2vec uses the encoding matrix 𝑭 as the final word embeddings § Other possibilities

• Mean of the vectors of 𝑭 and 𝑽 for each word
• Concatenation of the vectors in 𝑭 and 𝑽 for each word
• This results in vectors with 2𝑒 dimensions

𝑭𝑂×𝑒 𝑽%

𝑒×𝑂

42

word2vec (Skip-Gram) with Negative Sampling § Word2vec is an efficient and effective algorithm that proposes Negative Sampling method to define loss § In Negative Sampling instead of 𝑞 𝑑 𝑤 , the network estimates 𝑞 𝑧 = 1 𝑤, 𝑑 : the probability that the co-

• ccurrence of 𝑤, 𝑑 comes from a genuine distribution,

namely from training corpus § 𝑞 𝑧 = 1 𝑤, 𝑑 is defined using sigmoid 𝜏:

𝑞 𝑧 = 1 𝑤, 𝑑 = 𝜏(𝒇j C 𝒗k

m)

43

word2vec (Skip-Gram) with Negative Sampling § When two words 𝑤, 𝑑 appear in the training data (genuine distribution), it is a positive sample § Negative Sampling aims to distinguish between the co-occurrence probability of 𝑤 in a positive sample 𝑞 𝑧 = 1 𝑤, 𝑑 and the co-occurrences in 𝑙 negative samples with context words ̃ 𝑑 𝑞 𝑧 = 1 𝑤, ̃ 𝑑

44

word2vec (Skip-Gram) with Negative Sampling § Negative samples are drawn from the noisy distribution ] 𝒠

• q

𝒠 represents a randomly created corpus → words co-occur randomly!

• Why a random co-occurrence can be assumed as a negative

sample?

§ The noisy distribution ] 𝒠 is defined using a smooth unigram distribution of words in the corpus,

• In word2vec, q

𝒠 is smoothed by raising unigram counts to the power of 𝛽 = 0.75 → Context Distribution Smoothing

§ Number of negative samples 𝑙 is usually between 2 to 20

45

word2vec with Negative Sampling – Objective Function

§ The objective function

• increases the probability for the positive sample (𝑤, 𝑑)
• decreases the probability for the 𝑙 negative samples (𝑤, ̃

𝑑) § Loss function:

ℒ = −𝔽 @,? ~𝒠 log 𝑞 𝑧 = 1 𝑤, 𝑑 − `

̃ ?~I 𝒠 J K'#,1

log 𝑞 𝑧 = 1 𝑤, ̃ 𝑑 ℒ = −𝔽 @,? ~𝒠 log 𝜏 𝒇@𝒗?

A −

`

̃ ?~I 𝒠 J K'#,1

log 𝜏 𝒇@𝒗 ̃

? A positive samples negative samples

46

drink Tesgüino

• Train sample: (Tesgüino, drink)

Embedding vector Decoding vector

47

drink Tesgüino

• Train sample: (Tesgüino, drink)
• Sample 𝑙 negative context words

Embedding vector Decoding vector

48

drink Tesgüino

• Train sample: (Tesgüino, drink)
• Sample 𝑙 negative context words
• Update vectors to
• Maximize 𝑞 𝑧 = 1 Tesgüino, drink
• Minimize 𝑞(𝑧 = 1|Tesgüino, ̃

𝑑) Embedding vector Decoding vector

49

Negative Sampling – final words! § Negative Sampling turns the problem from multi-class classification to binary classification § Negative Sampling is a biased approximation of softmax

• Noisy Contrastive Estimation (the parent of Negative Sampling)

is an unbiased approximation of softmax

§ Softmax is a good choice for training language models, namely, to estimate 𝑞 𝑑 𝑤 § word2vec and Negative Sampling aim to train good embeddings