Tengyu Ma Joint works with Sanjeev Arora, Yuanzhi Li, Yingyu Liang, - - PowerPoint PPT Presentation

Tengyu Ma

Joint works with Sanjeev Arora, Yuanzhi Li, Yingyu Liang, and Andrej Risteski Princeton University

𝑦 ∈ 𝒴

𝑤& ∈ ℝ(

complicated space Euclidean space with meaningful inner products Ø Kernel methods

)*+,-.*/01, 23/03+4

Linearly separable Ø Neural nets

0.*3+1, +15.*2 +106

Multi-class linear classifier

Vocabulary= { 60k most frequent words } ℝ788 Goal: Embedding captures semantics information (via linear algebraic operations)

Ø inner products characterize similarity

Ø similar words have large inner products

Ø differences characterize relationship

Øanalogous pairs have similar differences

Ø more?

picture: Chris Olah’s blog

Meaning of a word is determined by words it co-occurs with. (Distributional hypothesis of meaning, [Harris’54], [Firth’57])

⋯ ⋮ ⋮ ⋱ ⋮ ⋮ ⋯

word 𝑦 → 𝑤& word 𝑧 ↓

Ø Pr 𝑦, 𝑧 ≜ prob. of co-occurrences

• f 𝑦, 𝑧 in a window of size 5

Ø

𝑤&,𝑤C - a good measure of similarity of (𝑦,𝑧) [Lund-Burgess’96]

Ø 𝑤& = row of entry-wise square-root of

co-occurrence matrix [Rohde et al’05]

Ø 𝑤& = row of PMI 𝑦, 𝑧 = log

L. [&,C]

• L. & L.[C]

matrix [Church-Hanks’90] Co-occurrence matrix Pr ⋅,⋅

Ø “Linear structure” in the found 𝑤&’s :

𝑤PQRST − 𝑤RST ≈ 𝑤WXYYT − 𝑤Z[T\ ≈ 𝑤XT]^Y − 𝑤SXT_ ≈ ⋯ aunt king uncle man woman queen Algorithm [Levy-Goldberg]: (dimension-reduction version of [Church-Hanks’90]) Ø Compute PMI 𝑦, 𝑧 = log

L. [&,C]

• L. & L.[C]

Ø Take rank-300 SVD (best rank-300 approximation) of PMI Ø ⇔ Fit PMI 𝑦,𝑧 ≈ 〈𝑤&, 𝑤C〉 (with squared loss), where 𝑤& ∈ ℝ788

Ø Questions:

woman: man queen: ?

,aunt: ?

Ø Answers:

𝑙𝑗𝑜𝑕 = argmink|| 𝑤WXYYT − 𝑤P − (𝑤PQRST−𝑤RST)|| 𝑏𝑣𝑜𝑢 = argmink|| 𝑤XT]^Y − 𝑤P − (𝑤PQRST−𝑤RST)|| aunt king uncle man woman queen

Ørecurrent neural network based model [Mikolov et al’12] Øword2vec [Mikolov et al’13] :

Pr 𝑦[pq 𝑦[pr,…,𝑦[pt ∝ exp〈𝑤&yz{,1 5 𝑤&yz~ + ⋯ + 𝑤&yz€ 〉

ØGloVe [Pennington et al’14] :

log Pr [𝑦,𝑧] ≈ 𝑤&,𝑤C + 𝑡& + 𝑡C + 𝐷

Ø [Levy-Goldberg’14] (Previous slide)

PMI 𝑦,𝑧 = log

L. [&,C]

• L. & L.[C] ≈ 𝑤&,𝑤C + 𝐷

Logarithm (or exponential) seems to exclude linear algebra!

Why co-occurrence statistics + log à linear structure

[Levy-Goldberg’13, Pennington et al’14, rephrased]

Ø For most of the words 𝜓:

Pr[𝜓 ∣ 𝑙𝑗𝑜𝑕] Pr[𝜓 ∣ 𝑟𝑣𝑓𝑓𝑜] ≈ Pr[𝜓 ∣ 𝑛𝑏𝑜] Pr 𝜓 𝑥𝑝𝑛𝑏𝑜]

§ For 𝜓 unrelated to gender: LHS, RHS ≈ 1 § for 𝜓 =dress, LHS, RHS ≪ 1 ; for 𝜓 = John, LHS, RHS ≫ 1

ØIt suggests

= • PMI 𝜓, 𝑙𝑗𝑜𝑕 − PMI 𝜓, 𝑟𝑣𝑓𝑓𝑜 − PMI 𝜓, 𝑛𝑏𝑜 − PMI 𝜓, 𝑥𝑝𝑛𝑏𝑜

Ž

• ≈ 0

Ø Rows of PMI matrix has “linear structure” Ø Empirically one can find 𝑤P’s s.t. PMI 𝜓, 𝑥 ≈ 〈𝑤Ž,𝑤P〉 Ø Suggestion: 𝑤P’s also have linear structure

• log Pr 𝜓

𝑙𝑗𝑜𝑕 Pr 𝜓 𝑟𝑣𝑓𝑓𝑜 − log Pr 𝜓 𝑛𝑏𝑜 Pr 𝜓 𝑥𝑝𝑛𝑏𝑜]

Ž

• ≈ 0

M1: Why do low-dim vectors capture essence of huge co-occurrence statistics? That is, why is a low-dim fit of PMI matrix even possible? PMI 𝑦, 𝑧 ≈ 𝑤&, 𝑤C (∗) M2: Why low-dim vectors solves analogy when (∗) is only roughly true?

Ø NB: solving analogy task requires inner products of 6 pairs of word vectors, and

that “king” survives against all other words – noise is potentially an issue! 𝑙𝑗𝑜𝑕 = argmaxk|| 𝑤WXYYT − 𝑤P − (𝑤PQRST−𝑤RST) ||•

Ø Fact: low-dim word vectors have more accurate linear structure than the

rows of PMI (therefore better analogy task performance).

↑ empirical fit has 17% error

Ø NB: PMI matrix is not necessarily PSD.

M1: Why do low-dim vectors capture essence of huge co-occurrence statistics? That is, why is a low-dim fit of PMI matrix even possible? PMI 𝑦, 𝑧 ≈ 𝑤&, 𝑤C (∗)

A1: Under a generative model (named RAND-WALK) , (*) provablyholds

M2: Why low-dim vectors solves analogy when (∗) is only roughly true?

A2: (*) + isotropy of word vectors ⇒ low-dim fitting reduces noise

(Quite intuitive, though doesn’t follow Occam’s bound for PAC-learning)

Ø Hidden Markov Model: § discourse vector 𝑑_ ∈ ℝ( governs the discourse/theme/context of time 𝑢 § words 𝑥_ (observable); embedding 𝑤P• ∈ ℝ( (parameters to learn) § log-linear observation model

Pr[𝑥_ ∣ 𝑑_] ∝ exp〈𝑤P•,𝑑_〉

Ø Closely related to [Mnih-Hinton’07]

𝑑_ 𝑑_pr 𝑑_p• 𝑑_p7 𝑥_ 𝑥_pr 𝑥_p• 𝑥_p7 𝑥_p– 𝑑_p–

Ø Ideally, 𝑑_,𝑤P ∈ ℝ( should contain semantic information in its coordinates

§ E.g. (0.5, -0.3, …) could mean “0.5 gender, -0.3 age,..”

Ø But, the whole system is rotational invariant: 𝑑_,𝑤P = 〈𝑆𝑑_,𝑆𝑤P〉 Ø There should exist a rotation so that the coordinates are meaningful (back to

this later) 𝑑_ 𝑑_pr 𝑑_p• 𝑑_p7 𝑥_ 𝑥_pr 𝑥_p• 𝑥_p7 𝑥_p– 𝑑_p–

Ø Assumptions: § {𝑤P} consists of vectors drawn from 𝑡 ⋅ 𝒪(0,Id); 𝑡 is bounded scalar r.v. § 𝑑_ does a slow random walk (doesn’t change much in a window of 5) § log-linear observation model: Pr[𝑥_ ∣ 𝑑_] ∝ exp〈𝑤P•,𝑑_〉 Ø Main Theorem:

(1) log Pr 𝑥,𝑥′ = 𝑤P + 𝑤P› •/𝑒 − 2 log 𝑎 ± 𝜗 (2) log Pr 𝑥 = 𝑤P •/𝑒 − log 𝑎 ± 𝜗 (3) PMI 𝑥,𝑥› = 𝑤P,𝑤P¢ /𝑒 ± 𝜗

Ø Norm determines frequency; spatial orientation determines “meaning”

𝑑_ 𝑑_pr 𝑑_p• 𝑑_p7 𝑥_ 𝑥_pr 𝑥_p• 𝑥_p7 𝑥_p– 𝑑_p–

Fact: (2) implies that the words have power law dist.

Øword2vec [Mikolov et al’13] :

Pr 𝑥[pq 𝑥[pr,… ,𝑥[pt ∝ exp〈𝑤Pyz{,1 5 𝑤Pyz~ + ⋯ + 𝑤Pyz€ 〉

ØGloVe [Pennington et al’14] :

log Pr [𝑥,𝑥′] ≈ 𝑤P, 𝑤P¢ + 𝑡P + 𝑡P› + 𝐷

• Eq. (1)

log Pr 𝑥,𝑥› = 𝑤P + 𝑤P¢

• /𝑒 − 2log 𝑎 ± 𝜗

Ø [Levy-Goldberg’14]

PMI 𝑥,𝑥› ≈ 𝑤P,𝑤P¢ + 𝐷

• Eq. (3) PMI 𝑥, 𝑥› = 𝑤P, 𝑤P¢ /𝑒 ± 𝜗

Øword2vec [Mikolov et al’13] :

Pr 𝑥[pq 𝑥[pr,…, 𝑥[pt ∝ exp〈𝑤Pyz{,1 5 𝑤Pyz~ + ⋯+ 𝑤Pyz€ 〉

Ø Under our model, § Random walk is slow: 𝑑[pr ≈ 𝑑[p• ≈ ⋯ ≈ 𝑑[pq ≈ 𝑑 § Best estimate for current discourse 𝑑[pq :

argmax

],||]||£r

Pr 𝑑 𝑥[pr,…,𝑥t] = 𝛽 𝑤Pyz~ + ⋯+ 𝑤Pyz€

§ Prob. distribution of next word given the best guess 𝑑:

Pr[𝑥[pq ∣ 𝑑[pq = 𝛽 𝑤Pyz~ + ⋯+ 𝑤Pyz€ ] ∝ exp〈𝑤Pyz{,𝛽 𝑤Pyz~ + ⋯+ 𝑤Pyz€ 〉 ↑ max-likelihood estimate of 𝑑[pq 𝑑[p– 𝑑[pt 𝑥[p– 𝑥[pt 𝑥[pq 𝑑[pq

Pr[𝑥,𝑥›] = ¥ Pr 𝑥 𝑑] Pr 𝑥› 𝑑′] 𝑞 𝑑,𝑑› 𝑒𝑑𝑒𝑑′ = ¥ 1 𝑎]𝑎]› ⋅ exp 𝑤P,𝑑 exp〈𝑤P¢,𝑑›〉 𝑞 𝑑, 𝑑› 𝑒𝑑𝑒𝑑′ Ø Assume 𝑑 = 𝑑′ with probability 1, = ¥exp〈𝑤P + 𝑤P¢, 𝑑〉𝑞 𝑑 𝑒𝑑 = exp 𝑤P + 𝑤P¢

• /𝑒

??

This talk: window of size 2 Pr[𝑥 ∣ 𝑑] ∝ exp〈𝑤P, 𝑑〉 𝑑 𝑑′ 𝑥 𝑥′ Pr[𝑥′ ∣ 𝑑′] ∝ exp〈𝑤P¢,𝑑′〉

Ø Pr[𝑥 ∣ 𝑑] =

r §¨ ⋅ exp〈𝑤P, 𝑑〉

Ø 𝑎] = ∑ exp

〈𝑤P,𝑑〉

P

partition function

• Eq. (1) log Pr 𝑥, 𝑥› =

𝑤P + 𝑤P¢

• /𝑒 − 2 log 𝑎 ± 𝜗

spherical Gaussian vector 𝑑 Ø 𝔽 exp 𝑤,𝑑 = exp 𝑤

• /𝑒

This talk: window of size 2 Pr[𝑥 ∣ 𝑑] ∝ exp〈𝑤P, 𝑑〉 𝑑 𝑑′ 𝑥 𝑥′ Pr[𝑥′ ∣ 𝑑′] ∝ exp〈𝑤P¢,𝑑′〉

Ø Pr[𝑥 ∣ 𝑑] =

r §¨ ⋅ exp〈𝑤P, 𝑑〉

Ø 𝑎] = ∑ exp

〈𝑤P,𝑑〉

P

partition function Lemma 1: for almost all c, almost all 𝑤P , 𝑎] = 1 + 𝑝 1 𝑎

Ø Proof (sketch) : § for most 𝑑, 𝑎] concentrates around its mean § mean of 𝑎] is determined by ||𝑑||, which in turn concentrates § caveat: exp〈𝑤,𝑑〉 for 𝑤 ∼ 𝒪(0,Id) is not subgaussian, nor sub-

• exponential. ( 𝛽-Orlicz norm is not bounded for any 𝛽 > 0)
• Eq. (1) log Pr 𝑥, 𝑥› =

𝑤P + 𝑤P¢

• /𝑒 − 2 log 𝑎 ± 𝜗

Ø Proof Sketch: Ø Fixing 𝑑, to show high probability over choices of 𝑤P’s

𝑎] = • exp〈𝑤P,𝑑〉

P

= 1 + 𝑝 1 𝔽[𝑎]]

Ø 𝑨P = 〈𝑤P,𝑑〉 scalar Gaussian random variable Ø ||𝑑|| governs the mean and variance of 𝑨P. Ø ||𝑑|| in turns is concentrated

Lemma 1: for almost all c, almost all 𝑤P , 𝑎] = 1 + 𝑝 1 𝑎

Ø Question: 𝑨r,… ,𝑨T ∼ 𝒪(0,1) 𝑎 = •exp(𝑨[)

T [£r

Ø How is 𝑎 concentrated ? Ø 𝔽 𝑎] = Θ(𝑜), and 𝕎𝑏𝑠 𝑎] = O 𝑜 Ø The tail of 𝑓𝑦𝑞(𝑨[) is bad! Ø Pr exp𝑨[ > 𝑢 ≈ 𝑢² 2³4 _ Ø Claim: Pr[𝑎 > 𝔽𝑎 + 𝐷 𝑜 ⋅ log 𝑜] ≤ exp(− log• 𝑜) Ø Trick: truncate 𝑨[ at log 𝑜 and deal with the tail by union bound

Ø (sub)-Gaussian tail Pr 𝑌 > 𝑢 ≤ exp(−𝑢•/2) Ø (sub)-exponential tail Pr 𝑌 > 𝑢 ≤ exp(−𝑢/2)

Ø Proof Sketch: Ø Fixing 𝑑, we have with high probability over choices of 𝑤P’s

𝑎] = • exp〈𝑤P,𝑑〉

P

= 1 + 𝑝 1 𝔽[𝑎]]

Ø 𝑨P = 〈𝑤P,𝑑〉 scalar Gaussian random variable Ø ||𝑑|| governs the mean and variance of 𝑨P. Ø ||𝑑|| in turns is concentrated

Lemma 1: for almost all c, almost all 𝑤P , 𝑎] = 1 + 𝑝 1 𝑎

Pr[𝑥,𝑥›] = ¥ 1 𝑎]𝑎]› ⋅ exp 𝑤P + 𝑤P¢, 𝑑 𝑞 𝑑 𝑒𝑑 = 1 ± 𝑝 1 1 𝑎• ¥ exp 𝑤P + 𝑤P¢, 𝑑 𝑞 𝑑 𝑒𝑑 = 1 ± 𝑝 1 1 𝑎• exp(||𝑤P + 𝑤P¢||•/𝑒) This talk: window of size 2 Pr[𝑥 ∣ 𝑑] ∝ exp〈𝑤P, 𝑑〉 𝑑 𝑑′ 𝑥 𝑥′ Pr[𝑥′ ∣ 𝑑′] ∝ exp〈𝑤P¢,𝑑′〉

• Eq. (1) log Pr 𝑥, 𝑥› =

𝑤P + 𝑤P¢

• /𝑒 − 2 log 𝑎 ± 𝜗

Ø Pr[𝑥 ∣ 𝑑] =

r §¨ ⋅ exp〈𝑤P, 𝑑〉

Ø 𝑎] = ∑ exp

〈𝑤P,𝑑〉

P

partition function Lemma 1: for almost all c, almost all 𝑤P , 𝑎] = 1 + 𝑝 1 𝑎

Ø Our theory predicts

• Eq. (1) log Pr 𝑥, 𝑥› =

𝑤P + 𝑤P¢

• /𝑒 − 2 log 𝑎 ± 𝜗

Ø (Approximate) maximum likelihood objective (SN)

min

{·¸},º • Pr

»[𝑥,𝑥›](logPr »[𝑥, 𝑥›] − 𝑤P + 𝑤P¢

• P,P›

− 𝑍)•

Simplest word embedding method yet (fewest “knobs” to turn) Comparable performance on analogy test

Ø Our theory predicts

• Eq. (2) log Pr 𝑥 =

𝑤P •/𝑒 − log 𝑎 ± 𝜗

Ø Our theory predicts

𝑎] = 1 ± 𝑝 1 𝑎

ØUnder generative model RANK-WALK

For most of the words 𝜓: Pr[𝜓 ∣ 𝑏] Pr[𝜓 ∣ 𝑐] ≈ Pr[𝜓 ∣ 𝑑] Pr 𝜓 𝑒]

⟺

𝑤S − 𝑤¿ ≈ 𝑤] − 𝑤( ↑ semantic def. of analogy ↑ algebraic def. of analogy

Ø Beyond only solving analogy task? Ø Extracting more information from analogy/embeddings?

Extracting different meanings from word embeddings

(same team: Arora, Li, Liang, M., Risteski)

Some recent work:

Ø “Tie” can mean article of clothing, or physical act ØTie represents unrelated words tie1, tie2, etc.

Quick experiment: Take two random/unrelatedwords w1, w2 where w1 is ~100 times more frequent than w2 . Declare these to be a single word and compute its embedding in our model. Result: close to something like 0.8𝑤P~ + 0.2𝑤PÂ

Ø Mathematical explanation Ø Merge 𝑥r,𝑥• as 𝑥. Let 𝑠 =

L.[P~] L.[PÂ] > 1

Ø Then 𝑤P ≈ 𝛽𝑤P~ + 𝛾𝑤PÂ, where §

𝛽 = 1 − 𝑑r log 1 +

r Ä

≈ 1

§

𝛾 = 1 − 𝑑• log 𝑠

Ø 𝛾 > .1 even if 𝑠 = 100 ! Ø Rare meaning is not swamped, thanks to the log !

which correspond to different representative “discourses” 𝑤_[Y = 0.8𝑏r + 0.2 𝑏• + noise

↑ discourse for 𝑢𝑗𝑓r ↑ discourse for 𝑢𝑗𝑓•

Ø “Tie” can mean article of clothing, or physical act ØTie represents unrelated words tie1, tie2, etc. Ø Sparse coding for extracting different meanings: § Find 𝑛 = 2000 “discourses” 𝑏r,𝑏•,… ∈ ℝ( such that each word vector

expressed as weighted sum of at most 5 of them, plus “noise vector.” 𝑤P = 𝑦P,r𝑏r + 𝑦P,•𝑏• + …+ 𝑜𝑝𝑗𝑡𝑓 𝑦P has only 5 non-zeros

Ø Training objective: min

Æ£[S~,…,SÇ] ÈÉSÄÈY &¸

¢ È

• 𝑤P − 𝐵𝑦P
• P

Ø local search algo. [EAB’05], provable algo. [SWW’12, AGM’14, AGMM’15..]

sparse coding, as well as its use to capture different senses of words.

Atom 1978 825 231 616 1638 149 330 drowning instagram stakes membrane slapping

• rchestra

conferences suicides twitter thoroughbred mitochondria pulling philharmonic meetings

• verdose

facebook guineas cytosol plucking philharmonia seminars murder tumblr preakness cytoplasm squeezing conductor workshops poisoning vimeo filly membranes twisting symphony exhibitions commits linkedin fillies

• rganelles

bowing

• rchestras
• rganizes

stabbing reddit epsom endoplasmic slamming toscanini concerts strangulation myspace racecourse proteins tossing concertgebouw lectures gunshot tweets sired vesicles grabbing solti presentations

Representative subset of 2000 discourses (represented using their nearest words)

↑

closest words to 𝑏•7r

5 atoms that express 𝑤_[Y

Ø Atoms of discourse found are fairly fine-grained Ø Maybe 𝑏¿[Q]ËYR[È_ÄC = 𝛽 ⋅ 𝑐¿[Q^Q\C + 𝛾 ⋅ 𝑐]ËYR[È_ÄC? Ø Another layer:

min

Ì,º ÈÉSÄÈY ||𝐵 − 𝐶𝑍||•

Ø Part I: new generative model that captures semantics. Ø Provable guarantee: § log of co-occurrence matrix has low rank structure § semantic analogy ⇔ linear algebraic structure for word vectors Ø Simplistic assumptions, but good fit to reality Ø Part II: automatic way of detect word meanings

§ Hierarchical basis in the embedding space

Ø Other applications of our model/method?

Ø Each ordinate of 𝑤P means something:

𝑤ÎÏÆ = […… ,0, ……… ,1, ……… ,1, ……… ,0, …… ] 𝑤(Q^^SÄ = […… ,1, ……… ,0, ……… ,1, ……… ,0, …… ] 𝑤ÐË[TS = […… ,0, ……… ,1, ……… ,0, ……… ,1, …… ] 𝑤ÑÒÌ = […… ,1, ……… ,0, ……… ,0, ……… ,1, …… ] currency

↓

country

↓

American

↓

Chinese

↓

𝑤ÎÏÆ − 𝑤(Q^^SÄ = […… ,−1, ……… ,1, ……… ,0 …… …,0,…… ] 𝑤ÐË[TS − 𝑤ÑÒÌ = [… …,−1,…… …, 1,…… …, 0,…… …, 0,… …]

Ø On other coordinates, the values are either very small or the supports are non-

• verlapping

Ø Problem: rotational invariance – rotation of word vectors doesn’t

change the model.

𝑤ÎÏÆ = […… ,0, ……… ,1, ……… ,1, ……… ,0, …… ] 𝑤(Q^^SÄ = […… ,1, ……… ,0, ……… ,1, ……… ,0, …… ] 𝑤ÐË[TS = […… ,0, ……… ,1, ……… ,0, ……… ,1, …… ] 𝑤ÑÒÌ = […… ,1, ……… ,0, ……… ,0, ……… ,1, …… ] currency

↓

country

↓

American

↓

Chinese

↓

⋅ 𝑆

↑

sparse coefficients

↑

basis vectors

ØWith sparsity, the model is identifiable; allows overcomplete basis; is tractable

under mild assumptions. [SWW’12] [AGM’13][AAJNT’13][AGMM’14]

min

Ó 6Ô*.61, Ñ ||𝑊 − 𝑌 ⋅ 𝑆||Ö

• Ø 𝑊 contains word vectors as rows (obtained from any embedding method)

Ø Sparsity of rows of X is chosen to be 5 Ø 𝑆 contains 2000 basis vectors (as rows), each of which is 300-dim

Assuming M1 was answered, PMI 𝑥, 𝑥› = 𝑤P, 𝑤P¢ + 𝜊 (*) with large 𝜊 M2: Why low-dim vectors solves analogy when (*) is only roughly true? A2: (*) + isotropy of word vectors ⇒ low-dim fitting reduces noise

(Quite intuitive, though doesn’t follow Occam’s bound for PAC-learning)

Ø Our theory assumes that 𝑑_ does a slow random walk Ø red dot: the estimate hidden

variable 𝑑_ at time 𝑢

Ø sentence at top: the window

• f size 10 at time 𝑢

Assuming M1 was answered, PMI 𝑥, 𝑥› = 𝑤P, 𝑤P¢ + 𝜊 (*) with large 𝜊 M2: Why low-dim vectors solves analogy when (*) is only roughly true? A2: (*) + isotropy of word vectors ⇒ low-dim fitting reduces noise

(Quite intuitive, though doesn’t follow Occam’s bound for PAC-learning)