Lecture 6: Representing Words Kai-Wei Chang CS @ UCLA - - PowerPoint PPT Presentation

Lecture 6: Representing Words

Kai-Wei Chang CS @ UCLA kw@kwchang.net Couse webpage: https://uclanlp.github.io/CS269-17/

1 ML in NLP

Bag-of-Words with N-grams

v N-grams: a contiguous sequence of n tokens from a given piece of text

CS 6501: Natural Language Processing 2

http://recognize-speech.com/language-model/n-gram-model/comparison

Language model

v Probability distributions over sentences (i.e., word sequences ) P(W) = P(𝑥"𝑥#𝑥$𝑥% … 𝑥') v Can use them to generate strings P(𝑥' ∣ 𝑥#𝑥$𝑥% … 𝑥')") v Rank possible sentences

v P(“Today is Tuesday”) > P(“Tuesday Today is”) v P(“Today is Tuesday”) > P(“Today is Los Angeles”)

CS 6501: Natural Language Processing 3

N-Gram Models

v Unigram model: 𝑄 𝑥" 𝑄 𝑥# 𝑄 𝑥$ … 𝑄(𝑥,) v Bigram model: 𝑄 𝑥" 𝑄 𝑥#|𝑥" 𝑄 𝑥$|𝑥# … 𝑄(𝑥,|𝑥,)") v Trigram model: 𝑄 𝑥" 𝑄 𝑥#|𝑥" 𝑄 𝑥$|𝑥#, 𝑥" … 𝑄(𝑥,|𝑥,)"𝑥,)#) v N-gram model: 𝑄 𝑥" 𝑄 𝑥#|𝑥" … 𝑄(𝑥,|𝑥,)"𝑥,)# … 𝑥,)0)

CS 6501: Natural Language Processing 4

Random language via n-gram

v http://www.cs.jhu.edu/~jason/465/PowerPo int/lect01,3tr-ngram-gen.pdf v https://research.googleblog.com/2006/08/al l-our-n-gram-are-belong-to-you.html

CS 6501: Natural Language Processing 5

Collection of n-gram

N-Gram Viewer

ML in NLP 6

https://books.google.com/ngrams

How to represent words?

v N-gram -- cannot capture word similarity v Word clusters

v Brown Clustering v Part-of-speech tagging

v Continuous space representation

v Word embedding

ML in NLP 7

Brown Clustering

v Similar to language model But, basic unit is “word clusters”

v Intuition: similar words appear in similar context

v Recap: Bigram Language Models

v 𝑄 𝑥1, 𝑥", 𝑥#, … , 𝑥, = 𝑄 𝑥" 𝑥1 𝑄 𝑥# 𝑥" … 𝑄 𝑥, 𝑥,)" = Π56"

7

P(w: ∣ 𝑥:)")

8 6501 Natural Language Processing

𝑥1 is a dummy word representing ”begin of a sentence”

Motivation example

v ”a dog is chasing a cat”

v 𝑄 𝑥1, “𝑏”, ”𝑒𝑝𝑕”, … , “𝑑𝑏𝑢” = 𝑄 ”𝑏” 𝑥1 𝑄 ”𝑒𝑝𝑕” ”𝑏” … 𝑄 ”𝑑𝑏𝑢” ”𝑏”

v Assume Every word belongs to a cluster

9 6501 Natural Language Processing

Cluster 3 a the Cluster 46 dog cat fox rabbit bird boy Cluster 64 is was Cluster 64 chasing following biting…

Motivation example

v Assume every word belongs to a cluster v “a dog is chasing a cat”

10 6501 Natural Language Processing

Cluster 3 a the Cluster 46 dog cat fox rabbit bird boy Cluster 64 is was Cluster 8 chasing following biting… C3 C46 C64 C8 C3 C46

Motivation example

v Assume every word belongs to a cluster v “a dog is chasing a cat”

11 6501 Natural Language Processing

Cluster 3 a the Cluster 46 dog cat fox rabbit bird boy Cluster 64 is was Cluster 8 chasing following biting… C3 C46 C64 C8 C3 C46 a dog is chasing a cat

Motivation example

v Assume every word belongs to a cluster v “the boy is following a rabbit”

12 6501 Natural Language Processing

Cluster 3 a the Cluster 46 dog cat fox rabbit bird boy Cluster 64 is was Cluster 8 chasing following biting… C3 C46 C64 C8 C3 C46 the boy is following a rabbit

Motivation example

v Assume every word belongs to a cluster v “a fox was chasing a bird”

13 6501 Natural Language Processing

Cluster 3 a the Cluster 46 dog cat fox rabbit bird boy Cluster 64 is was Cluster 8 chasing following biting… C3 C46 C64 C8 C3 C46 a fox was chasing a bird

Brown Clustering

v Let 𝐷 𝑥 denote the cluster that 𝑥 belongs to v “a dog is chasing a cat”

14 6501 Natural Language Processing

Cluster 3 a the Cluster 46 dog cat fox rabbit bird boy Cluster 64 is was Cluster 8 chasing following biting… C3 C46 C64 C8 C3 C46 a dog is chasing a cat

P(C(dog)|C(a)) P(cat|C(cat))

Brown clustering model

v P(“a dog is chasing a cat”)

= P(C(“a”)|𝐷1) P(C(“dog”)|C(“a”)) P(C(“dog”)|C(“a”))… P(“a”|C(“a”))P(“dog”|C(“dog”))...

15 6501 Natural Language Processing

Cluster 3 a the Cluster 46 dog cat fox rabbit bird boy Cluster 64 is was Cluster 8 chasing following biting… C3 C46 C64 C8 C3 C46 a dog is chasing a cat

P(C(dog)|C(a)) P(cat|C(cat))

Brown clustering model

v P(“a dog is chasing a cat”)

= P(C(“a”)|𝐷1) P(C(“dog”)|C(“a”)) P(C(“dog”)|C(“a”))… P(“a”|C(“a”))P(“dog”|C(“dog”))... v In general

𝑄 𝑥1, 𝑥", 𝑥#, … , 𝑥, = 𝑄 𝐷(𝑥") 𝐷 𝑥1 𝑄 𝐷(𝑥#) 𝐷(𝑥") … 𝑄 𝐷 𝑥, 𝐷 𝑥,)" 𝑄(𝑥"|𝐷 𝑥" 𝑄 𝑥# 𝐷 𝑥# … 𝑄(𝑥,|𝐷 𝑥, ) = Π56"

7

P 𝐷 w: 𝐷 𝑥:)" 𝑄(𝑥: ∣ 𝐷 𝑥: )

16 6501 Natural Language Processing

Model parameters

𝑄 𝑥1, 𝑥", 𝑥#, … , 𝑥, = Π56"

7

P 𝐷 w: 𝐷 𝑥:)" 𝑄(𝑥: ∣ 𝐷 𝑥: )

17 6501 Natural Language Processing Cluster 3 a the Cluster 46 dog cat fox rabbit bird boy Cluster 64 is was Cluster 8 chasing following biting… C3 C46 C64 C8 C3 C46 a dog is chasing a cat

Parameter set 1: 𝑄(𝐷(𝑥:)|𝐷 𝑥:)" ) Parameter set 2: 𝑄(𝑥:|𝐷 𝑥: ) Parameter set 3: 𝐷 𝑥:

Model parameters

𝑄 𝑥1, 𝑥", 𝑥#, … , 𝑥, = Π56"

7

P 𝐷 w: 𝐷 𝑥:)" 𝑄(𝑥: ∣ 𝐷 𝑥: ) v A vocabulary set 𝑋 v A function 𝐷: 𝑋 → {1, 2, 3, … 𝑙 }

v A partition of vocabulary into k classes

v Conditional probability 𝑄(𝑑′ ∣ 𝑑) for 𝑑, 𝑑N ∈ 1, … , 𝑙 v Conditional probability 𝑄(𝑥 ∣ 𝑑) for 𝑑, 𝑑N ∈ 1, … , 𝑙 , 𝑥 ∈ 𝑑

18 6501 Natural Language Processing

𝜄 represents the set of conditional probability parameters C represents the clustering

Log likelihood

LL(𝜄, 𝐷 ) = log 𝑄 𝑥1, 𝑥", 𝑥#, … , 𝑥, 𝜄, 𝐷 = log Π56"

7

P 𝐷 w: 𝐷 𝑥:)" 𝑄(𝑥: ∣ 𝐷 𝑥: ) = ∑56"

7

[log P 𝐷 w: 𝐷 𝑥:)" + log 𝑄(𝑥: ∣ 𝐷 𝑥: ) ]

v Maximizing LL(𝜄, 𝐷) can be done by

alternatively update 𝜄 and 𝐷

• 1. max

\∈] 𝑀𝑀(𝜄, 𝐷)

• 2. max

_ 𝑀𝑀(𝜄, 𝐷)

19 6501 Natural Language Processing

max

\∈] 𝑀𝑀(𝜄, 𝐷)

LL(𝜄, 𝐷 ) = log 𝑄 𝑥1, 𝑥", 𝑥#, … , 𝑥, 𝜄, 𝐷 = log Π56"

7

P 𝐷 w: 𝐷 𝑥:)" 𝑄(𝑥: ∣ 𝐷 𝑥: ) = ∑56"

7

[log P 𝐷 w: 𝐷 𝑥:)" + log 𝑄(𝑥: ∣ 𝐷 𝑥: ) ]

v 𝑄(𝑑′ ∣ 𝑑) = #(ab,a)

#a

v 𝑄(𝑥 ∣ 𝑑) = #(c,a)

#a

6501 Natural Language Processing 20

This part is the same as training a POS tagging model

See section 9.2: http://ciml.info/dl/v0_99/ciml-v0_99-ch09.pdf

max

_ 𝑀𝑀(𝜄, 𝐷)

max

_ ∑56" 7

[log P 𝐷 w: 𝐷 𝑥:)" + log 𝑄(𝑥: ∣ 𝐷 𝑥: ) ] = n ∑ ∑ 𝑞 𝑑, 𝑑N log

e a,ab e a e(ab) + 𝐻 ' aN6" ' a6"

where G is a constant

v Here,

𝑞 𝑑, 𝑑N =

# a,ab ∑ #(a,ab)

h,hb

, 𝑞 𝑑 =

# a ∑ #(a)

h

v 𝑑: cluster of w:, 𝑑′:cluster of w:)"

v

e a,ab e a e(ab) = e 𝑑 𝑑N e a

(mutual information)

21 6501 Natural Language Processing

See classnote here: http://web.cs.ucla.edu/~kwchang/teaching /NLP16/slides/classnote.pdf

Algorithm 1

v Start with |V| clusters each word is in its own cluster v The goal is to get k clusters v We run |V|-k merge steps:

v Pick 2 clusters and merge them v Each step pick the merge maximizing 𝑀𝑀(𝜄, 𝐷)

v Cost? (can be improved to 𝑃( 𝑊 $)) O(|V|-k) 𝑃( 𝑊 #) 𝑃 ( 𝑊 #) = 𝑃( 𝑊 k) #Iters #pairs compute LL

6501 Natural Language Processing 22

Algorithm 2

v m : a hyper-parameter, sort words by frequency v Take the top m most frequent words, put each of them in its own cluster 𝑑", 𝑑#, 𝑑$, … 𝑑l v For 𝑗 = 𝑛 + 1 … |𝑊|

v Create a new cluster 𝑑lo" (we have m+1 clusters) v Choose two cluster from m+1 clusters based on

𝑀𝑀 𝜄, 𝐷 and merge ⇒ back to m clusters

v Carry out (m-1) final merges ⇒ full hierarchy v Running time O 𝑊 𝑛# + 𝑜 , n=#words in corpus

6501 Natural Language Processing 23

Example clusters (Brown+1992)

6501 Natural Language Processing 24

Example Hierarchy(Miller+2004)

6501 Natural Language Processing 25