lecture 8 word clustering
play

Lecture 8: Word Clustering Kai-Wei Chang CS @ University of - PowerPoint PPT Presentation

Feb 16, 2024 β€’252 likes β€’493 views

Lecture 8: Word Clustering Kai-Wei Chang CS @ University of Virginia kw@kwchang.net Couse webpage: http://kwchang.net/teaching/NLP16 6501 Natural Language Processing 1 This lecture v Brown Clustering 6501 Natural Language Processing 2

  1. Lecture 8: Word Clustering Kai-Wei Chang CS @ University of Virginia kw@kwchang.net Couse webpage: http://kwchang.net/teaching/NLP16 6501 Natural Language Processing 1

  2. This lecture v Brown Clustering 6501 Natural Language Processing 2

  3. Brown Clustering v Similar to language model But, basic unit is β€œword clusters” v Intuition: again, similar words appear in similar context v Recap: Bigram Language Models v 𝑄 π‘₯ # , π‘₯ % , π‘₯ & , … , π‘₯ ( = 𝑄 π‘₯ % π‘₯ # 𝑄 π‘₯ & π‘₯ % … 𝑄 π‘₯ ( π‘₯ (+% / = Ξ  -.% P(w 3 ∣ π‘₯ 3+% ) π‘₯ # is a dummy word representing ”begin of a sentence” 6501 Natural Language Processing 3

  4. Motivation example v ”a dog is chasing a cat” v 𝑄 π‘₯ # , β€œπ‘β€, ”𝑒𝑝𝑕”, … , β€œπ‘‘π‘π‘’β€ = 𝑄 ”𝑏” π‘₯ # 𝑄 ”𝑒𝑝𝑕” ”𝑏” … 𝑄 ”𝑑𝑏𝑒” ”𝑏” v Assume Every word belongs to a cluster Cluster 46 Cluster 64 Cluster 3 Cluster 64 dog cat chasing a is fox rabbit following the was bird boy biting… 6501 Natural Language Processing 4

  5. Motivation example v Assume every word belongs to a cluster v β€œa dog is chasing a cat” C3 C8 C46 C3 C46 C64 Cluster 46 Cluster 8 Cluster 3 Cluster 64 dog cat chasing a is fox rabbit following the was bird boy biting… 6501 Natural Language Processing 5

  6. Motivation example v Assume every word belongs to a cluster v β€œa dog is chasing a cat” C3 C8 C46 C3 C46 C64 a dog is chasing a cat Cluster 46 Cluster 8 Cluster 3 Cluster 64 dog cat chasing a is fox rabbit following the was bird boy biting… 6501 Natural Language Processing 6

  7. Motivation example v Assume every word belongs to a cluster v β€œthe boy is following a rabbit” C3 C8 C46 C3 C46 C64 the boy is following a rabbit Cluster 46 Cluster 8 Cluster 3 Cluster 64 dog cat chasing a is fox rabbit following the was bird boy biting… 6501 Natural Language Processing 7

  8. Motivation example v Assume every word belongs to a cluster v β€œa fox was chasing a bird” C3 C8 C46 C3 C46 C64 a fox was chasing a bird Cluster 46 Cluster 8 Cluster 3 Cluster 64 dog cat chasing a is fox rabbit following the was bird boy biting… 6501 Natural Language Processing 8

  9. Brown Clustering v Let 𝐷 π‘₯ denote the cluster that π‘₯ belongs to v β€œa dog is chasing a cat” C3 C8 C46 C3 C46 C64 a dog is chasing a cat P(cat|C(cat)) P(C(dog)|C(a)) Cluster 46 Cluster 8 Cluster 3 Cluster 64 dog cat chasing a is fox rabbit following the was bird boy biting… 6501 Natural Language Processing 9

  10. Brown clustering model v P(β€œa dog is chasing a cat”) = P(C(β€œa”)| 𝐷 # ) P(C(β€œdog”)|C(β€œa”)) P(C(β€œdog”)|C(β€œa”))… P(β€œa”|C(β€œa”))P(β€œdog”|C(β€œdog”))... C3 C8 C46 C3 C46 C64 a dog is chasing a cat P(cat|C(cat)) P(C(dog)|C(a)) Cluster 46 Cluster 8 Cluster 3 Cluster 64 dog cat chasing a is fox rabbit following the was bird boy biting… 6501 Natural Language Processing 10

  11. Brown clustering model v P(β€œa dog is chasing a cat”) = P(C(β€œa”)| 𝐷 # ) P(C(β€œdog”)|C(β€œa”)) P(C(β€œdog”)|C(β€œa”))… P(β€œa”|C(β€œa”))P(β€œdog”|C(β€œdog”))... v In general 𝑄 π‘₯ # , π‘₯ % , π‘₯ & , … , π‘₯ ( = 𝑄 𝐷(π‘₯ % ) 𝐷 π‘₯ # 𝑄 𝐷(π‘₯ & ) 𝐷(π‘₯ % ) … 𝑄 𝐷 π‘₯ ( 𝐷 π‘₯ (+% 𝑄(π‘₯ % |𝐷 π‘₯ % 𝑄 π‘₯ & 𝐷 π‘₯ & … 𝑄(π‘₯ ( |𝐷 π‘₯ ( ) / = Ξ  -.% P 𝐷 w 3 𝐷 π‘₯ 3+% 𝑄(π‘₯ 3 ∣ 𝐷 π‘₯ 3 ) 6501 Natural Language Processing 11

  12. Model parameters 𝑄 π‘₯ # , π‘₯ % , π‘₯ & , … , π‘₯ ( / = Ξ  -.% P 𝐷 w 3 𝐷 π‘₯ 3+% 𝑄(π‘₯ 3 ∣ 𝐷 π‘₯ 3 ) Parameter set 2: Parameter set 1: 𝑄(π‘₯ 3 |𝐷 π‘₯ 3 ) 𝑄(𝐷(π‘₯ 3 )|𝐷 π‘₯ 3+% ) C8 C3 C46 C3 C46 C64 a dog is chasing a cat Parameter set 3: Cluster 46 Cluster 8 Cluster 3 Cluster 64 𝐷 π‘₯ 3 dog cat chasing a is fox rabbit following the was bird boy biting… 6501 Natural Language Processing 12

  13. Model parameters 𝑄 π‘₯ # , π‘₯ % , π‘₯ & , … , π‘₯ ( / = Ξ  -.% P 𝐷 w 3 𝐷 π‘₯ 3+% 𝑄(π‘₯ 3 ∣ 𝐷 π‘₯ 3 ) v A vocabulary set 𝑋 v A function 𝐷: 𝑋 β†’ {1, 2, 3, … 𝑙 } v A partition of vocabulary into k classes v Conditional probability 𝑄(𝑑′ ∣ 𝑑) for 𝑑, 𝑑 J ∈ 1,… , 𝑙 v Conditional probability 𝑄(π‘₯ ∣ 𝑑) for 𝑑, 𝑑 J ∈ 1,… , 𝑙 ,π‘₯ ∈ 𝑑 πœ„ represents the set of conditional probability parameters C represents the clustering 6501 Natural Language Processing 13

  14. Log likelihood LL( πœ„, 𝐷 ) = log 𝑄 π‘₯ # , π‘₯ % , π‘₯ & ,… , π‘₯ ( πœ„, 𝐷 / = log Ξ  -.% P 𝐷 w 3 𝐷 π‘₯ 3+% 𝑄(π‘₯ 3 ∣ 𝐷 π‘₯ 3 ) / = βˆ‘ -.% [log P 𝐷 w 3 𝐷 π‘₯ 3+% + log𝑄(π‘₯ 3 ∣ 𝐷 π‘₯ 3 ) ] v Maximizing LL( πœ„, 𝐷 ) can be done by alternatively update πœ„ and 𝐷 1. max X∈Y 𝑀𝑀(πœ„,𝐷) 2. max [ 𝑀𝑀(πœ„,𝐷) 6501 Natural Language Processing 14

  15. max X∈Y 𝑀𝑀(πœ„, 𝐷) LL( πœ„, 𝐷 ) = log 𝑄 π‘₯ # , π‘₯ % , π‘₯ & , … , π‘₯ ( πœ„, 𝐷 / = log Ξ  -.% P 𝐷 w 3 𝐷 π‘₯ 3+% 𝑄(π‘₯ 3 ∣ 𝐷 π‘₯ 3 ) / = βˆ‘ -.% [log P 𝐷 w 3 𝐷 π‘₯ 3+% + log𝑄(π‘₯ 3 ∣ 𝐷 π‘₯ 3 ) ] #(] ^ ,]) v 𝑄(𝑑′ ∣ 𝑑) = #] #(_,]) v 𝑄(π‘₯ ∣ 𝑑) = #] 6501 Natural Language Processing 15

  16. max [ 𝑀𝑀(πœ„, 𝐷) / max [ βˆ‘ -.% [log P 𝐷 w 3 𝐷 π‘₯ 3+% + log𝑄(π‘₯ 3 ∣ 𝐷 π‘₯ 3 ) ] π‘ž 𝑑, 𝑑 J log a ],] ^ c c = n βˆ‘ βˆ‘ a ] a(] ^ ) + 𝐻 ].% ]J.% where G is a constant v Here, # ],] ^ π‘ž 𝑑, 𝑑 J = # ] , π‘ž 𝑑 = #(],] ^ ) βˆ‘ βˆ‘ #(]) d d,d^ a 𝑑 𝑑 J a ],] ^ a ] a(] ^ ) = v (mutual information) a ] 6501 Natural Language Processing 16

  17. max [ 𝑀𝑀(πœ„, 𝐷) / max [ βˆ‘ -.% [log P 𝐷 w 3 𝐷 π‘₯ 3+% + log𝑄(π‘₯ 3 ∣ 𝐷 π‘₯ 3 ) ] π‘ž 𝑑, 𝑑 J log a ],] ^ c c = n βˆ‘ βˆ‘ a ] a(] ^ ) + 𝐻 ].% ]J.% 6501 Natural Language Processing 17

  18. Algorithm 1 v Start with |V| clusters each word is in its own cluster v The goal is to get k clusters v We run |V|-k merge steps: v Pick 2 clusters and merge them v Each step pick the merge maximizing 𝑀𝑀(πœ„, 𝐷) v Cost? (can be improved to 𝑃( π‘Š g ) ) O(|V|-k) 𝑃( π‘Š & ) 𝑃 ( π‘Š & ) = 𝑃( π‘Š h ) #Iters #pairs compute LL 6501 Natural Language Processing 18

  19. Algorithm 2 v m : a hyper-parameter, sort words by frequency v Take the top m most frequent words, put each of them in its own cluster 𝑑 % , 𝑑 & ,𝑑 g ,… 𝑑 i v For 𝑗 = 𝑛 + 1 … |π‘Š| v Create a new cluster 𝑑 il% (we have m+1 clusters) v Choose two cluster from m+1 clusters based on 𝑀𝑀 πœ„, 𝐷 and merge β‡’ back to m clusters v Carry out (m-1) final merges β‡’ full hierarchy v Running time O π‘Š 𝑛 & + π‘œ , n=#words in corpus 6501 Natural Language Processing 19

  20. Example clusters (Brown+1992) 6501 Natural Language Processing 20

  21. Example Hierarchy (Miller+2004) 6501 Natural Language Processing 21

  22. Quiz 1 v 30 min (9/20 Tue. 12:30pm-1:00pm) v Fill-in-the-blank, True/False v Short answer v Closed book, Closed notes, Closed laptop v Sample questions: v Add one smoothing v.s. Add-Lambda Smoothing v 𝑏 = 1,3,5 , 𝑐 = 2,3,6 what is the cosine similarity between a and 𝑐 ? 6501 Natural Language Processing 22

  23. 6501 Natural Language Processing 23

Recommend

Lecture 23: Spectral clustering Hierarchical clustering What is a good clustering?

Lecture 23: Spectral clustering Hierarchical clustering What is a good clustering?

Lecture 23: Spectral clustering Hierarchical clustering What is a good clustering? Aykut Erdem May 2016 Hacettepe University Last time K-Means An iterative clustering algorithm - Initialize: Pick K random points as cluster

933 views β€’ 64 slides
Lecture 20 Lecture 20 Nov 12 th 2008 Clustering with Mixture of Gaussians Clustering with Mixture

Lecture 20 Lecture 20 Nov 12 th 2008 Clustering with Mixture of Gaussians Clustering with Mixture

Lecture 20 Lecture 20 Nov 12 th 2008 Clustering with Mixture of Gaussians Clustering with Mixture of Gaussians Underlying model for data: assumes a generative process: Each cluster follows a Gaussian distribution First choose which

302 views β€’ 19 slides
Web Information Retrieval Lecture 15 Clustering Todays Topic: Clustering Document

Web Information Retrieval Lecture 15 Clustering Todays Topic: Clustering Document

Web Information Retrieval Lecture 15 Clustering Todays Topic: Clustering Document clustering Motivations Document representations Success criteria Clustering algorithms Partitional Hierarchical Ch. 16 What is

767 views β€’ 61 slides
LECTURE 7 Clustering The k-means algorithm Hierarchical Clustering The DBSCAN algorithm

LECTURE 7 Clustering The k-means algorithm Hierarchical Clustering The DBSCAN algorithm

DATA MINING LECTURE 7 Clustering The k-means algorithm Hierarchical Clustering The DBSCAN algorithm Clustering Evaluation What is a Clustering? In general a grouping of objects such that the objects in a group (cluster) are similar (or

1.39k views β€’ 110 slides
LECTURE 8 The EM Algorithm Clustering Validation Sequence segmentation CLUSTERING What is a

LECTURE 8 The EM Algorithm Clustering Validation Sequence segmentation CLUSTERING What is a

DATA MINING LECTURE 8 The EM Algorithm Clustering Validation Sequence segmentation CLUSTERING What is a Clustering? In general a grouping of objects such that the objects in a group (cluster) are similar (or related) to one another and

998 views β€’ 78 slides
Clustering Lecture 8 David Sontag New York University

Clustering Lecture 8 David Sontag New York University

Clustering Lecture 8 David Sontag New York University Slides adapted from Luke Zettlemoyer, Vibhav Gogate, Carlos Guestrin, Andrew Moore, Dan Klein Clustering Clustering: Unsupervised learning

1.22k views β€’ 56 slides
Clustering Lecture 14 David Sontag New York University

Clustering Lecture 14 David Sontag New York University

Clustering Lecture 14 David Sontag New York University Slides adapted from Luke Zettlemoyer, Vibhav Gogate, Carlos Guestrin, Andrew Moore, Dan Klein Clustering Clustering: Unsupervised learning

559 views β€’ 33 slides
Spectral Clustering Lecture 16 David Sontag New York

Spectral Clustering Lecture 16 David Sontag New York

Spectral Clustering Lecture 16 David Sontag New York University Slides adapted from James Hays, Alan Fern, and Tommi Jaakkola Spectral clustering K-means Spectral clustering twocircles, 2

848 views β€’ 19 slides
Hierarchical Clustering Lecture 15 David Sontag New York

Hierarchical Clustering Lecture 15 David Sontag New York

Hierarchical Clustering Lecture 15 David Sontag New York University Agglomerative Clustering Agglomerative clustering: First merge very similar instances Incrementally build larger clusters out

465 views β€’ 7 slides
DATA MINING LECTURE 9 The EM Algorithm Clustering Evaluation Sequence segmentation CLUSTERING

DATA MINING LECTURE 9 The EM Algorithm Clustering Evaluation Sequence segmentation CLUSTERING

DATA MINING LECTURE 9 The EM Algorithm Clustering Evaluation Sequence segmentation CLUSTERING What is a Clustering? In general a grouping of objects such that the objects in a group (cluster) are similar (or related) to one another and

905 views β€’ 79 slides
Word processing Lecture 9 COMPSCI111/111G Todays lecture u Storing information using ASCII

Word processing Lecture 9 COMPSCI111/111G Todays lecture u Storing information using ASCII

Word processing Lecture 9 COMPSCI111/111G Todays lecture u Storing information using ASCII u Word processor basics: u File formats u WYSIWYG u Basic features of a word processor: u Font and paragraphs u Styles u Headers, footers, footnotes,

886 views β€’ 34 slides
Lecture 10 Jan-Willem van de Meent Clustering Clustering Unsupervised learning (no labels

Lecture 10 Jan-Willem van de Meent Clustering Clustering Unsupervised learning (no labels

Unsupervised Machine Learning and Data Mining DS 5230 / DS 4420 - Fall 2018 Lecture 10 Jan-Willem van de Meent Clustering Clustering Unsupervised learning (no labels for training) Group data into similar classes that Maximize

1.29k views β€’ 69 slides
Lecture 11 Jan-Willem van de Meent Clustering Clustering Unsupervised learning (no labels

Lecture 11 Jan-Willem van de Meent Clustering Clustering Unsupervised learning (no labels

Unsupervised Machine Learning and Data Mining DS 5230 / DS 4420 - Fall 2018 Lecture 11 Jan-Willem van de Meent Clustering Clustering Unsupervised learning (no labels for training) Group data into similar classes that Maximize

833 views β€’ 26 slides
CSCE 478/878 Lecture 8: Stephen Scott Clustering Introduction Outline Clustering Stephen

CSCE 478/878 Lecture 8: Stephen Scott Clustering Introduction Outline Clustering Stephen

CSCE 478/878 Lecture 8: Clustering CSCE 478/878 Lecture 8: Stephen Scott Clustering Introduction Outline Clustering Stephen Scott k -Means Clustering Hierarchical Clustering sscott@cse.unl.edu 1 / 19 Introduction CSCE 478/878 If

200 views β€’ 19 slides
Lecture 12: Clustering Geoffrey Hinton Clustering We assume that the data was generated from

Lecture 12: Clustering Geoffrey Hinton Clustering We assume that the data was generated from

CSC321: Neural Networks Lecture 12: Clustering Geoffrey Hinton Clustering We assume that the data was generated from a number of different classes. The aim is to cluster data from the same class together. How do we decide the number of

529 views β€’ 21 slides
Lecture: k-means & mean-shift clustering Juan Carlos Niebles and Ranjay Krishna Stanford

Lecture: k-means & mean-shift clustering Juan Carlos Niebles and Ranjay Krishna Stanford

Clustering Lecture: k-means & mean-shift clustering Juan Carlos Niebles and Ranjay Krishna Stanford Vision and Learning Lab 22-Oct-2019 1 St Stanfor ord University CS 131 Roadmap Clustering Pixels Segments Images Videos Web

769 views β€’ 55 slides
Clustering Lecture notes Clustering is Exploratory, unsupervised method Data in cluster is

Clustering Lecture notes Clustering is Exploratory, unsupervised method Data in cluster is

Clustering Lecture notes Clustering is Exploratory, unsupervised method Data in cluster is similar to each other, and dissimilar to other cluster data Finding structure in data Hard vs. Soft/Fuzzy vs. Exclusive Understanding and/or summarise

400 views β€’ 26 slides
Hierarchical)&)Spectral)clustering) Lecture)13) David&Sontag&

Hierarchical)&)Spectral)clustering) Lecture)13) David&Sontag&

Hierarchical)&)Spectral)clustering) Lecture)13) David&Sontag& New&York&University& Slides adapted from Luke Zettlemoyer, Vibhav Gogate, Carlos Guestrin, Andrew Moore, Dan Klein Agglomerative Clustering Agglomerative

496 views β€’ 25 slides
Lecture 12: Clustering 1 6.0002 LECTURE 12 Re Reading Chapter 23 6.0002 LECTURE 12 2 Mach Ma

Lecture 12: Clustering 1 6.0002 LECTURE 12 Re Reading Chapter 23 6.0002 LECTURE 12 2 Mach Ma

Lecture 12: Clustering 1 6.0002 LECTURE 12 Re Reading Chapter 23 6.0002 LECTURE 12 2 Mach Ma chine e Lea earn rning Paradigm Observe set of examples: training data Infer something about process that generated that data Use

847 views β€’ 36 slides
Clustering Hierarchical clustering, k-mean clustering Genome 559: Introduction to Statistical and

Clustering Hierarchical clustering, k-mean clustering Genome 559: Introduction to Statistical and

Clustering Hierarchical clustering, k-mean clustering Genome 559: Introduction to Statistical and Computational Genomics Elhanan Borenstein A quick review The clustering problem: partition genes into distinct sets with high homogeneity

772 views β€’ 40 slides
1 K-means clustering The K-means clustering algorithm can be seen as applying the EM algorithm to

1 K-means clustering The K-means clustering algorithm can be seen as applying the EM algorithm to

Statistical Modeling and Analysis of Neural Data (NEU 560) Princeton University, Spring 2018 Jonathan Pillow Lecture 18 notes: K-means and Factor Analysis Tues, 4.17 1 K-means clustering The K-means clustering algorithm can be seen as

548 views β€’ 3 slides
Deconstructing Data Science David Bamman, UC Berkeley Info 290 Lecture 5: Clustering

Deconstructing Data Science David Bamman, UC Berkeley Info 290 Lecture 5: Clustering

Deconstructing Data Science David Bamman, UC Berkeley Info 290 Lecture 5: Clustering overview Feb 3, 2016 Clustering Clustering (and unsupervised learning more generally) finds structure in data, using just X X = a set of

768 views β€’ 59 slides
Deconstructing Data Science David Bamman, UC Berkeley Info 290 Lecture 5: Clustering

Deconstructing Data Science David Bamman, UC Berkeley Info 290 Lecture 5: Clustering

Deconstructing Data Science David Bamman, UC Berkeley Info 290 Lecture 5: Clustering overview Jan 31, 2016 Clustering Clustering (and unsupervised learning more generally) finds structure in data, using just X X = a set of

797 views β€’ 57 slides
Graph Clustering Graph Clustering What is clustering? What is clustering? Finding patterns

Graph Clustering Graph Clustering What is clustering? What is clustering? Finding patterns

Graph Clustering Graph Clustering What is clustering? What is clustering? Finding patterns in data, or grouping similar groups of data-points together into clusters . Clustering algorithms for numeric data: Lloyds K-means, EM

688 views β€’ 19 slides

More recommend