CS6220: DATA MINING TECHNIQUES Matrix Data: Clustering: Part 2 - - PowerPoint PPT Presentation

CS6220: DATA MINING TECHNIQUES

Instructor: Yizhou Sun

yzsun@ccs.neu.edu October 19, 2014

Matrix Data: Clustering: Part 2

Methods to Learn

Matrix Data Set Data Sequence Data Time Series Graph & Network Classification

Decision Tree; Naïve Bayes; Logistic Regression SVM; kNN HMM Label Propagation

Clustering

K-means; hierarchical clustering; DBSCAN; Mixture Models; kernel k-means SCAN; Spectral Clustering

Frequent Pattern Mining

Apriori; FP-growth GSP; PrefixSpan

Prediction

Linear Regression Autoregression

Similarity Search

DTW P-PageRank

Ranking

PageRank

2

Matrix Data: Clustering: Part 2

• Revisit K-means
• Mixture Model and EM algorithm
• Kernel K-means
• Summary

3

Recall K-Means

• Objective function
• 𝐾 = 𝑘=1

𝑙

𝐷 𝑗 =𝑘 ||𝑦𝑗 − 𝑑

𝑘||2

• Total within-cluster variance
• Re-arrange the objective function
• 𝐾 = 𝑘=1

𝑙

𝑗 𝑥𝑗𝑘||𝑦𝑗 − 𝑑

𝑘||2

• 𝑥𝑗𝑘 ∈ {0,1}
• 𝑥𝑗𝑘 = 1, 𝑗𝑔 𝑦𝑗 𝑐𝑓𝑚𝑝𝑜𝑕𝑡 𝑢𝑝 𝑑𝑚𝑣𝑡𝑢𝑓𝑠 𝑘; 𝑥𝑗𝑘 =

0, 𝑝𝑢ℎ𝑓𝑠𝑥𝑗𝑡𝑓

• Looking for:
• The best assignment 𝑥𝑗𝑘
• The best center 𝑑

𝑘

4

Solution of K-Means

• Iterations
• Step 1: Fix centers 𝑑

𝑘, find assignment 𝑥𝑗𝑘 that

minimizes 𝐾

• => 𝑥𝑗𝑘 = 1, 𝑗𝑔 ||𝑦𝑗 − 𝑑

𝑘||2 is the smallest

• Step 2: Fix assignment 𝑥𝑗𝑘, find centers that

minimize 𝐾

• => first derivative of 𝐾 = 0
• =>

𝜖𝐾 𝜖𝑑𝑘 = −2 𝑗 𝑥𝑗𝑘(𝑦𝑗 − 𝑑 𝑘) = 0

• =>𝑑

𝑘 = 𝑗 𝑥𝑗𝑘𝑦𝑗 𝑗 𝑥𝑗𝑘

• Note 𝑗 𝑥𝑗𝑘 is the total number of objects in cluster j

5

𝐾 =

𝑘=1 𝑙 𝑗

𝑥𝑗𝑘||𝑦𝑗 − 𝑑

𝑘||2

Converges! Why?

Limitations of K-Means

• K-means has problems when clusters are
• f differing
• Sizes
• Densities
• Non-Spherical Shapes

12

Limitations of K-Means: Different Density and Size

13

Limitations of K-Means: Non-Spherical Shapes

14

Demo

• http://webdocs.cs.ualberta.ca/~yaling/Clu

ster/Applet/Code/Cluster.html

15

Connections of K-means to Other Methods

16

K-means Gaussian Mixture Model Kernel K- means

Matrix Data: Clustering: Part 2

• Revisit K-means
• Mixture Model and EM algorithm
• Kernel K-means
• Summary

17

Fuzzy Set and Fuzzy Cluster

• Clustering methods discussed so far
• Every data object is assigned to exactly one cluster
• Some applications may need for fuzzy or

soft cluster assignment

• Ex. An e-game could belong to both entertainment

and software

• Methods: fuzzy clusters and probabilistic

model-based clusters

• Fuzzy cluster: A fuzzy set S: FS : X → [0, 1]

(value between 0 and 1)

18

Probabilistic Model-Based Clustering

• Cluster analysis is to find hidden categories.
• A hidden category (i.e., probabilistic cluster) is a distribution over the data

space, which can be mathematically represented using a probability density function (or distribution function).



• Ex. categories for digital cameras sold

 consumer line vs. professional line  density functions f1, f2 for C1, C2  obtained by probabilistic clustering



A mixture model assumes that a set of observed objects is a mixture

• f instances from multiple probabilistic clusters, and conceptually

each observed object is generated independently



Our task: infer a set of k probabilistic clusters that is mostly likely to generate D using the above data generation process

19

Mixture Model-Based Clustering

• A set C of k probabilistic clusters C1, …,Ck

with probability density functions f1, …, fk, respectively, and their probabilities w1, …, wk, 𝑘 𝑥

𝑘 = 1

• Probability of an object i generated by

cluster Cj is: 𝑄(𝑦𝑗, 𝑨𝑗 = 𝐷

𝑘) = 𝑥 𝑘𝑔 𝑘(𝑦𝑗)

• Probability of i generated by the set of

cluster C is: 𝑄 𝑦𝑗 = 𝑘 𝑥

𝑘𝑔 𝑘(𝑦𝑗)

20

Maximum Likelihood Estimation

• Since objects are assumed to be

generated independently, for a data set D = {x1, …, xn}, we have, 𝑄 𝐸 =

𝑗

𝑄 𝑦𝑗 =

𝑗 𝑘

𝑥

𝑘𝑔 𝑘(𝑦𝑗)

• Task: Find a set C of k probabilistic

clusters s.t. P(D) is maximized

21

The EM (Expectation Maximization) Algorithm

• The (EM) algorithm: A framework to approach maximum

likelihood or maximum a posteriori estimates of parameters in statistical models.

• E-st

step ep assigns objects to clusters according to the current fuzzy clustering or parameters of probabilistic clusters

• 𝑥𝑗𝑘

𝑢 = 𝑞 𝑨𝑗 = 𝑘 𝜄 𝑘 𝑢, 𝑦𝑗 ∝ 𝑞 𝑦𝑗 𝐷 𝑘 𝑢, 𝜄 𝑘 𝑢 𝑞(𝐷 𝑘 𝑢)

• M-st

step ep finds the new clustering or parameters that maximize the expected likelihood

22

Case 1: Gaussian Mixture Model

• Generative model
• For each object:
• Pick its distribution component:

𝑎~𝑁𝑣𝑚𝑢𝑗 𝑥1, … , 𝑥𝑙

• Sample a value from the selected distribution:

𝑌~𝑂 𝜈𝑎, 𝜏𝑎

2

• Overall likelihood function
• 𝑀 𝐸| 𝜄 = 𝑗 𝑘 𝑥

𝑘𝑞(𝑦𝑗|𝜈𝑘, 𝜏 𝑘 2)

• Q: What is 𝜄 here?

23

Estimating Parameters

• 𝑀 𝐸; 𝜄 = 𝑗 log 𝑘 𝑥

𝑘𝑞(𝑦𝑗|𝜈𝑘, 𝜏 𝑘 2)

• Considering the first derivative of 𝜈𝑘:
• 𝜖𝑀

𝜖𝑣𝑘 = 𝑗 𝑥𝑘 𝑘 𝑥𝑘𝑞(𝑦𝑗|𝜈𝑘,𝜏𝑘

2)

𝜖𝑞(𝑦𝑗|𝜈𝑘,𝜏𝑘

2)

𝜖𝜈𝑘

• = 𝑗

𝑥𝑘𝑞(𝑦𝑗|𝜈𝑘,𝜏𝑘

2)

𝑘 𝑥𝑘𝑞(𝑦𝑗|𝜈𝑘,𝜏𝑘

2)

1 𝑞(𝑦𝑗|𝜈𝑘,𝜏𝑘

2)

𝜖𝑞(𝑦𝑗|𝜈𝑘,𝜏𝑘

2)

𝜖𝜈𝑘

• = 𝑗

𝑥𝑘𝑞(𝑦𝑗|𝜈𝑘,𝜏𝑘

2)

𝑘 𝑥𝑘𝑞(𝑦𝑗|𝜈𝑘,𝜏𝑘

2)

𝜖𝑚𝑝𝑕𝑞(𝑦𝑗|𝜈𝑘,𝜏𝑘

2)

𝜖𝑣𝑘

24

𝑥𝑗𝑘 = 𝑄(𝑎 = 𝑘|𝑌 = 𝑦𝑗, 𝜄) 𝜖𝑚(𝑦𝑗)/𝜖𝜈𝑘

Intractable!

Like weighted likelihood estimation; But the weight is determined by the parameters!

Apply EM algorithm

• An iterative algorithm (at iteration t+1)
• E(expectation)-step
• Evaluate the weight 𝑥𝑗𝑘 when 𝜈𝑘, 𝜏

𝑘, 𝑥 𝑘are given

• 𝑥𝑗𝑘

𝑢 = 𝑥𝑘

𝑢𝑞(𝑦𝑗|𝜈𝑘 𝑢,(𝜏𝑘 2)𝑢)

𝑘 𝑥𝑘

𝑢𝑞(𝑦𝑗|𝜈𝑘 𝑢,(𝜏𝑘 2)𝑢)

• M(maximization)-step
• Evaluate 𝜈𝑘, 𝜏

𝑘, 𝜕𝑘 when 𝑥𝑗𝑘’s are given that maximize the

weighted likelihood

• It is equivalent to Gaussian distribution parameter

estimation when each point has a weight belonging to each distribution

• 𝜈𝑘

𝑢+1 = 𝑗 𝑥𝑗𝑘

𝑢 𝑦𝑗

𝑗 𝑥𝑗𝑘

𝑢 ; (𝜏𝑘

2)𝑢+1 = 𝑗 𝑥𝑗𝑘

𝑢

𝑦𝑗−𝜈𝑘

𝑢 2

𝑗 𝑥𝑗𝑘

𝑢

; 𝑥

𝑘 𝑢+1 ∝ 𝑗 𝑥𝑗𝑘 𝑢

25

K-Means: A Special Case of Gaussian Mixture Model

• When each Gaussian component with

covariance matrix 𝜏2𝐽

• Soft K-means
• 𝑞 𝑦𝑗 𝜈𝑘, 𝜏2

∝ exp{− 𝑦𝑗 − 𝜈𝑘

2/𝜏2}

• When 𝜏2 → 0
• Soft assignment becomes hard assignment
• 𝑥𝑗𝑘 → 1, 𝑗𝑔 𝑦𝑗 is closest to 𝜈𝑘 (why?)

26

Distance!

Case 2: Multinomial Mixture Model

• Generative model
• For each object:
• Pick its distribution component:

𝑎~𝑁𝑣𝑚𝑢𝑗 𝑥1, … , 𝑥𝑙

• Sample a value from the selected distribution:

𝑌~𝑁𝑣𝑚𝑢𝑗 𝛾𝑎1, 𝛾𝑎2, … , 𝛾𝑎𝑛

• Overall likelihood function
• 𝑀 𝐸| 𝜄 = 𝑗 𝑘 𝑥

𝑘𝑞(𝒚𝑗|𝜸𝑘)

• 𝑘 𝑥

𝑘 = 1; 𝑚 𝛾𝑘𝑚 = 1

• Q: What is 𝜄 here?

27

Application: Document Clustering

• A vocabulary containing m words
• Each document i:
• A m-dimensional vector: 𝑑𝑗1, 𝑑𝑗2, … , 𝑑𝑗𝑛
• 𝑑𝑗𝑚 is the number of occurrence of word l

appearing in document i

• Under unigram assumption
• 𝑞 𝒚𝑗 𝜸𝑘

=

( 𝑛 𝑑𝑗𝑚)! 𝑑𝑗1!…𝑑𝑗𝑛! 𝛾𝑘1 𝑑𝑗1 … 𝛾𝑘𝑛 𝑑𝑗𝑛

28

Length of document Constant to all parameters

Example

29

Estimating Parameters

• 𝑚 𝐸; 𝜄 = 𝑗 log 𝑘 𝜕𝑘 𝑚 𝑑𝑗𝑚𝑚𝑝𝑕𝛾𝑘𝑚
• Apply EM algorithm
• E-step:
• w𝑗𝑘 =

𝑥𝑘𝑞(𝒚𝑗|𝜸𝑘) 𝑘 𝑥𝑘𝑞(𝒚𝑗|𝜸𝑘)

• M-step: maximize weighted likelihood

𝑗 𝑥𝑗𝑘 𝑚 𝑑𝑗𝑚𝑚𝑝𝑕𝛾𝑘𝑚

• 𝛾𝑘𝑚 =

𝑗 𝑥𝑗𝑘𝑑𝑗𝑚 𝑚′ 𝑗 𝑥𝑗𝑘𝑑𝑗𝑚′ ; 𝜕𝑘 ∝ 𝑗 𝑥𝑗𝑘

30

Weighted percentage of word l in cluster j

Better Way for Topic Modeling

• Topic: a word distribution
• Unigram multinomial mixture model
• Once the topic of a document is decided, all its

words are generated from that topic

• PLSA (probabilistic latent semantic analysis)
• Every word of a document can be sampled from

different topics

• LDA (Latent Dirichlet Allocation)
• Assume priors on word distribution and/or

document cluster distribution

31

Why EM Works?

• E-Step: computing a tight lower bound f of the
• riginal objective function at 𝜄𝑝𝑚𝑒
• M-Step: find 𝜄𝑜𝑓𝑥 to maximize the lower bound
• 𝑚 𝜄𝑜𝑓𝑥 ≥ 𝑔 𝜄𝑜𝑓𝑥 ≥ 𝑔(𝜄𝑝𝑚𝑒) = 𝑚(𝜄𝑝𝑚𝑒)

32

*How to Find Tight Lower Bound?

• Jensen’s inequality
• When “=” holds to get a tight lower bound?
• 𝑟 ℎ = 𝑞(ℎ|𝑒, 𝜄) (why?)

33

𝑟 ℎ : 𝑢ℎ𝑓 𝑢𝑗𝑕ℎ𝑢 𝑚𝑝𝑥𝑓𝑠 𝑐𝑝𝑣𝑜𝑒 𝑥𝑓 𝑥𝑏𝑜𝑢 𝑢𝑝 𝑕𝑓𝑢

Advantages and Disadvantages of Mixture Models

• Strength
• Mixture models are more general than partitioning
• Clusters can be characterized by a small number of parameters
• The results may satisfy the statistical assumptions of the generative

models

• Weakness
• Converge to local optimal (overcome: run multi-times w. random

initialization)

• Computationally expensive if the number of distributions is large,
• r the data set contains very few observed data points
• Need large data sets
• Hard to estimate the number of clusters

34

Matrix Data: Clustering: Part 2

• Revisit K-means
• Mixture Model and EM algorithm
• Kernel K-means
• Summary

35

Kernel K-Means

• How to cluster the following data?
• A non-linear map: 𝜚: 𝑆𝑜 → 𝐺
• Map a data point into a higher/infinite dimensional space
• 𝑦 → 𝜚 𝑦
• Dot product matrix 𝐿𝑗𝑘
• 𝐿𝑗𝑘 =< 𝜚 𝑦𝑗 , 𝜚(𝑦𝑘) >

36

Typical Kernel Functions

• Recall kernel SVM:

37

Solution of Kernel K-Means

• Objective function under new feature space:
• 𝐾 = 𝑘=1

𝑙

𝑗 𝑥𝑗𝑘||𝜚(𝑦𝑗) − 𝑑

𝑘||2

• Algorithm
• By fixing assignment 𝑥𝑗𝑘
• 𝑑

𝑘 = 𝑗 𝑥𝑗𝑘 𝜚(𝑦𝑗)/ 𝑗 𝑥𝑗𝑘

• In the assignment step, assign the data points to the

closest center

• 𝑒 𝑦𝑗, 𝑑

𝑘 =

𝜚 𝑦𝑗 −

𝑗′ 𝑥𝑗′𝑘𝜚 𝑦𝑗′ 𝑗′ 𝑥𝑗′𝑘 2

= 𝜚 𝑦𝑗 ⋅ 𝜚 𝑦𝑗 − 2

𝑗′ 𝑥𝑗′𝑘𝜚 𝑦𝑗 ⋅𝜚 𝑦𝑗′ 𝑗′ 𝑥𝑗′𝑘

+

𝑗′ 𝑚 𝑥𝑗′𝑘𝑥𝑚𝑘𝜚 𝑦𝑗′ ⋅𝜚 𝑦𝑚 ( 𝑗′ 𝑥𝑗′𝑘)^2

38

Do not really need to know 𝝔 𝒚 , 𝒄𝒗𝒖 𝒑𝒐𝒎𝒛 𝑳𝒋𝒌

Advantages and Disadvantages of Kernel K-Means

• Advantages
• Algorithm is able to identify the non-linear structures.
• Disadvantages
• Number of cluster centers need to be predefined.
• Algorithm is complex in nature and time complexity is

large.

• References
• Kernel k-means and Spectral Clustering by Max

Welling.

• Kernel k-means, Spectral Clustering and Normalized

Cut by Inderjit S. Dhillon, Yuqiang Guan and Brian Kulis.

• An Introduction to kernel methods by Colin Campbell.

39

Matrix Data: Clustering: Part 2

• Revisit K-means
• Mixture Model and EM algorithm
• Kernel K-means
• Summary

40

Summary

• Revisit k-means
• Derivative
• Mixture models
• Gaussian mixture model; multinomial mixture

model; EM algorithm; Connection to k-means

• Kernel k-means
• Objective function; solution; connection to k-

means

41