CS145: INTRODUCTION TO DATA MINING Clustering Evaluation and - - PowerPoint PPT Presentation

CS145: INTRODUCTION TO DATA MINING

Instructor: Yizhou Sun

yzsun@cs.ucla.edu November 7, 2017

Clustering Evaluation and Practical Issues

Learnt Clustering Methods

2

Vector Data Set Data Sequence Data Text Data Classification

Logistic Regression; Decision Tree; KNN; SVM; NN Naïve Bayes for Text

Clustering

K-means; hierarchical clustering; DBSCAN; Mixture Models PLSA

Prediction

Linear Regression GLM*

Frequent Pattern Mining

Apriori; FP growth GSP; PrefixSpan

Similarity Search

DTW

Evaluation and Other Practical Issues

• Evaluation of Clustering
• Model Selection
• Summary

3

Measuring Clustering Quality

• Two methods: extrinsic vs. intrinsic
• Extrinsic: supervised, i.e., the ground truth is available
• Compare a clustering against the ground truth using certain

clustering quality measure

• Ex. Purity, precision and recall metrics, normalized mutual

information

• Intrinsic: unsupervised, i.e., the ground truth is unavailable
• Evaluate the goodness of a clustering by considering how well

the clusters are separated, and how compact the clusters are

• Ex. Silhouette coefficient

4

Purity

• Let 𝑫 = 𝑑1,… , 𝑑𝐿 be the output clustering

result, 𝜵 = 𝜕1,… , 𝜕𝐾 be the ground truth clustering result (ground truth class)

• 𝑑𝑙 𝑏𝑜𝑒 𝑥𝑙 are sets of data points
• 𝑞𝑣𝑠𝑗𝑢𝑧 𝐷,Ω =

1 𝑂 σ𝑙 max 𝑘

|𝑑𝑙 ∩ 𝜕𝑘|

• Match each output cluster 𝑑𝑙 to the best ground truth

cluster 𝜕𝑘

• Examine the overlap of data points between the two

matched clusters

• Purity is the proportion of data points that are matched

5

Example

• Clustering output: cluster 1, cluster 2, and cluster 3
• Ground truth clustering result: ×’s, ◊’s, and ○’s.
• cluster 1 vs. ×’s, cluster 2 vs. ○’s, and cluster 3 vs. ◊’s

6

Normalized Mutual Information

• 𝑂𝑁𝐽 𝐷, Ω =

𝐽(𝐷,Ω) 𝐼 𝐷 𝐼(Ω)

• 𝐽 Ω, 𝐷 = σ𝑙 σ𝑘 𝑄(𝑑𝑙 ∩ 𝜕𝑘) 𝑚𝑝𝑕

𝑄(𝑑𝑙∩𝜕𝑘) 𝑄 𝑑𝑙 𝑄(𝜕𝑘)

• 𝐼 Ω = − σ𝑘 𝑄 𝜕𝑘 𝑚𝑝𝑕𝑄 𝜕𝑘

= − ෍

𝑘

|𝜕𝑘| 𝑂 𝑚𝑝𝑕 |𝜕𝑘| 𝑂

7

= σ𝑙 σ𝑘

|𝑑𝑙∩𝜕𝑘| 𝑂

𝑚𝑝𝑕

𝑂|𝑑𝑙∩𝜕𝑘| 𝑑𝑙 ⋅|𝜕𝑘|

Example

Cluster 1 Cluster 2 Cluster 3 sum crosses 5 1 2 8 circles 1 4 5 diamonds 1 3 4 sum 6 6 5 N=17

8

|𝝏𝒍 ∩ 𝒅𝒌| |𝝏𝒍| |𝒅𝒌|

NMI=0.36

Precision and Recall

• Random Index (RI) = (TP+TN)/(TP+FP+FN+TN)
• F-measure: 2P*R/(P+R)
• P = TP/(TP+FP)
• R = TP/(TP+FN)
• Consider pairs of data points:
• hopefully, two data points that are in the same cluster will be

clustered into the same cluster (TP), and two data points that are in different clusters will be clustered into different clusters (TN).

9

Same cluster Different clusters Same class TP FN Different classes FP TN

Example

Data points Output clustering Ground truth clustering (class) a 1 2 b 1 2 c 2 2 d 2 1

10

• # pairs of data points: 6
• (a, b): same class, same cluster
• (a, c): same class, different cluster
• (a, d): different class, different cluster
• (b, c): same class, different cluster
• (b, d): different class, different cluster
• (c, d): different class, same cluster

TP = 1 FP = 1 FN = 2 TN = 2 RI = 0.5 P= ½, R= 1/3, F = 0.4

Question

• If we flip the ground truth cluster labels

(2->1 and 1->2), will the evaluation results be the same?

11

Data points Output clustering Ground truth clustering (class) a 1 2 b 1 2 c 2 2 d 2 1

Evaluation and Other Practical Issues

• Evaluation of Clustering
• Model Selection
• Summary

12

Selecting K in K-means and GMM

• Selecting K is a model selection problem
• Methods
• Heuristics-based methods
• Penalty method
• Cross-validation

13

Heuristic Approaches

• For K-means, plot sum of squared error

for different k

• Bigger k always leads to smaller cost
• Knee points suggest good candidates for k

14

Penalty Method: BIC

• For model-based clustering, e.g., GMM,

choose k that can maximizes BIC

• Larger k increases the likelihood, but also

increases the penalty term: a trade-off!

15

Cross-Validation Likelihood

• The likelihood of the training data will

increase when increasing k

• Compute the likelihood on unseen data
• For each possible k
• Partition the data into training and test
• Learn the GMM related parameters on training

dataset and compute the log-likelihood on test dataset

• Repeat this multiple times to get an average value
• Select k that maximizes the average log-likelihood
• n test dataset

16

Evaluation and Other Practical Issues

• Evaluation of Clustering
• Model Selection
• Summary

17

Summary

• Evaluation of Clustering
• Purity, NMI, F-measure
• Model selection
• How to select k for k-means and GMM

18