BBM406 Fundamentals of Machine Learning Lecture 21: Clustering - - PowerPoint PPT Presentation

Aykut Erdem // Hacettepe University // Fall 2019

Lecture 21:

Clustering K-Means

BBM406

Fundamentals of 
 Machine Learning

Photo by Unsplash user @foodiesfeed

Last time… Boosting

• Idea: given a weak learner, run it multiple times on (reweighted)

training data, then let the learned classifiers vote

• On each iteration t:
• weight each training example by how incorrectly it was classified
• Learn a hypothesis – ht
• A strength for this hypothesis – at
• Final classifier:
• A linear combination of the votes of the different classifiers

weighted by their strength

• Practically useful
• Theoretically interesting

2

Last time.. The AdaBoost Algorithm

3

Today

• What is clustering?
• K-means algorithm

4

What is clustering

5

Clustering

• Grouping data according to similarity

6

Clustering

• Grouping data according to similarity

7

Clustering

• Grouping data according to similarity

8

Clustering

• Grouping data according to similarity

9

e.g. archaeological dig

Clustering

• Grouping data according to similarity

10

Artifact location e.g. archaeological dig Artifact location Distance East Distance North

Clustering

• Grouping data according to similarity

11

Artifact location e.g. archaeological dig Artifact location Distance East Distance North

Clustering

• Grouping data according to similarity

12

Distance East Distance North Distance Nor e.g. archaeological dig

Clustering

• Grouping data according to similarity

13

Distance East Distance North Distance Nor e.g. archaeological dig

Clustering

• Grouping data according to similarity

14

Distance East Distance North Distance Nor e.g. archaeological dig

Clustering

• Grouping data according to similarity

15

Distance East Distance North Distance Nor e.g. archaeological dig

Clustering

• Grouping data according to similarity

16

Distance East Distance North Distance Nor e.g. archaeological dig

Clustering vs. Classification

• Grouping data according to similarity

17

Distance Nor

Predicting new labels from old labels

Clustering vs. Classification

• Grouping data according to similarity

18

Distance Nor

Predicting new labels from old labels

Family A Family B Family C

Clustering vs. Classification

• Grouping data according to similarity

19

Distance Nor

Predicting new labels from old labels

Family A Family B Family C Family A Family B Family C

Why use clustering… …instead of classification

• Exploratory data analysis

20

[Krivitsky Handcock 2008]

Why use clustering… …instead of classification

• Exploratory data analysis

21

• Classes are unspecified (unknown, changing

too quickly, expensive to label data, etc) ... when the cartoon looks so easy?

• High-dimensional data
• Big data
• Data not numerical

[Krivitsky Handcock 2008]

Why use clustering… …instead of classification

• Exploratory data analysis

22

• Classes are unspecified (unknown, changing

too quickly, expensive to label data, etc) ... when the cartoon looks so easy?

• High-dimensional data
• Big data
• Data not numerical

[Krivitsky Handcock 2008]

Datum: person Similarity: the number 


• f common interests of 


two people

Why use clustering… …instead of classification

• Exploratory data analysis

23

• Classes are unspecified (unknown, changing

too quickly, expensive to label data, etc) ... when the cartoon looks so easy?

• High-dimensional data
• Big data
• Data not numerical

[Krivitsky Handcock 2008]

Datum: a binary vector 
 specifying whether a 
 person has each
 interest Similarity: the number 


• f common interests of 


two people

Why use clustering… …instead of classification

• Exploratory data analysis
• Classes are unspecified (unknown, changing too

quickly, expensive to label data, etc)

24

Why use clustering… …instead of classification

• Exploratory data analysis
• Classes are unspecified (unknown, changing too

quickly, expensive to label data, etc)

25

Why use clustering… …instead of classification

• Exploratory data analysis
• Classes are unspecified (unknown, changing too

quickly, expensive to label data, etc)

26

... when the cartoon looks so easy?

• High-dimensional data
• Big data
• Data not numerical

[Blei 2003]

Topic Analysis

... when the cartoon looks so easy?

• High-dimensional data
• Big data
• Data not numerical

Why use clustering… …instead of classification

• Exploratory data analysis
• Classes are unspecified (unknown, changing too

quickly, expensive to label data, etc)

27

[Blei 2003]

Topic Analysis

... when the cartoon looks so easy?

• High-dimensional data
• Big data
• Data not numerical

Why use clustering… …instead of classification

• Exploratory data analysis
• Classes are unspecified (unknown, changing too

quickly, expensive to label data, etc)

28

[Blei 2003]

Topic Analysis

Datum: word Similarity: how many documents exist where two words co-occur

... when the cartoon looks so easy?

• High-dimensional data
• Big data
• Data not numerical

Why use clustering… …instead of classification

• Exploratory data analysis
• Classes are unspecified (unknown, changing too

quickly, expensive to label data, etc)

29

[Blei 2003]

Topic Analysis

Datum: binary vector indicating document

• ccurrence

Similarity: how many documents exist where two words co-occur

Why use clustering… …instead of classification

• Exploratory data analysis
• Classes are unspecified (unknown, changing too

quickly, expensive to label data, etc)

30

Why use clustering… …instead of classification

• Exploratory data analysis
• Classes are unspecified (unknown, changing too

quickly, expensive to label data, etc)

31

[Carpineto et al. 2009]

8

... when the cartoon looks so easy?

• High-dimensional data
• Big data
• Data not numerical

Document clustering

Why use clustering… …instead of classification

• Exploratory data analysis
• Classes are unspecified (unknown, changing too

quickly, expensive to label data, etc)

32

[Carpineto et al. 2009]

8

... when the cartoon looks so easy?

• High-dimensional data
• Big data
• Data not numerical

Document clustering

Datum: document Dissimilarity: distance between topic distributions

• f two documents

Why use clustering… …instead of classification

• Exploratory data analysis
• Classes are unspecified (unknown, changing too

quickly, expensive to label data, etc)

33

[Carpineto et al. 2009]

8

... when the cartoon looks so easy?

• High-dimensional data
• Big data
• Data not numerical

Document clustering

Datum: vector of topic

• ccurrences

Dissimilarity: distance between topic distributions

• f two documents

Why use clustering… …instead of classification

• Exploratory data analysis
• Classes are unspecified (unknown, changing too

quickly, expensive to label data, etc)

34

• High-dimensional data
• Big data
• Data not numerical

Why use clustering… …instead of classification

• Exploratory data analysis
• Classes are unspecified (unknown, changing too

quickly, expensive to label data, etc)

35

[Fei-Fei 2011]

Image segmentation

• High-dimensional data
• Big data
• Data not numerical

Why use clustering… …instead of classification

• Exploratory data analysis
• Classes are unspecified (unknown, changing too

quickly, expensive to label data, etc)

36

[Fei-Fei 2011]

Image segmentation

Datum: pixel Dissimilarity: difference in color + difference in location

• High-dimensional data
• Big data
• Data not numerical

Why use clustering… …instead of classification

• Exploratory data analysis
• Classes are unspecified (unknown, changing too

quickly, expensive to label data, etc)

37

[Fei-Fei 2011]

Image segmentation

Datum: pixel RGB values and pixel horizontal and vertical locations Dissimilarity: difference in color + difference in location

Clustering algorithms

• Partitioning algorithms
• Construct various partitions


and then evaluate them by 
 some criterion

• K-means
• Mixture of Gaussians
• Spectral Clustering
• Hierarchical algorithms
• Create a hierarchical decomposition 

• f the set of objects using some


criterion

• Bottom-up – agglomerative
• Top-down – divisive

38

• %;
• slide by Eric Xing

Desirable Properties of a Clustering Algorithm

• Scalability (in terms of both time and space)
• Ability to deal with different data types
• Minimal requirements for domain

knowledge to determine input parameters

• Ability to deal with noisy data
• Interpretability and usability
• Optional
• Incorporation of user-specified constraints

39

K-Means 
 Clustering

40

K-Means Clustering

Benefits

• Fast
• Conceptually straightforward
• Popular

41

K-Means: Preliminaries

42

K-Means: Preliminaries

43

Datum: Vector of continuous values

K-Means: Preliminaries

44

Datum: Vector of continuous values

Distance East Distance North

K-Means: Preliminaries

45

Datum: Vector of continuous values

1.5 6.2 Distance East Distance North

K-Means: Preliminaries

46

Datum: Vector of continuous values

1.5 6.2 Distance East Distance North x3 = (1.5, 6.2) 1.2 5.9 4.3 2.1 1.5 6.3 4.1 2.3 x1 x2 x3 xN Nor East ... Distance East North East

K-Means: Preliminaries

47

Datum: Vector of continuous values

1.5 6.2 Feature 1 Feature 2 x3 = (1.5, 6.2) 1.2 5.9 4.3 2.1 1.5 6.3 4.1 2.3 x1 x2 x3 xN Nor East ... Distance East

Feature 1 Feature 2

K-Means: Preliminaries

48

Datum: Vector of continuous values

Feature 1 Feature 2 x3 = (1.5, 6.2) 1.2 5.9 4.3 2.1 1.5 6.3 4.1 2.3 x1 x2 x3 xN Nor East ... Distance East

Feature 1 Feature 2

x3 = (x3,1, x3,2) x1 x2 x3 xN F F ... x1,1 x1,2 x2,1 x2,2 x3,1 x3,2 xN,1 xN,2 x3 = (x3,1, x3,2) Feature 1 Feature 2

K-Means: Preliminaries

49

Datum: Vector of D continuous values

Feature 1 Feature 2 1.2 5.9 4.3 2.1 1.5 6.3 4.1 2.3 x1 x2 x3 xN Nor East ... Distance East

Feature 1 Feature 2

x3 = (x3,1, x3,2) x1 x2 x3 xN F F ... x1,1 x1,2 x2,1 x2,2 x3,1 x3,2 xN,1 xN,2 x3 = (x3,1, x3,2) Feature 1 Feature 2

K-Means: Preliminaries

50

Feature 1 Feature 2 x3 = (x3,1, x3,2) x3 = (x3,1, x3,2) Feature 1 Feature 2

Datum: Vector of D continuous values

K-Means: Preliminaries

51

Feature 1 Feature 2 x3 = (x3,1, x3,2) x3 = (x3,1, x3,2) Feature 1 Feature 2

Dissimilarity: Distance as the crow flies

K-Means: Preliminaries

52

Feature 1 Feature 2 x3 = (x3,1, x3,2) x3 = (x3,1, x3,2) Feature 1 Feature 2

Dissimilarity: Distance as the crow flies

x3 x17

K-Means: Preliminaries

53

Feature 1 Feature 2 x3 = (x3,1, x3,2) x3 = (x3,1, x3,2) Feature 1 Feature 2

Dissimilarity: Distance as the crow flies

x3 x17

K-Means: Preliminaries

54

Feature 1 Feature 2 x3 = (x3,1, x3,2) x3 = (x3,1, x3,2) Feature 1 Feature 2

Dissimilarity: Euclidean distance

x3 x17

K-Means: Preliminaries

55

Feature 1 Feature 2 x3 = (x3,1, x3,2) x3 = (x3,1, x3,2) Feature 1 Feature 2

Dissimilarity: Squared Euclidean distance

x3 x17 dis(x3, x17) = (x3,1 − x17,1)2 + (x3,2 − x17,2)2

K-Means: Preliminaries

56

Feature 1 Feature 2 x3 = (x3,1, x3,2) x3 = (x3,1, x3,2) Feature 1 Feature 2

Dissimilarity: Squared Euclidean distance

x3 x17 dis(x3, x17) = (x3,1 − x17,1)2 + (x3,2 − x17,2)2 dis(x3, x17) =

D

• d=1

(x3,d − x17,d)2 For each feature For each feature

K-Means: Preliminaries

57

Feature 1 Feature 2 x3 = (x3,1, x3,2) x3 = (x3,1, x3,2) Feature 1 Feature 2

Dissimilarity

K-Means: Preliminaries

58

Feature 1 Feature 2 x3 = (x3,1, x3,2) x3 = (x3,1, x3,2) Feature 1 Feature 2

Cluster summary

K = number of clusters

K-Means: Preliminaries

59

Feature 1 Feature 2 x3 = (x3,1, x3,2) x3 = (x3,1, x3,2) Feature 1 Feature 2

Cluster summary

• K cluster centers

K-Means: Preliminaries

60

Feature 1 Feature 2 x3 = (x3,1, x3,2) x3 = (x3,1, x3,2) Feature 1 Feature 2

Cluster summary

• K cluster centers

K-Means: Preliminaries

61

Feature 1 Feature 2 x3 = (x3,1, x3,2) x3 = (x3,1, x3,2) Feature 1 Feature 2

Cluster summary

• K cluster centers

K-Means: Preliminaries

62

Feature 1 Feature 2 x3 = (x3,1, x3,2) x3 = (x3,1, x3,2) Feature 1 Feature 2

Cluster summary

• K cluster centers

µ2 µ3 µ1

K-Means: Preliminaries

63

Feature 1 Feature 2 x3 = (x3,1, x3,2) x3 = (x3,1, x3,2) Feature 1 Feature 2

Cluster summary

• K cluster centers

µ2 µ3 µ1

1 = (µ1,1, µ1,2)

K-Means: Preliminaries

64

Feature 1 Feature 2 x3 = (x3,1, x3,2) x3 = (x3,1, x3,2) Feature 1 Feature 2

Cluster summary

• K cluster centers

µ2 µ3 µ1

1 = (µ1,1, µ1,2)

µ1, µ2, . . . , µK

K-Means: Preliminaries

65

Feature 1 Feature 2 x3 = (x3,1, x3,2) x3 = (x3,1, x3,2) Feature 1 Feature 2

Cluster summary

µ1, µ2, . . . , µK

• K cluster centers
• Data assignments to clusters

K-Means: Preliminaries

66

Feature 1 Feature 2 x3 = (x3,1, x3,2) x3 = (x3,1, x3,2) Feature 1 Feature 2

Cluster summary

• Data assignments to clusters

µ1, µ

• K cluster centers
• Data assignments to clusters

µ1, µ2, . . . , µK

K-Means: Preliminaries

67

Feature 1 Feature 2 x3 = (x3,1, x3,2) x3 = (x3,1, x3,2) Feature 1 Feature 2

Cluster summary

• Data assignments to clusters

µ1, µ

• K cluster centers
• Data assignments to clusters

µ1, µ2, . . . , µK Sk = set of points in cluster k

= set of points in 
 cluster k

K-Means: Preliminaries

68

Feature 1 Feature 2 x3 = (x3,1, x3,2) x3 = (x3,1, x3,2) Feature 1 Feature 2

Cluster summary

• Data assignments to clusters

µ1, µ

• K cluster centers
• Data assignments to clusters

µ1, µ2, . . . , µK Sk = set of points in cluster k

= set of points in 
 cluster k

S1, S2, . . . , SK

K-Means: Preliminaries

69

Feature 1 Feature 2 x3 = (x3,1, x3,2) x3 = (x3,1, x3,2) Feature 1 Feature 2

Cluster summary

• Data assignments to clusters

µ1, µ

• K cluster centers
• Data assignments to clusters

µ1, µ2, . . . , µK S1, S2, . . . , SK µ1 µ3 µ2 Sk = set of points in cluster k

= set of points in 
 cluster k

K-Means: Preliminaries

70

Feature 1 Feature 2

Dissimilarity

Featur

K-Means: Preliminaries

71

Feature 1 Feature 2

Dissimilarity (global)

Featur disglobal =

K

• k=1
• n:xn∈Sk

D

• d=1

(xn,d − µk,d)2

K-Means: Preliminaries

72

Feature 1 Feature 2

Dissimilarity (global)

Featur disglobal =

K

• k=1
• n:xn∈Sk

D

• d=1

(xn,d − µk,d)2 For each cluster

K-Means: Preliminaries

73

Feature 1 Feature 2

Dissimilarity (global)

Featur disglobal =

K

• k=1
• n:xn∈Sk

D

• d=1

(xn,d − µk,d)2

• r each cluster
• r each data

For each cluster For each data
 point in the 
 kth cluster

K-Means: Preliminaries

74

Feature 1 Feature 2

Dissimilarity (global)

Featur disglobal =

K

• k=1
• n:xn∈Sk

D

• d=1

(xn,d − µk,d)2

• r each cluster
• r each data

For each cluster

• r each data

point in the kth cluster

• r each featur

For each data
 point in the 
 kth cluster For each feature

K-Means: Preliminaries

75

Feature 1 Feature 2

Dissimilarity (global)

Featur disglobal =

K

• k=1
• n:xn∈Sk

D

• d=1

(xn,d − µk,d)2

76

K-Means Algorithm

Featur

• Initialize K cluster centers
• Repeat until convergence:

✦ Assign each data point to


the cluster with the closest
 center.

✦ Assign each cluster 


center to be the mean of its cluster’s data points

77

K-Means Algorithm

Featur

• Initialize K cluster centers
• Repeat until convergence:

✦ Assign each data point to


the cluster with the closest
 center.

✦ Assign each cluster 


center to be the mean of its cluster’s data points

78

K-Means Algorithm

Featur

• Repeat until con

µk ← xn

• For k = 1,…, K

✦Randomly draw n from 


1,…,N without replacement

✦

• Repeat until convergence:

✦ Assign each data point to


the cluster with the closest
 center.

✦ Assign each cluster 


center to be the mean of its cluster’s data points

79

K-Means Algorithm

• Repeat until con

µk ← xn

• For k = 1,…, K

✦Randomly draw n from 


1,…,N without replacement

✦

• Repeat until convergence:

✦ Assign each data point to


the cluster with the closest
 center.

✦ Assign each cluster 


center to be the mean of its cluster’s data points

80

K-Means Algorithm

• Repeat until con

µk ← xn

• For k = 1,…, K

✦Randomly draw n from 


1,…,N without replacement

✦

• Repeat until convergence:

✦ Assign each data point to


the cluster with the closest
 center.

✦ Assign each cluster 


center to be the mean of its cluster’s data points

81

K-Means Algorithm

• For k = 1,…, K

✦ Randomly draw n from 


1,…,N without replacement

✦

• Repeat until convergence:

✦ Assign each data point to


the cluster with the closest
 center.

✦ Assign each cluster 


center to be the mean of its cluster’s data points

µk ← xn

• Repeat until con

82

K-Means Algorithm

• For k = 1,…, K

✦ Randomly draw n from 


1,…,N without replacement

✦

• Repeat until S1,…,Sk don’t

change:

✦ Assign each data point to


the cluster with the closest
 center.

✦ Assign each cluster 


center to be the mean of its cluster’s data points

µk ← xn

• Repeat until con

83

K-Means Algorithm

• For k = 1,…, K

✦ Randomly draw n from 


1,…,N without replacement

✦

• Repeat until S1,…,Sk don’t

change:

✦ Assign each data point to


the cluster with the closest
 center.

✦ Assign each cluster 


center to be the mean of its cluster’s data points

µk ← xn

• Repeat until con

Or no change Or no change 
 in Or no change in disglobal

84

K-Means Algorithm

• For k = 1,…, K

✦ Randomly draw n from 


1,…,N without replacement

✦

• Repeat until S1,…,Sk don’t

change:

✦ Assign each data point to


the cluster with the closest
 center.

✦ Assign each cluster 


center to be the mean of its cluster’s data points

µk ← xn

• Repeat until con

85

K-Means Algorithm

• For k = 1,…, K

✦ Randomly draw n from 


1,…,N without replacement

✦

• Repeat until S1,…,Sk don’t

change:

✦ For n = 1,…N ❖ Find k with smallest
 ❖ Put (and no 


• ther Sj)

✦ Assign each cluster 


center to be the mean of its cluster’s data points

µk ← xn

• Repeat until con

* Find k with smallest dis(xn, µk) * Put (and no xn ∈ Sk

86

K-Means Algorithm

µk ← xn

• Repeat until con

* Find k with smallest dis(xn, µk) * Put (and no xn ∈ Sk

• For k = 1,…, K

✦ Randomly draw n from 


1,…,N without replacement

✦

• Repeat until S1,…,Sk don’t

change:

✦ For n = 1,…N ❖ Find k with smallest
 ❖ Put (and no 


• ther Sj)

✦ Assign each cluster 


center to be the mean of its cluster’s data points

87

K-Means Algorithm

• For k = 1,…, K

✦ Randomly draw n from 


1,…,N without replacement

✦

• Repeat until S1,…,Sk don’t

change:

✦ For n = 1,…N ❖ Find k with smallest
 ❖ Put (and no 


• ther Sj)

✦ Assign each cluster 


center to be the mean of its cluster’s data points

µk ← xn

• Repeat until con

* Find k with smallest dis(xn, µk) * Put (and no xn ∈ Sk

88

K-Means Algorithm

• For k = 1,…, K

✦ Randomly draw n from 


1,…,N without replacement

✦

• Repeat until S1,…,Sk don’t

change:

✦ For n = 1,…N ✤ Find k with smallest
 ✤ Put (and no 


• ther Sj)

✦ Assign each cluster 


center to be the mean of its cluster’s data points

µk ← xn

• Repeat until con

xn ∈ Sk * Put (and no dis(xn, µk) * Find k with smallest

89

K-Means Algorithm

• For k = 1,…, K

✦ Randomly draw n from 


1,…,N without replacement

✦

• Repeat until S1,…,Sk don’t

change:

✦ For n = 1,…N ✤ Find k with smallest
 ✤ Put (and no 


• ther Sj)

✦ For k = 1,…,K ✤

µk ← xn

• Repeat until con

xn ∈ Sk * Put (and no dis(xn, µk) * Find k with smallest For k = 1,...,K * µk ← |Sk|−1

• n:n∈Sk

xn

90

K-Means Algorithm

• For k = 1,…, K

✦ Randomly draw n from 


1,…,N without replacement

✦

• Repeat until S1,…,Sk don’t

change:

✦ For n = 1,…N ✤ Find k with smallest
 ✤ Put (and no 


• ther Sj)

✦ For k = 1,…,K ✤

µk ← xn

• Repeat until con

xn ∈ Sk * Put (and no dis(xn, µk) * Find k with smallest For k = 1,...,K * µk ← |Sk|−1

• n:n∈Sk

xn

91

K-Means Algorithm

• For k = 1,…, K

✦ Randomly draw n from 


1,…,N without replacement

✦

• Repeat until S1,…,Sk don’t

change:

✦ For n = 1,…N ❖ Find k with smallest
 ❖ Put (and no 


• ther Sj)

✦ Assign each cluster 


center to be the mean of its cluster’s data points

µk ← xn

• Repeat until con

* Find k with smallest dis(xn, µk) * Put (and no xn ∈ Sk

92

K-Means Algorithm

• For k = 1,…, K

✦ Randomly draw n from 


1,…,N without replacement

✦

• Repeat until S1,…,Sk don’t

change:

✦ For n = 1,…N ❖ Find k with smallest
 ❖ Put (and no 


• ther Sj)

✦ Assign each cluster 


center to be the mean of its cluster’s data points

µk ← xn

• Repeat until con

* Find k with smallest dis(xn, µk) * Put (and no xn ∈ Sk

93

K-Means Algorithm

• For k = 1,…, K

✦ Randomly draw n from 


1,…,N without replacement

✦

• Repeat until S1,…,Sk don’t

change:

✦ For n = 1,…N ❖ Find k with smallest
 ❖ Put (and no 


• ther Sj)

✦ Assign each cluster 


center to be the mean of its cluster’s data points

µk ← xn

• Repeat until con

* Find k with smallest dis(xn, µk) * Put (and no xn ∈ Sk

94

K-Means Algorithm

• For k = 1,…, K

✦ Randomly draw n from 


1,…,N without replacement

✦

• Repeat until S1,…,Sk don’t

change:

✦ For n = 1,…N ✤ Find k with smallest
 ✤ Put (and no 


• ther Sj)

✦ For k = 1,…,K ✤

µk ← xn

• Repeat until con

xn ∈ Sk * Put (and no dis(xn, µk) * Find k with smallest For k = 1,...,K * µk ← |Sk|−1

• n:n∈Sk

xn

95

K-Means Algorithm

• For k = 1,…, K

✦ Randomly draw n from 


1,…,N without replacement

✦

• Repeat until S1,…,Sk don’t

change:

✦ For n = 1,…N ✤ Find k with smallest
 ✤ Put (and no 


• ther Sj)

✦ For k = 1,…,K ✤

µk ← xn

• Repeat until con

xn ∈ Sk * Put (and no dis(xn, µk) * Find k with smallest For k = 1,...,K * µk ← |Sk|−1

• n:n∈Sk

xn

96

K-Means Algorithm

• For k = 1,…, K

✦ Randomly draw n from 


1,…,N without replacement

✦

• Repeat until S1,…,Sk don’t

change:

✦ For n = 1,…N ✤ Find k with smallest
 ✤ Put (and no 


• ther Sj)

✦ For k = 1,…,K ✤

µk ← xn

• Repeat until con

xn ∈ Sk * Put (and no dis(xn, µk) * Find k with smallest For k = 1,...,K * µk ← |Sk|−1

• n:n∈Sk

xn

97

K-Means Algorithm

• For k = 1,…, K

✦ Randomly draw n from 


1,…,N without replacement

✦

• Repeat until S1,…,Sk don’t

change:

✦ For n = 1,…N ✤ Find k with smallest
 ✤ Put (and no 


• ther Sj)

✦ For k = 1,…,K ✤

µk ← xn

• Repeat until con

xn ∈ Sk * Put (and no dis(xn, µk) * Find k with smallest For k = 1,...,K * µk ← |Sk|−1

• n:n∈Sk

xn

98

K-Means Algorithm

• For k = 1,…, K

✦ Randomly draw n from 


1,…,N without replacement

✦

• Repeat until S1,…,Sk don’t

change:

✦ For n = 1,…N ✤ Find k with smallest
 ✤ Put (and no 


• ther Sj)

✦ For k = 1,…,K ✤

µk ← xn

• Repeat until con

xn ∈ Sk * Put (and no dis(xn, µk) * Find k with smallest For k = 1,...,K * µk ← |Sk|−1

• n:n∈Sk

xn

99

K-Means: Evaluation

100

K-Means: Evaluation

• Will it terminate?
• Yes. Always.

101

K-Means: Evaluation

• Will it terminate?
• Yes. Always.
• Is the clustering any good?

Global dissimilarity only useful for comparing clusterings.

• Guaranteed to converge in a finite number
• f iterations
• Running time per iteration:
• 1. Assign data points to closest cluster center 


O(KN) time

• 2. Change the cluster center to the average of its

assigned points 
 O(N) time

102

K-Means: Evaluation

Objective

• 1. Fix μ, optimize C:
• 2. Fix C, optimize μ:

– Take partial derivative of μi and set to zero, we have

103

K-Means takes an alternating optimization approach, each step is guaranteed to decrease the objective – thus guaranteed to converge

K-Means: Evaluation

Demo time…

104

• How to set k?
• Sensitive to initial centers
• Multiple initializations
• Sensitive to outliers
• Detects spherical clusters
• Assuming means can be computed
• It requires continuous, numerical features

105

K-Means Algorithm: Some Issues

Next Lecture:

K-Means Applications, Spectral clustering, Hierarchical clustering and What is a good clustering?

106