Lecture 22: Clustering Distance measures K-Means Aykut Erdem May - - PowerPoint PPT Presentation

Lecture 22:

− Clustering − Distance measures − K-Means

Aykut Erdem

May 2016 Hacettepe University

Last time… Boosting

• Idea: given a weak learner, run it multiple times on (reweighted)

training data, then let the learned classifiers vote

• On each iteration t:
• weight each training example by how incorrectly it was classified
• Learn a hypothesis – ht
• A strength for this hypothesis – at
• Final classifier:
• A linear combination of the votes of the different classifiers

weighted by their strength

• Practically useful
• Theoretically interesting

2

slide by Aarti Singh & Barnabas Poczos

Last time.. The AdaBoost Algorithm

3

slide by Jiri Matas and Jan Šochman

This week

• Clustering
• Distance measures
• K-Means
• Spectral clustering
• Hierarchical clustering
• What is a good clustering?

4

Distance measures

5

Distance measures

• In studying clustering techniques we will

assume that we are given a matrix of distances between all pairs of data points:

6

m 4 3 2 1

x x x x x

1

x

2

x

3

x

m

x

4

x

• • •

) x , d(x

j i

slide by Julia Hockenmeier

What is Similarity/Dissimilarity?

• The real meaning of similarity is a philosophical question. We will take

a more pragmatic approach.

• Depends on representation and algorithm. For many rep.//alg., easier

to think in terms of a distance (rather than similarity) between vectors.

7

Hard to define! But we know it when we see it

• slide by Eric Xing

Defining Distance Measures

8

0.23 3 342.7

gene2 gene1

• Definition: Let O1 and O2 be two objects from the

universe of possible objects. The distance (dissimilarity) between O1 and O2 is a real number denoted by D(O1, O2).

slide by Andrew Moore

• ance
• ฀

d(x,y) (xi yi)2

i

• ฀
• 9
• Similarity rather than distance
• Can determine similar trends

A few examples:

• Euclidean distance
• Correlation coefficient
• coefficient

฀

• ฀

s(x,y) (xi x)(yi y)

i

• x y
• slide by Andrew Moore

What properties should a distance measure have?

• Symmetric
• D(A,B) = D(B,A)
• Otherwise, we can say A looks like B but B does not look

like A

• Positivity, and self-similarity
• D(A,B) ≥ 0, and D(A,B) = 0 iff A = B
• Otherwise there will different objects that we cannot tell

apart

• Triangle inequality
• D(A,B) + D(B,C) ≥ D(A,C)
• Otherwise one can say “A is like B, B is like C, but A is not

like C at all”

10

slide by Alan Fern

hattan (L1) ity (Sup) Distance L

d(x, y) = x - y = xi − yi

i=1 d

∑

idean (L2) hattan (L )

d(x, y) = (xi − yi)2

i=1 d

∑

Distance measures

• Euclidean (L2) 


• Manhattan (L1)


• Infinity (Sup) Distance L∞


• Note that L∞ < L1 < L2, but different distances do

not induce the same ordering on points.

11

ity (Sup) Distance L∞

d(x, y) = max1≤i≤d xi − yi

slide by Julia Hockenmeier

Distance measures

12

x = (x1, x2) y = (x1–2, x2+4)

Euclidean: (42 + 22)1/2 = 4.47 Manhattan: 4+ 2 = 6 Sup: Max(4,2) = 4

2 4

slide by Julia Hockenmeier

Distance measures

• Different distances do not induce the same
• rdering on points

13 9

4

ε ε + = + = =

∞

5 ) (5 b) (a, L 5 b) (a, L

1/2 2 2 2

4

5

66 . 5 4 = = + = =

∞

2 4 ) (4 d) (c, L 4 d) (c, L

1/2 2 2 2

b) (a, L d) (c, L b) (a, L d) (c, L

2 2

> <

∞ ∞

slide by Julia Hockenmeier

Distance measures

• Clustering is sensitive to the distance measure.
• Sometimes it is beneficial to use a distance

measure that is invariant to transformations that are natural to the problem:

• Mahalanobis distance:

✓ Shift and scale invariance

14

slide by Julia Hockenmeier

Mahalanobis Distance

15

Σ is a (symmetric) Covariance Matrix: Translates all the axes to a mean = 0 and variance = 1 (shift and scale invariance)

d(x, y) = (x - y)T Σ(x − y)

µ = 1 m xi

i=1 m

∑

, (average of the data) Σ = 1 m (x −µ)(x −µ)T,

i=1 m

∑

a matrix of size m× m

slide by Julia Hockenmeier

Distance measures

• Some algorithms require distances between

a point x and a set of points A d(x, A) 


This might be defined e.g. as min/max/avg distance between x and any point in A. 


• Others require distances between two sets
• f points A, B, d(A, B). 


This might be defined e.g as min/max/avg distance between any point in A and any point in B.

16

slide by Julia Hockenmeier

Clustering algorithms

• Partitioning algorithms
• Construct various partitions


and then evaluate them by 
 some criterion

• K-means
• Mixture of Gaussians
• Spectral Clustering
• Hierarchical algorithms
• Create a hierarchical decomposition 

• f the set of objects using some


criterion

• Bottom-up – agglomerative
• Top-down – divisive

17

• %;
• slide by Eric Xing

Desirable Properties of a Clustering Algorithm

• Scalability (in terms of both time and space)
• Ability to deal with different data types
• Minimal requirements for domain

knowledge to determine input parameters

• Ability to deal with noisy data
• Interpretability and usability
• Optional
• Incorporation of user-specified constraints

18

slide by Andrew Moore

K-Means

19

K-Means

• An iterative

clustering algorithm

• Initialize: Pick K

random points as cluster centers (means)

• Alternate:
• Assign data instances

to closest mean

• Assign each mean to

the average of its assigned points

• Stop when no points’

assignments change

20

slide by David Sontag

K-Means

• An iterative

clustering algorithm

• Initialize: Pick K

random points as cluster centers (means)

• Alternate:
• Assign data instances

to closest mean

• Assign each mean to

the average of its assigned points

• Stop when no points’

assignments change

21

slide by David Sontag

K-Means Clustering: Example

• Pick K random

points as cluster centers (means)

Shown here for K=2

22

slide by David Sontag

Iterative Step 1

• Assign data points

to closest cluster centers

23

K-Means Clustering: Example

slide by David Sontag

Iterative Step 2

• Change the cluster

center to the average of the assigned points

24

K-Means Clustering: Example

slide by David Sontag

• Repeat until

convergence

25

K-Means Clustering: Example

slide by David Sontag

26

K-Means Clustering: Example

slide by David Sontag

27

K-Means Clustering: Example

slide by David Sontag

Properties of K-Means Algorithms

• Guaranteed to converge in a finite number
• f iterations
• Running time per iteration:
• 1. Assign data points to closest cluster center 


O(KN) time

• 2. Change the cluster center to the average of its

assigned points 
 O(N) time

28

slide by David Sontag

Objective

• 1. Fix μ, optimize C:
• 2. Fix C, optimize μ:

– Take partial derivative of μi and set to zero, we have

29

K-Means takes an alternating optimization approach, each step is guaranteed to decrease the objective – thus guaranteed to converge

K-Means Convergence

slide by Alan Fern

Demo time…

30

Example: K-Means for Segmentation

31

K = 2

K=2

Original image

Original

K = 3

K=3

K = 10

K=10

Goal of Segmentation is to partition an image into regions each of which has reasonably homogenous visual appearance.

slide by David Sontag

Example: K-Means for Segmentation

32

K = 2

K=2

Original image

Original

K = 3

K=3

K = 10

K=10

slide by David Sontag

Example: K-Means for Segmentation

33

K = 2

K=2

Original image

Original

K = 3

K=3

K = 10

K=10

slide by David Sontag

Example: Vector quantization

34

FIGURE 14.9. Sir Ronald A. Fisher (1890 − 1962) was one of the founders

• f modern day statistics, to whom we owe maximum-likelihood, sufficiency, and

many other fundamental concepts. The image on the left is a 1024×1024 grayscale image at 8 bits per pixel. The center image is the result of 2 × 2 block VQ, using 200 code vectors, with a compression rate of 1.9 bits/pixel. The right image uses

• nly four code vectors, with a compression rate of 0.50 bits/pixel

[Figure from Hastie et al. book]

slide by David Sontag

Bag of Words model

35

aardvark 0 about 2 all 2 Africa 1 apple anxious ... gas 1 ...

• il

1 … Zaire

slide by Carlos Guestrin

36

slide by Fei Fei Li

37

Object Bag of ‘words’

slide by Fei Fei Li

Interest Point Features

38

Normalize patch

Detect patches

[Mikojaczyk and Schmid ’02] [Matas et al. ’02] [Sivic et al. ’03]

Compute SIFT descriptor

[Lowe’99]

slide by Josef Sivic

Patch Features

39

…

slide by Josef Sivic

Dictionary Formation

40

…

slide by Josef Sivic

Clustering (usually K-means)

41

Vector quantization

…

slide by Josef Sivic

Clustered Image Patches

42

slide by Fei Fei Li

Visual synonyms and polysemy

43

Visual Polysemy. Single visual word occurring on different (but locally 
 similar) parts on different object categories. Visual Synonyms. Two different visual words representing a similar part of an object (wheel of a motorbike).

slide by Andrew Zisserman

Image Representation

44

…..

frequency

codewords

slide by Fei Fei Li

K-Means Clustering: Some Issues

• How to set k?
• Sensitive to initial centers
• Sensitive to outliers
• Detects spherical clusters
• Assuming means can be computed

45

slide by Kristen Grauman