Machine Learning Lecture Notes on Clustering (II) 2017-2018 Davide - - PowerPoint PPT Presentation

Machine Learning

Lecture Notes on Clustering (II) 2017-2018

Davide Eynard

davide.eynard@usi.ch

Institute of Computational Science Universit` a della Svizzera italiana

– p. 1/39

Today’s Outline

• K-Means limits
• K-Means extensions: K-Medoids and Fuzzy C-Means
• Hierarchical Clustering

– p. 2/39

K-Means limits

Importance of choosing initial centroids

– p. 3/39

K-Means limits

Importance of choosing initial centroids

– p. 4/39

K-Means limits

Differing sizes

– p. 5/39

K-Means limits

Differing density

– p. 6/39

K-Means limits

Non-globular shapes

– p. 7/39

K-Means: higher K

What if we tried to increase K to solve K-Means problems?

– p. 8/39

K-Means: higher K

What if we tried to increase K to solve K-Means problems?

– p. 9/39

K-Means: higher K

What if we tried to increase K to solve K-Means problems?

– p. 10/39

K-Medoids

• K-Means algorithm is too sensitive to outliers
• An object with an extremely large value may substantially distort

the distribution of the data

• Medoid: the most centrally located point in a cluster, as a

representative point of the cluster

• Note: while a medoid is always a point inside a cluster too, a centroid

could be not part of the cluster

• Analogy to using medians, instead of means, to describe the

representative point of a set

• Mean of 1, 3, 5, 7, 9 is 5
• Mean of 1, 3, 5, 7, 1009 is 205
• Median of 1, 3, 5, 7, 1009 is 5

– p. 11/39

PAM

PAM means Partitioning Around Medoids. The algorithm follows:

• 1. Given k
• 2. Randomly pick k instances as initial medoids
• 3. Assign each data point to the nearest medoid x
• 4. Calculate the objective function
• the sum of dissimilarities of all points to their nearest medoids.

(squared-error criterion)

• 5. For each non-medoid point y
• swap x and y and calculate the objective function
• 6. Select the configuration with the lowest cost
• 7. Repeat (3-6) until no change

– p. 12/39

PAM

• Pam is more robust than k-means in the presence of noise and
• utliers
• A medoid is less influenced by outliers or other extreme values

than a mean (can you tell why?)

• Pam works well for small data sets but does not scale well for large

data sets

• O(k(n − k)2) for each change where n is # of data objects, k is #
• f clusters
• NOTE: not having to calculate a mean, we do not need actual

positions of points but just their distances!

– p. 13/39

Fuzzy C-Means

Fuzzy C-Means (FCM, developed by Dunn in 1973 and improved by Bezdek in 1981) is a method of clustering which allows one piece of data to belong to two or more clusters.

• frequently used in pattern recognition
• based on minimization of the following objective function:

Jm =

N

• i=1

C

• j=1

um

ijxi − cj2, 1 < m < ∞

where: m is any real number greater than 1 (fuzziness coefficient), uij is the degree of membership of xi in the cluster j, xi is the i-th of d-dimensional measured data, cj is the d-dimension center of the cluster, · is any norm expressing the similarity between measured data and the center.

– p. 14/39

K-Means vs. FCM

• With K-Means, every piece of data either belongs to centroid A or to

centroid B

– p. 15/39

K-Means vs. FCM

• With FCM, data elements do not belong exclusively to one cluster, but

they may belong to several clusters (with different membership values)

– p. 16/39

Data representation

(KM)UN×C =        1 1 1 . . . . . . 1        (FCM)UN×C =        0.8 0.2 0.3 0.7 0.6 0.4 . . . . . . 0.9 0.1       

– p. 17/39

FCM Algorithm

The algorithm is composed of the following steps:

• 1. Initialize U = [uij] matrix, U (0)

– p. 18/39

FCM Algorithm

The algorithm is composed of the following steps:

• 1. Initialize U = [uij] matrix, U (0)
• 2. At t-step: calculate the centers vectors C(t) = [cj] with U (t):

cj = N

i=1 um ij · xi

N

i=1 um ij

– p. 19/39

FCM Algorithm

The algorithm is composed of the following steps:

• 1. Initialize U = [uij] matrix, U (0)
• 2. At t-step: calculate the centers vectors C(t) = [cj] with U (t):

cj = N

i=1 um ij · xi

N

i=1 um ij

• 3. Update U (t), U (t+1):

uij = 1 C

k=1

• xi−cj

xi−ck

• 2

m−1

– p. 20/39

FCM Algorithm

The algorithm is composed of the following steps:

• 1. Initialize U = [uij] matrix, U (0)
• 2. At t-step: calculate the centers vectors C(t) = [cj] with U (t):

cj = N

i=1 um ij · xi

N

i=1 um ij

• 3. Update U (t), U (t+1):

uij = 1 C

k=1

• xi−cj

xi−ck

• 2

m−1

• 4. If U (k+1) − U (k) < ε then STOP; otherwise return to step 2.

– p. 21/39

An Example

– p. 22/39

An Example

– p. 23/39

An Example

– p. 24/39

FCM Demo

Time for a demo!

– p. 25/39

Hierarchical Clustering

• Top-down vs Bottom-up
• Top-down (or divisive):
• Start with one universal cluster
• Split it into two clusters
• Proceed recursively on each subset
• Bottom-up (or agglomerative):
• Start with single-instance clusters ("every item is a cluster")
• At each step, join the two closest clusters
• (design decision: distance between clusters)

– p. 26/39

Agglomerative Hierarchical Clustering

Given a set of N items to be clustered, and an N*N distance (or dissimilarity) matrix, the basic process of agglomerative hierarchical clustering is the following:

• 1. Start by assigning each item to a cluster. Let the dissimilarities

between the clusters be the same as the dissimilarities between the items they contain.

• 2. Find the closest (most similar) pair of clusters and merge them into a

single cluster. Now, you have one cluster less.

• 3. Compute dissimilarities between the new cluster and each of the old
• nes.
• 4. Repeat Steps 2 and 3 until all items are clustered into a single cluster
• f size N.

– p. 27/39

Single Linkage (SL) clustering

• We consider the distance between two clusters to be equal to the

shortest distance from any member of one cluster to any member of the other one (greatest similarity).

– p. 28/39

Complete Linkage (CL) clustering

• We consider the distance between two clusters to be equal to the

greatest distance from any member of one cluster to any member of the other one (smallest similarity).

– p. 29/39

Group Average (GA) clustering

• We consider the distance between two clusters to be equal to the

average distance from any member of one cluster to any member of the other one.

– p. 30/39

About distances

If the data exhibit strong clustering tendency, all 3 methods produce similar results.

• SL: requires only a single dissimilarity to be small. Drawback:

produced clusters can violate the “compactness” property (cluster with large diameters)

• CL: opposite extreme (compact clusters with small diameters, but can

violate the “closeness” property)

• GA: compromise, it attempts to produce relatively compact clusters

and relatively far apart. BUT it depends on the dissimilarity scale.

– p. 31/39

Hierarchical algorithms limits

Strength of MIN

• Easily handles clusters of different sizes
• Can handle non elliptical shapes

– p. 32/39

Hierarchical algorithms limits

Limitations of MIN

• Sensitive to noise and outliers

– p. 33/39

Hierarchical algorithms limits

Strength of MAX

• Less sensitive to noise and outliers

– p. 34/39

Hierarchical algorithms limits

Limitations of MAX

• Tends to break large clusters
• Biased toward globular clusters

– p. 35/39

Hierarchical clustering: Summary

• Advantages
• It’s nice that you get a hierarchy instead of an amorphous

collection of groups

• If you want k groups, just cut the (k − 1) longest links
• Disadvantages
• It doesn’t scale well: time complexity of at least O(n2), where n is

the number of objects

– p. 36/39

Hierarchical Clustering Demo

Time for another demo!

– p. 37/39

Bibliography

• A Tutorial on Clustering Algorithms Online tutorial by M. Matteucci
• K-means and Hierarchical Clustering

Tutorial Slides by A. Moore

• "Metodologie per Sistemi Intelligenti" course - Clustering

Tutorial Slides by P .L. Lanzi

• K-Means Clustering Tutorials

Online tutorials by K. Teknomo

– p. 38/39

• The end

– p. 39/39