[PPT] - Machine Learning Lecture Notes on Clustering (IV) 2016-2017 Davide PowerPoint Presentation

SLIDE 1

Machine Learning

Lecture Notes on Clustering (IV) 2016-2017

Davide Eynard

davide.eynard@usi.ch

Institute of Computational Science Universit` a della Svizzera italiana

– p. 1/34

SLIDE 2

Lecture outline

Cluster Evaluation
Internal measures
External measures
Finding the correct number of clusters
Framework for cluster validity

– p. 2/34

SLIDE 3

Cluster Evaluation

Every algorithm has its pros and cons
(Not only about cluster quality: complexity, #clusters in advance, etc.)
For what concerns cluster quality, we can evaluate (or, better,

validate) clusters

For supervised classification we have a variety of measures to

evaluate how good our model is

Accuracy, precision, recall
For cluster analysis, the analogous question is: how can we evaluate

the "goodness" of the resulting clusters?

But most of all... why should we evaluate it?

– p. 3/34

SLIDE 4

Cluster found in random data

"Clusters are in the eye of the beholder"

– p. 4/34

SLIDE 5

Why evaluate?

To determine the clustering tendency of the dataset, that is

distinguish whether non-random structure actually exists in the data

To determine the correct number of clusters
To evaluate how well the results of a cluster analysis fit the data

without reference to external information

To compare the results of a cluster analysis to externally known

results, such as externally provided class labels

To compare two sets of clusters to determine which is better

Note:

the first three are unsupervised techniques, while the last two require external info
the last three can be applied to the entire clustering or just to individual clusters

– p. 5/34

SLIDE 6

Open challenges

Cluster evaluation has a number of challenges:

a measure of cluster validity may be quite limited in the scope of its

applicability

ie. dimensions of the problem: most work has been done only on

2- or 3-dimensional data

we need a framework to interpret any measure
How good is "10"?
if a measure is too complicated to apply or to understand, nobody will

use it

– p. 6/34

SLIDE 7

Measures of Cluster Validity

Numerical measures that are applied to judge various aspects of cluster validity are classified into the following three types:

Internal (unsupervised) Indices: Used to measure the goodness of

a clustering structure without respect to external information

cluster cohesion vs cluster separation
e.g. Sum of Squared Error (SSE)
External (supervised) Indices: Used to measure the extent to which

cluster labels match externally supplied class labels

e.g. entropy, purity, precision, accuracy, ...

Internal or external indices (e.g. SSE or entropy) can be used to evaluate a single clustering/cluster or to compare two different ones. In the latter case, they are used as relative indices.

– p. 7/34

SLIDE 8

External Measures

Entropy
The degree to which each cluster consists of objects of a single

class

For cluster i we compute pij, the probability that a member of

cluster i belongs to class j, as pij = mij/mi, where mi is the number of objects in cluster i and mij is the number of objects of class j in cluster i

The entropy of each cluster i is ei = − L

j=1 pijlog2pij, where L

is the number of classes

The total entropy is e = K

i=1 mi m ei, where K is the number of

clusters and m is the total number of data points

– p. 8/34

SLIDE 9

External Measures

Purity
Another measure of the extent to which a cluster contains objects
f a single class
Using the previous terminology, the purity of cluster i is

pi = max(pij) for all the j

The overall purity is purity = K

i=1 mi m pi

– p. 9/34

SLIDE 10

External Measures

Precision
The fraction of a cluster that consists of objects of a specified

class

The precision of cluster i with respect to class j is

precision(i, j) = pij

Recall
The extent to which a cluster contains all objects of a specified

class

The recall of cluster i with respect to class j is

recall(i, j) = mij/mj, where mj is the number of objects in class j

– p. 10/34

SLIDE 11

External Measures

F-measure
A combination of both precision and recall that measures the

extent to which a cluster contains only objects of a particular class and all objects of that class

The F-measure of cluster i with respect to class j is

F(i, j) = 2×precision(i,j)×recall(i,j)

precision(i,j)+recall(i,j)

– p. 11/34

SLIDE 12

External Measures: example

– p. 12/34

SLIDE 13

Internal measures: Cohesion and Separation

Graph-based view
Prototype-based view

– p. 13/34

SLIDE 14

Internal measures: Cohesion and Separation

Cluster Cohesion: Measures how closely related objects in a cluster

are

cohesion(Ci) =

x∈Ci,y∈Ci

proximity(x, y) cohesion(Ci) =

x∈Ci

proximity(x, ci)

Cluster Separation: Measure how distinct or well-separated a cluster

is from other clusters

separation(Ci, Cj) =

x∈Ci,y∈Cj

proximity(x, y) separation(Ci, Cj) = proximity(ci, cj) separation(Ci) = proximity(ci, c)

– p. 14/34

SLIDE 15

Cohesion and separation example

Cohesion is measured by the within cluster sum of squares (SSE)

WSS =

i
x∈Ci

(x − mi)2

Separation is measured by the between cluster sum of squares

BSS =

i

|Ci|(m − mi)2 where |Ci| is the size of cluster i

– p. 15/34

SLIDE 16

Cohesion and separation example

K=1 cluster:

WSS = (1 − 3)2 + (2 − 3)2 + (4 − 3)2 + (5 − 3)2 = 10 BSS = 4 × (3 − 3)2 = 0 Total = 10 + 0 = 10

K=2 clusters:

WSS = (1 − 1.5)2 + (2 − 1.5)2 + (4 − 4.5)2 + (5 − 4.5)2 = 1 BSS = 2 × (3 − 1.5)2 + 2 × (4.5 − 3)2 = 9 Total = 1 + 9 = 10

– p. 16/34

SLIDE 17

Evaluating individual clusters and Objects

So far, we have focused on evaluation of a group of clusters
Many of these measures, however, can also be used to evaluate

individual clusters and objects

For example, a cluster with a high cohesion may be considered

better than a cluster with a lower one

This information can often be used to improve the quality of the

clustering

Split not very cohesive clusters
Merge not very separated ones
We can also evaluate the objects within a cluster in terms of their

contribution to the overall cohesion or separation of the cluster

– p. 17/34

SLIDE 18

The Silhouette Coefficient

Silhouette Coefficient combines ideas of both cohesion and

separation, but for individual points, as well as clusters and clusterings

For an individual point, i
Calculate ai = average distance of i to the points in its cluster
Calculate bi = min (average distance of i to points in another

cluster)

The silhouette coefficient for a point is then given by

si = (bi − ai)/max(ai, bi)

– p. 18/34

SLIDE 19

The Silhouette Coefficient

Silhouette Coefficient combine ideas of both cohesion and separation,

but for individual points, as well as clusters and clusterings

– p. 19/34

SLIDE 20

Measuring Cluster Validity via Correlation

If we are given the similarity matrix for a data set and the cluster labels from a cluster analysis of the data set, then we can evaluate the "goodness" of the clustering by looking at the correlation between the similarity matrix and an ideal version of the similarity matrix based on the cluster labels

Similarity/Proximity Matrix
Ideal Matrix
One row and one column for each data point
An entry is 1 if the associated pair of points belongs to the same

cluster

An entry is 0 if the associated pair of points belongs to different

clusters

– p. 20/34

SLIDE 21

Measuring Cluster Validity via Correlation

Compute the correlation between the two matrices
Since the matrices are symmetric, only the correlation between n(n − 1)/2 entries

needs to be calculated

High correlation indicates that points that belong to the same cluster

are close to each other

– p. 21/34

SLIDE 22