Chapter 5-2: Clu lust ster erin ing Jilles Vreeken Revision 1, - - PowerPoint PPT Presentation

IRDM ‘15/16

Jilles Vreeken

Chapter 5-2: Clu lust ster erin ing

12 Nov 2015 Revision 1, November 20th typo’s fixed: dendrogram Revision 2, December 10th clarified: we do consider a point 𝑦 as a member of its own 𝜗-neighborhood

IRDM ‘15/16

Novem vember 19th

th 2015

2015

Th The Fir e First M Midt idterm T T es est

Wh Where: Günter-Hotz Hörsaal (E2.2) Material: the first four lectures, the first two homeworks You are a allo llowed to br brin ing o

• ne (1

(1) ) sheet o

• f A

A4 p pape per wit with handwr writ itten or pr prin inted notes o

• n bo

both s sid ides . . No other material (n l (notes, bo books, c course m materials) ) or devic ices (c (calc lculator, n notebook, c cell ph ll phone, t toothbrush, etc tc) a ) allo llowed. Br Brin ing a an ID; D; eit ither y your UdS UdS card, o

• r pa

passport.

V-2: 2

IRDM ‘15/16

Preliminary dates: Februar ary 15 15th

th an

and 16 16th

th 2016

2016

Th The Fin e Final Ex l Exam am

Oral e l exam. Can o

• nly

ly be be t taken wh when you passed tw two o

• ut o

t of th three mid id-term t tests. More details l later.

V-2: 3

IRDM ‘15/16

IRDM Chapter 5, overview

1.

Basic idea

2.

Representative-based clustering

3.

Probabilistic clustering

4.

Validation

5.

Hierarchical clustering

6.

Density-based clustering

7.

Clustering high-dimensional data

You’ll find this covered in Aggarwal Ch. 6, 7 Zaki & Meira, Ch. 13—15

V-2: 4

IRDM ‘15/16

IRDM Chapter 5, today

1.

Basic idea

2.

Representative-based clustering

3.

Probabilistic clustering

4.

Validation

5.

Hierarchical clustering

6.

Density-based clustering

7.

Clustering high-dimensional data

You’ll find this covered in Aggarwal Ch. 6, 7 Zaki & Meira, Ch. 13—15

V-2: 5

IRDM ‘15/16

Chapter 5.5:

Hier ierarchi hical C Clust lustering ng

Aggarwal Ch. 6.4

V-2: 6

IRDM ‘15/16

The basic idea

Create clustering for each number of clusters 𝑙 = 1,2, … , 𝑜 The clusterings must be hie ierarch chica ical

 every cluster of 𝑙-clustering is a union of some clusters in an 𝑚-

clustering for all 𝑙 < 𝑚

 i.e. for all 𝑚, and for all 𝑙 > 𝑚, every cluster in an 𝑚-clustering is a

subset of some cluster in the 𝑙-clustering Example:

V-2: 7

k = 6

IRDM ‘15/16

The basic idea

Create clustering for each number of clusters 𝑙 = 1,2, … , 𝑜 The clusterings must be hie ierarch chica ical

 every cluster of 𝑙-clustering is a union of some clusters in an 𝑚-

clustering for all 𝑙 < 𝑚

 i.e. for all 𝑚, and for all 𝑙 > 𝑚, every cluster in an 𝑚-clustering is a

subset of some cluster in the 𝑙-clustering Example:

V-2: 8

k = 5

IRDM ‘15/16

The basic idea

Create clustering for each number of clusters 𝑙 = 1,2, … , 𝑜 The clusterings must be hie ierarch chica ical

 every cluster of 𝑙-clustering is a union of some clusters in an 𝑚-

clustering for all 𝑙 < 𝑚

 i.e. for all 𝑚, and for all 𝑙 > 𝑚, every cluster in an 𝑚-clustering is a

subset of some cluster in the 𝑙-clustering Example:

V-2: 9

k = 4

IRDM ‘15/16

The basic idea

Create clustering for each number of clusters 𝑙 = 1,2, … , 𝑜 The clusterings must be hie ierarch chica ical

 every cluster of 𝑙-clustering is a union of some clusters in an 𝑚-

clustering for all 𝑙 < 𝑚

 i.e. for all 𝑚, and for all 𝑙 > 𝑚, every cluster in an 𝑚-clustering is a

subset of some cluster in the 𝑙-clustering Example:

V-2: 10

k = 3

IRDM ‘15/16

The basic idea

Create clustering for each number of clusters 𝑙 = 1,2, … , 𝑜 The clusterings must be hie ierarch chica ical

 every cluster of 𝑙-clustering is a union of some clusters in an 𝑚-

clustering for all 𝑙 < 𝑚

 i.e. for all 𝑚, and for all 𝑙 > 𝑚, every cluster in an 𝑚-clustering is a

subset of some cluster in the 𝑙-clustering Example:

V-2: 11

k = 2

IRDM ‘15/16

The basic idea

Create clustering for each number of clusters 𝑙 = 1,2, … , 𝑜 The clusterings must be hie ierarch chica ical

 every cluster of 𝑙-clustering is a union of some clusters in an 𝑚-

clustering for all 𝑙 < 𝑚

 i.e. for all 𝑚, and for all 𝑙 > 𝑚, every cluster in an 𝑚-clustering is a

subset of some cluster in the 𝑙-clustering Example:

V-2: 12

k = 1

IRDM ‘15/16

Dendrograms

The difference in height between the tree and its subtrees shows the distance between the two branches

V-2: 13

Distance is ≈0.7

IRDM ‘15/16

Dendrograms and clusters

V-2: 14

IRDM ‘15/16

Dendrograms, revisited

Dendrograms show the hierarchy of the clustering Number of clusters can be deduced from a dendrogram

 higher branches

Outliers can be detected from a dendrogram

 single points that are far from others

V-2: 15

IRDM ‘15/16

Agglomerative and Divisive

Agglome

• merative: bottom-up

 start with 𝑜 clusters  combine two closest clusters into a cluster of one bigger cluster

Div ivis isiv ive: top-down

 start with 1 cluster  divide the cluster into two

 divide the largest (per diameter) cluster into smaller clusters

V-2: 16

IRDM ‘15/16

Cluster distances

The distance between two points 𝑦 and 𝑧 is 𝑒(𝑦, 𝑧) What is the distance between two clusters? Many intuitive definitions – no universal truth

 different cluster distances yield different clusterings  the selection of cluster distance depends on application

Some distances between clusters 𝐶 and 𝐷:

 minimum distance

𝑒(𝐶, 𝐷) = min {𝑒(𝑦, 𝑧) ∶ 𝑦 ∈ 𝐶 𝑏𝑜𝑒 𝑧 ∈ 𝐷}

 maximum distance

𝑒(𝐶, 𝐷) = max {𝑒(𝑦, 𝑧) ∶ 𝑦 ∈ 𝐶 𝑏𝑜𝑒 𝑧 ∈ 𝐷}

 average distance

𝑒(𝐶, 𝐷) = 𝑏𝑏𝑏{𝑒(𝑦, 𝑧) ∶ 𝑦 ∈ 𝐶 𝑏𝑜𝑒 𝑧 ∈ 𝐷}

 distance of centroids

𝑒(𝐶, 𝐷) = 𝑒(𝜈𝐶, 𝜈𝐷), where 𝜈𝐶 is the centroid of 𝐶 and 𝜈𝐷 is the centroid of 𝐷

V-2: 17

IRDM ‘15/16

Single link

The distance between two clusters is the distance between the closest points

 𝑒(𝐶, 𝐷) = min

{𝑒(𝑦, 𝑧) ∶ 𝑦 ∈ 𝐶 𝑏𝑜𝑒 𝑧 ∈ 𝐷}

V-2: 18

IRDM ‘15/16

Strength of single-link

Can n ha hand ndle non non-spheric ical l clu clusters o

• f unequal s

l size ize

V-2: 19

IRDM ‘15/16

Weaknesses of single-link

Se Sens nsitive to

• noi

noise a and nd out

• utliers

Prod

• duc

uces e elong

• ngated clus

usters

V-2: 20

IRDM ‘15/16

Complete link

The distance between two clusters is the distance between the furthest points

 𝑒(𝐶, 𝐷) = max

{𝑒(𝑦, 𝑧) ∶ 𝑦 ∈ 𝐶 𝑏𝑜𝑒 𝑧 ∈ 𝐷}

V-2: 21

IRDM ‘15/16

Strengths of complete link

Le Less s sus usceptible t to

• noi

noise and nd out

• utliers

V-2: 22

IRDM ‘15/16

Weaknesses of complete-link

Break aks s largest c st clusters Bia iased t towards s spherica ical clu l clusters

V-2: 23

IRDM ‘15/16

Group average and Mean distance

Gr Group

• up average is the average of pairwise distances

 𝑒 𝐶, 𝐷 = avg 𝑒 𝑦, 𝑧 : 𝑦 ∈ 𝐶 𝑏𝑜𝑒 𝑧 ∈ 𝐷 = ∑

𝑒 𝑦,𝑧 𝐶 𝐷 𝑦∈𝐶,𝑧∈𝐷

Mean an di dista stance is the distance of the cluster centroids

 𝑒 𝐶, 𝐷 = 𝑒(𝜈𝐶, 𝜈𝐷)

V-2: 24

IRDM ‘15/16

Properties of group average

A compromise between single and complete link Le Less s sus usceptible t to

• noi

noise and nd out

• utliers

 similar to complete link

Bia iased t towards s spherica ical clu l clusters

 similar to complete link

V-2: 25

IRDM ‘15/16

Ward’s method

Ward’s dis istanc nce between clusters 𝐵 and 𝐶 is the increase in sum of squared errors (SSE) when the two clusters are merged

 SSE for cluster 𝐵 is 𝑇𝑇𝐹𝐵 = ∑

𝑦 − 𝜈𝐵 2

𝑦∈𝐵

 difference for merging clusters 𝐵 and 𝐶 into cluster 𝐷 is then

𝑒(𝐵, 𝐶) = Δ𝑇𝑇𝐹𝐷 = 𝑇𝑇𝐹𝐷 – 𝑇𝑇𝐹𝐵 – 𝑇𝑇𝐹𝐶

 or, equivalently, weighted mean distance

𝑒 𝐵, 𝐶 =

𝐵 𝐶 𝐵 +|𝐶| 𝜈A − 𝜈𝐶 2

V-2: 26

IRDM ‘15/16

Discussion on Ward’s method

Le Less s sus usceptible t to

• noi

noise and nd out

• utliers

Biase ases t s towar ards sp s spherical al clust sters Hierarchical analogue of 𝑙-means

 hence many shared pro’s and con’s  can be used to initialise 𝑙-means

V-2: 27

IRDM ‘15/16

Comparison

V-2: 28

Single link Group average Complete link Ward’s method

IRDM ‘15/16

Comparison

V-2: 29

Single link Group average Complete link Ward’s method

IRDM ‘15/16

Comparison

V-2: 30

Single link Group average Complete link Ward’s method

IRDM ‘15/16

Comparison

V-2: 31

Single link Group average Complete link Ward’s method

IRDM ‘15/16

Lance-Williams formula

After merging clusters 𝐵 and 𝐶 into cluster 𝐷 we need to compute 𝐷’s distance to another cluster 𝑎. The Lance- Williams formula provides a general equation for this:

𝑒 𝐷, 𝑎 = 𝛽𝐵𝑒 𝐵, 𝑎 + 𝛽𝐶𝑒 𝐶, 𝑎 + 𝛾𝑒 𝐵, 𝐶 + 𝛿|𝑒 𝐵, 𝑎 − 𝑒 𝐶, 𝑎 |

V-2: 32

𝛽𝐵 𝛽𝐶 𝛾 𝛿

Single link 1/2 1/2 – 1/2 Complete link 1/2 1/2 1/2 Group average |𝐵|/(|𝐵| + |𝐶|) |𝐶|/(|𝐵| + |𝐶|) Mean distance |𝐵|/(|𝐵| + |𝐶|) |𝐶|/(|𝐵| + |𝐶|) – |𝐵||𝐶|/(|𝐵| + |𝐶|)2 Ward’s method (|𝐵| + |𝑎|)/(|𝐵| + |𝐶| + |𝑎|) (|𝐶| + |𝑎|)/(|𝐵| + |𝐶| + |𝑎|) – |𝑎|/(|𝐵| + |𝐶| + |𝑎|)

IRDM ‘15/16

Computational complexity

Takes 𝑃(𝑜3) time in most cases

 𝑜 steps  in each step, 𝑜2 distance matrix must be updated and searched

𝑃(𝑜2 log (𝑜)) time for some approaches that use appropriate data structures

 e.g. keep distances in a heap  each step takes 𝑃(𝑜 log 𝑜) time

𝑃(𝑜2) space complexity

 have to store the distance matrix

V-2: 33

IRDM ‘15/16

Chapter 5.6:

Gri Grid and and Densit Density-bas based ed

Aggarwal Ch. 6.6

V-2: 34

IRDM ‘15/16

The idea

Representation-based clustering can find only convex clusters

 data may contain interesting

non-convex clusters

V-2: 35

In den ensit ity-based sed clust ster ering a cluster is a ‘dense area of points’

 how to define ‘dense area’?

IRDM ‘15/16

Grid-based Clustering

Alg lgorithm hm GENERICGRID(data 𝑬, num-ranges 𝑞, min-density 𝜐) :

 discretise each dimension of 𝑬 into 𝑞 ranges  determine those cells with density ≥ 𝜐  create a graph 𝐻 with a node per dense cell,

add an edge if the two cells are adjacent

 determine the connected components

re return points in each component as a cluster

V-2: 36

IRDM ‘15/16

Discussing Grid-based clustering

The G Good

• od

 we don’t have to specify 𝑙  we can find arbitrarily shaped clusters

Th The Ba Bad

 we have to specify a global minimal density 𝜐  only points in dense cells are part of clusters, all points in

neighbouring sparse cells are ignored

The Ugl e Ugly

 we consider only a single, global, rectangular-shaped grid  number of grid cells increases exponentially with dimensionality

V-2: 37

IRDM ‘15/16

Some definitions

An 𝝑-neighbo hbour urhood of point 𝒚 of data 𝑬 is the set of points of 𝑬 that are within 𝜗 distance from 𝒚

 𝑂𝜗 𝒚 = 𝒛 ∈ 𝑬: 𝑒 𝒚, 𝒛 ≤ 𝜗 -- note, we count x aswell!  parameter 𝜗 is set by the user

Point 𝒚 ∈ 𝑬 is a cor

• re p

poin

• int if 𝑂𝜗 𝒚

≥ 𝑛𝑛𝑜𝑞𝑛𝑛

 minpts

ts (aka 𝜐) is a user supplied parameter

Point 𝒚 ∈ 𝑬 is a border poin

• int if it is not a core point,

but 𝒚 ∈ 𝑂𝜗(𝒜) for some core point 𝒜 A point 𝒚 ∈ 𝑬 that is neither a core point nor a border point is called a nois

• ise p

poin

• int

(be aware: some definitions do count a point as a member of its own 𝜗-neighborhood, some do not. Here we do.) V-2: 38

IRDM ‘15/16

Example

(minpts was 5, now 6 to make clear we count x as an epsilon-neighbor of itself) V-2: 39

x z y min inpt pts = 6

Core point Noise point Border point

IRDM ‘15/16

Density reachability

Point 𝒚 ∈ 𝑬 is direc ectly d y density r y reacha chabl ble e from point 𝒛 ∈ 𝑬 if

 𝒛 is a core point  𝒚 ∈ 𝑂𝜗(𝒛)

Point 𝒚 ∈ 𝑬 is densi sity r y reach chable e from point 𝒛 ∈ 𝑬 if there is a chain of points 𝒚0, 𝒚1, … , 𝒚𝑚 s.t. 𝒚 = 𝒚0, 𝒛 = 𝒚𝑚, and 𝒚𝑗−𝟐 is directly density reachable from 𝒚𝑗 for all 𝑛 = 1, … , 𝑚

 not a symmetric relationship (!)

Points 𝒚, 𝒛 ∈ 𝑬 are densi sity c y conne nected ed if there exists a core point 𝒜 s.t. both 𝒚 and 𝒛 are density reachable from 𝒜

V-2: 40

IRDM ‘15/16

Density-based clusters

A de densi sity-based ed c cluster er is a maximal set of density connected points

(image from Wikipedia) V-2: 41

IRDM ‘15/16

The DBSCAN algorithm

 fo

for ea each unvisited point 𝒚 in the data

 compute 𝑂𝜗(𝒚)  if

if 𝑂𝜗 𝒚 ≥ 𝐧𝐧𝐧𝐧𝐧𝐧

 EXPANDCLUSTER(𝑦, ++clusterID)

 EXPANDCLUSTER(𝒚, ID)

 assign 𝒚 to cluster ID and set N ← 𝑂𝜗(𝒚)  fo

for ea each 𝒛 ∈ 𝑂

 if

if 𝒛 is not visited and 𝑂𝜗 𝒛 ≥ 𝐧𝐧𝐧𝐧𝐧𝐧

 𝑂 ← 𝑂 ∪ 𝑂𝜗(𝒛)  if

if 𝒛 does not belong to any cluster

 assign 𝒛 to cluster ID

V-2: 42

IRDM ‘15/16

More on DBSCAN

DBSCAN can return either overlapping or non-overlapping clusters

 ties are broken arbitrarily

The main time complexity comes from computing the neighborhoods

 total 𝑃(𝑜 log 𝑜) with spatial index structures

 won't work with high dimensions, worst case is 𝑃(𝑜2)

With the neighborhoods known, DBSCAN

• nly needs a si

singl gle p pass ass over the data

V-2: 43

IRDM ‘15/16

The parameters

DBSCAN requires two parameters, 𝜗 and 𝐧𝐧𝐧𝐧𝐧𝐧 𝐧𝐧𝐧𝐧𝐧𝐧 controls the minimum size of a cluster

 𝐧𝐧𝐧𝐧𝐧𝐧 = 1 allows singleton clusters  𝐧𝐧𝐧𝐧𝐧𝐧 = 2 makes DBSCAN essentially a single-link clustering  higher values avoid the long-and-narrow clusters of single link

𝜗 controls the required density

 a single 𝜗 is not enough if the clusters are of very different density

V-2: 44

IRDM ‘15/16

Chapter 5.7:

More C e Clustering ing M Models els

Aggarwal Ch. 6.7-6.8

V-2: 45

IRDM ‘15/16

More clustering models

So far we’ve seen

 representative-based clustering  model-based clustering  hierarchical clustering  density-based clustering

There are many more types of clustering, including

 co-clustering  graph clustering (Aggarwal Ch. 6.8)  non-negative matrix factorisation (NMF) (Aggarwal Ch. 6.9)

But we’re not going to discuss these in IRDM.

 phew!

V-2: 46

IRDM ‘15/16

Chapter 5.8:

Clust lustering H Hig igh-Dimens ensional nal Dat Data

Aggarwal Ch. 7.4—7.4.2

V-2: 47

IRDM ‘15/16

Clustering High Dimensional Data

If we compute similarity over many dimensions, all points will be roughly equi-distant. The here e exist no no clus usters ov

• ver ma

many ny dime mensions ns.

 or, are there?

Of course there are!

 data can have a much lower intrinsic dimensionality (SVD)

i.e. many dimensions are noisy, irrelevant, or copies

 data can have clusters embedded in subsets of it

its dim imens nsio ions

V-2: 48

IRDM ‘15/16

Spaces

The full s ll space ce of data 𝑬 is its set of attributes 𝒝 A su subsp space 𝑇 of 𝑬 is a subset of 𝒝, i.e. 𝑇 ⊆ 𝒝

 there exist 2 𝒝 − 1 non-empty subspaces

A su subsp space c clust ster is a cluster 𝐷 over a subspace 𝑇

 a group of points that is highly similar over subspace 𝑇

V-2: 49

IRDM ‘15/16

High-dimensional Grids

In full-dimensional grid-based methods, the grid cells are determined on the intersection of the discretization ranges 𝑞 across all ll dimensions. What happens for high-dimensional data?

 many many grid cells will be empty

CLIQUE is a generalisation of grid-based clustering to

• subspaces. In CLIQUE the ranges are determined over
• nly a sub

ubset o

• f dime

mens nsions

• ns with density greater than 𝜐.

V-2: 50

IRDM ‘15/16

CLustering In QUEst

CLIQUE is the first subspace clustering algorithm.

 partition each dimension into 𝑞 ranges  for each subspace we now have grid

cells of the same volume

 subspace clusters are connected

dense cells in the grid

(Agrawal et al. 1998) V-2: 51

IRDM ‘15/16

Finding dense cells

CLIQUE uses anti-monotonicity to find dense grid cells in subspaces: the higher the dimensionality, the sparser the cells

Main Idea:



every subspace we consider is a ‘transaction database’, every cell is then a ‘transaction’. If a cell is 𝜐-dense, the subspace ‘itemset’ has been ‘bought’.



we now mine frequent itemsets with minsup=1

V-2: 52

IRDM ‘15/16

Example

A-priori for subspace clusters:

For every level 𝑚 in the subspace lattice, we check, for all subspaces 𝑇 ∈ 𝒝 𝑚 whether 𝑇 contains dense cells; but only if all subspaces 𝑇′ ⊂ 𝑇 contain dense cells. If 𝑇 contains dense cells, we report each group of adjacent dense cells as a cluster 𝐷 over subspace 𝑇

V-2: 53 A

Dense cluster in subspace A

IRDM ‘15/16

Example

A-priori for subspace clusters:

For every level 𝑚 in the subspace lattice, we check, for all subspaces 𝑇 ∈ 𝒝 𝑚 whether 𝑇 contains dense cells; but only if all subspaces 𝑇′ ⊂ 𝑇 contain dense cells. If 𝑇 contains dense cells, we report each group of adjacent dense cells as a cluster 𝐷 over subspace 𝑇

V-2: 54 A B

Dense cluster in subspace B

IRDM ‘15/16

Example

A-priori for subspace clusters:

For every level 𝑚 in the subspace lattice, we check, for all subspaces 𝑇 ∈ 𝒝 𝑚 whether 𝑇 contains dense cells; but only if all subspaces 𝑇′ ⊂ 𝑇 contain dense cells. If 𝑇 contains dense cells, we report each group of adjacent dense cells as a cluster 𝐷 over subspace 𝑇

V-2: 55 A B

AB

Dense cluster in subspace AB Dense cluster in subspace AB

IRDM ‘15/16

Example

A-priori for subspace clusters:

For every level 𝑚 in the subspace lattice, we check, for all subspaces 𝑇 ∈ 𝒝 𝑚 whether 𝑇 contains dense cells; but only if all subspaces 𝑇′ ⊂ 𝑇 contain dense cells. If 𝑇 contains dense cells, we report each group of adjacent dense cells as a cluster 𝐷 over subspace 𝑇

V-2: 56 A B

To find dense clusters in a subspace, we only have to consider grid cells that are dense in all super-spaces

IRDM ‘15/16

Discussion of CLIQUE

CLIQUE was the first subspace clustering algorithm.

 and it shows

It produces an enormous amount of clusters

 just like frequent itemset mining

 nothing like ‘a summary of your data‘

This, however, is general problem of subspace clustering

 there are exponentially many subspaces  and for each subspace there are exponentially many clusters

V-2: 57

IRDM ‘15/16

Conclusions

Clustering is one of the most important and most used data analysis methods There exist many different types of clustering

 we’ve seen representative, hierarchical, probabilistic, and density-based

Analysis of clustering methods is often difficult Always think what you’re doing if you use clustering

 in fact, just always think what you’re doing

V-2: 58

IRDM ‘15/16

Thank you!

Clustering is one of the most important and most used data analysis methods There exist many different types of clustering

 we’ve seen representative, hierarchical, probabilistic, and density-based

Analysis of clustering methods is often difficult Always think what you’re doing if you use clustering

 in fact, just always think what you’re doing

V-2: 59