Clustering - - PowerPoint PPT Presentation

Clustering

Reference:http://www.elet.polimi.it/upload/matteucc/Clustering/tutorial_html/

Dr Ahmed Rafea

Outline

• Introduction
• Clustering Algorithms

– K-means – Fuzzy C-means – Hierarchical clustering

Introduction(1)

• Clustering can be considered the most important

unsupervised learning problem; so, as every other problem of this kind,

• it deals with finding a structure in a collection of

unlabeled data.

• A loose definition of clustering could be “the process of
• rganizing objects into groups whose members are

similar in some way”.

• A cluster is therefore a collection of objects which are

“similar” between them and are “dissimilar” to the objects belonging to other clusters.

Introduction(2)

• In this case we easily identify the 4 clusters into which the data can be divided;
• the similarity criterion is distance: two or more objects belong to the same

cluster if they are “close” according to a given distance (in this case geometrical distance). This is called distance-based clustering.

• Another kind of clustering is conceptual clustering: two or more objects belong

to the same cluster if this one defines a concept common to all that objects.

Introduction(3)

• The main requirements that a clustering

algorithm should satisfy are:

– scalability; – dealing with different types of attributes; – discovering clusters with arbitrary shape; – minimal requirements for domain knowledge to determine input parameters; – ability to deal with noise and outliers; – insensitivity to order of input records; – high dimensionality; – interpretability and usability.

Introduction(4)

• There are a number of problems with clustering. Among

them:

– current clustering techniques do not address all the requirements adequately – dealing with large number of dimensions and large number of data items can be problematic because of time complexity; – the effectiveness of the method depends on the definition of “distance” (for distance-based clustering); – if an obvious distance measure doesn’t exist we must “define” it, which is not always easy, especially in multi-dimensional spaces; – the result of the clustering algorithm can be interpreted in different ways.

Clustering Algorithms

• Exclusive Clustering

– Data are grouped in an exclusive way, so that if a certain datum belongs to a definite cluster then it could not be included in another cluster.

• Overlapping Clustering

– The overlapping clustering, uses fuzzy sets to cluster data, so that each point may belong to two or more clusters with different degrees of membership.

• Hierarchical Clustering

– A hierarchical clustering algorithm is based on the union between the two nearest clusters.

• Probabilistic Clustering

– This algorithm uses a completely probabilistic approach.

K-Means Clustering (1)

• K-means is one of the simplest unsupervised learning

algorithms that solve the well known clustering problem.

• The main idea is to define k centroids, one for each

cluster.

• These centroids should be placed as much as possible

far away from each other.

• Take each point belonging to a given data set and

associate it to the nearest centroid.

• Re-calculate k new centroids as barycenters of the

clusters resulting from the previous step.

• After we have these k new centroids, a new binding has

to be done between the same data set points and the nearest new centroid.

• Loop until no more changes are done.

K-Means Clustering (2)

• This algorithm aims at minimizing an objective function, in this case a

squared error function. The objective function

k n

J = Σ Σ || xi

(j) – cj ||2

j=1 i=1

Where || xi

(j) – cj ||2 is a chosen distance measure between a

data point xi

(j) and the cluster centre cj

• Although it can be proved that the procedure will always terminate, the k-

means algorithm does not necessarily find the most optimal configuration, corresponding to the global objective function minimum.

• The algorithm is also significantly sensitive to the initial randomly selected

cluster centers. The k-means algorithm can be run multiple times to reduce this effect.

Fuzzy C-Means Clustering(1)

• Fuzzy c-means (FCM) is a method of clustering which allows
• ne piece of data to belong to two or more clusters. It is based
• n minimization of the following objective function:

where m is any real number greater than 1, uij is the degree of membership of xi in the cluster j, xi is the ith of d-dimensional measured data, cj is the d-dimension center of the cluster, and ||*|| is any norm expressing the similarity between any measured data and the center.

Fuzzy C-Means Clustering(2)

• Fuzzy partitioning is carried out through an

iterative optimization of the objective function shown above, with the update of membership uij and the cluster centers cj by:

Look for the formula in the tutorial

• This iteration will stop when maxij{|uijk+1 - uijk|}< ε ,

where ε is a termination criterion between 0 and 1, whereas k are the iteration steps. This procedure converges to a local minimum or a saddle point of Jm.

Fuzzy C-Means Clustering(3)

1. Initialize U=[uij] matrix, U(0) 2. At k-step: calculate the centers vectors C(k)=[cj] with U(k)

Look for the formula in the tutorial

3. Update U(k) , U(k+1)

Look for the formula in the tutorial

4. If || U(k+1) - U(k)||< ε then STOP; otherwise return to step 2.

Fuzzy C-Means Clustering(4)

• For a better understanding, we may consider this simple mono-dimensional
• example. Given a certain data set, suppose to represent it as distributed on

an axis. The figure below shows this:

• Looking at the picture, we may identify two clusters in proximity of the

two data concentrations. We will refer to them using ‘A’ and ‘B’. In the k- means algorithm - we associated each datum to a specific centroid; therefore, this membership function looked like this:

Fuzzy C-Means Clustering(5)

• In the FCM approach, instead, the same given datum does not belong

exclusively to a well defined cluster, but it can be placed in a middle way. In this case, the membership function follows a smoother line to indicate that every datum may belong to several clusters with different values of the membership coefficient.

• In the figure above, the datum shown as a red marked spot belongs more to

the B cluster rather than the A cluster. The value 0.2 of ‘m’ indicates the degree of membership to A for such datum.

Fuzzy C-Means Clustering(6)

• Now, instead of using a graphical representation, we introduce a matrix U

whose factors are the ones taken from the membership functions:

Look for the matrices in the tutorial

(a) (b)

• We have C = 2 columns (C = 2 clusters) and N rows, where C is the total number of

clusters and N is the total number of data. The generic element is so indicated: uij.

• We can notice that in the first case (a) the coefficients are always unitary. It is so to

indicate the fact that each datum can belong only to one cluster. Other properties are shown below:

Hierarchical Clustering (1)

• Given a set of N items to be clustered, and an N*N distance (or

similarity) matrix, the basic process of hierarchical clustering is this:

– Start by assigning each item to a cluster, so that if you have N items, you now have N clusters, each containing just one item. Let the distances (similarities) between the clusters the same as the distances (similarities) between the items they contain. – Find the closest (most similar) pair of clusters and merge them into a single cluster, so that now you have one cluster less. – Compute distances (similarities) between the new cluster and each of the old clusters. – Repeat steps 2 and 3 until all items are clustered into a single cluster of size N.

• Of course there is no point in having all the N items grouped in a

single cluster but, once you have got the complete hierarchical tree, if you want k clusters you just have to cut the k-1 longest links

Hierarchical Clustering (2)

• Step 3 can be done in different ways, which is what

distinguishes single-linkage from complete-linkage and average-linkage clustering.

– In single-linkage clustering, we consider the distance between

• ne cluster and another cluster to be equal to the shortest

distance from any member of one cluster to any member of the

• ther cluster.

– In complete-linkage clustering, we consider the distance between one cluster and another cluster to be equal to the greatest distance from any member of one cluster to any member of the other cluster. – In average-linkage clustering, we consider the distance between

• ne cluster and another cluster to be equal to the average

distance from any member of one cluster to any member of the

• ther cluster.

Hierarchical Clustering (3)

• This kind of hierarchical clustering is called

agglomerative because it merges clusters

• iteratively. There is also a divisive

hierarchical clustering which does the reverse by starting with all objects in one cluster and subdividing them into smaller

• pieces. Divisive methods are not generally

available, and rarely have been applied.

single-linkage clustering (1)

1. Begin with the disjoint clustering having level L(0) = 0 and sequence number m = 0. 2. Find the least dissimilar pair of clusters in the current clustering, say pair (r), (s), according to d[(r),(s)] = min d[(i),(j)] where the minimum is over all pairs of clusters in the current clustering. 3. Increment the sequence number : m = m +1. Merge clusters (r) and (s) into a single cluster to form the next clustering m. Set the level of this clustering to L(m) = d[(r),(s)] 4. Update the proximity matrix, D, by deleting the rows and columns corresponding to clusters (r) and (s) and adding a row and column corresponding to the newly formed cluster. The proximity between the new cluster, denoted (r,s) and old cluster (k) is defined in this way: d[(k), (r,s)] = min d[(k),(r)], d[(k),(s)] 5. If all objects are in one cluster, stop. Else, go to step 2.

single-linkage clustering(2)

• A simple example: A

hierarchical clustering of distances in kilometers between some Italian cities.

• Input distance matrix

(L(0) = 0 for all the clusters)

• The nearest pair of cities

is MI and TO, at distance 138

BA FI MI NA RM TO BA 662 877 255 412 996 FI 662 295 468 268 400 MI 877 295 754 564 138 NA 255 468 754 219 869 RM 412 268 564 219 669 TO 996 400 138 869 669

single-linkage clustering(3)

BA FI MI/ TO NA RM

BA

662 877 255 412

FI

662 295 468 268

MI/ TO

877 295 754 564

NA

255 468 754 219

RM

412 268 564 219

• These are merged into a single cluster

called "MI/TO". The level of the new cluster is L(1) = 138 and the new sequence number is m = 1.

• Then we compute the distance from

this new compound object to all other

• bjects. So the distance from

"MI/TO" to RM is chosen to be 564, which is the distance from MI to RM, and so on. (RM to MI is shooter than the distance from RM to TO which is the second member in the cluster

single-linkage clustering(4)

BA FI MI/ TO NA/ RM BA 662 877 255 FI 662 295 268 MI/TO 877 295 564 NA/RM 255 268 564

• min d(i,j) = d(NA,RM) = 219 =>

merge NA and RM into a new cluster called NA/RM L(2) = 219 m = 2

• min d(i,j) = d(BA,NA/RM) = 255

=> merge BA and NA/RM into a new cluster called BA/NA/RM L(BA/NA/RM) = 255 m = 3

BA/N A/RM FI MI/ TO BA/NA/ RM 268 564 FI 268 295 MI/TO 564 295

single-linkage clustering(5)

BA/NA /RM/FI MI/ TO BA/NA/ RM/FI 295 MI/TO 295

• min d(i,j) = d(BA/NA/RM,FI) = 268

=> merge BA/NA/RM and FI into a new cluster called BA/FI/NA/RM L(4) = 268 m = 4

• Finally, we merge the last two clusters

at level 295.

• The process is summarized by the

following hierarchical tree: