Data Mining and Machine Learning: Fundamental Concepts and - PowerPoint PPT Presentation

Data Mining and Machine Learning: Fundamental Concepts and Algorithms dataminingbook.info Mohammed J. Zaki 1 Wagner Meira Jr. 2 1 Department of Computer Science Rensselaer Polytechnic Institute, Troy, NY, USA 2 Department of Computer Science Universidade Federal de Minas Gerais, Belo Horizonte, Brazil Chapter 16: Spectral & Graph Clustering Zaki & Meira Jr. (RPI and UFMG) Data Mining and Machine Learning Chapter 16: Spectral & Graph Clustering

Graphs and Matrices: Adjacency Matrix Given a dataset D = { x i } n i = 1 consisting of n points in R d , let A denote the n × n symmetric similarity matrix between the points, given as  ···  a 11 a 12 a 1 n ··· a 21 a 22 a 2 n   A =  . . .  . . .   . . ··· .   a n 1 a n 2 ··· a nn where A ( i , j ) = a ij denotes the similarity or affinity between points x i and x j . We require the similarity to be symmetric and non-negative, that is, a ij = a ji and a ij ≥ 0, respectively. The matrix A is the weighted adjacency matrix for the data graph. If all affinities are 0 or 1, then A represents the regular adjacency relationship between the vertices. Zaki & Meira Jr. (RPI and UFMG) Data Mining and Machine Learning Chapter 16: Spectral & Graph Clustering

bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC bC Iris Similarity Graph: Mutual Nearest Neighbors | V | = n = 150, | E | = m = 1730 Zaki & Meira Jr. (RPI and UFMG) Data Mining and Machine Learning Chapter 16: Spectral & Graph Clustering Edge weight given as

Graphs and Matrices: Degree Matrix For a vertex x i , let d i denote the degree of the vertex, defined as n � d i = a ij j = 1 We define the degree matrix ∆ of graph G as the n × n diagonal matrix: � n    j = 1 a 1 j 0 ··· 0  d 1 0 ··· 0 � n 0 d 2 ··· 0 0 j = 1 a 2 j ··· 0     ∆ =  = . . .  ...   . . .  ... . . . . . .     . . . . . .    � n 0 0 ··· d n 0 0 ··· j = 1 a nj ∆ can be compactly written as ∆( i , i ) = d i for all 1 ≤ i ≤ n . Zaki & Meira Jr. (RPI and UFMG) Data Mining and Machine Learning Chapter 16: Spectral & Graph Clustering

Graphs and Matrices: Normalized Adjacency Matrix The normalized adjacency matrix is obtained by dividing each row of the adjacency matrix by the degree of the corresponding node. Given the weighted adjacency matrix A for a graph G , its normalized adjacency matrix is defined as a 11 a 12 a 1 n ···   d 1 d 1 d 1 a 21 a 22 a 2 n ···   d 2 d 2 d 2 M = ∆ − 1 A =    . . .  ... . . .   . . .   a n 1 a n 2 a nn ··· d n d n d n Because A is assumed to have non-negative elements, this implies that each a ij element of M , namely m ij is also non-negative, as m ij = d i ≥ 0. Each row in M sums to 1, which implies that 1 is an eigenvalue of M . In fact, λ 1 = 1 is the largest eigenvalue of M , and the other eigenvalues satisfy the property that | λ i | ≤ 1. Because M is not symmetric, its eigenvectors are not necessarily orthogonal. Zaki & Meira Jr. (RPI and UFMG) Data Mining and Machine Learning Chapter 16: Spectral & Graph Clustering

Example Graph Adjacency and Degree Matrices 1 6 2 4 5 3 7 Its adjacency and degree matrices are given as  0 1 0 1 0 1 0   3 0 0 0 0 0 0  1 0 1 1 0 0 0 0 3 0 0 0 0 0         0 1 0 1 0 0 1 0 0 3 0 0 0 0         A = 1 1 1 0 1 0 0 ∆ = 0 0 0 4 0 0 0         0 0 0 1 0 1 1 0 0 0 0 3 0 0         1 0 0 0 1 0 1 0 0 0 0 0 3 0     0 0 1 0 1 1 0 0 0 0 0 0 0 3 Zaki & Meira Jr. (RPI and UFMG) Data Mining and Machine Learning Chapter 16: Spectral & Graph Clustering

Example Graph Normalized Adjacency Matrix 1 6 2 4 5 3 7 The normalized adjacency matrix is as follows:   0 0 . 33 0 0 . 33 0 0 . 33 0 0 . 33 0 0 . 33 0 . 33 0 0 0     0 0 . 33 0 0 . 33 0 0 0 . 33   M = ∆ − 1 A =   0 . 25 0 . 25 0 . 25 0 0 . 25 0 0     0 0 0 0 . 33 0 0 . 33 0 . 33     0 . 33 0 0 0 0 . 33 0 0 . 33   0 0 0 . 33 0 0 . 33 0 . 33 0 The eigenvalues of M are: λ 1 = 1, λ 2 = 0 . 483, λ 3 = 0 . 206, λ 4 = − 0 . 045, λ 5 = − 0 . 405, λ 6 = − 0 . 539, λ 7 = − 0 . 7 Zaki & Meira Jr. (RPI and UFMG) Data Mining and Machine Learning Chapter 16: Spectral & Graph Clustering

Graph Laplacian Matrix The Laplacian matrix of a graph is defined as L = ∆ − A � n  j = 1 a 1 j 0 ··· 0    a 11 a 12 ··· a 1 n � n 0 j = 1 a 2 j ··· 0 a 21 a 22 ··· a 2 n     =  −  . . .   . . .  ... . . . . . .     . . ··· . . . .    � n ··· 0 0 ··· j = 1 a nj a n 1 a n 2 a nn �  j � = 1 a 1 j − a 12 ··· − a 1 n  − a 21 � ··· − a 2 n j � = 2 a 2 j   =  . . .  . . .   . . ··· .   − a n 1 − a n 2 ··· � j � = n a nj L is a symmetric, positive semidefinite matrix. This means that L has n real, non-negative eigenvalues, which can be arranged in decreasing order as follows: λ 1 ≥ λ 2 ≥ ··· ≥ λ n ≥ 0. Because L is symmetric, its eigenvectors are orthonormal. The rank of L is at most n − 1, and the smallest eigenvalue is λ n = 0. Zaki & Meira Jr. (RPI and UFMG) Data Mining and Machine Learning Chapter 16: Spectral & Graph Clustering

Example Graph: Laplacian Matrix 1 6 2 4 5 3 7 The graph Laplacian is given as  3 − 1 0 − 1 0 − 1 0  − 1 3 − 1 − 1 0 0 0     0 − 1 3 − 1 0 0 − 1     L = ∆ − A = − 1 − 1 − 1 4 − 1 0 0     0 0 0 − 1 3 − 1 − 1     − 1 0 0 0 − 1 3 − 1   0 0 − 1 0 − 1 − 1 3 The eigenvalues of L are as follows: λ 1 = 5 . 618, λ 2 = 4 . 618, λ 3 = 4 . 414, λ 4 = 3 . 382, λ 5 = 2 . 382, λ 6 = 1 . 586, λ 7 = 0 Zaki & Meira Jr. (RPI and UFMG) Data Mining and Machine Learning Chapter 16: Spectral & Graph Clustering

Normalized Laplacian Matrices The normalized symmetric Laplacian matrix of a graph is defined as � j � = 1 a 1 j   a 12 a 1 n √ √ √ − ··· − d 1 d 1 d 1 d 2 d 1 d n   � j � = 2 a 2 j a 21 a 2 n √ √ √  − ··· −  L s = ∆ − 1 / 2 L ∆ − 1 / 2 =   d 2 d 1 d 2 d 2 d 2 d n   . . . ...   . . .  . . .    � j � = n a nj   a n 1 a n 2 √ √ − − ··· √ d n d n d n d 1 d n d 2 L s is a symmetric, positive semidefinite matrix, with rank at most n − 1. The smallest eigenvalue λ n = 0. The normalized asymmetric Laplacian matrix is defined as  � j � = 1 a 1 j  − a 12 − a 1 n ··· d 1 d 1 d 1   � j � = 2 a 2 j − a 21 − a 2 n   ··· L a = ∆ − 1 L =  d 2 d 2 d 2    . . . ... . . .   . . .     � j � = n a nj − a n 1 − a n 2 ··· d n d n d n L a is also a positive semi-definite matrix with n real eigenvalues λ 1 ≥ λ 2 ≥ ··· ≥ λ n = 0. Zaki & Meira Jr. (RPI and UFMG) Data Mining and Machine Learning Chapter 16: Spectral & Graph Clustering

Example Graph Normalized Symmetric Laplacian Matrix 1 6 2 4 5 3 7 The normalized symmetric Laplacian is given as  1 − 0 . 33 0 − 0 . 29 0 − 0 . 33 0  − 0 . 33 1 − 0 . 33 − 0 . 29 0 0 0     0 − 0 . 33 1 − 0 . 29 0 0 − 0 . 33   L s =   − 0 . 29 − 0 . 29 − 0 . 29 1 − 0 . 29 0 0     0 0 0 − 0 . 29 1 − 0 . 33 − 0 . 33     − 0 . 33 0 0 0 − 0 . 33 1 − 0 . 33   0 0 − 0 . 33 0 − 0 . 33 − 0 . 33 1 The eigenvalues of L s are as follows: λ 1 = 1 . 7, λ 2 = 1 . 539, λ 3 = 1 . 405, λ 4 = 1 . 045, λ 5 = 0 . 794, λ 6 = 0 . 517, λ 7 = 0 Zaki & Meira Jr. (RPI and UFMG) Data Mining and Machine Learning Chapter 16: Spectral & Graph Clustering