' ' ' ' ' ' Random'Walks'as'a'Stable'Analogue'of'Eigenvectors' - PowerPoint PPT Presentation

' ' ' ' ' ' Random'Walks'as'a'Stable'Analogue'of'Eigenvectors' ' '(with'Applications'to'Nearly?Linear?Time'Graph'Partitioning)' ' ' ' Lorenzo'Orecchia,'MIT'Math' Based'on'joint'works'with'Michael'Mahoney'(Stanford),'Sushant'Sachdeva'(Yale)'and' 'Nisheeth'Vishnoi'(MSR'India).' TexPoint'fonts'used'in'EMF.''

Why$Spectral$Algorithms$for$Graph$Problems$…$ …'in'practice?' • 'Simple'to'implement' • 'Can'exploit'very'efUicient'linear'algebra'routines' • 'Perform''well'in'practice'for'many'problems' ' …'in'theory? ' • 'Connections'between'spectral'and'combinatorial'objects'' • 'Connections'to'Markov'Chains'and'Probability'Theory' • ''Intuitive'geometric'viewpoint' ' RECENT'ADVANCES:'' Fast'algorithms'for'fundamental'combinatorial'problems' ' 'rely'on'spectral'and'optimization'ideas' '

Spectral$Algorithms$for$Graph$Par88oning$ Spectral'algorithms'are'widely'used'in'many'graph?partitioning'applications: 'clustering,'image'segmentation,'community?detection,'etc. ! CLASSICAL!!VIEW:!! '?'Based'on'Cheeger’s'Inequality'' '?'Eigenvectors'sweep?cuts'reveal'sparse'cuts'in'the'graph' '

Spectral$Algorithms$for$Graph$Par88oning$ Spectral'algorithms'are'widely'used'in'many'graph?partitioning'applications: 'clustering,'image'segmentation,'community?detection,'etc. ! CLASSICAL!!VIEW:!! '?'Based'on'Cheeger’s'Inequality'' '?'Eigenvectors'sweep?cuts'reveal'sparse'cuts'in'the'graph' NEW!TREND:! '?'Random'walk'vectors'replace'eigenvectors:' • 'Fast'Algorithms'for'Graph'Partitioning' • 'Local'Graph'Partitioning' • 'Real'Network'Analysis' '?'Different'random'walks:'PageRank,'Heat?Kernel,'etc.' ' '

Why$Random$Walks?$A$Prac88oner’s$View$ Advantages'of'Random'Walks:' 1) Quick'approximation'to'eigenvector'in'massive'graphs' A ='adjacency'matrix ' ' ' D '='diagonal'degree'matrix' W '=' AD A 1 =''natural'random'walk'matrix ' L '=' D –' A '='Laplacian'matrix '' Second'Eigenvector'of'the'Laplacian'can'be'computed'by'iterating' W :' For'random'' y 0 's.t.'''''''''''''''''''''''''','compute y T 0 D ¡ 1 1 = 0 D ¡ 1 W t y 0 '

Why$Random$Walks?$A$Prac88oner’s$View$ Advantages'of'Random'Walks:' 1) Quick'approximation'to'eigenvector'in'massive'graphs' A ='adjacency'matrix ' ' ' D '='diagonal'degree'matrix' W '=' AD A 1 =''natural'random'walk'matrix ' L '=' D –' A '='Laplacian'matrix '' Second'Eigenvector'of'the'Laplacian'can'be'computed'by'iterating' W :' For'random'' y 0 's.t.'''''''''''''''''''''''''','compute y T 0 D ¡ 1 1 = 0 D ¡ 1 W t y 0 D ¡ 1 W t y 0 x 2 ( L ) = lim t !1 In'the'limit,'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''.' || W t y 0 || D ¡ 1 '

Why$Random$Walks?$A$Prac88oner’s$View$ Advantages'of'Random'Walks:' 1) Quick'approximation'to'eigenvector'in'massive'graphs' A ='adjacency'matrix ' ' ' D '='diagonal'degree'matrix' W '=' AD A 1 =''natural'random'walk'matrix ' L '=' D –' A '='Laplacian'matrix '' Second'Eigenvector'of'the'Laplacian'can'be'computed'by'iterating' W :' For'random'' y 0 's.t.'''''''''''''''''''''''''','compute y T 0 D ¡ 1 1 = 0 D ¡ 1 W t y 0 D ¡ 1 W t y 0 In'the'limit,'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''.' x 2 ( L ) = lim t !1 || W t y 0 || D ¡ 1 Heuristic:'For'massive'graphs,'pick' t 'as'large'as'computationally'affordable.' '

Why$Random$Walks?$A$Prac88oner’s$View$ Advantages'of'Random'Walks:' 1) Quick'approximation'to'eigenvector'in'massive'graphs' 2) Statistical'robustness' 'Real?world'graphs'are'noisy' ' GROUND!TRUTH! GRAPH!

Why$Random$Walks?$A$Prac88oner’s$View$ Advantages'of'Random'Walks:' 1) Quick'approximation'to'eigenvector'in'massive'graphs' 2) Statistical'robustness' 'Real?world'graphs'are'noisy' ' NOISY! MEASUREMENT! GROUND4TRUTH! INPUT!GRAPH! GRAPH! GOAL :'estimate'eigenvector'of'ground? truth'graph.'

Why$Random$Walks?$A$Prac88oner’s$View$ Advantages'of'Random'Walks:' 1) Quick'approximation'to'eigenvector'in'massive'graphs' 2) Statistical'robustness' '' NOISY! MEASUREMENT! ' GROUND4TRUTH! INPUT!GRAPH! GRAPH! GOAL :'estimate'eigenvector'of'ground?truth'graph.' ' OBSERVATION :'eigenvector'of'input'graph'can'have'very'large'variance,'' ' 'as'it''can'be'very'sensitive'to'noise' ' RANDOM4WALK!VECTORS! provide'better,'more'stable'estimates. ! ' '

This$Talk$ QUESTION:'' Why'random?walk'vectors'in'the'design'of'fast'algorithms?' '' ' ' ' ' ' ' ' ' ' ''

This$Talk$ QUESTION:'' Why'random?walk'vectors'in'the'design'of'fast'algorithms?' 'ANSWER:'Stable,' regularized 'version'of'the'eigenvector' ' ' ' ' ' ' ' ' ' ''

This$Talk$ QUESTION:'' Why'random?walk'vectors'in'the'design'of'fast'algorithms?' 'ANSWER:'Stable,' regularized 'version'of'the'eigenvector' ' ' GOALS'OF'THIS'TALK:' ?'Show'optimization'perspective'on'why'random'walks'arise' ?'Application'to'nearly?linear?time'balanced'graph'partitioning ' ' ' ' '

' ' ' ' ' ' Random'Walks'' as'Regularized'Eigenvectors'

What$is$Regulariza8on?$ Regularization'is'a'fundamental'technique'in'optimization' ' ' ' WELL?BEHAVED' OPTIMIZATION' OPTIMIZATION' PROBLEM' PROBLEM' ' • 'Stable'optimum' • 'Unique'optimal'solution' • 'Smoothness'conditions' ' … '

What$is$Regulariza8on?$ Regularization'is'a'fundamental'technique'in'optimization' ' ' ' WELL?BEHAVED' OPTIMIZATION' OPTIMIZATION' PROBLEM' PROBLEM' ' Regularizer'' F Parameter' ¸ ' > ' 0 BeneUits'of'Regularization'in'Learning'and'Statistics : ' • 'Increases'stability' • 'Decreases'sensitivity'to'random'noise' • 'Prevents'overUitting'

Instability$of$Eigenvector$ EXPANDER!

Instability$of$Eigenvector$ − ✏ 1 − ✏ Current! eigenvector! − ✏ − ✏ EXPANDER!

Instability$of$Eigenvector$ − ✏ − ✏ 1 1 Small'perturbation' − ✏ Current! eigenvector! − ✏ − ✏ EXPANDER! Eigenvector!Changes!Completely!!

The$Laplacian$Eigenvalue$Problem$ 'Quadratic'Formulation ' 1 d min x T Lx s.t. || x || 2 = 1 x T 1 = 0 For'simplicity,'take'G'to'be'd?regular. $

The$Laplacian$Eigenvalue$Problem$ 'Quadratic'Formulation ' SDP'Formulation ' 1 1 d min x T Lx L • X d min s.t. || x || 2 = 1 I • X = 1 s.t. 11 T • X = 0 x T 1 = 0 X º 0

The$Laplacian$Eigenvalue$Problem$ 'Quadratic'Formulation ' SDP'Formulation ' 1 1 d min x T Lx L • X d min s.t. || x || 2 = 1 I • X = 1 s.t. 11 T • X = 0 x T 1 = 0 X º 0 Programs'have'same'optimum.'Take'optimal'solution'' $ X ¤ = x ¤ ( x ¤ ) T

Instability$of$Linear$Op8miza8on$ Consider'a'convex'set''''''''''''''''''and'a'linear'optimization'problem:' S ½ R n ' f ( c ) = argmin x 2 S c T x ' ' The'optimal'solution' f ( c )'may'be'very'unstable'under'perturbation''of'c':' ' k c 0 � c k  δ k f ( c 0 ) ¡ f ( c ) k >> ± '' and ' c 0 c f ( c 0 ) f ( c ) S

Regulariza8on$Helps$Stability$ S ½ R n Consider'a'convex'set''''''''''''''''''and'a !regularized! linear'optimization' problem' f ( c ) = argmin x 2 S c T x + F ( x ) ' '' where' F 'is' ¾ ?strongly'convex.'' ' k f ( c ) � f ( c 0 ) k  δ k c 0 � c k  δ Then:' implies ' σ ' ' '' c 0 T x + F ( x ) c T x + F ( x ) f ( c 0 ) f ( c )

Regulariza8on$Helps$Stability$ S ½ R n Consider'a'convex'set''''''''''''''''''and'a !regularized! linear'optimization' problem' f ( c ) = argmin x 2 S c T x + F ( x ) ' '' where' F 'is' ¾ ?strongly'convex.'' ' k f ( c ) � f ( c 0 ) k  δ k c 0 � c k  δ Then:' implies ' σ ' ' '' slope ≤ δ c 0 T x + F ( x ) c T x + F ( x ) f ( c 0 ) f ( c )

Regularized$Spectral$Op8miza8on$ SDP'Formulation ' 1 L • X d min I • X = 1 s.t. 11 T • X = 0 Density'Matrix' X º 0 Eigenvector'decomposition'of' X : 8 i,p i ¸ 0 , X = P p i v i v T P p i = 1 , i 8 i,v T i 1 = 0 . Eigenvalues'of' X deUine'probability'distribution

Regularized$Spectral$Op8miza8on$ SDP'Formulation ' 1 d min L • X s.t. I • X = 1 Density'Matrix' J • X = 0 X º 0 Eigenvalues'of' X 'deUine'probability'distribution 1 x ¤ X ¤ = x ¤ ( x ¤ ) T 0 TRIVIAL'DISTRIBUTION' 0

Regularized$Spectral$Op8miza8on$ 1 Regularizer' F L • X + ´ · F ( X ) d min Parameter' ´ I • X = 1 s.t. 11 T • X = 0 X º 0 The'regularizer' F ''forces'the'distribution'of'eigenvalues'of' X 'to'be'non? trivial' X ¤ = x ¤ ( x ¤ ) T 1 ¡ ² x ¤ ² 1 REGULARIZATION' X ¤ = P p i v i v T i ² 2

Regularizers$ Regularizers'are'SDP?versions'of'common'regularizers' ' ' • ''von'Neumann'Entropy' F H ( X ) = Tr( X log X ) = P p i log p i ' • 'p?Norm,' p > 1 P p p F p ( X ) = 1 p = 1 p Tr( X p ) = 1 p || X || p i p • 'And'more,'e.g.'log?determinant. ' '

Our$Main$Result$ Regularized''SDP ' 1 d min L • X + ´ · F ( X ) I • X = 1 s.t. J • X = 0 X º 0 RESULT:! '' Explicit'correspondence'between' regularizers!and!random!walks! REGULARIZER' OPTIMAL'SOLUTION'OF'REGULARIZED'PROGRAM' X ? / H t F = F H where't'depends'on' ´ Entropy' G X ? / ( qI + (1 ¡ q ) W ) 1 F = F p p ¡ 1 p ?Norm' where' q 'depends'on' ´

Our$Main$Result$ Regularized''SDP ' 1 d min L • X + ´ · F ( X ) I • X = 1 s.t. J • X = 0 X º 0 RESULT:! '' Explicit'correspondence'between' regularizers!and!random!walks! REGULARIZER' OPTIMAL'SOLUTION'OF'REGULARIZED'PROGRAM' X ? / H t F = F H where't'depends'on' ´ Entropy' G HEAT4KERNEL! X ? / ( qI + (1 ¡ q ) W ) 1 F = F p p ¡ 1 p ?Norm' LAZY!RANDOM!WALK! where' q 'depends'on' ´