Algorithms for Big Data (XIV) Chihao Zhang Shanghai Jiao Tong - PowerPoint PPT Presentation

Algorithms for Big Data (XIV) Chihao Zhang Shanghai Jiao Tong University Dec. 20, 2019 Algorithms for Big Data (XIV) 1/12

We defined the graph Laplacian Review Last week we studied electrical networks using matrices. : We also defined the notion of efgective resistance between two vertices in terms of : eff e e e e Algorithms for Big Data (XIV) 2/12

We defined the graph Laplacian Review Last week we studied electrical networks using matrices. : We also defined the notion of efgective resistance between two vertices in terms of : eff e e e e Algorithms for Big Data (XIV) 2/12

We also defined the notion of efgective resistance between two vertices in terms of Review Last week we studied electrical networks using matrices. : eff e e e e Algorithms for Big Data (XIV) 2/12 We defined the graph Laplacian L : L = U T WU.

Review Last week we studied electrical networks using matrices. Algorithms for Big Data (XIV) 2/12 We defined the graph Laplacian L : L = U T WU. We also defined the notion of efgective resistance between two vertices in terms of L : R eff ( u, v ) ≜ ( e u − e v ) T L + ( e u − e v ) .

Sparsification Given a graph , the goal of sparsification is to construct a sparse graph such that Similar Laplacian implies similar spectrum; similar efgective resistance between any two vertices; similar clustering; … Algorithms for Big Data (XIV) 3/12

Sparsification Similar Laplacian implies similar spectrum; similar efgective resistance between any two vertices; similar clustering; … Algorithms for Big Data (XIV) 3/12 Given a graph G , the goal of sparsification is to construct a sparse graph H such that ( 1 − ε ) L G ≼ L H ≼ ( 1 + ε ) L G .

Sparsification Similar Laplacian implies Algorithms for Big Data (XIV) 3/12 Given a graph G , the goal of sparsification is to construct a sparse graph H such that ( 1 − ε ) L G ≼ L H ≼ ( 1 + ε ) L G . ▶ similar spectrum; ▶ similar efgective resistance between any two vertices; ▶ similar clustering; ▶ …

The Construction We use to denote the Laplacian of the unweighted graph containing a single edge . For a graph , we have where is the weight on the edge . Let be a collection of probabilities on each pair of vertices. Algorithms for Big Data (XIV) 4/12

The Construction For a graph , we have where is the weight on the edge . Let be a collection of probabilities on each pair of vertices. Algorithms for Big Data (XIV) 4/12 We use L u,v to denote the Laplacian of the unweighted graph containing a single edge { u, v } .

The Construction Let be a collection of probabilities on each pair of vertices. Algorithms for Big Data (XIV) 4/12 We use L u,v to denote the Laplacian of the unweighted graph containing a single edge { u, v } . For a graph G = ( V, E ) , we have ∑ L G = w u,v · L u,v , { u,v } ∈ E where w u,v is the weight on the edge { u, v } ∈ E .

Algorithms for Big Data (XIV) The Construction 4/12 We use L u,v to denote the Laplacian of the unweighted graph containing a single edge { u, v } . For a graph G = ( V, E ) , we have ∑ L G = w u,v · L u,v , { u,v } ∈ E where w u,v is the weight on the edge { u, v } ∈ E . Let { p u,v } { u,v } ∈ E be a collection of probabilities on each pair of vertices.

contains the edge with probability for every pair independently. If an edge , we assign it with weight . It is easy to verify that E We will carefully choose to guarantee that is sparse with high probability; is well-concentrated to its expectation. Algorithms for Big Data (XIV) 5/12 Let H = ( V, E H ) be the sparse graph we are going to construct…

If an edge , we assign it with weight . It is easy to verify that E We will carefully choose to guarantee that is sparse with high probability; is well-concentrated to its expectation. Algorithms for Big Data (XIV) 5/12 Let H = ( V, E H ) be the sparse graph we are going to construct… H contains the edge { u, v } with probability p u,v for every pair { u, v } independently.

It is easy to verify that E We will carefully choose to guarantee that is sparse with high probability; is well-concentrated to its expectation. Algorithms for Big Data (XIV) 5/12 Let H = ( V, E H ) be the sparse graph we are going to construct… H contains the edge { u, v } with probability p u,v for every pair { u, v } independently. If an edge { u, v } ∈ E H , we assign it with weight w u,v /p u,v .

It is easy to verify that We will carefully choose to guarantee that is sparse with high probability; is well-concentrated to its expectation. Algorithms for Big Data (XIV) 5/12 Let H = ( V, E H ) be the sparse graph we are going to construct… H contains the edge { u, v } with probability p u,v for every pair { u, v } independently. If an edge { u, v } ∈ E H , we assign it with weight w u,v /p u,v . E [ L H ] = L G .

is sparse with high probability; It is easy to verify that is well-concentrated to its expectation. Algorithms for Big Data (XIV) 5/12 Let H = ( V, E H ) be the sparse graph we are going to construct… H contains the edge { u, v } with probability p u,v for every pair { u, v } independently. If an edge { u, v } ∈ E H , we assign it with weight w u,v /p u,v . E [ L H ] = L G . We will carefully choose { p u,v } to guarantee that

Algorithms for Big Data (XIV) It is easy to verify that 5/12 Let H = ( V, E H ) be the sparse graph we are going to construct… H contains the edge { u, v } with probability p u,v for every pair { u, v } independently. If an edge { u, v } ∈ E H , we assign it with weight w u,v /p u,v . E [ L H ] = L G . We will carefully choose { p u,v } to guarantee that ▶ H is sparse with high probability; ▶ L H is well-concentrated to its expectation.

is the projection onto the column space of A Transformation Sometimes it is more convenient to work with , the pseudo-inverse of . Note that The matrix . We will now study . Algorithms for Big Data (XIV) 6/12

is the projection onto the column space of A Transformation Note that The matrix . We will now study . Algorithms for Big Data (XIV) 6/12 Sometimes it is more convenient to work with L + G , the pseudo-inverse of L G .

is the projection onto the column space of A Transformation The matrix Algorithms for Big Data (XIV) . We will now study . 6/12 Note that Sometimes it is more convenient to work with L + G , the pseudo-inverse of L G . ⇒ L + /2 G L H L + /2 ≼ ( 1 + ε ) L + /2 G L G L + /2 L H ≼ ( 1 + ε ) L G ⇐ G . G

A Transformation We will now study Algorithms for Big Data (XIV) . 6/12 Note that Sometimes it is more convenient to work with L + G , the pseudo-inverse of L G . ⇒ L + /2 G L H L + /2 ≼ ( 1 + ε ) L + /2 G L G L + /2 L H ≼ ( 1 + ε ) L G ⇐ G . G The matrix L + /2 G L G L + /2 is the projection onto the column space of L G . G

A Transformation Algorithms for Big Data (XIV) 6/12 Note that Sometimes it is more convenient to work with L + G , the pseudo-inverse of L G . ⇒ L + /2 G L H L + /2 ≼ ( 1 + ε ) L + /2 G L G L + /2 L H ≼ ( 1 + ε ) L G ⇐ G . G The matrix L + /2 G L G L + /2 is the projection onto the column space of L G . G We will now study L + /2 G L H L + /2 G .

Chernoff Bound for Matrices and maximum eigenvalues of E Algorithms for Big Data (XIV) . , for Pr , and , for Pr respectively. Then be the minimum The main tool to establish concentration is the following analogue of Chernofg bound for and . Let almost surely. Let that be independent random positive semi-definite matrices such Let Theorem matrices. 7/12

Chernoff Bound for Matrices and maximum eigenvalues of E Algorithms for Big Data (XIV) . , for Pr , and , for Pr respectively. Then be the minimum The main tool to establish concentration is the following analogue of Chernofg bound for and . Let almost surely. Let that be independent random positive semi-definite matrices such Let Theorem matrices. 7/12

Chernoff Bound for Matrices The main tool to establish concentration is the following analogue of Chernofg bound for Algorithms for Big Data (XIV) 7/12 matrices. Theorem Let X 1 , . . . , X n ∈ R n × n be independent random positive semi-definite matrices such that λ max ( X i ) ≤ R almost surely. Let X = ∑ n i = 1 X i . Let µ min and µ max be the minimum and maximum eigenvalues of E [ X ] respectively. Then ) µ min /R ( ▶ Pr [ λ min ( X ) ≤ ( 1 − ε ) µ min ] ≤ n e − ε , for 0 < ε < 1 , and ( 1 − ε ) 1 − ε ) µ max /R ( e ε ▶ Pr [ λ max ( X ) ≥ ( 1 + ε ) µ max ] ≤ n , for ε > 0 . ( 1 + ε ) 1 + ε

For every pair of vertices and , we define Following our construction of , for every , define a random variable w.p. otherwise. Then and max Algorithms for Big Data (XIV) 8/12 Setting p u,v

Following our construction of , for every , define a random variable w.p. otherwise. Then and max Algorithms for Big Data (XIV) 8/12 Setting p u,v For every pair of vertices u and v , we define p u,v ≜ 1 Rw u,v ∥ L + /2 G L u,v L + /2 G ∥ .

8/12 otherwise. Algorithms for Big Data (XIV) max and Then Setting p u,v For every pair of vertices u and v , we define p u,v ≜ 1 Rw u,v ∥ L + /2 G L u,v L + /2 G ∥ . Following our construction of H , for every { u, v } , define a random variable { ( w u,v /p u,v ) L + /2 G L u,v L + /2 G , w.p. p u,v X u,v = 0,