On Sketching Quadratic Forms Bo Qin The Hong Kong University of - - PowerPoint PPT Presentation

On Sketching Quadratic Forms

Bo Qin

The Hong Kong University of Science and Technology

January 16, 2016 Joint with: Alexandr Andoni, Jiecao Chen, Robert Krauthgamer, David Woodruff and Qin Zhang

Bo Qin On Sketching Quadratic Forms

Outline

1 Sketching Quadratic Forms in Two Models 2 Sketches for General and PSD Matrices 3 Sketches for Laplacian Matrices 4 Cut and Spectral sketches for Laplacians

Bo Qin On Sketching Quadratic Forms

Sketching Quadratic Forms

Given a matrix A ∈ Rn×n, compute a sketch sk(A), which suffices to estimate the quadratic form xTAx for every query vector x ∈ Rn. (1 + ǫ)-approximation: Output ∈ (1 ± ǫ)xT Ax Goal: Sketch sk(A) of small size

Bo Qin On Sketching Quadratic Forms

Two Models

“For all” model: sk(A) succeeds on all queries x simul- taneously “For each” model: for every fixed query x, the sketch succeeds with constant (or high) probability

Bo Qin On Sketching Quadratic Forms

Outline

1 Sketching Quadratic Forms in Two Models 2 Sketches for General and PSD Matrices 3 Sketches for Laplacian Matrices 4 Cut and Spectral sketches for Laplacians

Bo Qin On Sketching Quadratic Forms

Lower Bounds for General and PSD Matrices

General and PSD Matrices: Any sketch sk(A) of A ∈ Rn×n satisfying the “for all” guaran- teemust use Ω(n2) bits of space

• For General Matrices, Ω(n2) bits is required even in the

“for each” model

Bo Qin On Sketching Quadratic Forms

PSD Matrices: “For Each” Model

• For a positive semidefinite (PSD) matrix A,

∀x ∈ Rn, xTAx = ||A1/2x||2 (If A is a PSD matrix, A has the unique square root)

Bo Qin On Sketching Quadratic Forms

PSD Matrices: “For Each” Model

• For a positive semidefinite (PSD) matrix A,

∀x ∈ Rn, xTAx = ||A1/2x||2 (If A is a PSD matrix, A has the unique square root)

• Johnson-Lindenstrauss lemma: There exists a random ε−2×

n matrix T of i.i.d. entries from {±ε} such that, for every fixed x ∈ Rn, (1 − ε)||A1/2x||2 ≤ ||TA1/2x||2 ≤ (1 + ε)||A1/2x||2, with probability at least 2/3.

Bo Qin On Sketching Quadratic Forms

PSD Matrices: “For Each” Model

• For a positive semidefinite (PSD) matrix A,

∀x ∈ Rn, xTAx = ||A1/2x||2 (If A is a PSD matrix, A has the unique square root)

• Johnson-Lindenstrauss lemma: There exists a random ε−2×

n matrix T of i.i.d. entries from {±ε} such that, for every fixed x ∈ Rn, (1 − ε)||A1/2x||2 ≤ ||TA1/2x||2 ≤ (1 + ε)||A1/2x||2, with probability at least 2/3.

• O(n/ε2)-size Sketch: sk(A) = TA1/2 (It is optimal!)

Bo Qin On Sketching Quadratic Forms

Main Results

“for all” model “for each” model Matrix family upper bound lower bound upper bound lower bound General ˜ O(n2) Ω(n2) ˜ O(n2) Ω(n2) PSD ˜ O(n2) Ω(n2) ˜ O(nǫ−2) Ω(nε−2)

Table: Here, ˜ O(f) denotes f · (log f)O(1). Our results are in bold.

Bo Qin On Sketching Quadratic Forms

Outline

1 Sketching Quadratic Forms in Two Models 2 Sketches for General and PSD Matrices 3 Sketches for Laplacian Matrices 4 Cut and Spectral sketches for Laplacians

Bo Qin On Sketching Quadratic Forms

Sketching Quadratic Forms of Laplacians

Laplacian: An important subclass of PSD matrices The Laplacian LG of a graph G = (V, E) is defined as LG = D − A, where D is the diagonal weighted degree matrix of G, and A is the weighted adjacency matrix of G.

Bo Qin On Sketching Quadratic Forms

Sketching Quadratic Forms of Laplacians

Laplacian: An important subclass of PSD matrices The Laplacian LG of a graph G = (V, E) is defined as LG = D − A, where D is the diagonal weighted degree matrix of G, and A is the weighted adjacency matrix of G.

• Spectral query: x ∈ Rn
• Cut query: x ∈ {0, 1}n, xTLGx = w(S, V \ S) where

S ⊂ V satisfying each vertex u ∈ S iff xu = 1

Bo Qin On Sketching Quadratic Forms

Laplacians: “For All” Model

Can one achieve smaller sketches for Laplacians?

• Graph Sparsificaiton: BK96, ST04, ST11, SS11, FHHP11,

KP12, BSS14, etc. Spectral Sparsifiers—Sketches for Laplacians in “For All” Model

Select a reweighted subgraph H of O(n/ε2) edges ∀x ∈ Rn, xT LHx ∈ (1 ± ε)xT LGx

[BSS14]: O(n/ε2) edges is optimal!

Bo Qin On Sketching Quadratic Forms

Laplacians: “For All” Model

Can one achieve smaller sketches for Laplacians?

• Graph Sparsificaiton: BK96, ST04, ST11, SS11, FHHP11,

KP12, BSS14, etc.

• Spectral Sparsifiers—Sketches for Laplacians in “For All”

Model

• Select a reweighted subgraph H of O(n/ε2) edges
• ∀x ∈ Rn, xT LHx ∈ (1 ± ε)xT LGx

[BSS14]: O(n/ε2) edges is optimal!

Bo Qin On Sketching Quadratic Forms

Laplacians: “For All” Model

Can one achieve smaller sketches for Laplacians?

• Graph Sparsificaiton: BK96, ST04, ST11, SS11, FHHP11,

KP12, BSS14, etc.

• Spectral Sparsifiers—Sketches for Laplacians in “For All”

Model

• Select a reweighted subgraph H of O(n/ε2) edges
• ∀x ∈ Rn, xT LHx ∈ (1 ± ε)xT LGx
• [BSS14]: O(n/ε2) edges is optimal!

Bo Qin On Sketching Quadratic Forms

Smaller Sketch?

Sketches for Laplacians:

• Arbitrary Data Structure: Beyond subgraphs?
• Cut Queries: Can cut-sparsifiers be smaller than spectral-

sparsifiers?

• “For Each” Model: Smaller sketches than those in “for all”

model?

Bo Qin On Sketching Quadratic Forms

Laplacians: Lower Bound in “For All” Model

Any Data Structure: Theorem (Informal) Any sketch sk(A) of A ∈ Rn×n satisfying the “for all” guarantee must use Ω(n/ε2) bits of space, even for cut queries.

• The lower bound holds for cut queries (for spectral queries

as well).

Bo Qin On Sketching Quadratic Forms

Laplacians: Lower Bound in “For All” Model

Previous bounds in the restricted versions:

• [Alon97] Ω(n/ε2)—The sparsifier H has regular degrees

and uniform edge weights.

• [BSS14] Ω(n/ε2)—H is a spectral sparsifier.

Bo Qin On Sketching Quadratic Forms

Laplacians: Lower Bound in “For All” Model

Previous bounds in the restricted versions:

• [Alon97] Ω(n/ε2)—The sparsifier H has regular degrees

and uniform edge weights.

• [BSS14] Ω(n/ε2)—H is a spectral sparsifier.

Our lower bound is the first lower bound without assump- tions!

Bo Qin On Sketching Quadratic Forms

Main Results

“for all” model “for each” model Matrix family upper bound lower bound upper bound lower bound General ˜ O(n2) Ω(n2) ˜ O(n2) Ω(n2) PSD ˜ O(n2) Ω(n2) ˜ O(nǫ−2) Ω(nε−2) Laplacian, SDD ˜ O(nε−2) [BSS14] Ω(nε−2) [BSS14] Laplacian+cut ˜ O(nε−2) [BSS14] Ω(nε−2)

Table: Here, ˜ O(f) denotes f · (log f)O(1). Our results are in bold.

Bo Qin On Sketching Quadratic Forms

Laplacians: “For Each” Model

We can do better in the “For Each” Model! Sketches for Laplacians: Cut Queries: Sketches of size ˜ O(nε−1) bits Spectral Queries: Sketches of size ˜ O(nε−1.6) bits

Bo Qin On Sketching Quadratic Forms

Main Results

“for all” model “for each” model Matrix family upper bound lower bound upper bound lower bound General ˜ O(n2) Ω(n2) ˜ O(n2) Ω(n2) PSD ˜ O(n2) Ω(n2) ˜ O(nǫ−2) Ω(nε−2) Laplacian, SDD ˜ O(nε−2) [BSS14] Ω(nε−2) [BSS14] ˜ O(nε−1.6) Ω(nε−1) Laplacian+cut ˜ O(nε−2) [BSS14] Ω(nε−2) ˜ O(nε−1) Ω(nε−1)

Table: Here, ˜ O(f) denotes f · (log f)O(1). Our results are in bold.

Separates the “for each” and “for all” models for Lapla- cians!

Bo Qin On Sketching Quadratic Forms

Main Results

“for all” model “for each” model Matrix family upper bound lower bound upper bound lower bound General ˜ O(n2) Ω(n2) ˜ O(n2) Ω(n2) PSD ˜ O(n2) Ω(n2) ˜ O(nǫ−2) Ω(nε−2) Laplacian, SDD ˜ O(nε−2) [BSS14] Ω(nε−2) [BSS14] ˜ O(nε−1.6) Ω(nε−1) Laplacian+cut ˜ O(nε−2) [BSS14] Ω(nε−2) ˜ O(nε−1) Ω(nε−1)

Table: Here, ˜ O(f) denotes f · (log f)O(1). Our results are in bold.

Separates the “for each” and “for all” models for Lapla- cians!

Bo Qin On Sketching Quadratic Forms

Outline

1 Sketching Quadratic Forms in Two Models 2 Sketches for General and PSD Matrices 3 Sketches for Laplacian Matrices 4 Cut and Spectral sketches for Laplacians

Bo Qin On Sketching Quadratic Forms

Cut Sketches “For Each”—First Attempt

Consider the complete graph

• Standard sampling scheme: Sample edges with probability

pe = 1/ε2

n

• Smaller probability fails: Even for “singleton cuts”,

w({u}, V \ {u}) ≈ 1 ε2 ± 1 ε

• Singleton cuts are the “most difficult” for concentration

Storing all vertex degrees—Only O(n) bits of space!

Bo Qin On Sketching Quadratic Forms

Constructing Cut Sketches “For Each”

Simple Graphs Assume an unweighted graph G = (V, E) satisfies ∀S ⊂ V, w(S, V \ S) min{|S|, |V \ S|} ≥ 1 ε Sketch for Answering Cut Queries s.t. w(S, V \ S) ≤ 1 ε2

Bo Qin On Sketching Quadratic Forms

Constructing Cut Sketches “For Each”

Note that w(S, V \ S) =

• u∈S

 du −

• v∈S,(u,v)∈E

w(u,v)  

• Store the degree du of every vertex u ∈ V
• For each u ∈ V , uniformly sample (with replacement) 1

ε

edges from those edges adjacent to u — ˜ O(n/ε) bits of space!

Bo Qin On Sketching Quadratic Forms

Constructing Cut Sketches “For Each”

Estimator: I =

u∈S

• du − duε

v∈S,(u,v)∈E 1(u,v) is sampled

• Analysis:
• I is an unbiased estimator, i.e., E[I] = w(S, V \ S)
• The standard deviation of I is small, i.e.,

σ[I] ≤ O(ε)w(S, V \ S)

• Chebyshev’s inequality =

⇒ (1+ε)-approximation of w(S, V \ S) with constant probability

Bo Qin On Sketching Quadratic Forms

Constructing Cut Sketches “For Each”

˜ O(n/ε)-bit Cut Sketch for General Graph! Key Idea: Process a graph into O(poly(log n

ε )) simple graphs

Bo Qin On Sketching Quadratic Forms

Constructing Cut Sketches “For Each”

˜ O(n/ε)-bit Cut Sketch for General Graph! Key Idea: Process a graph into O(poly(log n

ε )) simple graphs

One Important Step—Find the Sparsest Cut

• Find the sparsest cut (S, V \ S) in any connected compo-

nent s.t.

w(S,V \S) min{|S|,|V \S|} < 1 ε

• Store and remove all cut edges
• Repeat until all connected components have

∀S ⊂ V, w(S, V \ S) min{|S|, |V \ S|} ≥ 1 ε

Bo Qin On Sketching Quadratic Forms

Constructing Cut Sketches “For Each”

˜ O(n/ε)-bit Cut Sketch for General Graph! Key Idea: Process a graph into O(poly(log n

ε )) simple graphs

One Important Step—Find the Sparsest Cut

• Find the sparsest cut (S, V \ S) in any connected compo-

nent s.t.

w(S,V \S) min{|S|,|V \S|} < 1 ε

• Store and remove all cut edges
• Repeat until all connected components have

∀S ⊂ V, w(S, V \ S) min{|S|, |V \ S|} ≥ 1 ε —Only ˜ O(n/ε) edges needed to be stored!

Bo Qin On Sketching Quadratic Forms

Spectral Sketches and Extensions

• Spectral Sketches
• Similar idea but more complicated analysis
• ˜

O(n/ε1.6) bits of space

• Sketches for Symmetric Diagonally Dominant (SDD) Ma-

trices

Bo Qin On Sketching Quadratic Forms

Open Problems

Graphical sketch? Sketching of other combinatorial features (graphs)?

Bo Qin On Sketching Quadratic Forms

THANK YOU

Bo Qin On Sketching Quadratic Forms