Planted Cliques, Iterative Thresholding and Message Passing - - PowerPoint PPT Presentation

Planted Cliques, Iterative Thresholding and Message Passing Algorithms

Yash Deshpande and Andrea Montanari

Stanford University

November 5, 2013

Deshpande, Montanari Planted Cliques November 5, 2013 1 / 49

Problem Definition

Given distributions Q0❀ Q1, A Set ✦ S ✚ [n] Data ✦ Aij ✘

✭

Q1 if i❀ j ✷ S Q0

• therwise.

Aij = Aji Problem: Given A, identify S

Deshpande, Montanari Planted Cliques November 5, 2013 2 / 49

Problem Definition

Given distributions Q0❀ Q1, A Set ✦ S ✚ [n] Data ✦ Aij ✘

✭

Q1 if i❀ j ✷ S Q0

• therwise.

Aij = Aji

S S A = Q1 Q0

Problem: Given A, identify S

Deshpande, Montanari Planted Cliques November 5, 2013 2 / 49

Problem Definition

Given distributions Q0❀ Q1, A Set ✦ S ✚ [n] Data ✦ Aij ✘

✭

Q1 if i❀ j ✷ S Q0

• therwise.

Aij = Aji

S S A = Q1 Q0

Problem: Given A, identify S

Deshpande, Montanari Planted Cliques November 5, 2013 2 / 49

An Example

Q1 = N(✕❀ 1) Q0 = N(0❀ 1)✿

S S A = Q1 Q0

Deshpande, Montanari Planted Cliques November 5, 2013 3 / 49

An Example

A = ✕uSuT

S

+ Z

λ λ λ λ ∼ N(0, 1)

Data = Sparse, Low Rank + noise

Deshpande, Montanari Planted Cliques November 5, 2013 4 / 49

An Example

A = ✕uSuT

S

+ Z

λ λ λ λ ∼ N(0, 1)

Data = Sparse, Low Rank + noise

Deshpande, Montanari Planted Cliques November 5, 2013 4 / 49

Much work in statistics

Denoising: y = x + noise [Donoho, Jin 2004], [Arias-Castro, Candes, Durand 2011] [Arias-Castro, Bubeck, Lugosi 2012] Sparse signal recovery: y = Ax + noise [Candes, Romberg, Tao 2004], [Chen, Donoho 1998] The simple example combines different structures

Deshpande, Montanari Planted Cliques November 5, 2013 5 / 49

Much work in statistics

Denoising: y = x + noise [Donoho, Jin 2004], [Arias-Castro, Candes, Durand 2011] [Arias-Castro, Bubeck, Lugosi 2012] Sparse signal recovery: y = Ax + noise [Candes, Romberg, Tao 2004], [Chen, Donoho 1998] The simple example combines different structures

Deshpande, Montanari Planted Cliques November 5, 2013 5 / 49

Much work in statistics

Denoising: y = x + noise [Donoho, Jin 2004], [Arias-Castro, Candes, Durand 2011] [Arias-Castro, Bubeck, Lugosi 2012] Sparse signal recovery: y = Ax + noise [Candes, Romberg, Tao 2004], [Chen, Donoho 1998] The simple example combines different structures

Deshpande, Montanari Planted Cliques November 5, 2013 5 / 49

Our Running Example: Planted Cliques

A is “adjacency” matrix Q1 = ✍+1 Q0 = 1 2✍+1 + 1 2✍1✿ S forms a clique in the graph [Jerrum, 1992]

Deshpande, Montanari Planted Cliques November 5, 2013 6 / 49

Our Running Example: Planted Cliques

A is “adjacency” matrix Q1 = ✍+1 Q0 = 1 2✍+1 + 1 2✍1✿ S forms a clique in the graph [Jerrum, 1992]

Deshpande, Montanari Planted Cliques November 5, 2013 6 / 49

Our Running Example: Planted Cliques

◮ Average case version of

MAX-CLIQUE

◮ Communities in networks

Deshpande, Montanari Planted Cliques November 5, 2013 7 / 49

Our Running Example: Planted Cliques

◮ Average case version of

MAX-CLIQUE

◮ Communities in networks

Deshpande, Montanari Planted Cliques November 5, 2013 7 / 49

How large should S be?

Let ❥S❥ = k Size of largest clique in G

✏

n❀ 1

2

✑

Second moment calculation ✮ k ❃ 2 log2 n.

Deshpande, Montanari Planted Cliques November 5, 2013 8 / 49

How large should S be?

Let ❥S❥ = k Size of largest clique in G

✏

n❀ 1

2

✑

Second moment calculation ✮ k ❃ 2 log2 n.

Deshpande, Montanari Planted Cliques November 5, 2013 8 / 49

Progress(0)

Complexity k n2 log n 2 log2 n

Exhaustive search

Deshpande, Montanari Planted Cliques November 5, 2013 9 / 49

A Naive Algorithm

Pick k largest degree vertices of G as ❜ S.

Deshpande, Montanari Planted Cliques November 5, 2013 10 / 49

A Naive Algorithm

Pick k largest degree vertices of G as ❜ S.

Deshpande, Montanari Planted Cliques November 5, 2013 10 / 49

Analysis of NAIVE

If i ❂ ✷ S: deg(i) = Binomial

✒

n 1❀ 1 2

✓

✮ max

i ❂ ✷S deg(i) ✔ n

2 + O(

q

n log n) If i ✷ S: deg(i) = k 1 + Binomial

✒

n k + 1❀ 1 2

✓

✮ min

i✷S deg(i) ✕ k 1

2 + n 2 O(

q

n log n) NAIVE works if: k ✕ O(

♣

n log n)

Deshpande, Montanari Planted Cliques November 5, 2013 11 / 49

Analysis of NAIVE

If i ❂ ✷ S: deg(i) = Binomial

✒

n 1❀ 1 2

✓

✮ max

i ❂ ✷S deg(i) ✔ n

2 + O(

q

n log n) If i ✷ S: deg(i) = k 1 + Binomial

✒

n k + 1❀ 1 2

✓

✮ min

i✷S deg(i) ✕ k 1

2 + n 2 O(

q

n log n) NAIVE works if: k ✕ O(

♣

n log n)

Deshpande, Montanari Planted Cliques November 5, 2013 11 / 49

Analysis of NAIVE

If i ❂ ✷ S: deg(i) = Binomial

✒

n 1❀ 1 2

✓

✮ max

i ❂ ✷S deg(i) ✔ n

2 + O(

q

n log n) If i ✷ S: deg(i) = k 1 + Binomial

✒

n k + 1❀ 1 2

✓

✮ min

i✷S deg(i) ✕ k 1

2 + n 2 O(

q

n log n) NAIVE works if: k ✕ O(

♣

n log n)

Deshpande, Montanari Planted Cliques November 5, 2013 11 / 49

Analysis of NAIVE

If i ❂ ✷ S: deg(i) = Binomial

✒

n 1❀ 1 2

✓

✮ max

i ❂ ✷S deg(i) ✔ n

2 + O(

q

n log n) If i ✷ S: deg(i) = k 1 + Binomial

✒

n k + 1❀ 1 2

✓

✮ min

i✷S deg(i) ✕ k 1

2 + n 2 O(

q

n log n) NAIVE works if: k ✕ O(

♣

n log n)

Deshpande, Montanari Planted Cliques November 5, 2013 11 / 49

Progress(1)

Complexity k n2 log n 2 log2 n n2 √n log n

[Kučera, 1995]

Deshpande, Montanari Planted Cliques November 5, 2013 12 / 49

Spectral Method

A = uSuT

S

+ Z

1 1 1 1 ±1 ±1 ±1

Hopefully v1(A) ✙ uS

Deshpande, Montanari Planted Cliques November 5, 2013 13 / 49

Spectral Method

A = uSuT

S

+ Z

1 1 1 1 ±1 ±1 ±1

Hopefully v1(A) ✙ uS

Deshpande, Montanari Planted Cliques November 5, 2013 13 / 49

Analysis of SPECTRAL

1 ♣nA =

✒ k

♣n

✓ ⑤ ④③ ⑥

✔

eSeT

S + Z

♣n✿ By standard linear algebra: ✔ ✢

✌ ✌ ✌ ✌

Z ♣n

✌ ✌ ✌ ✌

2

✙ 2 = ✮ ❤v1(A)❀ eS✐ ✕ 1 ✍(✔) SPECTRAL works if k ✕ C♣n.

Deshpande, Montanari Planted Cliques November 5, 2013 14 / 49

Progress(2)

Complexity k n2 log n 2 log2 n n2 √n log n n2 log n C√n

[Alon, Krivelevich and Sudakov, 1998]

Deshpande, Montanari Planted Cliques November 5, 2013 15 / 49

Progress(2)

Complexity k n2 log n 2 log2 n n2 √n log n n2 log n C√n

[Ames, Vavasis 2009], [Feige, Krauthgamer 2000]

Deshpande, Montanari Planted Cliques November 5, 2013 16 / 49

Progress(3)

Complexity k n2 log n 2 log2 n n2 √n log n n2 log n C√n 1.65√n

[Dekel, Gurel-Gurevich and Peres, 2010]

Deshpande, Montanari Planted Cliques November 5, 2013 17 / 49

Progress(3)

Complexity k n2 log n 2 log2 n n2 √n log n n2 log n C√n 1.261√n

[Dekel, Gurel-Gurevich and Peres, 2010]

Deshpande, Montanari Planted Cliques November 5, 2013 18 / 49

Progress(3)

Complexity k n2 log n 2 log2 n n2 √n log n n2 log n C√n 1.261√n nr+2 C n

2r

[Alon, Krivelevich and Sudakov, 1998]

Deshpande, Montanari Planted Cliques November 5, 2013 19 / 49

Phase transition in spectrum of A

Deshpande, Montanari Planted Cliques November 5, 2013 20 / 49

Phase transition in spectrum of A

If k ✕ (1 + ✧)♣n

2 −2 Limiting Spectral Density

❤v1(A)❀ eS✐ ✕ ✍(✧)

Deshpande, Montanari Planted Cliques November 5, 2013 20 / 49

Phase transition in spectrum of A

If k ✕ (1 + ✧)♣n

2 −2 Limiting Spectral Density

❤v1(A)❀ eS✐ ✕ ✍(✧) If k ✔ (1 ✧)♣n

2 −2 Limiting Spectral Density

❤v1(A)❀ eS✐ ✔ n1❂2+✍✵(✧)

Deshpande, Montanari Planted Cliques November 5, 2013 20 / 49

Phase transition in spectrum of A

If k ✕ (1 + ✧)♣n

2 −2 Limiting Spectral Density

❤v1(A)❀ eS✐ ✕ ✍(✧) If k ✔ (1 ✧)♣n

2 −2 Limiting Spectral Density

❤v1(A)❀ eS✐ ✔ n1❂2+✍✵(✧) [Knowles, Yin, 2011]

Deshpande, Montanari Planted Cliques November 5, 2013 20 / 49

Seems much harder than it looks!

◮ “Statistical algorithms” fail if k = n1❂2✍: [Feldman et al., 2012] ◮ r-Lovász-Schrijver fails for k ✔

♣

n❂2r: [Feige, Krauthgamer, 2002]

◮ r-Lasserre fails for k ✔ ♣n (log n)r2 : [Widgerson and Meka, 2013]

Deshpande, Montanari Planted Cliques November 5, 2013 21 / 49

Seems much harder than it looks!

◮ “Statistical algorithms” fail if k = n1❂2✍: [Feldman et al., 2012] ◮ r-Lovász-Schrijver fails for k ✔

♣

n❂2r: [Feige, Krauthgamer, 2002]

◮ r-Lasserre fails for k ✔ ♣n (log n)r2 : [Widgerson and Meka, 2013]

Deshpande, Montanari Planted Cliques November 5, 2013 21 / 49

Seems much harder than it looks!

◮ “Statistical algorithms” fail if k = n1❂2✍: [Feldman et al., 2012] ◮ r-Lovász-Schrijver fails for k ✔

♣

n❂2r: [Feige, Krauthgamer, 2002]

◮ r-Lasserre fails for k ✔ ♣n (log n)r2 : [Widgerson and Meka, 2013]

Deshpande, Montanari Planted Cliques November 5, 2013 21 / 49

Our result

Theorem (Deshpande, Montanari, 2013)

If ❥S❥ = k ✕ (1 + ✧)

♣

n❂e, there exists an O(n2 log n) time algorithm that identifies S with high probability. I will present:

1 A (wrong) heuristic analysis 2 How to fix the heuristic 3 Lower bounds

Deshpande, Montanari Planted Cliques November 5, 2013 22 / 49

Our result

Theorem (Deshpande, Montanari, 2013)

If ❥S❥ = k ✕ (1 + ✧)

♣

n❂e, there exists an O(n2 log n) time algorithm that identifies S with high probability. I will present:

1 A (wrong) heuristic analysis 2 How to fix the heuristic 3 Lower bounds

Deshpande, Montanari Planted Cliques November 5, 2013 22 / 49

Our result

Theorem (Deshpande, Montanari, 2013)

If ❥S❥ = k ✕ (1 + ✧)

♣

n❂e, there exists an O(n2 log n) time algorithm that identifies S with high probability. I will present:

1 A (wrong) heuristic analysis 2 How to fix the heuristic 3 Lower bounds

Deshpande, Montanari Planted Cliques November 5, 2013 22 / 49

Iterative Thresholding

The power iteration: vt+1 = A vt✿ Improvement: vt+1 = AFt(vt)✿ where Ft(v) = (ft(v1)❀ ft(v2)❀ ✁ ✁ ✁ ❀ ft(vn))T Choose ft(✁) to exploit sparsity of eS

Deshpande, Montanari Planted Cliques November 5, 2013 23 / 49

Iterative Thresholding

The power iteration: vt+1 = A vt✿ Improvement: vt+1 = AFt(vt)✿ where Ft(v) = (ft(v1)❀ ft(v2)❀ ✁ ✁ ✁ ❀ ft(vn))T Choose ft(✁) to exploit sparsity of eS

Deshpande, Montanari Planted Cliques November 5, 2013 23 / 49

Iterative Thresholding

The power iteration: vt+1 = A vt✿ Improvement: vt+1 = AFt(vt)✿ where Ft(v) = (ft(v1)❀ ft(v2)❀ ✁ ✁ ✁ ❀ ft(vn))T Choose ft(✁) to exploit sparsity of eS

Deshpande, Montanari Planted Cliques November 5, 2013 23 / 49

Iterative Thresholding

The power iteration: vt+1 = A vt✿ Improvement: vt+1 = AFt(vt)✿ where Ft(v) = (ft(v1)❀ ft(v2)❀ ✁ ✁ ✁ ❀ ft(vn))T Choose ft(✁) to exploit sparsity of eS

Deshpande, Montanari Planted Cliques November 5, 2013 23 / 49

Iterative Thresholding

The power iteration: vt+1 = A vt✿ Improvement: vt+1 = AFt(vt)✿ where Ft(v) = (ft(v1)❀ ft(v2)❀ ✁ ✁ ✁ ❀ ft(vn))T Choose ft(✁) to exploit sparsity of eS

Deshpande, Montanari Planted Cliques November 5, 2013 23 / 49

(Wrong) Analysis

vt+1

i

= 1 ♣n

❳

j

Aijft(vt

j )✿

Aij are random ✝1 r.v. Use Central Limit Theorem for vt+1

i

Deshpande, Montanari Planted Cliques November 5, 2013 24 / 49

(Wrong) Analysis

vt+1

i

= 1 ♣n

❳

j

Aijft(vt

j )✿

Aij are random ✝1 r.v. Use Central Limit Theorem for vt+1

i

Deshpande, Montanari Planted Cliques November 5, 2013 24 / 49

(Wrong) Analysis

If i ❂ ✷ S: vt+1

i

= 1 ♣n

❳

j

Aijft(vt

j )

✙ N

✒

0❀ 1 n

❳

j

ft(vt

j )2

✓

Letting vt

i ✙ N(0❀ ✛2 t ). . .

✛2

t+1 = 1

n

❳

j

ft(vt

j )2

= E❢ft(✛t✘)2❣❀ ✘ ✘ N(0❀ 1)

Deshpande, Montanari Planted Cliques November 5, 2013 25 / 49

(Wrong) Analysis

If i ❂ ✷ S: vt+1

i

= 1 ♣n

❳

j

Aijft(vt

j )

✙ N

✒

0❀ 1 n

❳

j

ft(vt

j )2

✓

Letting vt

i ✙ N(0❀ ✛2 t ). . .

✛2

t+1 = 1

n

❳

j

ft(vt

j )2

= E❢ft(✛t✘)2❣❀ ✘ ✘ N(0❀ 1)

Deshpande, Montanari Planted Cliques November 5, 2013 25 / 49

(Wrong) Analysis

If i ❂ ✷ S: vt+1

i

= 1 ♣n

❳

j

Aijft(vt

j )

✙ N

✒

0❀ 1 n

❳

j

ft(vt

j )2

✓

Letting vt

i ✙ N(0❀ ✛2 t ). . .

✛2

t+1 = 1

n

❳

j

ft(vt

j )2

= E❢ft(✛t✘)2❣❀ ✘ ✘ N(0❀ 1)

Deshpande, Montanari Planted Cliques November 5, 2013 25 / 49

(Wrong) Analysis

If i ❂ ✷ S: vt+1

i

= 1 ♣n

❳

j

Aijft(vt

j )

✙ N

✒

0❀ 1 n

❳

j

ft(vt

j )2

✓

Letting vt

i ✙ N(0❀ ✛2 t ). . .

✛2

t+1 = 1

n

❳

j

ft(vt

j )2

= E❢ft(✛t✘)2❣❀ ✘ ✘ N(0❀ 1)

Deshpande, Montanari Planted Cliques November 5, 2013 25 / 49

(Wrong) Analysis

If i ✷ S: vt+1

i

= 1 ♣n

❳

j✷S

ft(vt

j )

⑤ ④③ ⑥

✖t+1

+ 1 ♣n

❳

j ❂ ✷S

Aijft(vt

j )

⑤ ④③ ⑥

✙N(0❀✛2

t+1)

where ✖t+1 = 1 ♣n

❳

j✷S

ft(vt

j )

=

✒ k

♣n

✓ ✒1

k

❳

j✷S

ft(vt

j )

✓

= ✔ E❢ft(✖t + ✛t✘)❣❀ ✘ ✘ N(0❀ 1)

Deshpande, Montanari Planted Cliques November 5, 2013 26 / 49

(Wrong) Analysis

If i ✷ S: vt+1

i

= 1 ♣n

❳

j✷S

ft(vt

j )

⑤ ④③ ⑥

✖t+1

+ 1 ♣n

❳

j ❂ ✷S

Aijft(vt

j )

⑤ ④③ ⑥

✙N(0❀✛2

t+1)

where ✖t+1 = 1 ♣n

❳

j✷S

ft(vt

j )

=

✒ k

♣n

✓ ✒1

k

❳

j✷S

ft(vt

j )

✓

= ✔ E❢ft(✖t + ✛t✘)❣❀ ✘ ✘ N(0❀ 1)

Deshpande, Montanari Planted Cliques November 5, 2013 26 / 49

(Wrong) Analysis

If i ✷ S: vt+1

i

= 1 ♣n

❳

j✷S

ft(vt

j )

⑤ ④③ ⑥

✖t+1

+ 1 ♣n

❳

j ❂ ✷S

Aijft(vt

j )

⑤ ④③ ⑥

✙N(0❀✛2

t+1)

where ✖t+1 = 1 ♣n

❳

j✷S

ft(vt

j )

=

✒ k

♣n

✓ ✒1

k

❳

j✷S

ft(vt

j )

✓

= ✔ E❢ft(✖t + ✛t✘)❣❀ ✘ ✘ N(0❀ 1)

Deshpande, Montanari Planted Cliques November 5, 2013 26 / 49

(Wrong) Analysis

If i ✷ S: vt+1

i

= 1 ♣n

❳

j✷S

ft(vt

j )

⑤ ④③ ⑥

✖t+1

+ 1 ♣n

❳

j ❂ ✷S

Aijft(vt

j )

⑤ ④③ ⑥

✙N(0❀✛2

t+1)

where ✖t+1 = 1 ♣n

❳

j✷S

ft(vt

j )

=

✒ k

♣n

✓ ✒1

k

❳

j✷S

ft(vt

j )

✓

= ✔ E❢ft(✖t + ✛t✘)❣❀ ✘ ✘ N(0❀ 1)

Deshpande, Montanari Planted Cliques November 5, 2013 26 / 49

Summarizing . . .

−4 −2 2 4 0.1 0.2 0.3 vt

i, i ∈ S

vt

i, i /

∈ S

Deshpande, Montanari Planted Cliques November 5, 2013 27 / 49

State Evolution

✖t+1 = ✔ E ❢ft(✖t + ✛t✘)❣ ✛2

t+1 = E❢ft(✛t✘)2❣✿

Using the optimal function ft(x) = e✖tx✖2

t

✖t+1 = ✔e✖2

t ❂2

✛2

t+1 = 1

Deshpande, Montanari Planted Cliques November 5, 2013 28 / 49

State Evolution

✖t+1 = ✔ E ❢ft(✖t + ✛t✘)❣ ✛2

t+1 = E❢ft(✛t✘)2❣✿

Using the optimal function ft(x) = e✖tx✖2

t

✖t+1 = ✔e✖2

t ❂2

✛2

t+1 = 1

Deshpande, Montanari Planted Cliques November 5, 2013 28 / 49

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

If ✔ ❁

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

If ✔ ❁

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

If ✔ ❁

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

If ✔ ❁

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

If ✔ ❁

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

Analysis is wrong but. . .

Theorem (Deshpande, Montanari, 2013)

If ❥S❥ = k ✕ (1 + ✧)

♣

n❂e, there exists an O(n2 log n) time algorithm that identifies S with high probability. . . . so we modify the algorithm.

Deshpande, Montanari Planted Cliques November 5, 2013 30 / 49

What algorithm?

Slight modification to iterative scheme: (vt

i )i✷[n] ✦ (vt i✦j)i❀j✷[n]

vt+1

i✦j =

1 ♣n

❳

❵✻=i❀j

Ai❵ ft(vt

❵✦i)✿

Analysis is exact as n ✦ ✶.

Deshpande, Montanari Planted Cliques November 5, 2013 31 / 49

What algorithm?

Slight modification to iterative scheme: (vt

i )i✷[n] ✦ (vt i✦j)i❀j✷[n]

vt+1

i✦j =

1 ♣n

❳

❵✻=i❀j

Ai❵ ft(vt

❵✦i)✿

Analysis is exact as n ✦ ✶.

Deshpande, Montanari Planted Cliques November 5, 2013 31 / 49

What algorithm?

Slight modification to iterative scheme: (vt

i )i✷[n] ✦ (vt i✦j)i❀j✷[n]

vt+1

i✦j =

1 ♣n

❳

❵✻=i❀j

Ai❵ ft(vt

❵✦i)✿

Analysis is exact as n ✦ ✶.

Deshpande, Montanari Planted Cliques November 5, 2013 31 / 49

What algorithm?

Slight modification to iterative scheme: (vt

i )i✷[n] ✦ (vt i✦j)i❀j✷[n]

vt+1

i✦j =

1 ♣n

❳

❵✻=i❀j

Ai❵ ft(vt

❵✦i)✿

Analysis is exact as n ✦ ✶.

Deshpande, Montanari Planted Cliques November 5, 2013 31 / 49

Fixing the heuristic

Lemma

Let (ft(z))t✕0 be a sequence of polynomials. Then, for every fixed t, and bounded, continuous function ✥ : R ✦ R the following limit holds in probability: lim

n✦✶

1 ♣n

❳

i✷S

✥(vt

i✦j) = ✔ E❢✥(✖t + ✛t✘)❣❀

lim

n✦✶

1 n

❳

i✷[n]♥S

✥(vt

i✦j) = E❢✥(✛t✘)❣❀

where ✘ ✘ N(0❀ 1).

Deshpande, Montanari Planted Cliques November 5, 2013 32 / 49

Proof Technique

Key ideas: Expand vt

i✦j for polynomial ft(✁)

Wrong analysis works if A ✦ At

Deshpande, Montanari Planted Cliques November 5, 2013 33 / 49

Proof Technique

Key ideas: Expand vt

i✦j for polynomial ft(✁)

Wrong analysis works if A ✦ At

Deshpande, Montanari Planted Cliques November 5, 2013 33 / 49

Proof Technique

Key ideas: Expand vt

i✦j for polynomial ft(✁)

Wrong analysis works if A ✦ At

Deshpande, Montanari Planted Cliques November 5, 2013 33 / 49

Proof Technique - Expanding v t

Let ft(x) = x2❀ v0

i✦j = 1

v1

i✦j =

❳

k✻=j

Aik✿

i k j

Deshpande, Montanari Planted Cliques November 5, 2013 34 / 49

Proof Technique - Expanding v t

v2

i✦j =

❳

k✻=j

Aik(v1

k✦i)2

=

❳

k✻=j

Aik

✒ ❳

❵✻=i

Ak❵

✓✒ ❳

m✻=i

Akm

✓

=

❳

k✻=j

❳

❵✻=i

❳

m✻=i

AikAk❵Akm✿

i j k ℓ m

Deshpande, Montanari Planted Cliques November 5, 2013 35 / 49

Proof Technique - Expanding v t

v2

i✦j =

❳

k✻=j

Aik(v1

k✦i)2

=

❳

k✻=j

Aik

✒ ❳

❵✻=i

Ak❵

✓✒ ❳

m✻=i

Akm

✓

=

❳

k✻=j

❳

❵✻=i

❳

m✻=i

AikAk❵Akm✿

i j k ℓ m

Deshpande, Montanari Planted Cliques November 5, 2013 35 / 49

Proof Technique - Expanding v t

v2

i✦j =

❳

k✻=j

Aik(v1

k✦i)2

=

❳

k✻=j

Aik

✒ ❳

❵✻=i

Ak❵

✓✒ ❳

m✻=i

Akm

✓

=

❳

k✻=j

❳

❵✻=i

❳

m✻=i

AikAk❵Akm✿

i j k ℓ m

Deshpande, Montanari Planted Cliques November 5, 2013 35 / 49

Proof Technique

vt+1

i✦j =

❳

k✻=i

Aikft(vt

k✦i)✿

i k ℓ m

• p

q r s w

✘t+1

i✦j =

❳

k✻=i

At

ikft(✘t k✦i)✿

i k ℓ m

• p

q r s w

Deshpande, Montanari Planted Cliques November 5, 2013 36 / 49

Proof Technique - a Combinatorial Lemma

Lemma

vt

i✦j =

❳

T✷❚ t

i✦j

A(T)Γ(T)v0(T) where ❚ t

i✦j consists rooted, labeled trees that:

1 have maximum depth t. 2 do not backtrack.

(Similarly for the ✘t

i✦j)

Deshpande, Montanari Planted Cliques November 5, 2013 37 / 49

Proof Technique - Moment Method

vt+1

i✦j =

❳

k✻=i

Aikft(vt

k✦i)✿

i k ℓ m

• p

q r s w

✘t+1

i✦j =

❳

k✻=i

At

ikft(✘t k✦i)✿

i k ℓ m

• p

q r s w

limn✦✶ Moments of vt+1 = Moments of ✘t+1.

Deshpande, Montanari Planted Cliques November 5, 2013 38 / 49

Proof Technique - Moment Method

vt+1

i✦j =

❳

k✻=i

Aikft(vt

k✦i)✿

i k ℓ m

• p

q r s w

✘t+1

i✦j =

❳

k✻=i

At

ikft(✘t k✦i)✿

i k ℓ m

• p

q r s w

limn✦✶ Moments of vt+1 = Moments of ✘t+1.

Deshpande, Montanari Planted Cliques November 5, 2013 38 / 49

Progress(4)

Complexity k n2 log n 2 log2 n n2 2√n log n n2 log n C√n 1.261√n Spectral threshold = √n n

e

Deshpande, Montanari Planted Cliques November 5, 2013 39 / 49

Is this threshold fundamental?

Rest of the talk: perhaps

Deshpande, Montanari Planted Cliques November 5, 2013 40 / 49

The “Hidden Set” Problem

Given Gn = ([n]❀ En) A Set ✦ S ✚ [n] Data ✦ Aij ✘

✭

Q1 if i❀ j ✷ S❀ Q0

• therwise.

Aij ∼ Q1 Aij ∼ Q0

Problem: Given edge labels (Aij)(i❀j)✷En, identify S

Deshpande, Montanari Planted Cliques November 5, 2013 41 / 49

The “Hidden Set” Problem

Given Gn = ([n]❀ En) A Set ✦ S ✚ [n] Data ✦ Aij ✘

✭

Q1 if i❀ j ✷ S❀ Q0

• therwise.

Aij ∼ Q1 Aij ∼ Q0

Problem: Given edge labels (Aij)(i❀j)✷En, identify S

Deshpande, Montanari Planted Cliques November 5, 2013 41 / 49

The “Hidden Set” Problem

Given Gn = ([n]❀ En) A Set ✦ S ✚ [n] Data ✦ Aij ✘

✭

Q1 if i❀ j ✷ S❀ Q0

• therwise.

S S A =

Problem: Given edge labels (Aij)(i❀j)✷En, identify S

Deshpande, Montanari Planted Cliques November 5, 2013 42 / 49

“Local” Algorithms

A t-local algorithm computes: Estimate at i:

❜

u(i) = F(ABall(i❀t))

Deshpande, Montanari Planted Cliques November 5, 2013 43 / 49

The Sparse Graph Analogue

Gn = ([n]❀ En)❀ n ✕ 1 satisfies:

◮ locally tree-like ◮ regular degree ∆

Further

◮ Q1 = ✍+1, Q0 = 1 2✍+1 + 1 2✍1 Aij ∼ Q1 Aij ∼ Q0

What can local algorithms achieve?

Deshpande, Montanari Planted Cliques November 5, 2013 44 / 49

The Sparse Graph Analogue

Gn = ([n]❀ En)❀ n ✕ 1 satisfies:

◮ locally tree-like ◮ regular degree ∆

Further

◮ Q1 = ✍+1, Q0 = 1 2✍+1 + 1 2✍1 Aij ∼ Q1 Aij ∼ Q0

What can local algorithms achieve?

Deshpande, Montanari Planted Cliques November 5, 2013 44 / 49

The Sparse Graph Analogue

Gn = ([n]❀ En)❀ n ✕ 1 satisfies:

◮ locally tree-like ◮ regular degree ∆

Further

◮ Q1 = ✍+1, Q0 = 1 2✍+1 + 1 2✍1 Aij ∼ Q1 Aij ∼ Q0

What can local algorithms achieve?

Deshpande, Montanari Planted Cliques November 5, 2013 44 / 49

The Sparse Graph Analogue

Gn = ([n]❀ En)❀ n ✕ 1 satisfies:

◮ locally tree-like ◮ regular degree ∆

Further

◮ Q1 = ✍+1, Q0 = 1 2✍+1 + 1 2✍1 Aij ∼ Q1 Aij ∼ Q0

What can local algorithms achieve?

Deshpande, Montanari Planted Cliques November 5, 2013 44 / 49

What can we hope for?

If ❥S❥ = C

n ♣ ∆:

❜

Snaive = Random set of size ❥S❥ ✮ 1 nE❢ ❜ Snaive✹S❣ = Θ

✒ 1

♣ ∆

✓

✿

Deshpande, Montanari Planted Cliques November 5, 2013 45 / 49

What can we hope for?

If ❥S❥ = C

n ♣ ∆:

Poisson bound ✮ for any local algorithm: 1 nE❢ ❜ S✹S❣ ✕ eC ✵♣

∆✿

Deshpande, Montanari Planted Cliques November 5, 2013 46 / 49

A result for local algorithms. . .

Theorem (Deshpande, Montanari, 2013)

Let Gn converge locally to ∆regular tree: If ❥S❥ ✕ (1 + ✧)

n ♣ e∆ there exists a local algorithm achieving

1 nE❢S✹ ❜ S❣ ✔ eΘ(

♣ ∆)✿

Conversely, if ❥S❥ ✔ (1 ✧)

n ♣ e∆ every local algorithm suffers

1 nE❢S✹ ❜ S❣ ✕ Θ

✒ 1

♣ ∆

✓

With ∆ = n 1 we recover the complete graph result!

Deshpande, Montanari Planted Cliques November 5, 2013 47 / 49

A result for local algorithms. . .

Theorem (Deshpande, Montanari, 2013)

Let Gn converge locally to ∆regular tree: If ❥S❥ ✕ (1 + ✧)

n ♣ e∆ there exists a local algorithm achieving

1 nE❢S✹ ❜ S❣ ✔ eΘ(

♣ ∆)✿

Conversely, if ❥S❥ ✔ (1 ✧)

n ♣ e∆ every local algorithm suffers

1 nE❢S✹ ❜ S❣ ✕ Θ

✒ 1

♣ ∆

✓

With ∆ = n 1 we recover the complete graph result!

Deshpande, Montanari Planted Cliques November 5, 2013 47 / 49

A result for local algorithms. . .

Theorem (Deshpande, Montanari, 2013)

Let Gn converge locally to ∆regular tree: If ❥S❥ ✕ (1 + ✧)

n ♣ e∆ there exists a local algorithm achieving

1 nE❢S✹ ❜ S❣ ✔ eΘ(

♣ ∆)✿

Conversely, if ❥S❥ ✔ (1 ✧)

n ♣ e∆ every local algorithm suffers

1 nE❢S✹ ❜ S❣ ✕ Θ

✒ 1

♣ ∆

✓

With ∆ = n 1 we recover the complete graph result!

Deshpande, Montanari Planted Cliques November 5, 2013 47 / 49

To conclude. . .

◮ Message-passing algorithm performs “weighted” counts of

non-reversing trees

◮ Such structures have been used elsewhere:

✍ Clustering sparse networks: [Krzakala et al. 2013] ✍ Compressed sensing: [Bayati, Lelarge, Montanari 2013]

Deshpande, Montanari Planted Cliques November 5, 2013 48 / 49

To conclude. . .

◮ Message-passing algorithm performs “weighted” counts of

non-reversing trees

◮ Such structures have been used elsewhere:

✍ Clustering sparse networks: [Krzakala et al. 2013] ✍ Compressed sensing: [Bayati, Lelarge, Montanari 2013]

Deshpande, Montanari Planted Cliques November 5, 2013 48 / 49

To conclude. . .

◮ Message-passing algorithm performs “weighted” counts of

non-reversing trees

◮ Such structures have been used elsewhere:

✍ Clustering sparse networks: [Krzakala et al. 2013] ✍ Compressed sensing: [Bayati, Lelarge, Montanari 2013]

Deshpande, Montanari Planted Cliques November 5, 2013 48 / 49

To conclude. . .

◮ What is the “dense” analogue for local algorithms? ◮ What about other structural properties?

Thank you!

Deshpande, Montanari Planted Cliques November 5, 2013 49 / 49

To conclude. . .

◮ What is the “dense” analogue for local algorithms? ◮ What about other structural properties?

Thank you!

Deshpande, Montanari Planted Cliques November 5, 2013 49 / 49

To conclude. . .

◮ What is the “dense” analogue for local algorithms? ◮ What about other structural properties?

Thank you!

Deshpande, Montanari Planted Cliques November 5, 2013 49 / 49