Planted Cliques, Iterative Thresholding and Message Passing - - PowerPoint PPT Presentation

planted cliques iterative thresholding and message
SMART_READER_LITE
LIVE PREVIEW

Planted Cliques, Iterative Thresholding and Message Passing - - PowerPoint PPT Presentation

Planted Cliques, Iterative Thresholding and Message Passing Algorithms Yash Deshpande and Andrea Montanari Stanford University November 5, 2013 Deshpande, Montanari Planted Cliques November 5, 2013 1 / 49 Problem Definition Given


slide-1
SLIDE 1

Planted Cliques, Iterative Thresholding and Message Passing Algorithms

Yash Deshpande and Andrea Montanari

Stanford University

November 5, 2013

Deshpande, Montanari Planted Cliques November 5, 2013 1 / 49

slide-2
SLIDE 2

Problem Definition

Given distributions Q0❀ Q1, A Set ✦ S ✚ [n] Data ✦ Aij ✘

Q1 if i❀ j ✷ S Q0

  • therwise.

Aij = Aji Problem: Given A, identify S

Deshpande, Montanari Planted Cliques November 5, 2013 2 / 49

slide-3
SLIDE 3

Problem Definition

Given distributions Q0❀ Q1, A Set ✦ S ✚ [n] Data ✦ Aij ✘

Q1 if i❀ j ✷ S Q0

  • therwise.

Aij = Aji

S S A = Q1 Q0

Problem: Given A, identify S

Deshpande, Montanari Planted Cliques November 5, 2013 2 / 49

slide-4
SLIDE 4

Problem Definition

Given distributions Q0❀ Q1, A Set ✦ S ✚ [n] Data ✦ Aij ✘

Q1 if i❀ j ✷ S Q0

  • therwise.

Aij = Aji

S S A = Q1 Q0

Problem: Given A, identify S

Deshpande, Montanari Planted Cliques November 5, 2013 2 / 49

slide-5
SLIDE 5

An Example

Q1 = N(✕❀ 1) Q0 = N(0❀ 1)✿

S S A = Q1 Q0

Deshpande, Montanari Planted Cliques November 5, 2013 3 / 49

slide-6
SLIDE 6

An Example

A = ✕uSuT

S

+ Z

λ λ λ λ ∼ N(0, 1)

Data = Sparse, Low Rank + noise

Deshpande, Montanari Planted Cliques November 5, 2013 4 / 49

slide-7
SLIDE 7

An Example

A = ✕uSuT

S

+ Z

λ λ λ λ ∼ N(0, 1)

Data = Sparse, Low Rank + noise

Deshpande, Montanari Planted Cliques November 5, 2013 4 / 49

slide-8
SLIDE 8

Much work in statistics

Denoising: y = x + noise [Donoho, Jin 2004], [Arias-Castro, Candes, Durand 2011] [Arias-Castro, Bubeck, Lugosi 2012] Sparse signal recovery: y = Ax + noise [Candes, Romberg, Tao 2004], [Chen, Donoho 1998] The simple example combines different structures

Deshpande, Montanari Planted Cliques November 5, 2013 5 / 49

slide-9
SLIDE 9

Much work in statistics

Denoising: y = x + noise [Donoho, Jin 2004], [Arias-Castro, Candes, Durand 2011] [Arias-Castro, Bubeck, Lugosi 2012] Sparse signal recovery: y = Ax + noise [Candes, Romberg, Tao 2004], [Chen, Donoho 1998] The simple example combines different structures

Deshpande, Montanari Planted Cliques November 5, 2013 5 / 49

slide-10
SLIDE 10

Much work in statistics

Denoising: y = x + noise [Donoho, Jin 2004], [Arias-Castro, Candes, Durand 2011] [Arias-Castro, Bubeck, Lugosi 2012] Sparse signal recovery: y = Ax + noise [Candes, Romberg, Tao 2004], [Chen, Donoho 1998] The simple example combines different structures

Deshpande, Montanari Planted Cliques November 5, 2013 5 / 49

slide-11
SLIDE 11

Our Running Example: Planted Cliques

A is “adjacency” matrix Q1 = ✍+1 Q0 = 1 2✍+1 + 1 2✍1✿ S forms a clique in the graph [Jerrum, 1992]

Deshpande, Montanari Planted Cliques November 5, 2013 6 / 49

slide-12
SLIDE 12

Our Running Example: Planted Cliques

A is “adjacency” matrix Q1 = ✍+1 Q0 = 1 2✍+1 + 1 2✍1✿ S forms a clique in the graph [Jerrum, 1992]

Deshpande, Montanari Planted Cliques November 5, 2013 6 / 49

slide-13
SLIDE 13

Our Running Example: Planted Cliques

◮ Average case version of

MAX-CLIQUE

◮ Communities in networks

Deshpande, Montanari Planted Cliques November 5, 2013 7 / 49

slide-14
SLIDE 14

Our Running Example: Planted Cliques

◮ Average case version of

MAX-CLIQUE

◮ Communities in networks

Deshpande, Montanari Planted Cliques November 5, 2013 7 / 49

slide-15
SLIDE 15

How large should S be?

Let ❥S❥ = k Size of largest clique in G

n❀ 1

2

Second moment calculation ✮ k ❃ 2 log2 n.

Deshpande, Montanari Planted Cliques November 5, 2013 8 / 49

slide-16
SLIDE 16

How large should S be?

Let ❥S❥ = k Size of largest clique in G

n❀ 1

2

Second moment calculation ✮ k ❃ 2 log2 n.

Deshpande, Montanari Planted Cliques November 5, 2013 8 / 49

slide-17
SLIDE 17

Progress(0)

Complexity k n2 log n 2 log2 n

Exhaustive search

Deshpande, Montanari Planted Cliques November 5, 2013 9 / 49

slide-18
SLIDE 18

A Naive Algorithm

Pick k largest degree vertices of G as ❜ S.

Deshpande, Montanari Planted Cliques November 5, 2013 10 / 49

slide-19
SLIDE 19

A Naive Algorithm

Pick k largest degree vertices of G as ❜ S.

Deshpande, Montanari Planted Cliques November 5, 2013 10 / 49

slide-20
SLIDE 20

Analysis of NAIVE

If i ❂ ✷ S: deg(i) = Binomial

n 1❀ 1 2

✮ max

i ❂ ✷S deg(i) ✔ n

2 + O(

q

n log n) If i ✷ S: deg(i) = k 1 + Binomial

n k + 1❀ 1 2

✮ min

i✷S deg(i) ✕ k 1

2 + n 2 O(

q

n log n) NAIVE works if: k ✕ O(

n log n)

Deshpande, Montanari Planted Cliques November 5, 2013 11 / 49

slide-21
SLIDE 21

Analysis of NAIVE

If i ❂ ✷ S: deg(i) = Binomial

n 1❀ 1 2

✮ max

i ❂ ✷S deg(i) ✔ n

2 + O(

q

n log n) If i ✷ S: deg(i) = k 1 + Binomial

n k + 1❀ 1 2

✮ min

i✷S deg(i) ✕ k 1

2 + n 2 O(

q

n log n) NAIVE works if: k ✕ O(

n log n)

Deshpande, Montanari Planted Cliques November 5, 2013 11 / 49

slide-22
SLIDE 22

Analysis of NAIVE

If i ❂ ✷ S: deg(i) = Binomial

n 1❀ 1 2

✮ max

i ❂ ✷S deg(i) ✔ n

2 + O(

q

n log n) If i ✷ S: deg(i) = k 1 + Binomial

n k + 1❀ 1 2

✮ min

i✷S deg(i) ✕ k 1

2 + n 2 O(

q

n log n) NAIVE works if: k ✕ O(

n log n)

Deshpande, Montanari Planted Cliques November 5, 2013 11 / 49

slide-23
SLIDE 23

Analysis of NAIVE

If i ❂ ✷ S: deg(i) = Binomial

n 1❀ 1 2

✮ max

i ❂ ✷S deg(i) ✔ n

2 + O(

q

n log n) If i ✷ S: deg(i) = k 1 + Binomial

n k + 1❀ 1 2

✮ min

i✷S deg(i) ✕ k 1

2 + n 2 O(

q

n log n) NAIVE works if: k ✕ O(

n log n)

Deshpande, Montanari Planted Cliques November 5, 2013 11 / 49

slide-24
SLIDE 24

Progress(1)

Complexity k n2 log n 2 log2 n n2 √n log n

[Kučera, 1995]

Deshpande, Montanari Planted Cliques November 5, 2013 12 / 49

slide-25
SLIDE 25

Spectral Method

A = uSuT

S

+ Z

1 1 1 1 ±1 ±1 ±1

Hopefully v1(A) ✙ uS

Deshpande, Montanari Planted Cliques November 5, 2013 13 / 49

slide-26
SLIDE 26

Spectral Method

A = uSuT

S

+ Z

1 1 1 1 ±1 ±1 ±1

Hopefully v1(A) ✙ uS

Deshpande, Montanari Planted Cliques November 5, 2013 13 / 49

slide-27
SLIDE 27

Analysis of SPECTRAL

1 ♣nA =

✒ k

♣n

✓ ⑤ ④③ ⑥

eSeT

S + Z

♣n✿ By standard linear algebra: ✔ ✢

✌ ✌ ✌ ✌

Z ♣n

✌ ✌ ✌ ✌

2

✙ 2 = ✮ ❤v1(A)❀ eS✐ ✕ 1 ✍(✔) SPECTRAL works if k ✕ C♣n.

Deshpande, Montanari Planted Cliques November 5, 2013 14 / 49

slide-28
SLIDE 28

Progress(2)

Complexity k n2 log n 2 log2 n n2 √n log n n2 log n C√n

[Alon, Krivelevich and Sudakov, 1998]

Deshpande, Montanari Planted Cliques November 5, 2013 15 / 49

slide-29
SLIDE 29

Progress(2)

Complexity k n2 log n 2 log2 n n2 √n log n n2 log n C√n

[Ames, Vavasis 2009], [Feige, Krauthgamer 2000]

Deshpande, Montanari Planted Cliques November 5, 2013 16 / 49

slide-30
SLIDE 30

Progress(3)

Complexity k n2 log n 2 log2 n n2 √n log n n2 log n C√n 1.65√n

[Dekel, Gurel-Gurevich and Peres, 2010]

Deshpande, Montanari Planted Cliques November 5, 2013 17 / 49

slide-31
SLIDE 31

Progress(3)

Complexity k n2 log n 2 log2 n n2 √n log n n2 log n C√n 1.261√n

[Dekel, Gurel-Gurevich and Peres, 2010]

Deshpande, Montanari Planted Cliques November 5, 2013 18 / 49

slide-32
SLIDE 32

Progress(3)

Complexity k n2 log n 2 log2 n n2 √n log n n2 log n C√n 1.261√n nr+2 C n

2r

[Alon, Krivelevich and Sudakov, 1998]

Deshpande, Montanari Planted Cliques November 5, 2013 19 / 49

slide-33
SLIDE 33

Phase transition in spectrum of A

Deshpande, Montanari Planted Cliques November 5, 2013 20 / 49

slide-34
SLIDE 34

Phase transition in spectrum of A

If k ✕ (1 + ✧)♣n

2 −2 Limiting Spectral Density

❤v1(A)❀ eS✐ ✕ ✍(✧)

Deshpande, Montanari Planted Cliques November 5, 2013 20 / 49

slide-35
SLIDE 35

Phase transition in spectrum of A

If k ✕ (1 + ✧)♣n

2 −2 Limiting Spectral Density

❤v1(A)❀ eS✐ ✕ ✍(✧) If k ✔ (1 ✧)♣n

2 −2 Limiting Spectral Density

❤v1(A)❀ eS✐ ✔ n1❂2+✍✵(✧)

Deshpande, Montanari Planted Cliques November 5, 2013 20 / 49

slide-36
SLIDE 36

Phase transition in spectrum of A

If k ✕ (1 + ✧)♣n

2 −2 Limiting Spectral Density

❤v1(A)❀ eS✐ ✕ ✍(✧) If k ✔ (1 ✧)♣n

2 −2 Limiting Spectral Density

❤v1(A)❀ eS✐ ✔ n1❂2+✍✵(✧) [Knowles, Yin, 2011]

Deshpande, Montanari Planted Cliques November 5, 2013 20 / 49

slide-37
SLIDE 37

Seems much harder than it looks!

◮ “Statistical algorithms” fail if k = n1❂2✍: [Feldman et al., 2012] ◮ r-Lovász-Schrijver fails for k ✔

n❂2r: [Feige, Krauthgamer, 2002]

◮ r-Lasserre fails for k ✔ ♣n (log n)r2 : [Widgerson and Meka, 2013]

Deshpande, Montanari Planted Cliques November 5, 2013 21 / 49

slide-38
SLIDE 38

Seems much harder than it looks!

◮ “Statistical algorithms” fail if k = n1❂2✍: [Feldman et al., 2012] ◮ r-Lovász-Schrijver fails for k ✔

n❂2r: [Feige, Krauthgamer, 2002]

◮ r-Lasserre fails for k ✔ ♣n (log n)r2 : [Widgerson and Meka, 2013]

Deshpande, Montanari Planted Cliques November 5, 2013 21 / 49

slide-39
SLIDE 39

Seems much harder than it looks!

◮ “Statistical algorithms” fail if k = n1❂2✍: [Feldman et al., 2012] ◮ r-Lovász-Schrijver fails for k ✔

n❂2r: [Feige, Krauthgamer, 2002]

◮ r-Lasserre fails for k ✔ ♣n (log n)r2 : [Widgerson and Meka, 2013]

Deshpande, Montanari Planted Cliques November 5, 2013 21 / 49

slide-40
SLIDE 40

Our result

Theorem (Deshpande, Montanari, 2013)

If ❥S❥ = k ✕ (1 + ✧)

n❂e, there exists an O(n2 log n) time algorithm that identifies S with high probability. I will present:

1 A (wrong) heuristic analysis 2 How to fix the heuristic 3 Lower bounds

Deshpande, Montanari Planted Cliques November 5, 2013 22 / 49

slide-41
SLIDE 41

Our result

Theorem (Deshpande, Montanari, 2013)

If ❥S❥ = k ✕ (1 + ✧)

n❂e, there exists an O(n2 log n) time algorithm that identifies S with high probability. I will present:

1 A (wrong) heuristic analysis 2 How to fix the heuristic 3 Lower bounds

Deshpande, Montanari Planted Cliques November 5, 2013 22 / 49

slide-42
SLIDE 42

Our result

Theorem (Deshpande, Montanari, 2013)

If ❥S❥ = k ✕ (1 + ✧)

n❂e, there exists an O(n2 log n) time algorithm that identifies S with high probability. I will present:

1 A (wrong) heuristic analysis 2 How to fix the heuristic 3 Lower bounds

Deshpande, Montanari Planted Cliques November 5, 2013 22 / 49

slide-43
SLIDE 43

Iterative Thresholding

The power iteration: vt+1 = A vt✿ Improvement: vt+1 = AFt(vt)✿ where Ft(v) = (ft(v1)❀ ft(v2)❀ ✁ ✁ ✁ ❀ ft(vn))T Choose ft(✁) to exploit sparsity of eS

Deshpande, Montanari Planted Cliques November 5, 2013 23 / 49

slide-44
SLIDE 44

Iterative Thresholding

The power iteration: vt+1 = A vt✿ Improvement: vt+1 = AFt(vt)✿ where Ft(v) = (ft(v1)❀ ft(v2)❀ ✁ ✁ ✁ ❀ ft(vn))T Choose ft(✁) to exploit sparsity of eS

Deshpande, Montanari Planted Cliques November 5, 2013 23 / 49

slide-45
SLIDE 45

Iterative Thresholding

The power iteration: vt+1 = A vt✿ Improvement: vt+1 = AFt(vt)✿ where Ft(v) = (ft(v1)❀ ft(v2)❀ ✁ ✁ ✁ ❀ ft(vn))T Choose ft(✁) to exploit sparsity of eS

Deshpande, Montanari Planted Cliques November 5, 2013 23 / 49

slide-46
SLIDE 46

Iterative Thresholding

The power iteration: vt+1 = A vt✿ Improvement: vt+1 = AFt(vt)✿ where Ft(v) = (ft(v1)❀ ft(v2)❀ ✁ ✁ ✁ ❀ ft(vn))T Choose ft(✁) to exploit sparsity of eS

Deshpande, Montanari Planted Cliques November 5, 2013 23 / 49

slide-47
SLIDE 47

Iterative Thresholding

The power iteration: vt+1 = A vt✿ Improvement: vt+1 = AFt(vt)✿ where Ft(v) = (ft(v1)❀ ft(v2)❀ ✁ ✁ ✁ ❀ ft(vn))T Choose ft(✁) to exploit sparsity of eS

Deshpande, Montanari Planted Cliques November 5, 2013 23 / 49

slide-48
SLIDE 48

(Wrong) Analysis

vt+1

i

= 1 ♣n

j

Aijft(vt

j )✿

Aij are random ✝1 r.v. Use Central Limit Theorem for vt+1

i

Deshpande, Montanari Planted Cliques November 5, 2013 24 / 49

slide-49
SLIDE 49

(Wrong) Analysis

vt+1

i

= 1 ♣n

j

Aijft(vt

j )✿

Aij are random ✝1 r.v. Use Central Limit Theorem for vt+1

i

Deshpande, Montanari Planted Cliques November 5, 2013 24 / 49

slide-50
SLIDE 50

(Wrong) Analysis

If i ❂ ✷ S: vt+1

i

= 1 ♣n

j

Aijft(vt

j )

✙ N

0❀ 1 n

j

ft(vt

j )2

Letting vt

i ✙ N(0❀ ✛2 t ). . .

✛2

t+1 = 1

n

j

ft(vt

j )2

= E❢ft(✛t✘)2❣❀ ✘ ✘ N(0❀ 1)

Deshpande, Montanari Planted Cliques November 5, 2013 25 / 49

slide-51
SLIDE 51

(Wrong) Analysis

If i ❂ ✷ S: vt+1

i

= 1 ♣n

j

Aijft(vt

j )

✙ N

0❀ 1 n

j

ft(vt

j )2

Letting vt

i ✙ N(0❀ ✛2 t ). . .

✛2

t+1 = 1

n

j

ft(vt

j )2

= E❢ft(✛t✘)2❣❀ ✘ ✘ N(0❀ 1)

Deshpande, Montanari Planted Cliques November 5, 2013 25 / 49

slide-52
SLIDE 52

(Wrong) Analysis

If i ❂ ✷ S: vt+1

i

= 1 ♣n

j

Aijft(vt

j )

✙ N

0❀ 1 n

j

ft(vt

j )2

Letting vt

i ✙ N(0❀ ✛2 t ). . .

✛2

t+1 = 1

n

j

ft(vt

j )2

= E❢ft(✛t✘)2❣❀ ✘ ✘ N(0❀ 1)

Deshpande, Montanari Planted Cliques November 5, 2013 25 / 49

slide-53
SLIDE 53

(Wrong) Analysis

If i ❂ ✷ S: vt+1

i

= 1 ♣n

j

Aijft(vt

j )

✙ N

0❀ 1 n

j

ft(vt

j )2

Letting vt

i ✙ N(0❀ ✛2 t ). . .

✛2

t+1 = 1

n

j

ft(vt

j )2

= E❢ft(✛t✘)2❣❀ ✘ ✘ N(0❀ 1)

Deshpande, Montanari Planted Cliques November 5, 2013 25 / 49

slide-54
SLIDE 54

(Wrong) Analysis

If i ✷ S: vt+1

i

= 1 ♣n

j✷S

ft(vt

j )

⑤ ④③ ⑥

✖t+1

+ 1 ♣n

j ❂ ✷S

Aijft(vt

j )

⑤ ④③ ⑥

✙N(0❀✛2

t+1)

where ✖t+1 = 1 ♣n

j✷S

ft(vt

j )

=

✒ k

♣n

✓ ✒1

k

j✷S

ft(vt

j )

= ✔ E❢ft(✖t + ✛t✘)❣❀ ✘ ✘ N(0❀ 1)

Deshpande, Montanari Planted Cliques November 5, 2013 26 / 49

slide-55
SLIDE 55

(Wrong) Analysis

If i ✷ S: vt+1

i

= 1 ♣n

j✷S

ft(vt

j )

⑤ ④③ ⑥

✖t+1

+ 1 ♣n

j ❂ ✷S

Aijft(vt

j )

⑤ ④③ ⑥

✙N(0❀✛2

t+1)

where ✖t+1 = 1 ♣n

j✷S

ft(vt

j )

=

✒ k

♣n

✓ ✒1

k

j✷S

ft(vt

j )

= ✔ E❢ft(✖t + ✛t✘)❣❀ ✘ ✘ N(0❀ 1)

Deshpande, Montanari Planted Cliques November 5, 2013 26 / 49

slide-56
SLIDE 56

(Wrong) Analysis

If i ✷ S: vt+1

i

= 1 ♣n

j✷S

ft(vt

j )

⑤ ④③ ⑥

✖t+1

+ 1 ♣n

j ❂ ✷S

Aijft(vt

j )

⑤ ④③ ⑥

✙N(0❀✛2

t+1)

where ✖t+1 = 1 ♣n

j✷S

ft(vt

j )

=

✒ k

♣n

✓ ✒1

k

j✷S

ft(vt

j )

= ✔ E❢ft(✖t + ✛t✘)❣❀ ✘ ✘ N(0❀ 1)

Deshpande, Montanari Planted Cliques November 5, 2013 26 / 49

slide-57
SLIDE 57

(Wrong) Analysis

If i ✷ S: vt+1

i

= 1 ♣n

j✷S

ft(vt

j )

⑤ ④③ ⑥

✖t+1

+ 1 ♣n

j ❂ ✷S

Aijft(vt

j )

⑤ ④③ ⑥

✙N(0❀✛2

t+1)

where ✖t+1 = 1 ♣n

j✷S

ft(vt

j )

=

✒ k

♣n

✓ ✒1

k

j✷S

ft(vt

j )

= ✔ E❢ft(✖t + ✛t✘)❣❀ ✘ ✘ N(0❀ 1)

Deshpande, Montanari Planted Cliques November 5, 2013 26 / 49

slide-58
SLIDE 58

Summarizing . . .

−4 −2 2 4 0.1 0.2 0.3 vt

i, i ∈ S

vt

i, i /

∈ S

Deshpande, Montanari Planted Cliques November 5, 2013 27 / 49

slide-59
SLIDE 59

State Evolution

✖t+1 = ✔ E ❢ft(✖t + ✛t✘)❣ ✛2

t+1 = E❢ft(✛t✘)2❣✿

Using the optimal function ft(x) = e✖tx✖2

t

✖t+1 = ✔e✖2

t ❂2

✛2

t+1 = 1

Deshpande, Montanari Planted Cliques November 5, 2013 28 / 49

slide-60
SLIDE 60

State Evolution

✖t+1 = ✔ E ❢ft(✖t + ✛t✘)❣ ✛2

t+1 = E❢ft(✛t✘)2❣✿

Using the optimal function ft(x) = e✖tx✖2

t

✖t+1 = ✔e✖2

t ❂2

✛2

t+1 = 1

Deshpande, Montanari Planted Cliques November 5, 2013 28 / 49

slide-61
SLIDE 61

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

slide-62
SLIDE 62

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

slide-63
SLIDE 63

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

slide-64
SLIDE 64

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

slide-65
SLIDE 65

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

slide-66
SLIDE 66

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

slide-67
SLIDE 67

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

slide-68
SLIDE 68

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

slide-69
SLIDE 69

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

slide-70
SLIDE 70

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

slide-71
SLIDE 71

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

If ✔ ❁

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

slide-72
SLIDE 72

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

If ✔ ❁

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

slide-73
SLIDE 73

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

If ✔ ❁

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

slide-74
SLIDE 74

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

If ✔ ❁

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

slide-75
SLIDE 75

Fixed points develop below threshold!

If ✔ ❃

1 ♣e

µt+1 µt

If ✔ ❁

1 ♣e

µt+1 µt

Deshpande, Montanari Planted Cliques November 5, 2013 29 / 49

slide-76
SLIDE 76

Analysis is wrong but. . .

Theorem (Deshpande, Montanari, 2013)

If ❥S❥ = k ✕ (1 + ✧)

n❂e, there exists an O(n2 log n) time algorithm that identifies S with high probability. . . . so we modify the algorithm.

Deshpande, Montanari Planted Cliques November 5, 2013 30 / 49

slide-77
SLIDE 77

What algorithm?

Slight modification to iterative scheme: (vt

i )i✷[n] ✦ (vt i✦j)i❀j✷[n]

vt+1

i✦j =

1 ♣n

❵✻=i❀j

Ai❵ ft(vt

❵✦i)✿

Analysis is exact as n ✦ ✶.

Deshpande, Montanari Planted Cliques November 5, 2013 31 / 49

slide-78
SLIDE 78

What algorithm?

Slight modification to iterative scheme: (vt

i )i✷[n] ✦ (vt i✦j)i❀j✷[n]

vt+1

i✦j =

1 ♣n

❵✻=i❀j

Ai❵ ft(vt

❵✦i)✿

Analysis is exact as n ✦ ✶.

Deshpande, Montanari Planted Cliques November 5, 2013 31 / 49

slide-79
SLIDE 79

What algorithm?

Slight modification to iterative scheme: (vt

i )i✷[n] ✦ (vt i✦j)i❀j✷[n]

vt+1

i✦j =

1 ♣n

❵✻=i❀j

Ai❵ ft(vt

❵✦i)✿

Analysis is exact as n ✦ ✶.

Deshpande, Montanari Planted Cliques November 5, 2013 31 / 49

slide-80
SLIDE 80

What algorithm?

Slight modification to iterative scheme: (vt

i )i✷[n] ✦ (vt i✦j)i❀j✷[n]

vt+1

i✦j =

1 ♣n

❵✻=i❀j

Ai❵ ft(vt

❵✦i)✿

Analysis is exact as n ✦ ✶.

Deshpande, Montanari Planted Cliques November 5, 2013 31 / 49

slide-81
SLIDE 81

Fixing the heuristic

Lemma

Let (ft(z))t✕0 be a sequence of polynomials. Then, for every fixed t, and bounded, continuous function ✥ : R ✦ R the following limit holds in probability: lim

n✦✶

1 ♣n

i✷S

✥(vt

i✦j) = ✔ E❢✥(✖t + ✛t✘)❣❀

lim

n✦✶

1 n

i✷[n]♥S

✥(vt

i✦j) = E❢✥(✛t✘)❣❀

where ✘ ✘ N(0❀ 1).

Deshpande, Montanari Planted Cliques November 5, 2013 32 / 49

slide-82
SLIDE 82

Proof Technique

Key ideas: Expand vt

i✦j for polynomial ft(✁)

Wrong analysis works if A ✦ At

Deshpande, Montanari Planted Cliques November 5, 2013 33 / 49

slide-83
SLIDE 83

Proof Technique

Key ideas: Expand vt

i✦j for polynomial ft(✁)

Wrong analysis works if A ✦ At

Deshpande, Montanari Planted Cliques November 5, 2013 33 / 49

slide-84
SLIDE 84

Proof Technique

Key ideas: Expand vt

i✦j for polynomial ft(✁)

Wrong analysis works if A ✦ At

Deshpande, Montanari Planted Cliques November 5, 2013 33 / 49

slide-85
SLIDE 85

Proof Technique - Expanding v t

Let ft(x) = x2❀ v0

i✦j = 1

v1

i✦j =

k✻=j

Aik✿

i k j

Deshpande, Montanari Planted Cliques November 5, 2013 34 / 49

slide-86
SLIDE 86

Proof Technique - Expanding v t

v2

i✦j =

k✻=j

Aik(v1

k✦i)2

=

k✻=j

Aik

✒ ❳

❵✻=i

Ak❵

✓✒ ❳

m✻=i

Akm

=

k✻=j

❵✻=i

m✻=i

AikAk❵Akm✿

i j k ℓ m

Deshpande, Montanari Planted Cliques November 5, 2013 35 / 49

slide-87
SLIDE 87

Proof Technique - Expanding v t

v2

i✦j =

k✻=j

Aik(v1

k✦i)2

=

k✻=j

Aik

✒ ❳

❵✻=i

Ak❵

✓✒ ❳

m✻=i

Akm

=

k✻=j

❵✻=i

m✻=i

AikAk❵Akm✿

i j k ℓ m

Deshpande, Montanari Planted Cliques November 5, 2013 35 / 49

slide-88
SLIDE 88

Proof Technique - Expanding v t

v2

i✦j =

k✻=j

Aik(v1

k✦i)2

=

k✻=j

Aik

✒ ❳

❵✻=i

Ak❵

✓✒ ❳

m✻=i

Akm

=

k✻=j

❵✻=i

m✻=i

AikAk❵Akm✿

i j k ℓ m

Deshpande, Montanari Planted Cliques November 5, 2013 35 / 49

slide-89
SLIDE 89

Proof Technique

vt+1

i✦j =

k✻=i

Aikft(vt

k✦i)✿

i k ℓ m

  • p

q r s w

✘t+1

i✦j =

k✻=i

At

ikft(✘t k✦i)✿

i k ℓ m

  • p

q r s w

Deshpande, Montanari Planted Cliques November 5, 2013 36 / 49

slide-90
SLIDE 90

Proof Technique - a Combinatorial Lemma

Lemma

vt

i✦j =

T✷❚ t

i✦j

A(T)Γ(T)v0(T) where ❚ t

i✦j consists rooted, labeled trees that:

1 have maximum depth t. 2 do not backtrack.

(Similarly for the ✘t

i✦j)

Deshpande, Montanari Planted Cliques November 5, 2013 37 / 49

slide-91
SLIDE 91

Proof Technique - Moment Method

vt+1

i✦j =

k✻=i

Aikft(vt

k✦i)✿

i k ℓ m

  • p

q r s w

✘t+1

i✦j =

k✻=i

At

ikft(✘t k✦i)✿

i k ℓ m

  • p

q r s w

limn✦✶ Moments of vt+1 = Moments of ✘t+1.

Deshpande, Montanari Planted Cliques November 5, 2013 38 / 49

slide-92
SLIDE 92

Proof Technique - Moment Method

vt+1

i✦j =

k✻=i

Aikft(vt

k✦i)✿

i k ℓ m

  • p

q r s w

✘t+1

i✦j =

k✻=i

At

ikft(✘t k✦i)✿

i k ℓ m

  • p

q r s w

limn✦✶ Moments of vt+1 = Moments of ✘t+1.

Deshpande, Montanari Planted Cliques November 5, 2013 38 / 49

slide-93
SLIDE 93

Progress(4)

Complexity k n2 log n 2 log2 n n2 2√n log n n2 log n C√n 1.261√n Spectral threshold = √n n

e

Deshpande, Montanari Planted Cliques November 5, 2013 39 / 49

slide-94
SLIDE 94

Is this threshold fundamental?

Rest of the talk: perhaps

Deshpande, Montanari Planted Cliques November 5, 2013 40 / 49

slide-95
SLIDE 95

The “Hidden Set” Problem

Given Gn = ([n]❀ En) A Set ✦ S ✚ [n] Data ✦ Aij ✘

Q1 if i❀ j ✷ S❀ Q0

  • therwise.

Aij ∼ Q1 Aij ∼ Q0

Problem: Given edge labels (Aij)(i❀j)✷En, identify S

Deshpande, Montanari Planted Cliques November 5, 2013 41 / 49

slide-96
SLIDE 96

The “Hidden Set” Problem

Given Gn = ([n]❀ En) A Set ✦ S ✚ [n] Data ✦ Aij ✘

Q1 if i❀ j ✷ S❀ Q0

  • therwise.

Aij ∼ Q1 Aij ∼ Q0

Problem: Given edge labels (Aij)(i❀j)✷En, identify S

Deshpande, Montanari Planted Cliques November 5, 2013 41 / 49

slide-97
SLIDE 97

The “Hidden Set” Problem

Given Gn = ([n]❀ En) A Set ✦ S ✚ [n] Data ✦ Aij ✘

Q1 if i❀ j ✷ S❀ Q0

  • therwise.

S S A =

Problem: Given edge labels (Aij)(i❀j)✷En, identify S

Deshpande, Montanari Planted Cliques November 5, 2013 42 / 49

slide-98
SLIDE 98

“Local” Algorithms

A t-local algorithm computes: Estimate at i:

u(i) = F(ABall(i❀t))

Deshpande, Montanari Planted Cliques November 5, 2013 43 / 49

slide-99
SLIDE 99

The Sparse Graph Analogue

Gn = ([n]❀ En)❀ n ✕ 1 satisfies:

◮ locally tree-like ◮ regular degree ∆

Further

◮ Q1 = ✍+1, Q0 = 1 2✍+1 + 1 2✍1 Aij ∼ Q1 Aij ∼ Q0

What can local algorithms achieve?

Deshpande, Montanari Planted Cliques November 5, 2013 44 / 49

slide-100
SLIDE 100

The Sparse Graph Analogue

Gn = ([n]❀ En)❀ n ✕ 1 satisfies:

◮ locally tree-like ◮ regular degree ∆

Further

◮ Q1 = ✍+1, Q0 = 1 2✍+1 + 1 2✍1 Aij ∼ Q1 Aij ∼ Q0

What can local algorithms achieve?

Deshpande, Montanari Planted Cliques November 5, 2013 44 / 49

slide-101
SLIDE 101

The Sparse Graph Analogue

Gn = ([n]❀ En)❀ n ✕ 1 satisfies:

◮ locally tree-like ◮ regular degree ∆

Further

◮ Q1 = ✍+1, Q0 = 1 2✍+1 + 1 2✍1 Aij ∼ Q1 Aij ∼ Q0

What can local algorithms achieve?

Deshpande, Montanari Planted Cliques November 5, 2013 44 / 49

slide-102
SLIDE 102

The Sparse Graph Analogue

Gn = ([n]❀ En)❀ n ✕ 1 satisfies:

◮ locally tree-like ◮ regular degree ∆

Further

◮ Q1 = ✍+1, Q0 = 1 2✍+1 + 1 2✍1 Aij ∼ Q1 Aij ∼ Q0

What can local algorithms achieve?

Deshpande, Montanari Planted Cliques November 5, 2013 44 / 49

slide-103
SLIDE 103

What can we hope for?

If ❥S❥ = C

n ♣ ∆:

Snaive = Random set of size ❥S❥ ✮ 1 nE❢ ❜ Snaive✹S❣ = Θ

✒ 1

♣ ∆

Deshpande, Montanari Planted Cliques November 5, 2013 45 / 49

slide-104
SLIDE 104

What can we hope for?

If ❥S❥ = C

n ♣ ∆:

Poisson bound ✮ for any local algorithm: 1 nE❢ ❜ S✹S❣ ✕ eC ✵♣

∆✿

Deshpande, Montanari Planted Cliques November 5, 2013 46 / 49

slide-105
SLIDE 105

A result for local algorithms. . .

Theorem (Deshpande, Montanari, 2013)

Let Gn converge locally to ∆regular tree: If ❥S❥ ✕ (1 + ✧)

n ♣ e∆ there exists a local algorithm achieving

1 nE❢S✹ ❜ S❣ ✔ eΘ(

♣ ∆)✿

Conversely, if ❥S❥ ✔ (1 ✧)

n ♣ e∆ every local algorithm suffers

1 nE❢S✹ ❜ S❣ ✕ Θ

✒ 1

♣ ∆

With ∆ = n 1 we recover the complete graph result!

Deshpande, Montanari Planted Cliques November 5, 2013 47 / 49

slide-106
SLIDE 106

A result for local algorithms. . .

Theorem (Deshpande, Montanari, 2013)

Let Gn converge locally to ∆regular tree: If ❥S❥ ✕ (1 + ✧)

n ♣ e∆ there exists a local algorithm achieving

1 nE❢S✹ ❜ S❣ ✔ eΘ(

♣ ∆)✿

Conversely, if ❥S❥ ✔ (1 ✧)

n ♣ e∆ every local algorithm suffers

1 nE❢S✹ ❜ S❣ ✕ Θ

✒ 1

♣ ∆

With ∆ = n 1 we recover the complete graph result!

Deshpande, Montanari Planted Cliques November 5, 2013 47 / 49

slide-107
SLIDE 107

A result for local algorithms. . .

Theorem (Deshpande, Montanari, 2013)

Let Gn converge locally to ∆regular tree: If ❥S❥ ✕ (1 + ✧)

n ♣ e∆ there exists a local algorithm achieving

1 nE❢S✹ ❜ S❣ ✔ eΘ(

♣ ∆)✿

Conversely, if ❥S❥ ✔ (1 ✧)

n ♣ e∆ every local algorithm suffers

1 nE❢S✹ ❜ S❣ ✕ Θ

✒ 1

♣ ∆

With ∆ = n 1 we recover the complete graph result!

Deshpande, Montanari Planted Cliques November 5, 2013 47 / 49

slide-108
SLIDE 108

To conclude. . .

◮ Message-passing algorithm performs “weighted” counts of

non-reversing trees

◮ Such structures have been used elsewhere:

✍ Clustering sparse networks: [Krzakala et al. 2013] ✍ Compressed sensing: [Bayati, Lelarge, Montanari 2013]

Deshpande, Montanari Planted Cliques November 5, 2013 48 / 49

slide-109
SLIDE 109

To conclude. . .

◮ Message-passing algorithm performs “weighted” counts of

non-reversing trees

◮ Such structures have been used elsewhere:

✍ Clustering sparse networks: [Krzakala et al. 2013] ✍ Compressed sensing: [Bayati, Lelarge, Montanari 2013]

Deshpande, Montanari Planted Cliques November 5, 2013 48 / 49

slide-110
SLIDE 110

To conclude. . .

◮ Message-passing algorithm performs “weighted” counts of

non-reversing trees

◮ Such structures have been used elsewhere:

✍ Clustering sparse networks: [Krzakala et al. 2013] ✍ Compressed sensing: [Bayati, Lelarge, Montanari 2013]

Deshpande, Montanari Planted Cliques November 5, 2013 48 / 49

slide-111
SLIDE 111

To conclude. . .

◮ What is the “dense” analogue for local algorithms? ◮ What about other structural properties?

Thank you!

Deshpande, Montanari Planted Cliques November 5, 2013 49 / 49

slide-112
SLIDE 112

To conclude. . .

◮ What is the “dense” analogue for local algorithms? ◮ What about other structural properties?

Thank you!

Deshpande, Montanari Planted Cliques November 5, 2013 49 / 49

slide-113
SLIDE 113

To conclude. . .

◮ What is the “dense” analogue for local algorithms? ◮ What about other structural properties?

Thank you!

Deshpande, Montanari Planted Cliques November 5, 2013 49 / 49