Performance of Local Algorithms in Random Structures. Power and - - PowerPoint PPT Presentation

performance of local algorithms in random structures
SMART_READER_LITE
LIVE PREVIEW

Performance of Local Algorithms in Random Structures. Power and - - PowerPoint PPT Presentation

Performance of Local Algorithms in Random Structures. Power and limitations David Gamarnik MIT Sub-Linear Algorithms Bootcamp June, 2018 D.Gamarnik Local Algorithms in Random Structures June, 2018 1 / 29 Outline Part I. Challenges Part


slide-1
SLIDE 1

Performance of Local Algorithms in Random

  • Structures. Power and limitations

David Gamarnik MIT Sub-Linear Algorithms Bootcamp June, 2018

D.Gamarnik Local Algorithms in Random Structures June, 2018 1 / 29

slide-2
SLIDE 2

Outline

Part I. Challenges Part II. Solution Space Geometry Part III. Local Algorithms

D.Gamarnik Local Algorithms in Random Structures June, 2018 2 / 29

slide-3
SLIDE 3

Part I. Challenges

D.Gamarnik Local Algorithms in Random Structures June, 2018 3 / 29

slide-4
SLIDE 4

(Arguably) Most Embarrassing Algorithmic Problem in Random Graphs

D.Gamarnik Local Algorithms in Random Structures June, 2018 4 / 29

slide-5
SLIDE 5

(Arguably) Most Embarrassing Algorithmic Problem in Random Graphs Consider G(n, p).

D.Gamarnik Local Algorithms in Random Structures June, 2018 4 / 29

slide-6
SLIDE 6

(Arguably) Most Embarrassing Algorithmic Problem in Random Graphs Consider G(n, p). The largest clique (fully connected subgraph) is 2(1 + o(1)) log 1

p n. D.Gamarnik Local Algorithms in Random Structures June, 2018 4 / 29

slide-7
SLIDE 7

(Arguably) Most Embarrassing Algorithmic Problem in Random Graphs Consider G(n, p). The largest clique (fully connected subgraph) is 2(1 + o(1)) log 1

p n.

A trivial greedy algorithm finds a clique of size (1 + o(1)) log 1

p n. D.Gamarnik Local Algorithms in Random Structures June, 2018 4 / 29

slide-8
SLIDE 8

(Arguably) Most Embarrassing Algorithmic Problem in Random Graphs Consider G(n, p). The largest clique (fully connected subgraph) is 2(1 + o(1)) log 1

p n.

A trivial greedy algorithm finds a clique of size (1 + o(1)) log 1

p n.

Karp [1976] Find a better algorithm.

D.Gamarnik Local Algorithms in Random Structures June, 2018 4 / 29

slide-9
SLIDE 9

(Arguably) Most Embarrassing Algorithmic Problem in Random Graphs Consider G(n, p). The largest clique (fully connected subgraph) is 2(1 + o(1)) log 1

p n.

A trivial greedy algorithm finds a clique of size (1 + o(1)) log 1

p n.

Karp [1976] Find a better algorithm. Still open. This is embarrassing...

D.Gamarnik Local Algorithms in Random Structures June, 2018 4 / 29

slide-10
SLIDE 10

More Examples: Ind Sets in Sparse Random Graphs

D.Gamarnik Local Algorithms in Random Structures June, 2018 5 / 29

slide-11
SLIDE 11

More Examples: Ind Sets in Sparse Random Graphs Erd¨

  • s-R´

enyi G(n, d

n ) or random d-regular graph Gd(n).

D.Gamarnik Local Algorithms in Random Structures June, 2018 5 / 29

slide-12
SLIDE 12

More Examples: Ind Sets in Sparse Random Graphs Erd¨

  • s-R´

enyi G(n, d

n ) or random d-regular graph Gd(n).

The largest independent set (a largest subset of nodes with no edges in between) is 2(1 + od(1)) log d d n, Frieze, Luczak [1992]

D.Gamarnik Local Algorithms in Random Structures June, 2018 5 / 29

slide-13
SLIDE 13

More Examples: Ind Sets in Sparse Random Graphs Erd¨

  • s-R´

enyi G(n, d

n ) or random d-regular graph Gd(n).

The largest independent set (a largest subset of nodes with no edges in between) is 2(1 + od(1)) log d d n, Frieze, Luczak [1992] Greedy algorithm finds an independent set of size (1 + od(1)) log d d n.

D.Gamarnik Local Algorithms in Random Structures June, 2018 5 / 29

slide-14
SLIDE 14

More Examples: Ind Sets in Sparse Random Graphs Erd¨

  • s-R´

enyi G(n, d

n ) or random d-regular graph Gd(n).

The largest independent set (a largest subset of nodes with no edges in between) is 2(1 + od(1)) log d d n, Frieze, Luczak [1992] Greedy algorithm finds an independent set of size (1 + od(1)) log d d n. Nothing better is known.

D.Gamarnik Local Algorithms in Random Structures June, 2018 5 / 29

slide-15
SLIDE 15

More Examples: Coloring of Sparse Random Graphs

D.Gamarnik Local Algorithms in Random Structures June, 2018 6 / 29

slide-16
SLIDE 16

More Examples: Coloring of Sparse Random Graphs Erd¨

  • s-R´

enyi G(n, d

n ) or random d-regular graph Gd(n).

D.Gamarnik Local Algorithms in Random Structures June, 2018 6 / 29

slide-17
SLIDE 17

More Examples: Coloring of Sparse Random Graphs Erd¨

  • s-R´

enyi G(n, d

n ) or random d-regular graph Gd(n).

Chromatic number is χ(G) = (1+od(1))d

2 log d

.

D.Gamarnik Local Algorithms in Random Structures June, 2018 6 / 29

slide-18
SLIDE 18

More Examples: Coloring of Sparse Random Graphs Erd¨

  • s-R´

enyi G(n, d

n ) or random d-regular graph Gd(n).

Chromatic number is χ(G) = (1+od(1))d

2 log d

. Greedy can color with twice as many colors (1+od(1))d

log d

.

D.Gamarnik Local Algorithms in Random Structures June, 2018 6 / 29

slide-19
SLIDE 19

More Examples: Coloring of Sparse Random Graphs Erd¨

  • s-R´

enyi G(n, d

n ) or random d-regular graph Gd(n).

Chromatic number is χ(G) = (1+od(1))d

2 log d

. Greedy can color with twice as many colors (1+od(1))d

log d

. Nothing better is known.

D.Gamarnik Local Algorithms in Random Structures June, 2018 6 / 29

slide-20
SLIDE 20

More Examples: Random K-SAT

D.Gamarnik Local Algorithms in Random Structures June, 2018 7 / 29

slide-21
SLIDE 21

More Examples: Random K-SAT Generate an instance Φ(n, dn) of the K-SAT problem u.a.r. on n boolean 0, 1 variables and dn clauses. (x1,1 ∨ · · · ∨ ¯ x1,K) ∧ (¯ x2,1 ∨ · · · ¯ ∨x2,K) ∧ · · · ∧ (¯ xdn,1 ∨ · · · ∨ xdn,K)

D.Gamarnik Local Algorithms in Random Structures June, 2018 7 / 29

slide-22
SLIDE 22

More Examples: Random K-SAT Generate an instance Φ(n, dn) of the K-SAT problem u.a.r. on n boolean 0, 1 variables and dn clauses. (x1,1 ∨ · · · ∨ ¯ x1,K) ∧ (¯ x2,1 ∨ · · · ¯ ∨x2,K) ∧ · · · ∧ (¯ xdn,1 ∨ · · · ∨ xdn,K) Satisfiable iff d ≤ (1 + oK(1)2K log 2 ≡ d∗(K) Achlioptas & Moore [2002], Achlioptas & Peres [2003], Coja-Oghlan [2014], Ding, Sly & Sun [2015]

D.Gamarnik Local Algorithms in Random Structures June, 2018 7 / 29

slide-23
SLIDE 23

More Examples: Random K-SAT Generate an instance Φ(n, dn) of the K-SAT problem u.a.r. on n boolean 0, 1 variables and dn clauses. (x1,1 ∨ · · · ∨ ¯ x1,K) ∧ (¯ x2,1 ∨ · · · ¯ ∨x2,K) ∧ · · · ∧ (¯ xdn,1 ∨ · · · ∨ xdn,K) Satisfiable iff d ≤ (1 + oK(1)2K log 2 ≡ d∗(K) Achlioptas & Moore [2002], Achlioptas & Peres [2003], Coja-Oghlan [2014], Ding, Sly & Sun [2015] Algorithmically can find a satisfying assignment only when d ≤ log K K d∗(K). Coja-Oghlan [2011]

D.Gamarnik Local Algorithms in Random Structures June, 2018 7 / 29

slide-24
SLIDE 24

More Examples: Random K-SAT Generate an instance Φ(n, dn) of the K-SAT problem u.a.r. on n boolean 0, 1 variables and dn clauses. (x1,1 ∨ · · · ∨ ¯ x1,K) ∧ (¯ x2,1 ∨ · · · ¯ ∨x2,K) ∧ · · · ∧ (¯ xdn,1 ∨ · · · ∨ xdn,K) Satisfiable iff d ≤ (1 + oK(1)2K log 2 ≡ d∗(K) Achlioptas & Moore [2002], Achlioptas & Peres [2003], Coja-Oghlan [2014], Ding, Sly & Sun [2015] Algorithmically can find a satisfying assignment only when d ≤ log K K d∗(K). Coja-Oghlan [2011] Nothing better is known.

D.Gamarnik Local Algorithms in Random Structures June, 2018 7 / 29

slide-25
SLIDE 25

Examples in Statistics. Largest submatrix problem

D.Gamarnik Local Algorithms in Random Structures June, 2018 8 / 29

slide-26
SLIDE 26

Examples in Statistics. Largest submatrix problem Cn =    C11 . . . C1n . . . ... . . . Cn1 . . . Cnn    , Ci,j

d

= i.i.d. N(0, 1)

D.Gamarnik Local Algorithms in Random Structures June, 2018 8 / 29

slide-27
SLIDE 27

Examples in Statistics. Largest submatrix problem Cn =    C11 . . . C1n . . . ... . . . Cn1 . . . Cnn    , Ci,j

d

= i.i.d. N(0, 1) Given k, find max

I,J⊂[n],|I|=|J|=k Ave(CI,J).

D.Gamarnik Local Algorithms in Random Structures June, 2018 8 / 29

slide-28
SLIDE 28

Examples in Statistics. Largest submatrix problem Cn =    C11 . . . C1n . . . ... . . . Cn1 . . . Cnn    , Ci,j

d

= i.i.d. N(0, 1) Given k, find max

I,J⊂[n],|I|=|J|=k Ave(CI,J).

Bhamidi, Dey & Nobel [2017]. Largest value is max Ave(CI,J) = 2(1 + o(1))

  • log n

k .

D.Gamarnik Local Algorithms in Random Structures June, 2018 8 / 29

slide-29
SLIDE 29

Examples in Statistics. Largest submatrix problem Cn =    C11 . . . C1n . . . ... . . . Cn1 . . . Cnn    , Ci,j

d

= i.i.d. N(0, 1) Given k, find max

I,J⊂[n],|I|=|J|=k Ave(CI,J).

Bhamidi, Dey & Nobel [2017]. Largest value is max Ave(CI,J) = 2(1 + o(1))

  • log n

k . Algorithmically (G & Li [2017]) AveALG(CI,J) = 4(1 + o(1)) 3 √ 2

  • log n

k .

D.Gamarnik Local Algorithms in Random Structures June, 2018 8 / 29

slide-30
SLIDE 30

Examples in Statistics. Sparse Regression

D.Gamarnik Local Algorithms in Random Structures June, 2018 9 / 29

slide-31
SLIDE 31

Examples in Statistics. Sparse Regression X ∈ Rn×p, W ∈ Rn i.i.d. N(0, σ2).

D.Gamarnik Local Algorithms in Random Structures June, 2018 9 / 29

slide-32
SLIDE 32

Examples in Statistics. Sparse Regression X ∈ Rn×p, W ∈ Rn i.i.d. N(0, σ2).    Y1 . . . Yn    =    X11 X12 . . . X1p . . . . . . ... . . . Xn1 Xn2 . . . Xnp       β∗

1

. . . β∗

p

   +    W1 . . . Wn    .

D.Gamarnik Local Algorithms in Random Structures June, 2018 9 / 29

slide-33
SLIDE 33

Examples in Statistics. Sparse Regression X ∈ Rn×p, W ∈ Rn i.i.d. N(0, σ2).    Y1 . . . Yn    =    X11 X12 . . . X1p . . . . . . ... . . . Xn1 Xn2 . . . Xnp       β∗

1

. . . β∗

p

   +    W1 . . . Wn    . Goal: Recover β∗ from observed X and Y when β∗0 ≤ k by solving min

β0≤k Y − Xβ2.

D.Gamarnik Local Algorithms in Random Structures June, 2018 9 / 29

slide-34
SLIDE 34

Examples in Statistics. Sparse Regression X ∈ Rn×p, W ∈ Rn i.i.d. N(0, σ2).    Y1 . . . Yn    =    X11 X12 . . . X1p . . . . . . ... . . . Xn1 Xn2 . . . Xnp       β∗

1

. . . β∗

p

   +    W1 . . . Wn    . Goal: Recover β∗ from observed X and Y when β∗0 ≤ k by solving min

β0≤k Y − Xβ2.

Brute force works iff n ≥ n∗

Info Ω

  • k log p

log k

  • . Wang, Wainwright &

Ramchandran [2010], Rad [2011], G & Zadik [2017]

D.Gamarnik Local Algorithms in Random Structures June, 2018 9 / 29

slide-35
SLIDE 35

Examples in Statistics. Sparse Regression X ∈ Rn×p, W ∈ Rn i.i.d. N(0, σ2).    Y1 . . . Yn    =    X11 X12 . . . X1p . . . . . . ... . . . Xn1 Xn2 . . . Xnp       β∗

1

. . . β∗

p

   +    W1 . . . Wn    . Goal: Recover β∗ from observed X and Y when β∗0 ≤ k by solving min

β0≤k Y − Xβ2.

Brute force works iff n ≥ n∗

Info Ω

  • k log p

log k

  • . Wang, Wainwright &

Ramchandran [2010], Rad [2011], G & Zadik [2017] Efficiently (convex optimization) only when n ≥ n∗

Convex = Ω (k log p).

Donoho, Tanner, Wainwright, Hastie, Tibshirani, Candes, Tao [1996–]

D.Gamarnik Local Algorithms in Random Structures June, 2018 9 / 29

slide-36
SLIDE 36

Geometry

Part II. Geometry of the solution space

D.Gamarnik Local Algorithms in Random Structures June, 2018 10 / 29

slide-37
SLIDE 37

Overlap Gap Property

D.Gamarnik Local Algorithms in Random Structures June, 2018 11 / 29

slide-38
SLIDE 38

Overlap Gap Property Optimization with random input X min

θ∈Θ L(θ, X).

D.Gamarnik Local Algorithms in Random Structures June, 2018 11 / 29

slide-39
SLIDE 39

Overlap Gap Property Optimization with random input X min

θ∈Θ L(θ, X).

OGP holds if there exists R > 0, such that the set {θ : L(θ, X) ≤ min

θ∈Θ L(θ, X) + R}

is disconnected.

D.Gamarnik Local Algorithms in Random Structures June, 2018 11 / 29

slide-40
SLIDE 40

Overlap Gap Property Optimization with random input X min

θ∈Θ L(θ, X).

OGP holds if there exists R > 0, such that the set {θ : L(θ, X) ≤ min

θ∈Θ L(θ, X) + R}

is disconnected. The set of R-optimal solutions is partitioned into separated disconnected components.

D.Gamarnik Local Algorithms in Random Structures June, 2018 11 / 29

slide-41
SLIDE 41

Overlap Gap Property

min

θ

L(θ, X)

θ

min L(θ, X)

D.Gamarnik Local Algorithms in Random Structures June, 2018 12 / 29

slide-42
SLIDE 42

Overlap Gap Property

min

θ

L(θ, X) R

θ

min L(θ, X)

D.Gamarnik Local Algorithms in Random Structures June, 2018 13 / 29

slide-43
SLIDE 43

Overlap Gap Property

min

θ

L(θ, X) R

θ

min L(θ, X)

D.Gamarnik Local Algorithms in Random Structures June, 2018 14 / 29

slide-44
SLIDE 44

OGP for Independent Sets in Random Graphs

D.Gamarnik Local Algorithms in Random Structures June, 2018 15 / 29

slide-45
SLIDE 45

OGP for Independent Sets in Random Graphs For either G(n, p) or G(n, d

n ), let I(β), β ∈ (0, 1) be the set of ind sets

with multiplicative approximation ratio β. Then the set of overlaps |I ∩ J|, I, J ∈ I(β) is

D.Gamarnik Local Algorithms in Random Structures June, 2018 15 / 29

slide-46
SLIDE 46

OGP for Independent Sets in Random Graphs For either G(n, p) or G(n, d

n ), let I(β), β ∈ (0, 1) be the set of ind sets

with multiplicative approximation ratio β. Then the set of overlaps |I ∩ J|, I, J ∈ I(β) is disconnected, for all pairs when β > 1

2 + 1 2 √ 2, and for ”most” pairs

when β > 1

2, Coja–Oghlan & Efthymiou [2013], G & Sudan [2014]

D.Gamarnik Local Algorithms in Random Structures June, 2018 15 / 29

slide-47
SLIDE 47

OGP for Independent Sets in Random Graphs For either G(n, p) or G(n, d

n ), let I(β), β ∈ (0, 1) be the set of ind sets

with multiplicative approximation ratio β. Then the set of overlaps |I ∩ J|, I, J ∈ I(β) is disconnected, for all pairs when β > 1

2 + 1 2 √ 2, and for ”most” pairs

when β > 1

2, Coja–Oghlan & Efthymiou [2013], G & Sudan [2014]

connected for β < 1

2, Coja–Oghlan & Efthymiou [2013].

D.Gamarnik Local Algorithms in Random Structures June, 2018 15 / 29

slide-48
SLIDE 48

OGP for Independent Sets in Random Graphs For either G(n, p) or G(n, d

n ), let I(β), β ∈ (0, 1) be the set of ind sets

with multiplicative approximation ratio β. Then the set of overlaps |I ∩ J|, I, J ∈ I(β) is disconnected, for all pairs when β > 1

2 + 1 2 √ 2, and for ”most” pairs

when β > 1

2, Coja–Oghlan & Efthymiou [2013], G & Sudan [2014]

connected for β < 1

2, Coja–Oghlan & Efthymiou [2013].

Extension to multi-overlaps |I1 ∩ · · · ∩ Im| by Rahman & Virag [2015].

D.Gamarnik Local Algorithms in Random Structures June, 2018 15 / 29

slide-49
SLIDE 49

OGP for Random K-SAT Problem Let SAT n be the set of satisfying assignments of a random formula Φ(n, dn).

D.Gamarnik Local Algorithms in Random Structures June, 2018 16 / 29

slide-50
SLIDE 50

OGP for Random K-SAT Problem Let SAT n be the set of satisfying assignments of a random formula Φ(n, dn). Then the set of multi-overlaps for a fixed m (d(σℓ, σr), σℓ ∈ SAT n, 1 ≤ ℓ, r ≤ m) .

D.Gamarnik Local Algorithms in Random Structures June, 2018 16 / 29

slide-51
SLIDE 51

OGP for Random K-SAT Problem Let SAT n be the set of satisfying assignments of a random formula Φ(n, dn). Then the set of multi-overlaps for a fixed m (d(σℓ, σr), σℓ ∈ SAT n, 1 ≤ ℓ, r ≤ m) . is connected when d < 2K log 2

K

= d∗

K for m = 2, general case open.

D.Gamarnik Local Algorithms in Random Structures June, 2018 16 / 29

slide-52
SLIDE 52

OGP for Random K-SAT Problem Let SAT n be the set of satisfying assignments of a random formula Φ(n, dn). Then the set of multi-overlaps for a fixed m (d(σℓ, σr), σℓ ∈ SAT n, 1 ≤ ℓ, r ≤ m) . is connected when d < 2K log 2

K

= d∗

K for m = 2, general case open.

disconnected when d > 2K log2 K log 2

K

= log2 K

K

d∗, Achlioptas, Coja-Oghlan & Ricci-Tersenghi [2011], G & Sudan [2014] (for NAE-K-SAT).

D.Gamarnik Local Algorithms in Random Structures June, 2018 16 / 29

slide-53
SLIDE 53

OGP in Sparse Regression

D.Gamarnik Local Algorithms in Random Structures June, 2018 17 / 29

slide-54
SLIDE 54

OGP in Sparse Regression Assume binary case β∗

i = 0, 1.

D.Gamarnik Local Algorithms in Random Structures June, 2018 17 / 29

slide-55
SLIDE 55

OGP in Sparse Regression Assume binary case β∗

i = 0, 1.

Consider the optimization problem parametrized by ζ ∈ (0, 1) Γ∗(ζ) min

β

Y − Xβ2 Subject to: β0 = k, β∗ − β0 = 2kζ. Then Γ(ζ) is

D.Gamarnik Local Algorithms in Random Structures June, 2018 17 / 29

slide-56
SLIDE 56

OGP in Sparse Regression Assume binary case β∗

i = 0, 1.

Consider the optimization problem parametrized by ζ ∈ (0, 1) Γ∗(ζ) min

β

Y − Xβ2 Subject to: β0 = k, β∗ − β0 = 2kζ. Then Γ(ζ) is increasing when n > Cn∗

Convex,

D.Gamarnik Local Algorithms in Random Structures June, 2018 17 / 29

slide-57
SLIDE 57

OGP in Sparse Regression Assume binary case β∗

i = 0, 1.

Consider the optimization problem parametrized by ζ ∈ (0, 1) Γ∗(ζ) min

β

Y − Xβ2 Subject to: β0 = k, β∗ − β0 = 2kζ. Then Γ(ζ) is increasing when n > Cn∗

Convex,

non-monotone when n < cn∗

Convex, G & Zadik [2017].

D.Gamarnik Local Algorithms in Random Structures June, 2018 17 / 29

slide-58
SLIDE 58

OGP in Sparse Regression

D.Gamarnik Local Algorithms in Random Structures June, 2018 18 / 29

slide-59
SLIDE 59

OGP in Sparse Regression When n > Cn∗

Convex

D.Gamarnik Local Algorithms in Random Structures June, 2018 18 / 29

slide-60
SLIDE 60

OGP in Sparse Regression

D.Gamarnik Local Algorithms in Random Structures June, 2018 19 / 29

slide-61
SLIDE 61

OGP in Sparse Regression when n < cn∗

Convex – OGP

.

ζ Γ(ζ)

0 1

0 1

τ1 τ2

R

D.Gamarnik Local Algorithms in Random Structures June, 2018 19 / 29

slide-62
SLIDE 62

Part III. Local algorithms

D.Gamarnik Local Algorithms in Random Structures June, 2018 20 / 29

slide-63
SLIDE 63

Three Types of Local Algorithms

D.Gamarnik Local Algorithms in Random Structures June, 2018 21 / 29

slide-64
SLIDE 64

Three Types of Local Algorithms

  • Greedy. Example for ind sets: select a node into I. Delete
  • neighbors. Repeat.

D.Gamarnik Local Algorithms in Random Structures June, 2018 21 / 29

slide-65
SLIDE 65

Three Types of Local Algorithms

  • Greedy. Example for ind sets: select a node into I. Delete
  • neighbors. Repeat.

D.Gamarnik Local Algorithms in Random Structures June, 2018 21 / 29

slide-66
SLIDE 66

Three Types of Local Algorithms

  • Greedy. Example for ind sets: select a node into I. Delete
  • neighbors. Repeat.

Local-in-Parallel – Factors of I.I.D (FIID). Make randomized decision based on the radius r neighborhood B(u, r) for each node u.

D.Gamarnik Local Algorithms in Random Structures June, 2018 21 / 29

slide-67
SLIDE 67

Three Types of Local Algorithms

  • Greedy. Example for ind sets: select a node into I. Delete
  • neighbors. Repeat.

Local-in-Parallel – Factors of I.I.D (FIID). Make randomized decision based on the radius r neighborhood B(u, r) for each node u. Local-Sequential. Make decision on ui based on ”implied” decisions for u1, . . . , ui−1.

D.Gamarnik Local Algorithms in Random Structures June, 2018 21 / 29

slide-68
SLIDE 68

Three Types of Local Algorithms

  • Greedy. Example for ind sets: select a node into I. Delete
  • neighbors. Repeat.

Local-in-Parallel – Factors of I.I.D (FIID). Make randomized decision based on the radius r neighborhood B(u, r) for each node u. Local-Sequential. Make decision on ui based on ”implied” decisions for u1, . . . , ui−1.

Markov Chain Monte Carlo (MCMC). Example for ind sets. Fix β – inverse temperature.

D.Gamarnik Local Algorithms in Random Structures June, 2018 21 / 29

slide-69
SLIDE 69

Three Types of Local Algorithms

  • Greedy. Example for ind sets: select a node into I. Delete
  • neighbors. Repeat.

Local-in-Parallel – Factors of I.I.D (FIID). Make randomized decision based on the radius r neighborhood B(u, r) for each node u. Local-Sequential. Make decision on ui based on ”implied” decisions for u1, . . . , ui−1.

Markov Chain Monte Carlo (MCMC). Example for ind sets. Fix β – inverse temperature. Pick u u.a.r.

D.Gamarnik Local Algorithms in Random Structures June, 2018 21 / 29

slide-70
SLIDE 70

Three Types of Local Algorithms

  • Greedy. Example for ind sets: select a node into I. Delete
  • neighbors. Repeat.

Local-in-Parallel – Factors of I.I.D (FIID). Make randomized decision based on the radius r neighborhood B(u, r) for each node u. Local-Sequential. Make decision on ui based on ”implied” decisions for u1, . . . , ui−1.

Markov Chain Monte Carlo (MCMC). Example for ind sets. Fix β – inverse temperature. Pick u u.a.r.

If u ∈ I delete with probability

1 1+eβ .

D.Gamarnik Local Algorithms in Random Structures June, 2018 21 / 29

slide-71
SLIDE 71

Three Types of Local Algorithms

  • Greedy. Example for ind sets: select a node into I. Delete
  • neighbors. Repeat.

Local-in-Parallel – Factors of I.I.D (FIID). Make randomized decision based on the radius r neighborhood B(u, r) for each node u. Local-Sequential. Make decision on ui based on ”implied” decisions for u1, . . . , ui−1.

Markov Chain Monte Carlo (MCMC). Example for ind sets. Fix β – inverse temperature. Pick u u.a.r.

If u ∈ I delete with probability

1 1+eβ .

If u / ∈ I and can be added without conflict, add it with prob

eβ 1+eβ .

D.Gamarnik Local Algorithms in Random Structures June, 2018 21 / 29

slide-72
SLIDE 72

Three Types of Local Algorithms

  • Greedy. Example for ind sets: select a node into I. Delete
  • neighbors. Repeat.

Local-in-Parallel – Factors of I.I.D (FIID). Make randomized decision based on the radius r neighborhood B(u, r) for each node u. Local-Sequential. Make decision on ui based on ”implied” decisions for u1, . . . , ui−1.

Markov Chain Monte Carlo (MCMC). Example for ind sets. Fix β – inverse temperature. Pick u u.a.r.

If u ∈ I delete with probability

1 1+eβ .

If u / ∈ I and can be added without conflict, add it with prob

eβ 1+eβ .

Gradient Descent (GD): keep improving while you can (MCMC with β = ∞)

D.Gamarnik Local Algorithms in Random Structures June, 2018 21 / 29

slide-73
SLIDE 73

Properties

D.Gamarnik Local Algorithms in Random Structures June, 2018 22 / 29

slide-74
SLIDE 74

Properties Greedy and Local-in-Parallel are special cases of Local-Sequential (Obvious).

D.Gamarnik Local Algorithms in Random Structures June, 2018 22 / 29

slide-75
SLIDE 75

Properties Greedy and Local-in-Parallel are special cases of Local-Sequential (Obvious). Greedy for ind set can be well approximated by Local-in-Parallel, G & Goldberg [2010]

D.Gamarnik Local Algorithms in Random Structures June, 2018 22 / 29

slide-76
SLIDE 76

Properties Greedy and Local-in-Parallel are special cases of Local-Sequential (Obvious). Greedy for ind set can be well approximated by Local-in-Parallel, G & Goldberg [2010]

Proof sketch: assign weights R(u1), . . . , R(un) i.i.d. u.a.r. from [0, 1].

D.Gamarnik Local Algorithms in Random Structures June, 2018 22 / 29

slide-77
SLIDE 77

Properties Greedy and Local-in-Parallel are special cases of Local-Sequential (Obvious). Greedy for ind set can be well approximated by Local-in-Parallel, G & Goldberg [2010]

Proof sketch: assign weights R(u1), . . . , R(un) i.i.d. u.a.r. from [0, 1]. For each node u for the decision not to be r-local there should exist a path u, u1, . . . , ur such that R(u1) < R(u2) < · · · < R(ur).

D.Gamarnik Local Algorithms in Random Structures June, 2018 22 / 29

slide-78
SLIDE 78

Properties Greedy and Local-in-Parallel are special cases of Local-Sequential (Obvious). Greedy for ind set can be well approximated by Local-in-Parallel, G & Goldberg [2010]

Proof sketch: assign weights R(u1), . . . , R(un) i.i.d. u.a.r. from [0, 1]. For each node u for the decision not to be r-local there should exist a path u, u1, . . . , ur such that R(u1) < R(u2) < · · · < R(ur). The probability of this is O 1

r!

  • – decays faster than the exponential

growth rate dr of the neighborhood of r.

D.Gamarnik Local Algorithms in Random Structures June, 2018 22 / 29

slide-79
SLIDE 79

Properties Greedy and Local-in-Parallel are special cases of Local-Sequential (Obvious). Greedy for ind set can be well approximated by Local-in-Parallel, G & Goldberg [2010]

Proof sketch: assign weights R(u1), . . . , R(un) i.i.d. u.a.r. from [0, 1]. For each node u for the decision not to be r-local there should exist a path u, u1, . . . , ur such that R(u1) < R(u2) < · · · < R(ur). The probability of this is O 1

r!

  • – decays faster than the exponential

growth rate dr of the neighborhood of r. As a result, for most nodes the decision is r-localized.

D.Gamarnik Local Algorithms in Random Structures June, 2018 22 / 29

slide-80
SLIDE 80

Properties

D.Gamarnik Local Algorithms in Random Structures June, 2018 23 / 29

slide-81
SLIDE 81

Properties MCMC has a simple to describe (Gibbs) steady state: for every ind set P(I) = eβ|I|

  • J eβ|I| .

D.Gamarnik Local Algorithms in Random Structures June, 2018 23 / 29

slide-82
SLIDE 82

Properties MCMC has a simple to describe (Gibbs) steady state: for every ind set P(I) = eβ|I|

  • J eβ|I| .

MCMC always produces a near-optimal solution for large enough β, (but possibly after exponential time), for the following reason: |I∗| ≤ 1 β log

  • J

eβ|I|

  • ≤ |I∗| + nlog 2

β .

D.Gamarnik Local Algorithms in Random Structures June, 2018 23 / 29

slide-83
SLIDE 83

Properties MCMC has a simple to describe (Gibbs) steady state: for every ind set P(I) = eβ|I|

  • J eβ|I| .

MCMC always produces a near-optimal solution for large enough β, (but possibly after exponential time), for the following reason: |I∗| ≤ 1 β log

  • J

eβ|I|

  • ≤ |I∗| + nlog 2

β . As a result, most of the Gibbs ”mass” is concentrated on ind sets I with |I| ≈ |I∗|.

D.Gamarnik Local Algorithms in Random Structures June, 2018 23 / 29

slide-84
SLIDE 84

Summary Punchline: For all of the discussed models, below the OGP one of the versions of the local algorithms finds a solution. Above the threshold no algorithms are known and possibly no poly-time algorithms exist.

D.Gamarnik Local Algorithms in Random Structures June, 2018 24 / 29

slide-85
SLIDE 85

Examples: Ind set in sparse random graphs In either G(n, d

n ) or Gd(n), local parallel (FIID) algorithms can construct

ind sets with size (1 + od(1)) log d d n Lauer & Wormald [2007], G & Goldberg [2010].

D.Gamarnik Local Algorithms in Random Structures June, 2018 25 / 29

slide-86
SLIDE 86

Examples: Ind set in sparse random graphs In either G(n, d

n ) or Gd(n), local parallel (FIID) algorithms can construct

ind sets with size (1 + od(1)) log d d n Lauer & Wormald [2007], G & Goldberg [2010]. and no Local Parallel algorithms exist which construct ind set with larger size w.h.p., G & Sudan, Rahman & Virag [2015]. The OGP is used as an obstruction to such algorithms.

D.Gamarnik Local Algorithms in Random Structures June, 2018 25 / 29

slide-87
SLIDE 87

Examples: Ind set in sparse random graphs In either G(n, d

n ) or Gd(n), local parallel (FIID) algorithms can construct

ind sets with size (1 + od(1)) log d d n Lauer & Wormald [2007], G & Goldberg [2010]. and no Local Parallel algorithms exist which construct ind set with larger size w.h.p., G & Sudan, Rahman & Virag [2015]. The OGP is used as an obstruction to such algorithms. Also MCMC fail finding ind sets of the same size Coja–Oghlan & Efthymiou [2013] (MCMC mixes in exponential time).

D.Gamarnik Local Algorithms in Random Structures June, 2018 25 / 29

slide-88
SLIDE 88

Examples: Sparse Regression For sparse regression problem Y = Xβ∗ + W, Gradient Descent reconstructs β∗ when n > CnConvex = Ω (k log p) ,

D.Gamarnik Local Algorithms in Random Structures June, 2018 26 / 29

slide-89
SLIDE 89

Examples: Sparse Regression For sparse regression problem Y = Xβ∗ + W, Gradient Descent reconstructs β∗ when n > CnConvex = Ω (k log p) , and Gradient Descent provably fails for certain starting points when n < cnConvex = Ω (k log p) .

D.Gamarnik Local Algorithms in Random Structures June, 2018 26 / 29

slide-90
SLIDE 90

Examples: Sparse Regression For sparse regression problem Y = Xβ∗ + W, Gradient Descent reconstructs β∗ when n > CnConvex = Ω (k log p) , and Gradient Descent provably fails for certain starting points when n < cnConvex = Ω (k log p) . OGP serves as an obstruction to the GD algorithm, G & Zadik [2017]

D.Gamarnik Local Algorithms in Random Structures June, 2018 26 / 29

slide-91
SLIDE 91

Conclusion In conclusion, we have a fairly complete theory: optimiza- tion in random graphs is solvable by local algorithms be- low the OGP threshold. Above the threshold local algo- rithms provably fails, but no other poly-time algorithms are known, and possibly none exist.

D.Gamarnik Local Algorithms in Random Structures June, 2018 27 / 29

slide-92
SLIDE 92

Future Work

D.Gamarnik Local Algorithms in Random Structures June, 2018 28 / 29

slide-93
SLIDE 93

Future Work Planted clique problem G(n, 1

2, m) exhibits non-monotonicity

implying OGP iff the size of the planted clique m = o(√n), G & Zadik [201?]

D.Gamarnik Local Algorithms in Random Structures June, 2018 28 / 29

slide-94
SLIDE 94

Future Work Planted clique problem G(n, 1

2, m) exhibits non-monotonicity

implying OGP iff the size of the planted clique m = o(√n), G & Zadik [201?] Sparse Planted XOR-K-SAT formula Φ(n, dn) exhibits non-monotonicity implying OGP .

D.Gamarnik Local Algorithms in Random Structures June, 2018 28 / 29

slide-95
SLIDE 95

Thank you

D.Gamarnik Local Algorithms in Random Structures June, 2018 29 / 29