Performance of Local Algorithms in Random Structures. Power and - - PowerPoint PPT Presentation

Performance of Local Algorithms in Random

• Structures. Power and limitations

David Gamarnik MIT Sub-Linear Algorithms Bootcamp June, 2018

D.Gamarnik Local Algorithms in Random Structures June, 2018 1 / 29

Outline

Part I. Challenges Part II. Solution Space Geometry Part III. Local Algorithms

D.Gamarnik Local Algorithms in Random Structures June, 2018 2 / 29

Part I. Challenges

D.Gamarnik Local Algorithms in Random Structures June, 2018 3 / 29

(Arguably) Most Embarrassing Algorithmic Problem in Random Graphs

D.Gamarnik Local Algorithms in Random Structures June, 2018 4 / 29

(Arguably) Most Embarrassing Algorithmic Problem in Random Graphs Consider G(n, p).

D.Gamarnik Local Algorithms in Random Structures June, 2018 4 / 29

(Arguably) Most Embarrassing Algorithmic Problem in Random Graphs Consider G(n, p). The largest clique (fully connected subgraph) is 2(1 + o(1)) log 1

p n. D.Gamarnik Local Algorithms in Random Structures June, 2018 4 / 29

(Arguably) Most Embarrassing Algorithmic Problem in Random Graphs Consider G(n, p). The largest clique (fully connected subgraph) is 2(1 + o(1)) log 1

p n.

A trivial greedy algorithm finds a clique of size (1 + o(1)) log 1

p n. D.Gamarnik Local Algorithms in Random Structures June, 2018 4 / 29

(Arguably) Most Embarrassing Algorithmic Problem in Random Graphs Consider G(n, p). The largest clique (fully connected subgraph) is 2(1 + o(1)) log 1

p n.

A trivial greedy algorithm finds a clique of size (1 + o(1)) log 1

p n.

Karp [1976] Find a better algorithm.

D.Gamarnik Local Algorithms in Random Structures June, 2018 4 / 29

(Arguably) Most Embarrassing Algorithmic Problem in Random Graphs Consider G(n, p). The largest clique (fully connected subgraph) is 2(1 + o(1)) log 1

p n.

A trivial greedy algorithm finds a clique of size (1 + o(1)) log 1

p n.

Karp [1976] Find a better algorithm. Still open. This is embarrassing...

D.Gamarnik Local Algorithms in Random Structures June, 2018 4 / 29

More Examples: Ind Sets in Sparse Random Graphs

D.Gamarnik Local Algorithms in Random Structures June, 2018 5 / 29

More Examples: Ind Sets in Sparse Random Graphs Erd¨

• s-R´

enyi G(n, d

n ) or random d-regular graph Gd(n).

D.Gamarnik Local Algorithms in Random Structures June, 2018 5 / 29

More Examples: Ind Sets in Sparse Random Graphs Erd¨

• s-R´

enyi G(n, d

n ) or random d-regular graph Gd(n).

The largest independent set (a largest subset of nodes with no edges in between) is 2(1 + od(1)) log d d n, Frieze, Luczak [1992]

D.Gamarnik Local Algorithms in Random Structures June, 2018 5 / 29

More Examples: Ind Sets in Sparse Random Graphs Erd¨

• s-R´

enyi G(n, d

n ) or random d-regular graph Gd(n).

The largest independent set (a largest subset of nodes with no edges in between) is 2(1 + od(1)) log d d n, Frieze, Luczak [1992] Greedy algorithm finds an independent set of size (1 + od(1)) log d d n.

D.Gamarnik Local Algorithms in Random Structures June, 2018 5 / 29

More Examples: Ind Sets in Sparse Random Graphs Erd¨

• s-R´

enyi G(n, d

n ) or random d-regular graph Gd(n).

The largest independent set (a largest subset of nodes with no edges in between) is 2(1 + od(1)) log d d n, Frieze, Luczak [1992] Greedy algorithm finds an independent set of size (1 + od(1)) log d d n. Nothing better is known.

D.Gamarnik Local Algorithms in Random Structures June, 2018 5 / 29

More Examples: Coloring of Sparse Random Graphs

D.Gamarnik Local Algorithms in Random Structures June, 2018 6 / 29

More Examples: Coloring of Sparse Random Graphs Erd¨

• s-R´

enyi G(n, d

n ) or random d-regular graph Gd(n).

D.Gamarnik Local Algorithms in Random Structures June, 2018 6 / 29

More Examples: Coloring of Sparse Random Graphs Erd¨

• s-R´

enyi G(n, d

n ) or random d-regular graph Gd(n).

Chromatic number is χ(G) = (1+od(1))d

2 log d

.

D.Gamarnik Local Algorithms in Random Structures June, 2018 6 / 29

More Examples: Coloring of Sparse Random Graphs Erd¨

• s-R´

enyi G(n, d

n ) or random d-regular graph Gd(n).

Chromatic number is χ(G) = (1+od(1))d

2 log d

. Greedy can color with twice as many colors (1+od(1))d

log d

.

D.Gamarnik Local Algorithms in Random Structures June, 2018 6 / 29

More Examples: Coloring of Sparse Random Graphs Erd¨

• s-R´

enyi G(n, d

n ) or random d-regular graph Gd(n).

Chromatic number is χ(G) = (1+od(1))d

2 log d

. Greedy can color with twice as many colors (1+od(1))d

log d

. Nothing better is known.

D.Gamarnik Local Algorithms in Random Structures June, 2018 6 / 29

More Examples: Random K-SAT

D.Gamarnik Local Algorithms in Random Structures June, 2018 7 / 29

More Examples: Random K-SAT Generate an instance Φ(n, dn) of the K-SAT problem u.a.r. on n boolean 0, 1 variables and dn clauses. (x1,1 ∨ · · · ∨ ¯ x1,K) ∧ (¯ x2,1 ∨ · · · ¯ ∨x2,K) ∧ · · · ∧ (¯ xdn,1 ∨ · · · ∨ xdn,K)

D.Gamarnik Local Algorithms in Random Structures June, 2018 7 / 29

More Examples: Random K-SAT Generate an instance Φ(n, dn) of the K-SAT problem u.a.r. on n boolean 0, 1 variables and dn clauses. (x1,1 ∨ · · · ∨ ¯ x1,K) ∧ (¯ x2,1 ∨ · · · ¯ ∨x2,K) ∧ · · · ∧ (¯ xdn,1 ∨ · · · ∨ xdn,K) Satisfiable iff d ≤ (1 + oK(1)2K log 2 ≡ d∗(K) Achlioptas & Moore [2002], Achlioptas & Peres [2003], Coja-Oghlan [2014], Ding, Sly & Sun [2015]

D.Gamarnik Local Algorithms in Random Structures June, 2018 7 / 29

More Examples: Random K-SAT Generate an instance Φ(n, dn) of the K-SAT problem u.a.r. on n boolean 0, 1 variables and dn clauses. (x1,1 ∨ · · · ∨ ¯ x1,K) ∧ (¯ x2,1 ∨ · · · ¯ ∨x2,K) ∧ · · · ∧ (¯ xdn,1 ∨ · · · ∨ xdn,K) Satisfiable iff d ≤ (1 + oK(1)2K log 2 ≡ d∗(K) Achlioptas & Moore [2002], Achlioptas & Peres [2003], Coja-Oghlan [2014], Ding, Sly & Sun [2015] Algorithmically can find a satisfying assignment only when d ≤ log K K d∗(K). Coja-Oghlan [2011]

D.Gamarnik Local Algorithms in Random Structures June, 2018 7 / 29

More Examples: Random K-SAT Generate an instance Φ(n, dn) of the K-SAT problem u.a.r. on n boolean 0, 1 variables and dn clauses. (x1,1 ∨ · · · ∨ ¯ x1,K) ∧ (¯ x2,1 ∨ · · · ¯ ∨x2,K) ∧ · · · ∧ (¯ xdn,1 ∨ · · · ∨ xdn,K) Satisfiable iff d ≤ (1 + oK(1)2K log 2 ≡ d∗(K) Achlioptas & Moore [2002], Achlioptas & Peres [2003], Coja-Oghlan [2014], Ding, Sly & Sun [2015] Algorithmically can find a satisfying assignment only when d ≤ log K K d∗(K). Coja-Oghlan [2011] Nothing better is known.

D.Gamarnik Local Algorithms in Random Structures June, 2018 7 / 29

Examples in Statistics. Largest submatrix problem

D.Gamarnik Local Algorithms in Random Structures June, 2018 8 / 29

Examples in Statistics. Largest submatrix problem Cn =    C11 . . . C1n . . . ... . . . Cn1 . . . Cnn    , Ci,j

d

= i.i.d. N(0, 1)

D.Gamarnik Local Algorithms in Random Structures June, 2018 8 / 29

Examples in Statistics. Largest submatrix problem Cn =    C11 . . . C1n . . . ... . . . Cn1 . . . Cnn    , Ci,j

d

= i.i.d. N(0, 1) Given k, find max

I,J⊂[n],|I|=|J|=k Ave(CI,J).

D.Gamarnik Local Algorithms in Random Structures June, 2018 8 / 29

Examples in Statistics. Largest submatrix problem Cn =    C11 . . . C1n . . . ... . . . Cn1 . . . Cnn    , Ci,j

d

= i.i.d. N(0, 1) Given k, find max

I,J⊂[n],|I|=|J|=k Ave(CI,J).

Bhamidi, Dey & Nobel [2017]. Largest value is max Ave(CI,J) = 2(1 + o(1))

• log n

k .

D.Gamarnik Local Algorithms in Random Structures June, 2018 8 / 29

Examples in Statistics. Largest submatrix problem Cn =    C11 . . . C1n . . . ... . . . Cn1 . . . Cnn    , Ci,j

d

= i.i.d. N(0, 1) Given k, find max

I,J⊂[n],|I|=|J|=k Ave(CI,J).

Bhamidi, Dey & Nobel [2017]. Largest value is max Ave(CI,J) = 2(1 + o(1))

• log n

k . Algorithmically (G & Li [2017]) AveALG(CI,J) = 4(1 + o(1)) 3 √ 2

• log n

k .

D.Gamarnik Local Algorithms in Random Structures June, 2018 8 / 29

Examples in Statistics. Sparse Regression

D.Gamarnik Local Algorithms in Random Structures June, 2018 9 / 29

Examples in Statistics. Sparse Regression X ∈ Rn×p, W ∈ Rn i.i.d. N(0, σ2).

D.Gamarnik Local Algorithms in Random Structures June, 2018 9 / 29

Examples in Statistics. Sparse Regression X ∈ Rn×p, W ∈ Rn i.i.d. N(0, σ2).    Y1 . . . Yn    =    X11 X12 . . . X1p . . . . . . ... . . . Xn1 Xn2 . . . Xnp       β∗

1

. . . β∗

p

   +    W1 . . . Wn    .

D.Gamarnik Local Algorithms in Random Structures June, 2018 9 / 29

Examples in Statistics. Sparse Regression X ∈ Rn×p, W ∈ Rn i.i.d. N(0, σ2).    Y1 . . . Yn    =    X11 X12 . . . X1p . . . . . . ... . . . Xn1 Xn2 . . . Xnp       β∗

1

. . . β∗

p

   +    W1 . . . Wn    . Goal: Recover β∗ from observed X and Y when β∗0 ≤ k by solving min

β0≤k Y − Xβ2.

D.Gamarnik Local Algorithms in Random Structures June, 2018 9 / 29

Examples in Statistics. Sparse Regression X ∈ Rn×p, W ∈ Rn i.i.d. N(0, σ2).    Y1 . . . Yn    =    X11 X12 . . . X1p . . . . . . ... . . . Xn1 Xn2 . . . Xnp       β∗

1

. . . β∗

p

   +    W1 . . . Wn    . Goal: Recover β∗ from observed X and Y when β∗0 ≤ k by solving min

β0≤k Y − Xβ2.

Brute force works iff n ≥ n∗

Info Ω

• k log p

log k

• . Wang, Wainwright &

Ramchandran [2010], Rad [2011], G & Zadik [2017]

D.Gamarnik Local Algorithms in Random Structures June, 2018 9 / 29

Examples in Statistics. Sparse Regression X ∈ Rn×p, W ∈ Rn i.i.d. N(0, σ2).    Y1 . . . Yn    =    X11 X12 . . . X1p . . . . . . ... . . . Xn1 Xn2 . . . Xnp       β∗

1

. . . β∗

p

   +    W1 . . . Wn    . Goal: Recover β∗ from observed X and Y when β∗0 ≤ k by solving min

β0≤k Y − Xβ2.

Brute force works iff n ≥ n∗

Info Ω

• k log p

log k

• . Wang, Wainwright &

Ramchandran [2010], Rad [2011], G & Zadik [2017] Efficiently (convex optimization) only when n ≥ n∗

Convex = Ω (k log p).

Donoho, Tanner, Wainwright, Hastie, Tibshirani, Candes, Tao [1996–]

D.Gamarnik Local Algorithms in Random Structures June, 2018 9 / 29

Geometry

Part II. Geometry of the solution space

D.Gamarnik Local Algorithms in Random Structures June, 2018 10 / 29

Overlap Gap Property

D.Gamarnik Local Algorithms in Random Structures June, 2018 11 / 29

Overlap Gap Property Optimization with random input X min

θ∈Θ L(θ, X).

D.Gamarnik Local Algorithms in Random Structures June, 2018 11 / 29

Overlap Gap Property Optimization with random input X min

θ∈Θ L(θ, X).

OGP holds if there exists R > 0, such that the set {θ : L(θ, X) ≤ min

θ∈Θ L(θ, X) + R}

is disconnected.

D.Gamarnik Local Algorithms in Random Structures June, 2018 11 / 29

Overlap Gap Property Optimization with random input X min

θ∈Θ L(θ, X).

OGP holds if there exists R > 0, such that the set {θ : L(θ, X) ≤ min

θ∈Θ L(θ, X) + R}

is disconnected. The set of R-optimal solutions is partitioned into separated disconnected components.

D.Gamarnik Local Algorithms in Random Structures June, 2018 11 / 29

Overlap Gap Property

min

θ

L(θ, X)

θ

min L(θ, X)

D.Gamarnik Local Algorithms in Random Structures June, 2018 12 / 29

Overlap Gap Property

min

θ

L(θ, X) R

θ

min L(θ, X)

D.Gamarnik Local Algorithms in Random Structures June, 2018 13 / 29

Overlap Gap Property

min

θ

L(θ, X) R

θ

min L(θ, X)

D.Gamarnik Local Algorithms in Random Structures June, 2018 14 / 29

OGP for Independent Sets in Random Graphs

D.Gamarnik Local Algorithms in Random Structures June, 2018 15 / 29

OGP for Independent Sets in Random Graphs For either G(n, p) or G(n, d

n ), let I(β), β ∈ (0, 1) be the set of ind sets

with multiplicative approximation ratio β. Then the set of overlaps |I ∩ J|, I, J ∈ I(β) is

D.Gamarnik Local Algorithms in Random Structures June, 2018 15 / 29

OGP for Independent Sets in Random Graphs For either G(n, p) or G(n, d

n ), let I(β), β ∈ (0, 1) be the set of ind sets

with multiplicative approximation ratio β. Then the set of overlaps |I ∩ J|, I, J ∈ I(β) is disconnected, for all pairs when β > 1

2 + 1 2 √ 2, and for ”most” pairs

when β > 1

2, Coja–Oghlan & Efthymiou [2013], G & Sudan [2014]

D.Gamarnik Local Algorithms in Random Structures June, 2018 15 / 29

OGP for Independent Sets in Random Graphs For either G(n, p) or G(n, d

n ), let I(β), β ∈ (0, 1) be the set of ind sets

with multiplicative approximation ratio β. Then the set of overlaps |I ∩ J|, I, J ∈ I(β) is disconnected, for all pairs when β > 1

2 + 1 2 √ 2, and for ”most” pairs

when β > 1

2, Coja–Oghlan & Efthymiou [2013], G & Sudan [2014]

connected for β < 1

2, Coja–Oghlan & Efthymiou [2013].

D.Gamarnik Local Algorithms in Random Structures June, 2018 15 / 29

OGP for Independent Sets in Random Graphs For either G(n, p) or G(n, d

n ), let I(β), β ∈ (0, 1) be the set of ind sets

with multiplicative approximation ratio β. Then the set of overlaps |I ∩ J|, I, J ∈ I(β) is disconnected, for all pairs when β > 1

2 + 1 2 √ 2, and for ”most” pairs

when β > 1

2, Coja–Oghlan & Efthymiou [2013], G & Sudan [2014]

connected for β < 1

2, Coja–Oghlan & Efthymiou [2013].

Extension to multi-overlaps |I1 ∩ · · · ∩ Im| by Rahman & Virag [2015].

D.Gamarnik Local Algorithms in Random Structures June, 2018 15 / 29

OGP for Random K-SAT Problem Let SAT n be the set of satisfying assignments of a random formula Φ(n, dn).

D.Gamarnik Local Algorithms in Random Structures June, 2018 16 / 29

OGP for Random K-SAT Problem Let SAT n be the set of satisfying assignments of a random formula Φ(n, dn). Then the set of multi-overlaps for a fixed m (d(σℓ, σr), σℓ ∈ SAT n, 1 ≤ ℓ, r ≤ m) .

D.Gamarnik Local Algorithms in Random Structures June, 2018 16 / 29

OGP for Random K-SAT Problem Let SAT n be the set of satisfying assignments of a random formula Φ(n, dn). Then the set of multi-overlaps for a fixed m (d(σℓ, σr), σℓ ∈ SAT n, 1 ≤ ℓ, r ≤ m) . is connected when d < 2K log 2

K

= d∗

K for m = 2, general case open.

D.Gamarnik Local Algorithms in Random Structures June, 2018 16 / 29

OGP for Random K-SAT Problem Let SAT n be the set of satisfying assignments of a random formula Φ(n, dn). Then the set of multi-overlaps for a fixed m (d(σℓ, σr), σℓ ∈ SAT n, 1 ≤ ℓ, r ≤ m) . is connected when d < 2K log 2

K

= d∗

K for m = 2, general case open.

disconnected when d > 2K log2 K log 2

K

= log2 K

K

d∗, Achlioptas, Coja-Oghlan & Ricci-Tersenghi [2011], G & Sudan [2014] (for NAE-K-SAT).

D.Gamarnik Local Algorithms in Random Structures June, 2018 16 / 29

OGP in Sparse Regression

D.Gamarnik Local Algorithms in Random Structures June, 2018 17 / 29

OGP in Sparse Regression Assume binary case β∗

i = 0, 1.

D.Gamarnik Local Algorithms in Random Structures June, 2018 17 / 29

OGP in Sparse Regression Assume binary case β∗

i = 0, 1.

Consider the optimization problem parametrized by ζ ∈ (0, 1) Γ∗(ζ) min

β

Y − Xβ2 Subject to: β0 = k, β∗ − β0 = 2kζ. Then Γ(ζ) is

D.Gamarnik Local Algorithms in Random Structures June, 2018 17 / 29

OGP in Sparse Regression Assume binary case β∗

i = 0, 1.

Consider the optimization problem parametrized by ζ ∈ (0, 1) Γ∗(ζ) min

β

Y − Xβ2 Subject to: β0 = k, β∗ − β0 = 2kζ. Then Γ(ζ) is increasing when n > Cn∗

Convex,

D.Gamarnik Local Algorithms in Random Structures June, 2018 17 / 29

OGP in Sparse Regression Assume binary case β∗

i = 0, 1.

Consider the optimization problem parametrized by ζ ∈ (0, 1) Γ∗(ζ) min

β

Y − Xβ2 Subject to: β0 = k, β∗ − β0 = 2kζ. Then Γ(ζ) is increasing when n > Cn∗

Convex,

non-monotone when n < cn∗

Convex, G & Zadik [2017].

D.Gamarnik Local Algorithms in Random Structures June, 2018 17 / 29

OGP in Sparse Regression

D.Gamarnik Local Algorithms in Random Structures June, 2018 18 / 29

OGP in Sparse Regression When n > Cn∗

Convex

D.Gamarnik Local Algorithms in Random Structures June, 2018 18 / 29

OGP in Sparse Regression

D.Gamarnik Local Algorithms in Random Structures June, 2018 19 / 29

OGP in Sparse Regression when n < cn∗

Convex – OGP

.

ζ Γ(ζ)

0 1

0 1

τ1 τ2

R

D.Gamarnik Local Algorithms in Random Structures June, 2018 19 / 29

Part III. Local algorithms

D.Gamarnik Local Algorithms in Random Structures June, 2018 20 / 29

Three Types of Local Algorithms

D.Gamarnik Local Algorithms in Random Structures June, 2018 21 / 29

Three Types of Local Algorithms

• Greedy. Example for ind sets: select a node into I. Delete
• neighbors. Repeat.

D.Gamarnik Local Algorithms in Random Structures June, 2018 21 / 29

Three Types of Local Algorithms

• Greedy. Example for ind sets: select a node into I. Delete
• neighbors. Repeat.

D.Gamarnik Local Algorithms in Random Structures June, 2018 21 / 29

Three Types of Local Algorithms

• Greedy. Example for ind sets: select a node into I. Delete
• neighbors. Repeat.

Local-in-Parallel – Factors of I.I.D (FIID). Make randomized decision based on the radius r neighborhood B(u, r) for each node u.

D.Gamarnik Local Algorithms in Random Structures June, 2018 21 / 29

Three Types of Local Algorithms

• Greedy. Example for ind sets: select a node into I. Delete
• neighbors. Repeat.

Local-in-Parallel – Factors of I.I.D (FIID). Make randomized decision based on the radius r neighborhood B(u, r) for each node u. Local-Sequential. Make decision on ui based on ”implied” decisions for u1, . . . , ui−1.

D.Gamarnik Local Algorithms in Random Structures June, 2018 21 / 29

Three Types of Local Algorithms

• Greedy. Example for ind sets: select a node into I. Delete
• neighbors. Repeat.

Local-in-Parallel – Factors of I.I.D (FIID). Make randomized decision based on the radius r neighborhood B(u, r) for each node u. Local-Sequential. Make decision on ui based on ”implied” decisions for u1, . . . , ui−1.

Markov Chain Monte Carlo (MCMC). Example for ind sets. Fix β – inverse temperature.

D.Gamarnik Local Algorithms in Random Structures June, 2018 21 / 29

Three Types of Local Algorithms

• Greedy. Example for ind sets: select a node into I. Delete
• neighbors. Repeat.

Local-in-Parallel – Factors of I.I.D (FIID). Make randomized decision based on the radius r neighborhood B(u, r) for each node u. Local-Sequential. Make decision on ui based on ”implied” decisions for u1, . . . , ui−1.

Markov Chain Monte Carlo (MCMC). Example for ind sets. Fix β – inverse temperature. Pick u u.a.r.

D.Gamarnik Local Algorithms in Random Structures June, 2018 21 / 29

Three Types of Local Algorithms

• Greedy. Example for ind sets: select a node into I. Delete
• neighbors. Repeat.

Local-in-Parallel – Factors of I.I.D (FIID). Make randomized decision based on the radius r neighborhood B(u, r) for each node u. Local-Sequential. Make decision on ui based on ”implied” decisions for u1, . . . , ui−1.

Markov Chain Monte Carlo (MCMC). Example for ind sets. Fix β – inverse temperature. Pick u u.a.r.

If u ∈ I delete with probability

1 1+eβ .

D.Gamarnik Local Algorithms in Random Structures June, 2018 21 / 29

Three Types of Local Algorithms

• Greedy. Example for ind sets: select a node into I. Delete
• neighbors. Repeat.

Local-in-Parallel – Factors of I.I.D (FIID). Make randomized decision based on the radius r neighborhood B(u, r) for each node u. Local-Sequential. Make decision on ui based on ”implied” decisions for u1, . . . , ui−1.

Markov Chain Monte Carlo (MCMC). Example for ind sets. Fix β – inverse temperature. Pick u u.a.r.

If u ∈ I delete with probability

1 1+eβ .

If u / ∈ I and can be added without conflict, add it with prob

eβ 1+eβ .

D.Gamarnik Local Algorithms in Random Structures June, 2018 21 / 29

Three Types of Local Algorithms

• Greedy. Example for ind sets: select a node into I. Delete
• neighbors. Repeat.

Local-in-Parallel – Factors of I.I.D (FIID). Make randomized decision based on the radius r neighborhood B(u, r) for each node u. Local-Sequential. Make decision on ui based on ”implied” decisions for u1, . . . , ui−1.

Markov Chain Monte Carlo (MCMC). Example for ind sets. Fix β – inverse temperature. Pick u u.a.r.

If u ∈ I delete with probability

1 1+eβ .

If u / ∈ I and can be added without conflict, add it with prob

eβ 1+eβ .

Gradient Descent (GD): keep improving while you can (MCMC with β = ∞)

D.Gamarnik Local Algorithms in Random Structures June, 2018 21 / 29

Properties

D.Gamarnik Local Algorithms in Random Structures June, 2018 22 / 29

Properties Greedy and Local-in-Parallel are special cases of Local-Sequential (Obvious).

D.Gamarnik Local Algorithms in Random Structures June, 2018 22 / 29

Properties Greedy and Local-in-Parallel are special cases of Local-Sequential (Obvious). Greedy for ind set can be well approximated by Local-in-Parallel, G & Goldberg [2010]

D.Gamarnik Local Algorithms in Random Structures June, 2018 22 / 29

Properties Greedy and Local-in-Parallel are special cases of Local-Sequential (Obvious). Greedy for ind set can be well approximated by Local-in-Parallel, G & Goldberg [2010]

Proof sketch: assign weights R(u1), . . . , R(un) i.i.d. u.a.r. from [0, 1].

D.Gamarnik Local Algorithms in Random Structures June, 2018 22 / 29

Properties Greedy and Local-in-Parallel are special cases of Local-Sequential (Obvious). Greedy for ind set can be well approximated by Local-in-Parallel, G & Goldberg [2010]

Proof sketch: assign weights R(u1), . . . , R(un) i.i.d. u.a.r. from [0, 1]. For each node u for the decision not to be r-local there should exist a path u, u1, . . . , ur such that R(u1) < R(u2) < · · · < R(ur).

D.Gamarnik Local Algorithms in Random Structures June, 2018 22 / 29

Properties Greedy and Local-in-Parallel are special cases of Local-Sequential (Obvious). Greedy for ind set can be well approximated by Local-in-Parallel, G & Goldberg [2010]

Proof sketch: assign weights R(u1), . . . , R(un) i.i.d. u.a.r. from [0, 1]. For each node u for the decision not to be r-local there should exist a path u, u1, . . . , ur such that R(u1) < R(u2) < · · · < R(ur). The probability of this is O 1

r!

• – decays faster than the exponential

growth rate dr of the neighborhood of r.

D.Gamarnik Local Algorithms in Random Structures June, 2018 22 / 29

Properties Greedy and Local-in-Parallel are special cases of Local-Sequential (Obvious). Greedy for ind set can be well approximated by Local-in-Parallel, G & Goldberg [2010]

Proof sketch: assign weights R(u1), . . . , R(un) i.i.d. u.a.r. from [0, 1]. For each node u for the decision not to be r-local there should exist a path u, u1, . . . , ur such that R(u1) < R(u2) < · · · < R(ur). The probability of this is O 1

r!

• – decays faster than the exponential

growth rate dr of the neighborhood of r. As a result, for most nodes the decision is r-localized.

D.Gamarnik Local Algorithms in Random Structures June, 2018 22 / 29

Properties

D.Gamarnik Local Algorithms in Random Structures June, 2018 23 / 29

Properties MCMC has a simple to describe (Gibbs) steady state: for every ind set P(I) = eβ|I|

• J eβ|I| .

D.Gamarnik Local Algorithms in Random Structures June, 2018 23 / 29

Properties MCMC has a simple to describe (Gibbs) steady state: for every ind set P(I) = eβ|I|

• J eβ|I| .

MCMC always produces a near-optimal solution for large enough β, (but possibly after exponential time), for the following reason: |I∗| ≤ 1 β log

• J

eβ|I|

• ≤ |I∗| + nlog 2

β .

D.Gamarnik Local Algorithms in Random Structures June, 2018 23 / 29

Properties MCMC has a simple to describe (Gibbs) steady state: for every ind set P(I) = eβ|I|

• J eβ|I| .

MCMC always produces a near-optimal solution for large enough β, (but possibly after exponential time), for the following reason: |I∗| ≤ 1 β log

• J

eβ|I|

• ≤ |I∗| + nlog 2

β . As a result, most of the Gibbs ”mass” is concentrated on ind sets I with |I| ≈ |I∗|.

D.Gamarnik Local Algorithms in Random Structures June, 2018 23 / 29

Summary Punchline: For all of the discussed models, below the OGP one of the versions of the local algorithms finds a solution. Above the threshold no algorithms are known and possibly no poly-time algorithms exist.

D.Gamarnik Local Algorithms in Random Structures June, 2018 24 / 29

Examples: Ind set in sparse random graphs In either G(n, d

n ) or Gd(n), local parallel (FIID) algorithms can construct

ind sets with size (1 + od(1)) log d d n Lauer & Wormald [2007], G & Goldberg [2010].

D.Gamarnik Local Algorithms in Random Structures June, 2018 25 / 29

Examples: Ind set in sparse random graphs In either G(n, d

n ) or Gd(n), local parallel (FIID) algorithms can construct

ind sets with size (1 + od(1)) log d d n Lauer & Wormald [2007], G & Goldberg [2010]. and no Local Parallel algorithms exist which construct ind set with larger size w.h.p., G & Sudan, Rahman & Virag [2015]. The OGP is used as an obstruction to such algorithms.

D.Gamarnik Local Algorithms in Random Structures June, 2018 25 / 29

Examples: Ind set in sparse random graphs In either G(n, d

n ) or Gd(n), local parallel (FIID) algorithms can construct

ind sets with size (1 + od(1)) log d d n Lauer & Wormald [2007], G & Goldberg [2010]. and no Local Parallel algorithms exist which construct ind set with larger size w.h.p., G & Sudan, Rahman & Virag [2015]. The OGP is used as an obstruction to such algorithms. Also MCMC fail finding ind sets of the same size Coja–Oghlan & Efthymiou [2013] (MCMC mixes in exponential time).

D.Gamarnik Local Algorithms in Random Structures June, 2018 25 / 29

Examples: Sparse Regression For sparse regression problem Y = Xβ∗ + W, Gradient Descent reconstructs β∗ when n > CnConvex = Ω (k log p) ,

D.Gamarnik Local Algorithms in Random Structures June, 2018 26 / 29

Examples: Sparse Regression For sparse regression problem Y = Xβ∗ + W, Gradient Descent reconstructs β∗ when n > CnConvex = Ω (k log p) , and Gradient Descent provably fails for certain starting points when n < cnConvex = Ω (k log p) .

D.Gamarnik Local Algorithms in Random Structures June, 2018 26 / 29

Examples: Sparse Regression For sparse regression problem Y = Xβ∗ + W, Gradient Descent reconstructs β∗ when n > CnConvex = Ω (k log p) , and Gradient Descent provably fails for certain starting points when n < cnConvex = Ω (k log p) . OGP serves as an obstruction to the GD algorithm, G & Zadik [2017]

D.Gamarnik Local Algorithms in Random Structures June, 2018 26 / 29

Conclusion In conclusion, we have a fairly complete theory: optimiza- tion in random graphs is solvable by local algorithms be- low the OGP threshold. Above the threshold local algo- rithms provably fails, but no other poly-time algorithms are known, and possibly none exist.

D.Gamarnik Local Algorithms in Random Structures June, 2018 27 / 29

Future Work

D.Gamarnik Local Algorithms in Random Structures June, 2018 28 / 29

Future Work Planted clique problem G(n, 1

2, m) exhibits non-monotonicity

implying OGP iff the size of the planted clique m = o(√n), G & Zadik [201?]

D.Gamarnik Local Algorithms in Random Structures June, 2018 28 / 29

Future Work Planted clique problem G(n, 1

2, m) exhibits non-monotonicity

implying OGP iff the size of the planted clique m = o(√n), G & Zadik [201?] Sparse Planted XOR-K-SAT formula Φ(n, dn) exhibits non-monotonicity implying OGP .

D.Gamarnik Local Algorithms in Random Structures June, 2018 28 / 29

Thank you

D.Gamarnik Local Algorithms in Random Structures June, 2018 29 / 29