Solving SDPs for synchronization and MaxCut problems via the - - PowerPoint PPT Presentation

▶

Dec 18, 2023 373 likes •672 views

Solving SDPs for synchronization and MaxCut problems via the Grothendieck inequality Song Mei Stanford University July 8, 2017 Joint work with Theodor Misiakiewicz, Andrea Montanari, and Roberto I. Oliveira Song Mei (Stanford University) The

SLIDE 1

Solving SDPs for synchronization and MaxCut problems via the Grothendieck inequality

Song Mei

Stanford University

July 8, 2017

Joint work with Theodor Misiakiewicz, Andrea Montanari, and Roberto I. Oliveira

Song Mei (Stanford University) The landscape of non-convex SDP July 8, 2017 1 / 13

SLIDE 2

The MaxCut SDP problem

◮ ❆ ✷ R♥✂♥ symmetric. ◮ MaxCut SDP:

♠❛①✐♠✐③❡

❳✷R♥✂♥

❤❆❀ ❳✐ s✉❜❥❡❝t t♦ ❳✐✐ ❂ ✶❀ ✐ ✷ ❬♥❪❀ ❳ ✗ ✵✿ (SDP)

◮ Applications: MaxCut problem, Z✷ synchronization, Stochastic

block model...

Song Mei (Stanford University) The landscape of non-convex SDP July 8, 2017 2 / 13

SLIDE 3

The MaxCut SDP problem

◮ ❆ ✷ R♥✂♥ symmetric. ◮ MaxCut SDP:

♠❛①✐♠✐③❡

❳✷R♥✂♥

❤❆❀ ❳✐ s✉❜❥❡❝t t♦ ❳✐✐ ❂ ✶❀ ✐ ✷ ❬♥❪❀ ❳ ✗ ✵✿ (SDP)

◮ Applications: MaxCut problem, Z✷ synchronization, Stochastic

block model...

Song Mei (Stanford University) The landscape of non-convex SDP July 8, 2017 2 / 13

SLIDE 4

The MaxCut SDP problem

◮ ❆ ✷ R♥✂♥ symmetric. ◮ MaxCut SDP:

♠❛①✐♠✐③❡

❳✷R♥✂♥

❤❆❀ ❳✐ s✉❜❥❡❝t t♦ ❳✐✐ ❂ ✶❀ ✐ ✷ ❬♥❪❀ ❳ ✗ ✵✿ (SDP)

◮ Applications: MaxCut problem, Z✷ synchronization, Stochastic

block model...

Song Mei (Stanford University) The landscape of non-convex SDP July 8, 2017 2 / 13

SLIDE 5

MaxCut Problem

◮ ●: a positively weighted graph. ❆●: its adjacency matrix. ◮ MaxCut of ●: known to be NP-hard

♠❛①✐♠✐③❡

①✷❢✝✶❣♥

✶ ✹

♥

❳

✐❀❥❂✶

❆●❀✐❥✭✶ ①✐①❥✮✿ (MaxCut)

◮ SDP relaxation: ✵✿✽✼✽-approximate guarantee [Goemanns and

Williamson, 1995] ♠❛①✐♠✐③❡

❳✷R♥✂♥

✶ ✹

♥

❳

✐❀❥❂✶

❆●❀✐❥✭✶ ❳✐❥✮❀ s✉❜❥❡❝t t♦ ❳✐✐ ❂ ✶❀ ❳ ✗ ✵✿ (SDPCut)

Song Mei (Stanford University) The landscape of non-convex SDP July 8, 2017 3 / 13

SLIDE 6

MaxCut Problem

◮ ●: a positively weighted graph. ❆●: its adjacency matrix. ◮ MaxCut of ●: known to be NP-hard

♠❛①✐♠✐③❡

①✷❢✝✶❣♥

✶ ✹

♥

❳

✐❀❥❂✶

❆●❀✐❥✭✶ ①✐①❥✮✿ (MaxCut)

◮ SDP relaxation: ✵✿✽✼✽-approximate guarantee [Goemanns and

Williamson, 1995] ♠❛①✐♠✐③❡

❳✷R♥✂♥

✶ ✹

♥

❳

✐❀❥❂✶

❆●❀✐❥✭✶ ❳✐❥✮❀ s✉❜❥❡❝t t♦ ❳✐✐ ❂ ✶❀ ❳ ✗ ✵✿ (SDPCut)

Song Mei (Stanford University) The landscape of non-convex SDP July 8, 2017 3 / 13

SLIDE 7

MaxCut Problem

◮ ●: a positively weighted graph. ❆●: its adjacency matrix. ◮ MaxCut of ●: known to be NP-hard

♠❛①✐♠✐③❡

①✷❢✝✶❣♥

✶ ✹

♥

❳

✐❀❥❂✶

❆●❀✐❥✭✶ ①✐①❥✮✿ (MaxCut)

◮ SDP relaxation: ✵✿✽✼✽-approximate guarantee [Goemanns and

Williamson, 1995] ♠❛①✐♠✐③❡

❳✷R♥✂♥

✶ ✹

♥

❳

✐❀❥❂✶

❆●❀✐❥✭✶ ❳✐❥✮❀ s✉❜❥❡❝t t♦ ❳✐✐ ❂ ✶❀ ❳ ✗ ✵✿ (SDPCut)

Song Mei (Stanford University) The landscape of non-convex SDP July 8, 2017 3 / 13

SLIDE 8

Burer-Monteiro approach

◮ Convex formulation: ♥ up to ✶✵✸ using interior point method

♠❛①✐♠✐③❡

❳✷R♥✂♥

❤❆❀ ❳✐ s✉❜❥❡❝t t♦ ❳✐✐ ❂ ✶❀ ✐ ✷ ❬♥❪❀ ❳ ✗ ✵✿ (SDP)

◮ Change variable ❳ ❂ ✛ ✁ ✛T, ✛ ✷ R♥✂❦, ❦ ✜ ♥. ◮ Non-convex formulation: ♥ up to ✶✵✺

♠❛①✐♠✐③❡

✛✷R♥✂❦

❤✛❀ ❆✛✐ s✉❜❥❡❝t t♦ ✛ ❂ ❬✛✶❀ ✿ ✿ ✿ ❀ ✛♥❪T❀ ❦✛✐❦✷ ❂ ✶❀ ✐ ✷ ❬♥❪✿ (❦-Ncvx-SDP)

Song Mei (Stanford University) The landscape of non-convex SDP July 8, 2017 4 / 13

SLIDE 9

Burer-Monteiro approach

◮ Convex formulation: ♥ up to ✶✵✸ using interior point method

♠❛①✐♠✐③❡

❳✷R♥✂♥

❤❆❀ ❳✐ s✉❜❥❡❝t t♦ ❳✐✐ ❂ ✶❀ ✐ ✷ ❬♥❪❀ ❳ ✗ ✵✿ (SDP)

◮ Change variable ❳ ❂ ✛ ✁ ✛T, ✛ ✷ R♥✂❦, ❦ ✜ ♥. ◮ Non-convex formulation: ♥ up to ✶✵✺

♠❛①✐♠✐③❡

✛✷R♥✂❦

❤✛❀ ❆✛✐ s✉❜❥❡❝t t♦ ✛ ❂ ❬✛✶❀ ✿ ✿ ✿ ❀ ✛♥❪T❀ ❦✛✐❦✷ ❂ ✶❀ ✐ ✷ ❬♥❪✿ (❦-Ncvx-SDP)

Song Mei (Stanford University) The landscape of non-convex SDP July 8, 2017 4 / 13

SLIDE 10

Burer-Monteiro approach

◮ Convex formulation: ♥ up to ✶✵✸ using interior point method

♠❛①✐♠✐③❡

❳✷R♥✂♥

❤❆❀ ❳✐ s✉❜❥❡❝t t♦ ❳✐✐ ❂ ✶❀ ✐ ✷ ❬♥❪❀ ❳ ✗ ✵✿ (SDP)

◮ Change variable ❳ ❂ ✛ ✁ ✛T, ✛ ✷ R♥✂❦, ❦ ✜ ♥. ◮ Non-convex formulation: ♥ up to ✶✵✺

♠❛①✐♠✐③❡

✛✷R♥✂❦

❤✛❀ ❆✛✐ s✉❜❥❡❝t t♦ ✛ ❂ ❬✛✶❀ ✿ ✿ ✿ ❀ ✛♥❪T❀ ❦✛✐❦✷ ❂ ✶❀ ✐ ✷ ❬♥❪✿ (❦-Ncvx-SDP)

Song Mei (Stanford University) The landscape of non-convex SDP July 8, 2017 4 / 13

SLIDE 11

Related literatures

◮ As ❦ ✕

♣ ✷♥, the global maxima of the Non Convex formulation coincide with the global maximizer of the Convex formulation [Pataki, 1998], [Barviok, 2001], [Burer and Monteiro, 2003].

◮ As ❦ ✕

♣ ✷♥, Non Convex formulation has no spurious local maxima [Boumal, et al., 2016].

◮ What if ❦ remains of order ✶, as ♥ ✦ ✶? Is there spurious local

maxima? Sadly, yes.

◮ How is these local maxima? Empirically, they are good!

Song Mei (Stanford University) The landscape of non-convex SDP July 8, 2017 5 / 13

SLIDE 12

Related literatures

◮ As ❦ ✕

♣ ✷♥, the global maxima of the Non Convex formulation coincide with the global maximizer of the Convex formulation [Pataki, 1998], [Barviok, 2001], [Burer and Monteiro, 2003].

◮ As ❦ ✕

♣ ✷♥, Non Convex formulation has no spurious local maxima [Boumal, et al., 2016].

◮ What if ❦ remains of order ✶, as ♥ ✦ ✶? Is there spurious local

maxima? Sadly, yes.

◮ How is these local maxima? Empirically, they are good!

Song Mei (Stanford University) The landscape of non-convex SDP July 8, 2017 5 / 13

SLIDE 13

Related literatures

◮ As ❦ ✕

♣ ✷♥, the global maxima of the Non Convex formulation coincide with the global maximizer of the Convex formulation [Pataki, 1998], [Barviok, 2001], [Burer and Monteiro, 2003].

◮ As ❦ ✕

♣ ✷♥, Non Convex formulation has no spurious local maxima [Boumal, et al., 2016].

◮ What if ❦ remains of order ✶, as ♥ ✦ ✶? Is there spurious local

maxima? Sadly, yes.

◮ How is these local maxima? Empirically, they are good!

Song Mei (Stanford University) The landscape of non-convex SDP July 8, 2017 5 / 13

SLIDE 14

Related literatures

◮ As ❦ ✕

♣ ✷♥, the global maxima of the Non Convex formulation coincide with the global maximizer of the Convex formulation [Pataki, 1998], [Barviok, 2001], [Burer and Monteiro, 2003].

◮ As ❦ ✕

♣ ✷♥, Non Convex formulation has no spurious local maxima [Boumal, et al., 2016].

◮ What if ❦ remains of order ✶, as ♥ ✦ ✶? Is there spurious local

maxima? Sadly, yes.

◮ How is these local maxima? Empirically, they are good!

Song Mei (Stanford University) The landscape of non-convex SDP July 8, 2017 5 / 13

SLIDE 15

Related literatures

◮ As ❦ ✕

♣ ✷♥, the global maxima of the Non Convex formulation coincide with the global maximizer of the Convex formulation [Pataki, 1998], [Barviok, 2001], [Burer and Monteiro, 2003].

◮ As ❦ ✕

♣ ✷♥, Non Convex formulation has no spurious local maxima [Boumal, et al., 2016].

◮ What if ❦ remains of order ✶, as ♥ ✦ ✶? Is there spurious local

maxima? Sadly, yes.

◮ How is these local maxima? Empirically, they are good!

Song Mei (Stanford University) The landscape of non-convex SDP July 8, 2017 5 / 13

SLIDE 16

Related literatures

◮ As ❦ ✕

♣ ✷♥, the global maxima of the Non Convex formulation coincide with the global maximizer of the Convex formulation [Pataki, 1998], [Barviok, 2001], [Burer and Monteiro, 2003].

◮ As ❦ ✕

♣ ✷♥, Non Convex formulation has no spurious local maxima [Boumal, et al., 2016].

◮ What if ❦ remains of order ✶, as ♥ ✦ ✶? Is there spurious local

maxima? Sadly, yes.

◮ How is these local maxima? Empirically, they are good!

Song Mei (Stanford University) The landscape of non-convex SDP July 8, 2017 5 / 13

SLIDE 17

Geometry

♠❛①✐♠✐③❡

✛✷R♥✂❦

❤✛❀ ❆✛✐ ✿❂ ❢✭✛✮ s✉❜❥❡❝t t♦ ❦✛✐❦✷ ❂ ✶✿

♦

▼❦ ❂ ❢✛ ✷ R♥✂❦ ✿ ❦✛✐❦✷ ❂ ✶❣✿

Definition (✧-approximate concave point)

We call ✛ ✷ ▼❦ an ✧-approximate concave point of ❢ on ▼❦, if for any tangent vector ✉ ✷ ❚✛▼❦, we have ❤✉❀ Hess❢✭✛✮❬✉❪✐ ✔ ✧❤✉❀ ✉✐✿ (1)

Remark

A local maximizer is ✵-approximate concave. An ✧-approximate concave point is nearly locally optimal.

Song Mei (Stanford University) The landscape of non-convex SDP July 8, 2017 6 / 13

SLIDE 18

Geometry

♠❛①✐♠✐③❡

✛✷R♥✂❦

❤✛❀ ❆✛✐ ✿❂ ❢✭✛✮ s✉❜❥❡❝t t♦ ❦✛✐❦✷ ❂ ✶✿

♦

▼❦ ❂ ❢✛ ✷ R♥✂❦ ✿ ❦✛✐❦✷ ❂ ✶❣✿

Definition (✧-approximate concave point)

We call ✛ ✷ ▼❦ an ✧-approximate concave point of ❢ on ▼❦, if for any tangent vector ✉ ✷ ❚✛▼❦, we have ❤✉❀ Hess❢✭✛✮❬✉❪✐ ✔ ✧❤✉❀ ✉✐✿ (1)

Remark

A local maximizer is ✵-approximate concave. An ✧-approximate concave point is nearly locally optimal.

Song Mei (Stanford University) The landscape of non-convex SDP July 8, 2017 6 / 13

SLIDE 19

Geometry

♠❛①✐♠✐③❡

✛✷R♥✂❦

❤✛❀ ❆✛✐ ✿❂ ❢✭✛✮ s✉❜❥❡❝t t♦ ❦✛✐❦✷ ❂ ✶✿

♦

▼❦ ❂ ❢✛ ✷ R♥✂❦ ✿ ❦✛✐❦✷ ❂ ✶❣✿

Definition (✧-approximate concave point)

We call ✛ ✷ ▼❦ an ✧-approximate concave point of ❢ on ▼❦, if for any tangent vector ✉ ✷ ❚✛▼❦, we have ❤✉❀ Hess❢✭✛✮❬✉❪✐ ✔ ✧❤✉❀ ✉✐✿ (1)

Remark

A local maximizer is ✵-approximate concave. An ✧-approximate concave point is nearly locally optimal.

Song Mei (Stanford University) The landscape of non-convex SDP July 8, 2017 6 / 13

SLIDE 20

Landscape Theorem

Theorem (A Grothendieck-type inequality)

For any ✧-approximate concave point ✛ ✷ ▼❦ of the rank-❦ non-convex problem, we have ❢✭✛✮ ✕ ❙❉P✭❆✮ ✶ ❦ ✶✭❙❉P✭❆✮ ✰ ❙❉P✭❆✮✮ ♥ ✷ ✧✿ (2) ❙❉P✭❆✮: the maximum value of SDP with input matrix ❆. Geometric iterpretation: the function value for all local maxima are within a gap of order ❖✭✶❂❦✮ within the global maximum.

Song Mei (Stanford University) The landscape of non-convex SDP July 8, 2017 7 / 13

SLIDE 21

Landscape of non-convex SDP

◮ ❢✭✛✮ ✕ ❙❉P✭❆✮ ✶ ❦✶✭❙❉P✭❆✮ ✰ ❙❉P✭❆✮✮ ♥ ✷ ✧.

Gap =

1 k−1

SDP(A) + SDP(−A)
kSDP(A)

SDP(A) + SDP(−A)

nε/2

a saddle point with ε curvature global optimizer a local optimizer

SDP(A) −SDP(−A)

Gap Song Mei (Stanford University) The landscape of non-convex SDP July 8, 2017 8 / 13

SLIDE 22

Efficient Algorithm

◮ Guaranteed converge rate using Riemannian trust region method. ◮ Getting absolute error ♥✧ ✰ ✭❙❉P✭❆✮ ✰ ❙❉P✭❆✮✮❂✭❦ ✶✮ within

❝ ✁ ♥❦❆❦✷

✶❂✧✷ trust region steps. ◮ Empirically, gradient descent converges faster than what is

guaranteed.

Song Mei (Stanford University) The landscape of non-convex SDP July 8, 2017 9 / 13

SLIDE 23

Approximate MaxCut Guarantee

Theorem (Approximate MaxCut Guarantee)

For any ❦ ✕ ✸, if ✛❄ is a local maximizer of corresponding rank-❦ non-convex problem, then we can use ✛❄ to find a ✵✿✽✼✽ ✂ ✭✶ ✶❂✭❦ ✶✮✮-approximate MaxCut. The global maximizer: ✵✿✽✼✽-approximate MaxCut. Any Local maximizers: ✵✿✽✼✽ ✂ ✭✶ ✶❂✭❦ ✶✮✮-approximate MaxCut.

Song Mei (Stanford University) The landscape of non-convex SDP July 8, 2017 10 / 13

SLIDE 24

Approximate MaxCut Guarantee

Theorem (Approximate MaxCut Guarantee)

For any ❦ ✕ ✸, if ✛❄ is a local maximizer of corresponding rank-❦ non-convex problem, then we can use ✛❄ to find a ✵✿✽✼✽ ✂ ✭✶ ✶❂✭❦ ✶✮✮-approximate MaxCut. The global maximizer: ✵✿✽✼✽-approximate MaxCut. Any Local maximizers: ✵✿✽✼✽ ✂ ✭✶ ✶❂✭❦ ✶✮✮-approximate MaxCut.

Song Mei (Stanford University) The landscape of non-convex SDP July 8, 2017 10 / 13

SLIDE 25

Group Synchronization

◮ ❙❖✭❞✮ synchronization, Orthogonal Cut SDP

♠❛①✐♠✐③❡

❳✷R♥❦✂♥❦

❤❆❀ ❳✐ s✉❜❥❡❝t t♦ ❳✐✐ ❂ I❦❀ ❳ ✗ ✵✿ (3)

◮ Similar guarantee.

Song Mei (Stanford University) The landscape of non-convex SDP July 8, 2017 11 / 13

SLIDE 26

Conclusion

◮ Non-convex MaxCut SDP: all local maxima are near global

maxima.

◮ An alternate algorithm for approximating MaxCut. ◮ Conclusion generalizable to general SDP problem.

What I did not go into detail

◮ Z✷ synchronization and ❙❖✭❞✮ synchronization. ◮ The one page proof for the Grothendieck-type inequality.

Song Mei (Stanford University) The landscape of non-convex SDP July 8, 2017 12 / 13

SLIDE 27

Questions?

❢✭✛✮ ✕ ❙❉P✭❆✮ ✶ ❦ ✶✭❙❉P✭❆✮ ✰ ❙❉P✭❆✮✮ ♥ ✷ ✧✿

◮ ❙❉P✭❆✮? Typically has no relationship with ❙❉P✭❆✮. You can

think of it has the same order as ❙❉P✭❆✮. Fit well in the MaxCut problem.

◮ ✶❂❦ tight? We believe Yes.

Song Mei (Stanford University) The landscape of non-convex SDP July 8, 2017 13 / 13

SLIDE 28

Questions?

❢✭✛✮ ✕ ❙❉P✭❆✮ ✶ ❦ ✶✭❙❉P✭❆✮ ✰ ❙❉P✭❆✮✮ ♥ ✷ ✧✿

◮ ❙❉P✭❆✮? Typically has no relationship with ❙❉P✭❆✮. You can

think of it has the same order as ❙❉P✭❆✮. Fit well in the MaxCut problem.

◮ ✶❂❦ tight? We believe Yes.

Song Mei (Stanford University) The landscape of non-convex SDP July 8, 2017 13 / 13