Lecture 3: Semidefinite Programming Lecture Outline Part I: - - PowerPoint PPT Presentation

Lecture 3: Semidefinite Programming

Lecture Outline

• Part I: Semidefinite programming, examples,

canonical form, and duality

• Part II: Strong Duality Failure Examples
• Part III: Conditions for strong duality
• Part IV: Solving convex optimization problems

Part I: Semidefinite Programming, Examples, Canonical Form, and Duality

Semidefinite Programming

• Semidefinite Programming: Want to optimize a

linear function, can now have matrix positive semidefiniteness (PSD) constraints as well as linear equalities and inequalities

• Example: Maximize 𝑦 subject to

1 𝑦 𝑦 2 + 𝑦 ≽ 0

• Answer: 𝑦 = 2

Example: Goemans-Williamson

• First approximation algorithm using a

semiefinite program (SDP)

• MAX-CUT reformulation: Have a variable 𝑦𝑗 for

each vertex i, will set 𝑦𝑗 = ±1 depending on which side of the cut 𝑗 is on.

• Want to maximize σ𝑗,𝑘:𝑗<𝑘, 𝑗,𝑘 ∈𝐹(𝐻)

1−𝑦𝑗𝑦𝑘 2

where 𝑦𝑗 ∈ {−1, +1} for all 𝑗.

Example: Goemans-Williamson

• Idea: Take 𝑁 so that 𝑁𝑗𝑘 = 𝑦𝑗𝑦𝑘
• Want to maximize σ𝑗,𝑘:𝑗<𝑘, 𝑗,𝑘 ∈𝐹(𝐻)

1−𝑁𝑗𝑘 2

where 𝑁𝑗𝑗 = 1 for all 𝑗 and 𝑁 = xxT.

• Relaxation: Maximize σ𝑗,𝑘:𝑗<𝑘, 𝑗,𝑘 ∈𝐹(𝐻)

1−𝑁𝑗𝑘 2

subject to

1. ∀𝑗, 𝑁𝑗𝑗 = 1 2. 𝑁 ≽ 0

Example: SOS Hierarchy

• Goal: Minimize a polynomial ℎ(𝑦1, … , 𝑦𝑜)

subject to constraints 𝑡1 𝑦1, … , 𝑦𝑜 = 0, 𝑡2 𝑦1, … , 𝑦𝑜 = 0, etc.

• Relaxation: Minimize Ẽ[ℎ] where Ẽ is a linear

map from polynomials of degree ≤ 𝑒 to ℝ satisfying:

1. Ẽ 1 = 1

• 2. Ẽ 𝑔𝑡𝑗 = 0 whenever deg 𝑔 + deg 𝑡𝑗 ≤ 𝑒
• 3. Ẽ 𝑕2 ≥ 0 whenever deg 𝑕 ≤

𝑒 2

The Moment Matrix

• Indexed by monomials of degree ≤

𝑒 2

• 𝑁𝑞𝑟 = ෨

𝐹[𝑞𝑟]

• Each 𝑕 of degree ≤

𝑒 2 corresponds to a vector

• ෨

𝐹 𝑕2 = 𝑕𝑈𝑁𝑕

• ∀𝑕, ෨

𝐹 𝑕2 ≥ 0 ⇔ 𝑁 is PSD

𝑟 𝑞 Ẽ[pq] 𝑁 p,q are monomials of degree at most

𝑒 2.

Semidefinite Program for SOS

• Program: Minimize Ẽ[ℎ] where Ẽ satisfies:

1. Ẽ 1 = 1

• 2. Ẽ 𝑔𝑡𝑗 = 0 whenever deg 𝑔 + deg 𝑡𝑗 ≤ 𝑒
• 3. Ẽ 𝑕2 ≥ 0 whenever deg 𝑕 ≤

𝑒 2

• Expressible as semidefinite program using 𝑁:
• 1. ∀ℎ, Ẽ[ℎ] is a linear function of entries of 𝑁
• 2. Constraints that Ẽ 1 = 1 and Ẽ 𝑔𝑡𝑗 = 0 give

linear constraints on entries of 𝑁

• 3. Ẽ 𝑕2 ≥ 0 whenever deg 𝑕 ≤

𝑒 2 ⬄𝑁 ≽ 0

• 4. Also have SOS symmetry constraints

SOS symmetry

• Define 𝑦𝐽 = ς𝑗∈𝐽 𝑦𝑗 where 𝐽 is a multi-set
• SOS symmetry constraints: 𝑁𝑦𝐽𝑦𝐾 = 𝑁𝑦𝐽′𝑦𝐾′

whenever 𝐽 ∪ 𝐾 = 𝐽′ ∪ 𝐾′

• Example:

1 𝑏 𝑐 𝑑 𝑒 𝑓 𝑏 𝑑 𝑒 𝑔 𝑕 ℎ 𝑐 𝑒 𝑓 𝑕 ℎ 𝑗 𝑑 𝑔 𝑕 𝑘 𝑙 𝑚 𝑒 𝑕 ℎ 𝑙 𝑚 𝑛 𝑓 ℎ 𝑗 𝑚 𝑛 𝑜 1 𝑦 𝑧 𝑦2 𝑦𝑧 𝑧2 1 𝑦 𝑧 𝑦2 𝑦𝑧 𝑧2

Canonical Form

• Def: Define 𝑌⦁𝑍 = σ𝑗,𝑘 𝑌𝑗𝑘𝑍

𝑗𝑘 = 𝑢𝑠(𝑌𝑍𝑈) to be

the entry-wise dot product of 𝑌 and 𝑍

• Canonical form: Minimize 𝐷⦁𝑌 subject to

1. ∀𝑗, 𝐵𝑗⦁𝑌 = 𝑐𝑗 where the 𝐵𝑗 are symmetric 2. 𝑌 ≽ 0

Putting Things Into Canonical Form

• Canonical form: Minimize 𝐷⦁𝑌 subject to

1. ∀𝑗, 𝐵𝑗⦁𝑌 = 𝑐𝑗 where the 𝐵𝑗 are symmetric 2. 𝑌 ≽ 0

• Ideas for obtaining canonical form:

1. 𝑌 ≽ 0, 𝑍 ≽ 0⬄ 𝑌 𝑍 ≽ 0

• 2. Slack variables: 𝐵𝑗⦁𝑌 ≤ 𝑐𝑗 ⬄𝐵𝑗⦁𝑌 = 𝑐𝑗 + 𝑡𝑗, 𝑡𝑗 ≥ 0
• 3. Can enforce 𝑡𝑗 ≥ 0 by putting 𝑡𝑗 on the diagonal of

𝑌

Semidefinite Programming Dual

• Primal: Minimize 𝐷⦁𝑌 subject to

1. ∀𝑗, 𝐵𝑗⦁𝑌 = 𝑐𝑗 where the 𝐵𝑗 are symmetric 2. 𝑌 ≽ 0

• Dual: Maximize σ𝑗 𝑧𝑗𝑐𝑗 subject to

1. σ𝑗 𝑧𝑗𝐵𝑗 ≼ 𝐷

• Value for dual lower bounds value for primal:

𝐷⦁𝑌 = 𝐷 − σ𝑗 𝑧𝑗𝐵𝑗 ⦁𝑌 + σ𝑗 𝑧𝑗𝐵𝑗 ⦁𝑌 ≥ σ𝑗 𝑧𝑗𝑐𝑗

Explanation for Duality

• Primal: Minimize 𝐷⦁𝑌 subject to

1. ∀𝑗, 𝐵𝑗⦁𝑌 = 𝑐𝑗 where the 𝐵𝑗 are symmetric 2. 𝑌 ≽ 0

• = min

𝑌≽0 max 𝑧

𝐷⦁𝑌 + σ𝑗 𝑧𝑗 𝑐𝑗 − 𝐵𝑗⦁𝑌

• = max

𝑧

min

𝑌≽0 σ𝑗 𝑧𝑗𝑐𝑗 + (𝐷 − σ𝑗 𝑧𝑗𝐵𝑗)⦁𝑌

• Dual: Maximize σ𝑗 𝑧𝑗𝑐𝑗 subject to

1. σ𝑗 𝑧𝑗𝐵𝑗 ≼ 𝐷

In class exercise: SOS duality

• Exercise: What is the dual of the semidefinite

program for SOS?

• Primal: Minimize Ẽ[ℎ] where Ẽ is a linear map

from polynomials of degree ≤ 𝑒 to ℝ such that:

1. Ẽ 1 = 1

• 2. Ẽ 𝑔𝑡𝑗 = 0 whenever deg 𝑔 + deg 𝑡𝑗 ≤ 𝑒
• 3. Ẽ 𝑕2 ≥ 0 whenever deg 𝑕 ≤

𝑒 2

In class exercise solution

• Definition: Given a symmetric matrix 𝑅 indexed

by monomials 𝑦𝐽, we say that 𝑅 represents the polynomial 𝑞𝑅 = σ𝐾 σ𝐽,𝐽′:𝐽∪𝐽′=𝐾 𝑅𝑦𝐽𝑦𝐽′𝑦𝐾

• Proposition 1: If 𝑅 ≽ 0 then 𝑞𝑅 is a sum of
• squares. Conversely, if 𝑞 is a sum of squares

then ∃𝑅 ≽ 0: 𝑞 = 𝑞𝑅

• Proposition 2: If 𝑁 is a moment matrix then

𝑁⦁𝑅 = Ẽ 𝑞𝑅

In class exercise solution continued

• 𝐷 = 𝐼 where 𝑞𝐼 = ℎ
• Constraint that ෨

𝐹 1 = 1 gives matrix 𝐵1 = 1 ⋯ ⋯ ⋮ ⋮ ⋱ and 𝑐1 = 1

• Constraints that ෨

𝐹 𝑔𝑡𝑗 = 0 give matrices 𝐵𝑘 where 𝑞𝐵𝑘 = 𝑔𝑡𝑗 and 𝑐

𝑘 = 0

• SOS symmetry constraints give matrices 𝐵𝑙

such that 𝑞𝐵𝑙 = 0 and 𝑐𝑙 = 0

In class exercise solution continued

• Recall dual: Maximize σ𝑗 𝑧𝑗𝑐𝑗 subject to

1. σ𝑗 𝑧𝑗𝐵𝑗 ≼ 𝐷

• Here: Maximize 𝑑 such that

𝑑𝐵1 + σ𝑘 𝑧𝑘𝐵𝑘 + σ𝑙 𝑧𝑙𝐵𝑙 ≼ 𝐼

• This is the answer, but let’s simplify it into a

more intuitive form.

• Let 𝑅 = 𝐼 − 𝑑𝐵1 + σ𝑘 𝑧𝑘𝐵𝑘 + σ𝑙 𝑧𝑙𝐵𝑙
• 𝑅 ≽ 0

In class exercise solution continued

• 𝐼 = 𝑑𝐵1 + σ𝑘 𝑧𝑘𝐵𝑘 + σ𝑙 𝑧𝑙𝐵𝑙 + 𝑅,

A1 = 1 ⋯ ⋯ ⋮ ⋮ ⋱ , 𝑅 ≽ 0

• View everything in terms of polynomials.
• 𝑞𝐼 = ℎ, 𝑞𝐵1 = 1, 𝑞(σ𝑘 𝑧𝑘𝐵𝑘) = σ𝑗 𝑔

𝑗𝑡𝑗 for some

𝑔

𝑗, 𝑞σ𝑙 𝑧𝑙𝐵𝑙 = 0, 𝑞𝑅 = σ𝑘 𝑕𝑘 2 for some 𝑕𝑘

• ℎ = 𝑑 + σ𝑗 𝑔

𝑗𝑡𝑗 + σ𝑘 𝑕𝑘 2

In class exercise solution continued

• Simplified Dual: Maximize 𝑑 such that

ℎ = 𝑑 + σ𝑗 𝑔

𝑗𝑡𝑗 + σ𝑘 𝑕𝑘 2

• This is a Positivstellensatz proof that ℎ ≥ 𝑑 (see

Lectures 1 and 5)

Part II: Strong Duality Failure Examples

Strong Duality Failure

• Unlike linear programming, it is not always the

case that the values of the primal and dual are the same.

• However, almost never an issue in practice,

have to be trying in order to break strong duality.

• We’ll give this issue its due here then ignore it

for the rest of the seminar.

Non-attainability Example

• Primal: Minimize 𝑦2 subject to 𝑦1

1 1 𝑦2 ≽ 0

• Dual: Maximize 2𝑧 subject to

1. 𝑧 𝑧 0 ≼ 0 1

• Duality demonstration:

1 − 0 𝑧 𝑧 ⦁ 𝑦1 1 1 𝑦2 = 𝑦2 − 2𝑧 ≥ 0

• Dual has optimal value 0, this is not attainable

in the primal (we can only get arbitrarily close)

Duality Gap Example

• Primal: Minimize 𝑦2 + 1 subject to

1 + 𝑦2 𝑦1 𝑦2 𝑦2 ≽ 0

• Dual: Maximize 2𝑧 subject to

2𝑧 𝑧1 𝑧2 𝑧1 −𝑧 𝑧2 −𝑧 𝑧3 ≼ 1

• Duality demonstration

1 − 2𝑧 𝑧1 𝑧2 𝑧1 −𝑧 𝑧2 −𝑧 𝑧3 ⦁ 1 + 𝑦2 𝑦1 𝑦2 𝑦2 = 𝑦2 + 1 − 2𝑧 ≥ 0

Duality Gap Example

• Primal: Minimize 𝑦2 + 1 subject to

1 + 𝑦2 𝑦1 𝑦2 𝑦2 ≽ 0

• Has optimal value 1 as we must have 𝑦2 = 0
• Dual: Maximize 2𝑧 subject to

2𝑧 𝑧1 𝑧2 𝑧1 −𝑧 𝑧2 −𝑧 𝑧3 ≼ 1

• Has optimal value 0 as we must have 𝑧 = 0.
• Note: This example was taken from Lecture 13,

EE227A at Berkeley given on October 14, 2008.

Part III: Conditions for strong duality

Sufficient Strong Duality Conditions

• How can we rule out such a gap?
• Slater’s Condition (informal):

– If the feasible region for the primal has an interior point (in the subspace defined by the linear equalities) then the duality gap is 0. Moreover, if the optimal value is finite then it is attainable in the dual.

• Also sufficient if either the primal or the dual is

feasible and bounded (i.e. any very large point violates the constraints)

Recall Minimax Theorem

• Von Neumann [1928]: If 𝑌 and 𝑍 are convex

compact subsets of 𝑆𝑛 and 𝑆𝑜 and 𝑔: 𝑌 × 𝑍 → 𝑆 is a continuous function which is convex in 𝑌 and concave in 𝑍 then max

𝑧∈𝑍 min 𝑦∈𝑌 𝑔(𝑦, 𝑧) = min 𝑦∈𝑌 max 𝑧∈𝑍 𝑔(𝑦, 𝑧)

• Issue: 𝑌 and 𝑍 are unbounded in our setting.

Minimax in the limit

• Idea: Minimax applies for arbitrarily large 𝑌, 𝑍 so

long as they are bounded

• Can take the limit as 𝑌, 𝑍 get larger and larger
• Question: Do we bound 𝑌 or 𝑍 first?
• If 𝑌 is bounded first, get primal: min

𝑦∈𝑌 max 𝑧∈𝑍 𝑔(𝑦, 𝑧)

• If 𝑍 is bounded first, get dual: max

𝑧∈𝑍 min 𝑦∈𝑌 𝑔(𝑦, 𝑧)

• If we can show it doesn’t matter which is

bounded first, have duality gap 0.

Formal Statements

• Def: Define 𝐶 𝑆 = {𝑦: 𝑦

≤ 𝑆}

• Minimax Theorem: For all finite 𝑆1, 𝑆2

max

𝑧∈𝑍∩𝐶(𝑆2)

min

𝑦∈𝑌∩𝐶(𝑆1) 𝑔(𝑦, 𝑧) =

min

𝑦∈𝑌∩𝐶(𝑆1)

max

𝑧∈𝑍∩𝐶(𝑆2) 𝑔(𝑦, 𝑧)

• Let’s call this value 𝑝𝑞𝑢(𝑆1, 𝑆2)
• Primal value: lim

𝑆1→∞ lim 𝑆2→∞ 𝑝𝑞𝑢(𝑆1, 𝑆2)

• Dual value: lim

𝑆2→∞ lim 𝑆1→∞ 𝑝𝑞𝑢(𝑆1, 𝑆2)

• Sufficient for 0 gap: Show that

∃𝑆: ∀𝑆1, 𝑆2 ≥ 𝑆, 𝑝𝑞𝑢 𝑆1, 𝑆2 = 𝑝𝑞𝑢(𝑆, 𝑆2)

Boundedness Condition Intuition

• Assume ∃ feasible 𝑦 ∈ 𝐶(𝑆) but when 𝑦

> 𝑆, constraints on primal are violated.

• Want to show that ∃𝑆′

∀𝑆1, 𝑆2 ≥ 𝑆′, 𝑝𝑞𝑢 𝑆1, 𝑆2 = 𝑝𝑞𝑢(𝑆′, 𝑆2)

• We are in the finite setting, so we can assume 𝑦

player plays first.

• Intuition: 𝑧 player can heavily punish 𝑦 player

for violated constraints, so 𝑦 player should always choose an 𝑦 ∈ 𝐶(𝑆′).

• Similar logic applies to the dual.

Slater’s Condition Intuition

• Idea: Strictly feasible point 𝑦 shows it is bad for

𝑧 player to play a very large 𝑧.

• Primal: Minimize ℎ(𝑦) subject to {𝑕𝑗 𝑦 ≤ 𝑑𝑗}

where the 𝑕𝑗 are convex.

• 𝑔 𝑦, 𝑧 = σ𝑗 𝑧𝑗 𝑕𝑗 𝑦 − 𝑑𝑗 + ℎ(𝑦) (we’ll

restrict ourselves to non-negative 𝑧)

• Dual: Doesn’t seem to have a nicer form than

max

𝑧≥0 min 𝑦

σ𝑗 𝑧𝑗 𝑕𝑗 𝑦 − 𝑑𝑗 + ℎ 𝑦

Slater’s Condition Intuition

• Primal: Minimize ℎ(𝑦) subject to {𝑕𝑗 𝑦 ≤ 𝑑𝑗}
• Dual: max

𝑧≥0 min 𝑦

σ𝑗 𝑧𝑗 𝑕𝑗 𝑦 − 𝑑𝑗 + ℎ 𝑦

• Key observation: If ∀𝑗, 𝑕𝑗 𝑦 < 𝑑𝑗, 𝑦 punishes

very large 𝑧. Thus, 𝑧 is effectively bounded.

Strong Duality Conditions Summary

• Strong duality may fail for semidefinite

programs.

• However, strong duality holds if the program is

at all robust (Slater’s condition is satisfied) or either the primal or dual is feasible and bounded (any very large point violates the constraints)

• Note: working over the hypercube satisfies

boundedness.

Part III: Solving convex

• ptimization problems

Solving Convex Optimization Problems

• In practice: simplex methods or interior point

methods work best

• First polynomial time guarantee: ellipsoid

method

• This seminar: We’ll use algorithms as a black-

box and assume that semidefinite programs of size 𝑜𝑒 can be solved in time 𝑜𝑃(𝑒).

• Note: Can fail to be polynomial time in

pathological cases (see Ryan O’Donnell’s note), almost never an issue in practice

Usefulness of Convexity

• Want to minimize a convex function 𝑔 over a

convex set 𝑌.

• All local minima are global minima: If 𝑔 𝑦 <

𝑔(𝑧) then 𝑔(𝑧) is not a local minima as ∀𝜗 > 0, 𝑔 𝜗𝑦 + 1 − 𝜗 𝑧 ≤ 𝜗𝑔 𝑦 + 1 − 𝜗 f(y)

• Can always go from the current point 𝑦 towards

a global minima.

Reduction to Feasibility Testing

• Want to minimize a convex function 𝑔 over a

convex set 𝑌

• Testing whether we can achieve 𝑔 𝑦 ≤ 𝑑 is

equivalent to finding a point in the convex set Xc = 𝑌 ∩ {𝑦: 𝑔 𝑦 ≤ 𝑑}

• If we can solve this feasibility problem, we can

use binary search to approximate the optimal value.

Cutting-plane Oracles

• Let 𝑌 be a convex set. Given a point 𝑦0 ∉ 𝑌, a

cutting-plane oracle returns a hyperplane 𝐼 passing through 𝑦0 such that 𝑌 is entirely on

• ne side of 𝐼.
• Intuition for obtaining a cutting-plane oracle: If

𝑦0 ∉ 𝑌 then there is a constraint 𝑦0 violates. This constraint is of the form 𝑔 𝑦0 < 𝑑 where 𝑔 is convex. 𝑌 must be inside the half-space 𝛂𝑔 ⋅ 𝑦 − 𝑦0 ≤ 0

Ellipsoid Method Sketch

• Algorithm: Let 𝑌 be the feasible set
• 1. Keep track of an ellipsoid 𝑇 containing 𝑌
• 2. At each step, query center 𝑑 of 𝑇
• 3. If 𝑑 ∈ 𝑌, output 𝑑. Otherwise, cutting-plane oracle

gives a hyperplane 𝐼 passing through c and 𝑌 is on

• ne side of 𝐼. Use 𝐼 to find a smaller ellipsoid 𝑇′.
• Initial Guarantees:

1. 𝑌 ⊆ 𝐶(𝑆) where 𝐶 𝑆 = {𝑦 ∈ 𝑆𝑜: 𝑦 ≤ 𝑆} 2. 𝑌 contains a ball of radius 𝑠.

• Note: Not polynomial time if 𝑆

𝑠 is 2(𝑜𝜕(1))

References

• L. E. Ghaoui. Lecture 13: SDP Duality. Berkeley EE227A Convex Optimization and
• Applications. 2008
• [GW95] M. X. Goemans and D. P. Williamson. Improved Approximation Algorithms

for Maximum Cut and Satisfiability Problems Using Semidefinite Programming. J. ACM, 42(6):1115–1145, 1995.

• [O’Don17] R. O’Donnell. SOS is not Obviously Automatizable, Even Approximately.

ITCS 2017.

• [Sla50] M. Slater. Lagrange Multipliers Revisited. Cowles Commission Discussion

Paper: Mathematics 403, 1950.