Lecture 3: Semidefinite Programming Lecture Outline Part I: - PowerPoint PPT Presentation

Lecture 3: Semidefinite Programming

Lecture Outline • Part I: Semidefinite programming, examples, canonical form, and duality • Part II: Strong Duality Failure Examples • Part III: Conditions for strong duality • Part IV: Solving convex optimization problems

Part I: Semidefinite Programming, Examples, Canonical Form, and Duality

Semidefinite Programming • Semidefinite Programming: Want to optimize a linear function, can now have matrix positive semidefiniteness (PSD) constraints as well as linear equalities and inequalities • Example: Maximize 𝑦 subject to 1 𝑦 2 + 𝑦 ≽ 0 𝑦 • Answer: 𝑦 = 2

Example: Goemans-Williamson • First approximation algorithm using a semiefinite program (SDP) • MAX-CUT reformulation: Have a variable 𝑦 𝑗 for each vertex i, will set 𝑦 𝑗 = ±1 depending on which side of the cut 𝑗 is on. 1−𝑦 𝑗 𝑦 𝑘 • Want to maximize σ 𝑗,𝑘:𝑗<𝑘, 𝑗,𝑘 ∈𝐹(𝐻) where 2 𝑦 𝑗 ∈ {−1, +1} for all 𝑗 .

Example: Goemans-Williamson • Idea: Take 𝑁 so that 𝑁 𝑗𝑘 = 𝑦 𝑗 𝑦 𝑘 1−𝑁 𝑗𝑘 • Want to maximize σ 𝑗,𝑘:𝑗<𝑘, 𝑗,𝑘 ∈𝐹(𝐻) where 2 𝑁 𝑗𝑗 = 1 for all 𝑗 and 𝑁 = xx T . 1−𝑁 𝑗𝑘 • Relaxation: Maximize σ 𝑗,𝑘:𝑗<𝑘, 𝑗,𝑘 ∈𝐹(𝐻) 2 subject to ∀𝑗, 𝑁 𝑗𝑗 = 1 1. 𝑁 ≽ 0 2.

Example: SOS Hierarchy • Goal: Minimize a polynomial ℎ(𝑦 1 , … , 𝑦 𝑜 ) subject to constraints 𝑡 1 𝑦 1 , … , 𝑦 𝑜 = 0 , 𝑡 2 𝑦 1 , … , 𝑦 𝑜 = 0 , etc. • Relaxation: Minimize Ẽ[ℎ] where Ẽ is a linear map from polynomials of degree ≤ 𝑒 to ℝ satisfying: Ẽ 1 = 1 1. 2. Ẽ 𝑔𝑡 𝑗 = 0 whenever deg 𝑔 + deg 𝑡 𝑗 ≤ 𝑒 3. Ẽ 𝑕 2 ≥ 0 whenever deg 𝑕 ≤ 𝑒 2

The Moment Matrix 𝑟 p,q are monomials of 𝑒 degree at most 2 . 𝑞 Ẽ[p q] 𝑁 𝑒 • Indexed by monomials of degree ≤ 2 • 𝑁 𝑞𝑟 = ෨ 𝐹[𝑞𝑟] 𝑒 • Each 𝑕 of degree ≤ 2 corresponds to a vector 𝐹 𝑕 2 = 𝑕 𝑈 𝑁𝑕 • ෨ 𝐹 𝑕 2 ≥ 0 ⇔ 𝑁 is PSD • ∀𝑕, ෨

Semidefinite Program for SOS • Program: Minimize Ẽ[ℎ] where Ẽ satisfies: Ẽ 1 = 1 1. 2. Ẽ 𝑔𝑡 𝑗 = 0 whenever deg 𝑔 + deg 𝑡 𝑗 ≤ 𝑒 𝑒 3. Ẽ 𝑕 2 ≥ 0 whenever deg 𝑕 ≤ 2 • Expressible as semidefinite program using 𝑁 : 1. ∀ℎ, Ẽ[ℎ] is a linear function of entries of 𝑁 2. Constraints that Ẽ 1 = 1 and Ẽ 𝑔𝑡 𝑗 = 0 give linear constraints on entries of 𝑁 𝑒 3. Ẽ 𝑕 2 ≥ 0 whenever deg 𝑕 ≤ 2 ⬄𝑁 ≽ 0 4. Also have SOS symmetry constraints

SOS symmetry • Define 𝑦 𝐽 = ς 𝑗∈𝐽 𝑦 𝑗 where 𝐽 is a multi-set • SOS symmetry constraints: 𝑁 𝑦 𝐽 𝑦 𝐾 = 𝑁 𝑦 𝐽′ 𝑦 𝐾′ whenever 𝐽 ∪ 𝐾 = 𝐽 ′ ∪ 𝐾′ • Example: 𝑦 2 𝑧 2 1 𝑦 𝑧 𝑦𝑧 1 1 𝑏 𝑐 𝑑 𝑒 𝑓 𝑦 𝑏 𝑑 𝑒 𝑔 𝑕 ℎ 𝑧 𝑐 𝑒 𝑓 𝑕 ℎ 𝑗 𝑦 2 𝑑 𝑔 𝑕 𝑘 𝑙 𝑚 𝑦𝑧 𝑒 𝑕 ℎ 𝑙 𝑚 𝑛 𝑧 2 𝑓 ℎ 𝑗 𝑚 𝑛 𝑜

Canonical Form 𝑗𝑘 = 𝑢𝑠(𝑌𝑍 𝑈 ) to be • Def: Define 𝑌⦁𝑍 = σ 𝑗,𝑘 𝑌 𝑗𝑘 𝑍 the entry-wise dot product of 𝑌 and 𝑍 • Canonical form: Minimize 𝐷⦁𝑌 subject to 1. ∀𝑗, 𝐵 𝑗 ⦁𝑌 = 𝑐 𝑗 where the 𝐵 𝑗 are symmetric 𝑌 ≽ 0 2.

Putting Things Into Canonical Form • Canonical form: Minimize 𝐷⦁𝑌 subject to ∀𝑗, 𝐵 𝑗 ⦁𝑌 = 𝑐 𝑗 where the 𝐵 𝑗 are symmetric 1. 𝑌 ≽ 0 2. • Ideas for obtaining canonical form: 𝑌 ≽ 0, 𝑍 ≽ 0⬄ 𝑌 0 𝑍 ≽ 0 1. 0 2. Slack variables: 𝐵 𝑗 ⦁𝑌 ≤ 𝑐 𝑗 ⬄𝐵 𝑗 ⦁𝑌 = 𝑐 𝑗 + 𝑡 𝑗 , 𝑡 𝑗 ≥ 0 3. Can enforce 𝑡 𝑗 ≥ 0 by putting 𝑡 𝑗 on the diagonal of 𝑌

Semidefinite Programming Dual • Primal: Minimize 𝐷⦁𝑌 subject to ∀𝑗, 𝐵 𝑗 ⦁𝑌 = 𝑐 𝑗 where the 𝐵 𝑗 are symmetric 1. 𝑌 ≽ 0 2. • Dual: Maximize σ 𝑗 𝑧 𝑗 𝑐 𝑗 subject to σ 𝑗 𝑧 𝑗 𝐵 𝑗 ≼ 𝐷 1. • Value for dual lower bounds value for primal: 𝐷⦁𝑌 = 𝐷 − σ 𝑗 𝑧 𝑗 𝐵 𝑗 ⦁𝑌 + σ 𝑗 𝑧 𝑗 𝐵 𝑗 ⦁𝑌 ≥ σ 𝑗 𝑧 𝑗 𝑐 𝑗

Explanation for Duality • Primal: Minimize 𝐷⦁𝑌 subject to ∀𝑗, 𝐵 𝑗 ⦁𝑌 = 𝑐 𝑗 where the 𝐵 𝑗 are symmetric 1. 𝑌 ≽ 0 2. • = min 𝐷⦁𝑌 + σ 𝑗 𝑧 𝑗 𝑐 𝑗 − 𝐵 𝑗 ⦁𝑌 𝑌≽0 max 𝑧 • = max 𝑌≽0 σ 𝑗 𝑧 𝑗 𝑐 𝑗 + (𝐷 − σ 𝑗 𝑧 𝑗 𝐵 𝑗 )⦁𝑌 min 𝑧 • Dual: Maximize σ 𝑗 𝑧 𝑗 𝑐 𝑗 subject to σ 𝑗 𝑧 𝑗 𝐵 𝑗 ≼ 𝐷 1.

In class exercise: SOS duality • Exercise: What is the dual of the semidefinite program for SOS? • Primal: Minimize Ẽ[ℎ] where Ẽ is a linear map from polynomials of degree ≤ 𝑒 to ℝ such that: Ẽ 1 = 1 1. 2. Ẽ 𝑔𝑡 𝑗 = 0 whenever deg 𝑔 + deg 𝑡 𝑗 ≤ 𝑒 3. Ẽ 𝑕 2 ≥ 0 whenever deg 𝑕 ≤ 𝑒 2

In class exercise solution • Definition: Given a symmetric matrix 𝑅 indexed by monomials 𝑦 𝐽 , we say that 𝑅 represents the polynomial 𝑞 𝑅 = σ 𝐾 σ 𝐽,𝐽 ′ :𝐽∪𝐽 ′ =𝐾 𝑅 𝑦 𝐽 𝑦 𝐽′ 𝑦 𝐾 • Proposition 1: If 𝑅 ≽ 0 then 𝑞 𝑅 is a sum of squares. Conversely, if 𝑞 is a sum of squares then ∃𝑅 ≽ 0: 𝑞 = 𝑞 𝑅 • Proposition 2: If 𝑁 is a moment matrix then 𝑁⦁𝑅 = Ẽ 𝑞 𝑅

In class exercise solution continued • 𝐷 = 𝐼 where 𝑞 𝐼 = ℎ • Constraint that ෨ 𝐹 1 = 1 gives matrix 1 0 ⋯ 𝐵 1 = and 𝑐 1 = 1 0 0 ⋯ ⋮ ⋮ ⋱ • Constraints that ෨ 𝐹 𝑔𝑡 𝑗 = 0 give matrices 𝐵 𝑘 where 𝑞 𝐵 𝑘 = 𝑔𝑡 𝑗 and 𝑐 𝑘 = 0 • SOS symmetry constraints give matrices 𝐵 𝑙 such that 𝑞 𝐵 𝑙 = 0 and 𝑐 𝑙 = 0

In class exercise solution continued • Recall dual: Maximize σ 𝑗 𝑧 𝑗 𝑐 𝑗 subject to σ 𝑗 𝑧 𝑗 𝐵 𝑗 ≼ 𝐷 1. • Here: Maximize 𝑑 such that 𝑑𝐵 1 + σ 𝑘 𝑧 𝑘 𝐵 𝑘 + σ 𝑙 𝑧 𝑙 𝐵 𝑙 ≼ 𝐼 • This is the answer, but let’s simplify it into a more intuitive form. • Let 𝑅 = 𝐼 − 𝑑𝐵 1 + σ 𝑘 𝑧 𝑘 𝐵 𝑘 + σ 𝑙 𝑧 𝑙 𝐵 𝑙 • 𝑅 ≽ 0

In class exercise solution continued • 𝐼 = 𝑑𝐵 1 + σ 𝑘 𝑧 𝑘 𝐵 𝑘 + σ 𝑙 𝑧 𝑙 𝐵 𝑙 + 𝑅 , 1 0 ⋯ A 1 = , 𝑅 ≽ 0 0 0 ⋯ ⋮ ⋮ ⋱ • View everything in terms of polynomials. • 𝑞 𝐼 = ℎ , 𝑞 𝐵 1 = 1 , 𝑞 (σ 𝑘 𝑧 𝑘 𝐵 𝑘 ) = σ 𝑗 𝑔 𝑗 𝑡 𝑗 for some 2 for some 𝑕 𝑘 𝑗 , 𝑞 σ 𝑙 𝑧 𝑙 𝐵 𝑙 = 0 , 𝑞 𝑅 = σ 𝑘 𝑕 𝑘 𝑔 2 • ℎ = 𝑑 + σ 𝑗 𝑔 𝑗 𝑡 𝑗 + σ 𝑘 𝑕 𝑘

In class exercise solution continued • Simplified Dual: Maximize 𝑑 such that 2 ℎ = 𝑑 + σ 𝑗 𝑔 𝑗 𝑡 𝑗 + σ 𝑘 𝑕 𝑘 • This is a Positivstellensatz proof that ℎ ≥ 𝑑 (see Lectures 1 and 5)

Part II: Strong Duality Failure Examples

Strong Duality Failure • Unlike linear programming, it is not always the case that the values of the primal and dual are the same. • However, almost never an issue in practice, have to be trying in order to break strong duality. • We’ll give this issue its due here then ignore it for the rest of the seminar.

Non-attainability Example • Primal: Minimize 𝑦 2 subject to 𝑦 1 1 𝑦 2 ≽ 0 1 • Dual: Maximize 2𝑧 subject to 0 𝑧 0 ≼ 0 0 1. 𝑧 0 1 • Duality demonstration: 1 − 0 𝑧 ⦁ 𝑦 1 1 0 0 𝑦 2 = 𝑦 2 − 2𝑧 ≥ 0 𝑧 0 1 0 • Dual has optimal value 0 , this is not attainable in the primal (we can only get arbitrarily close)

Duality Gap Example • Primal: Minimize 𝑦 2 + 1 subject to 1 + 𝑦 2 0 0 0 𝑦 1 𝑦 2 ≽ 0 0 𝑦 2 0 • Dual: Maximize 2𝑧 subject to 2𝑧 𝑧 1 𝑧 2 1 0 0 𝑧 1 0 −𝑧 ≼ 0 0 0 𝑧 2 −𝑧 𝑧 3 0 0 0 • Duality demonstration 2𝑧 𝑧 1 𝑧 2 1 + 𝑦 2 0 0 1 0 0 𝑧 1 0 −𝑧 0 𝑦 1 𝑦 2 − ⦁ = 𝑦 2 + 1 − 2𝑧 ≥ 0 0 0 0 𝑧 2 −𝑧 𝑧 3 0 𝑦 2 0 0 0 0

Duality Gap Example • Primal: Minimize 𝑦 2 + 1 subject to 1 + 𝑦 2 0 0 0 𝑦 1 𝑦 2 ≽ 0 0 𝑦 2 0 • Has optimal value 1 as we must have 𝑦 2 = 0 • Dual: Maximize 2𝑧 subject to 2𝑧 𝑧 1 𝑧 2 1 0 0 𝑧 1 0 −𝑧 ≼ 0 0 0 𝑧 2 −𝑧 𝑧 3 0 0 0 • Has optimal value 0 as we must have 𝑧 = 0 . • Note: This example was taken from Lecture 13, EE227A at Berkeley given on October 14, 2008.

Part III: Conditions for strong duality

Sufficient Strong Duality Conditions • How can we rule out such a gap? • Slater’s Condition (informal): – If the feasible region for the primal has an interior point (in the subspace defined by the linear equalities) then the duality gap is 0 . Moreover, if the optimal value is finite then it is attainable in the dual. • Also sufficient if either the primal or the dual is feasible and bounded (i.e. any very large point violates the constraints)