Lecture 9: SOS Lower Bound for Knapsack Lecture Outline Part I: - PowerPoint PPT Presentation

Lecture 9: SOS Lower Bound for Knapsack

Lecture Outline • Part I: Knapsack Eqations and Pseudo- expectation Values • Part II: Johnson Scheme • Part III: Proving PSDness • Part IV: Further Work

Part I: Knapsack Eqations and Pseudo-expectation Values

Knapsack Problem • Knapsack problem: Given weights 𝑥 1 , … , 𝑥 𝑜 and a knapsack with total capacity 𝐷 , what is the maximum weight that can be carried? • In other words, defining 𝑥 𝐽 = σ 𝑗∈𝐽 𝑥 𝑗 for each subset 𝐽 ⊆ [1, 𝑜] , what is max{𝑥 𝐽 : 𝐽 ⊆ 1, 𝑜 , 𝑥 𝐽 ≤ 𝐷} ? • Here we’ll consider the simple case where 𝑥 𝑗 = 1 for all 𝑗 and 𝐷 ∈ [0, 𝑜] is not an integer. • Answer is 𝐷 , but can SOS prove it?

Knapsack Equations • Want 𝑦 𝑗 = 1 if 𝑗 ∈ 𝐽 and 𝑦 𝑗 = 0 otherwise. • Knapsack equations: 2 = 𝑦 𝑗 ∀𝑗, 𝑦 𝑗 1. 𝑜 σ 𝑗=1 𝑦 𝑗 = 𝑙 2. • Here we take 𝑙 ∈ [0, 𝑜] to be a non-integer. 𝑜 • Equations are infeasible because σ 𝑗=1 𝑦 𝑗 ∈ ℤ

SOS Lower Bound for Knapsack • Theorem[Gri01]: SOS needs degree at least 2min{𝑙, 𝑜 − 𝑙} to refute these equations • We’ll follow the presentation of [MPW15] and show a lower bound of min{𝑙, 𝑜 − 𝑙} • Note: This presentation was already in the retracted paper [MW13]

Review: SOS Lower Bound Strategy • Recall: To prove an SOS lower bound, we generally do the following: 1. Come up with pseudo-expectation values ෨ 𝐹 which obey the required linear equations 2. Show that the moment matrix 𝑁 is PSD • Here we’ll use symmetry for part 1 and some combinatorics for part 2.

Pseudo-expectation Values • Define 𝑦 𝐽 = ς 𝑗∈𝐽 𝑦 𝑗 𝑜 • ∀𝐽, (σ 𝑘=1 𝑦 𝑘 )𝑦 𝐽 = σ 𝑘∈𝐽 𝑦 𝑘 𝑦 𝐽 + σ 𝑘∉𝐽 𝑦 𝑘 𝑦 𝐽 = 𝑙𝑦 𝐽 • If ෨ 𝐹[𝑦 𝐽 ] only depends on |𝐽| , ∀𝐽, 𝑘 ∉ 𝐽, |𝐽| ෨ 𝐹[𝑦 𝐽 ] + 𝑜 − |𝐽| ෨ 𝐹[𝑦 𝐽∪{𝑘} ] = 𝑙 ෨ 𝐹[𝑦 𝐽 ] 𝐹 𝑦 𝐽∪{𝑘} = 𝑙 − |𝐽| ∀𝐽, 𝑘 ∉ 𝐽, ෨ ෨ 𝐹[𝑦 𝐽 ] 𝑜 − |𝐽| 𝑙 𝐹[𝑦 𝐽 ] = 𝑙 𝑙−1 …(𝑙−|𝐽|+1) • Thus, ෨ |𝐽| 𝑜 𝑜−1 …(𝑜−|𝐽|+1) = 𝑜 |𝐽|

Viewing ෨ 𝐹 as an Expectation 𝑙 • ෨ |𝐽| 𝐹[𝑦 𝐽 ] = 𝑜 |𝐽| • Could have predicted this as follows: If we had 𝑜 a set 𝐵 of 1s of size 𝑙 , then of the |𝐽| 𝑙 possible sets of size |𝐽| , |𝐽| of them will be contained in 𝐵 . • Bayesian view: ෨ 𝐹[𝑦 𝐽 ] is the expected value of 𝑦 𝐽 given what we can compute (in SOS). • Here it is a true expectation if 𝑙 ∈ ℤ

Reduction to Multilinear Indices • Recall from last lecture: If we have 2 = 𝑦 𝑗 or 𝑦 𝑗 2 = 1 , it is sufficient constraints 𝑦 𝑗 to consider ෨ 𝐹[𝑕 2 ] for multilinear 𝑕 . • Reason: For every polynomial 𝑕 , there is a multilinear polynomial 𝑕′ with deg 𝑕 ′ ≤ 𝐹 𝑕 ′2 = ෨ deg(𝑕) such that ෨ 𝐹[𝑕 2 ] . • Thus, it is sufficient to consider the restriction of 𝑁 to multilinear indices.

𝑒 Reduction to Degree 2 Indices • Lemma: If we also have the constraint 𝑜 σ 𝑗=1 𝑦 𝑗 = 𝑙 , for every polynomial 𝑕 of degree 𝑒 at most 2 , there is a homogeneous, multilinear 𝑒 polynomial 𝑕′ of degree exactly 2 such that 𝐹 𝑕 ′2 = ෨ ෨ 𝐹[𝑕 2 ] . • Proof idea: Use the following reductions: 2 𝑔 = 𝑦 𝑗 𝑔 1. ∀𝑗, 𝑦 𝑗 𝑜 σ 𝑗∉𝐽 𝑦 𝐽∪{𝑗} 𝑒 ∀𝐽 ⊆ 1, 𝑜 : 𝐽 < 2 , 𝑦 𝐽 = 2. . To see this, 𝑙−|𝐽| 𝑜 𝑦 𝐽∪{𝑗} 𝑜 note that σ 𝑗=1 𝑦 𝑗 𝑦 𝐽 = 𝑙𝑦 𝐽 = 𝐽 𝑦 𝐽 + σ 𝑗∉𝐽

𝑒 Reduction to Degree 2 Indices • Corollary: To prove that 𝑁 ≽ 0 , it is sufficient to prove that the submatrix of 𝑁 with multilinear 𝑒 entries of degree exactly 2 is PSD.

Part II: Johnson Scheme

Johnson Scheme • Algebra of matrices 𝑁 such that: 1. The rows and columns of 𝑁 are indexed by subsets of [1, 𝑜] of size 𝑠 for some 𝑠 . 2. 𝑁 𝐽𝐾 only depends on |𝐽 ∩ 𝐾| • Equivalently, the Johnson Scheme is the algebra of matrices which are invariant under permutations of [1, 𝑜] . • Claim: The matrices 𝑁 in the Johnson scheme are all symmetric and commute with each other

Johnson Scheme Claim Proof • Claim: For all 𝐵, 𝐶 in the Johnson scheme, 𝐵 𝑈 = 𝐵 , 𝐵𝐶 is in the Johnson scheme as well, and 𝐵𝐶 = 𝐶𝐵 • Proof: For the first part, ∀𝐽, 𝐾 , 𝐵 𝐽𝐾 = 𝐵 𝐾𝐽 because 𝐽 ∩ 𝐾 = |𝐾 ∩ 𝐽| . For the second part, 𝐵𝐶 𝐽𝐿 = σ 𝐾∈ 𝑜 𝑠 𝐵 𝐽𝐾 𝐶 𝐾𝐿 . Now observe that for any permutation 𝜏 of [1, 𝑜] , 𝐵𝐶 𝐽𝐿 = σ 𝐾∈ 𝑜 𝑠 𝐵 𝐽𝐾 𝐶 𝐾𝐿 = σ 𝐾∈ 𝑜 𝑠 𝐵 𝜏(𝐽)𝐾 𝐶 𝐾𝜏(𝐿) = 𝐵𝐶 𝜏 𝐽 𝜏(𝐿) • For the third part, 𝐵𝐶 = 𝐵𝐶 𝑈 = 𝐶 𝑈 𝐵 𝑈 = 𝐶𝐵

Johnson Scheme Picture for 𝑠 = 1 1 2 3 4 5 6 1 𝐽 ∩ 𝐾 = 1 2 3 4 𝐽 ∩ 𝐾 = 0 5 6

Johnson Scheme Picture for 𝑠 = 2 12 13 14 15 16 23 24 25 26 34 35 36 45 46 56 12 13 14 15 𝐽 ∩ 𝐾 = 2 16 23 24 𝐽 ∩ 𝐾 = 1 25 26 34 𝐽 ∩ 𝐾 = 0 35 36 45 46 56

Basis for Johnson Scheme • Natural basis for Johnson Scheme: Define 𝑜 𝑠 × 𝑜 𝑠 to have entries 𝐸 𝑏 𝐽𝐾 = 1 if 𝐸 𝑏 ∈ ℝ 𝐽 ∩ 𝐾 = 𝑏 and 𝐸 𝑗 𝐽𝐾 = 0 if 𝐽 ∩ 𝐾 ≠ 𝑏 . • Easy to express matrices in this basis, but not so easy to show PSDness

PSD Basis for Johnson Scheme • Want a convenient basis of PSD matrices. • Building block: Define 𝑤 𝐵 so that 𝑤 𝐵 𝐽 = 1 if A ⊆ 𝐽 and 0 otherwise • PSD basis for Johnson Scheme: Define 𝑄 𝑏 ∈ 𝑜 𝑠 × 𝑜 𝑈 𝑠 to be 𝑄 𝑏 = σ 𝐵⊆ 1,𝑜 : 𝐵 =𝑏 𝑤 𝐵 𝑤 𝐵 ℝ |𝐽∩𝐾| • 𝑄 𝑏 has entries 𝑄 𝑏 𝐽𝐾 = because 𝑏 𝑈 = 1 if and only if 𝐵 ⊆ 𝐽 ∩ 𝐾 and there 𝑤 𝐵 𝑤 𝐵 are |𝐽∩𝐾| such 𝐵 ⊆ [1, 𝑜] of size 𝑏 . 𝑏

Basis for 𝑠 = 1 1 2 3 4 5 6 1 𝐸 0 𝐽𝐾 = 0 2 3 4 𝐸 0 𝐽𝐾 = 1 5 6

Basis for 𝑠 = 1 1 2 3 4 5 6 1 𝐸 1 𝐽𝐾 = 1 2 3 4 𝐸 1 𝐽𝐾 = 0 5 6

PSD Basis for 𝑠 = 1 1 2 3 4 5 6 1 1 𝑄 0 𝐽𝐾 = = 1 2 0 3 4 0 𝑄 0 𝐽𝐾 = = 1 5 0 6

PSD Basis for 𝑠 = 1 1 2 3 4 5 6 1 1 𝑄 1 𝐽𝐾 = = 1 2 1 3 4 0 𝑄 1 𝐽𝐾 = = 0 5 1 6

PSD Basis for 𝑠 = 2 12 13 14 15 16 23 24 25 26 34 35 36 45 46 56 12 2 13 𝑄 0 𝐽𝐾 = = 1 14 0 15 16 1 23 𝑄 0 𝐽𝐾 = = 1 0 24 25 26 0 𝑄 0 𝐽𝐾 = = 1 34 0 35 36 45 46 56

PSD Basis for 𝑠 = 2 12 13 14 15 16 23 24 25 26 34 35 36 45 46 56 12 2 13 𝑄 1 𝐽𝐾 = = 2 14 1 15 16 1 23 𝑄 1 𝐽𝐾 = = 1 1 24 25 26 0 𝑄 1 𝐽𝐾 = = 0 34 1 35 36 45 46 56

PSD Basis for 𝑠 = 2 12 13 14 15 16 23 24 25 26 34 35 36 45 46 56 12 2 13 𝑄 2 𝐽𝐾 = = 1 14 2 15 16 1 23 𝑄 2 𝐽𝐾 = = 0 2 24 25 26 0 𝑄 2 𝐽𝐾 = = 0 34 2 35 36 45 46 56

Shifting Between Bases • Basis for Johnson Scheme: 𝐸 𝑏 𝐽𝐾 = 𝜀 𝑏|𝐽∩𝐾| |𝐽∩𝐾| • PSD Basis for Johnson Scheme : 𝑄 𝑏 𝐽𝐾 = 𝑏 • Want to shift between bases. • Lemma: 𝑐 𝑠 𝑏 = σ 𝑐=𝑏 𝑄 𝑏 𝐸 𝑐 1. −1 𝑐−𝑏 𝑐 𝑠 𝐸 𝑏 = σ 𝑐=𝑏 𝑏 𝑄 𝑐 2. • First part is trivial, second part follows from a bit of combinatorics.

Shifting Between Bases Proof • Lemma: 𝑐 𝑠 𝑏 = σ 𝑐=𝑏 𝑄 𝑏 𝐸 𝑐 1. −1 𝑐−𝑏 𝑐 𝑠 𝐸 𝑏 = σ 𝑐=𝑏 𝑏 𝑄 𝑐 2. • Proof of the second part: Observe that 𝑏 ′ −1 𝑐−𝑏 𝑐 𝑠 −1 𝑐−𝑏 𝑐 𝑠 σ 𝑐=𝑏 𝑏 𝑄 𝑐 = σ 𝑏 ′ =𝑏 σ 𝑐=𝑏 𝑏 𝐸 𝑐 • Must show that for all 𝑏 ′ ≥ 𝑏 , 𝑏 ′ 𝑐 𝑏′ −1 𝑐−𝑏 σ 𝑐=𝑏 𝑏 = 𝜀 𝑏 ′ 𝑏 𝑐 • In-class exercise: Prove this

Shifting Between Bases Proof 𝑏 ′ 𝑏 ′ 𝑐 −1 𝑐−𝑏 • Need to show: σ 𝑐=𝑏 𝑏 = 𝜀 𝑏 ′ 𝑏 𝑐 • Answer: Observe that 𝑏 ′ !𝑐! 𝑏 ′ ! 𝑏 ′ −𝑏 ! 𝑏 ′ 𝑐 𝑏 = 𝑐!(𝑏 ′ −𝑐)!𝑏!(𝑐−𝑏)! = 𝑐 𝑏! 𝑏 ′ −𝑏 ! 𝑏 ′ −𝑐 ! 𝑐−𝑏 ! • Our expression is equal to 𝑏 ′ ! where 𝑛 = 𝑏 ′ − 𝑏 𝑛 𝑛 −1 𝑘 𝑏! 𝑏 ′ −𝑏 ! σ 𝑘=0 𝑘 𝑛 , 𝑛 𝑛 −1 𝑘 • Now note that σ 𝑘=0 = 1 + −1 𝑘 which equals 1 if 𝑛 = 0 and 0 if 𝑛 > 0 .

Part III: Proving PSDness

Decomposition of 𝑁 𝑙 • Recall that ෨ |𝐽| 𝐹[𝑦 𝐽 ] = 𝑜 |𝐽| 𝑙 |𝐽∪𝐾| • 𝑁 𝐽𝐾 = 𝑜 |𝐽∪𝐾| 𝑙 2𝑠−𝑏 𝑠 • Thus, 𝑁 = σ 𝑏=0 2𝑠−𝑏 𝐸 𝑏 𝑜

PSD Decomposition • To prove 𝑁 ≽ 0 , it is sufficient to express 𝑁 as a non-negative linear combination of the matrices 𝑄 𝑏 .