Lecture 14: Planted Sparse Vector Lecture Outline Part I: Planted - - PowerPoint PPT Presentation

Lecture 14: Planted Sparse Vector

Lecture Outline

• Part I: Planted Sparse Vector and 2 to 4 Norm
• Part II: SOS and 2 to 4 Norm on Random

Subspaces

• Part III: Warmup: Showing 𝑦

≈ 1

• Part IV: 4-Norm Analysis
• Part V: SOS-symmetry to the Rescue
• Part VI: Observations and Loose Ends
• Part VII: Open Problems

Part I: Planted Sparse Vector and 2 to 4 Norm

• Planted Sparse Vector problem: Given the span
• f 𝑒 − 1 random vectors in ℝ𝑜 and one unit

vector 𝑤 ∈ ℝ𝑜 of sparsity 𝑙, can we recover 𝑤?

• More precisely, let 𝑊 be an n × 𝑒 matrix where:

1. 𝑒 − 1 columns of 𝑊 are vectors of length ≈ 1 chosen randomly from ℝ𝑜

• 2. One column of 𝑊 is a unit vector 𝑤 with ≤ 𝑙

nonzero entries.

• Given 𝑊𝑆 where 𝑆 is an arbitrary invertible 𝑒 × 𝑒

matrix, can we recover 𝑤?

Planted Sparse Vector

• Theorem 1.4 [BKS14]: There is a constant 𝑑 > 0

and an algorithm based on constant degree SOS such that for every vector 𝑤0 supported on at most 𝑑𝑜 ⋅ min{1, 𝑜/𝑒2} coordinates, if 𝑤1, … , 𝑤𝑒 are chosen independently at random from the Gaussian distribution on 𝑆𝑜, then given any basis for 𝑊 = 𝑡𝑞𝑏𝑜{𝑤0, … , 𝑤𝑒}, the algorithm

• utputs an 𝜗-approximation to 𝑤0 in

𝑞𝑝𝑚𝑧(𝑜, log(1/𝜗)) time.

Theorem Statement

• Random Distribution: We choose each entry of 𝑊

independently from 𝑂 0,

1 𝑜 , the normal

distribution with mean 0 and standard deviation

1 𝑜

• We then choose 𝑆 to be a random 𝑒 × 𝑒
• rthogonal/rotation matrix and take 𝑊𝑆 to be
• ur input matrix.

Random Distribution

• Remark: If 𝑆 is any 𝑒 × 𝑒 orthogonal/rotation

matrix then 𝑊𝑆 can also be chosen by taking each entry of 𝑊 independently from 𝑂 0,

1 𝑜 .

• Idea: Each row of 𝑊 comes from a multivariate

normal distribution with covariance matrix 1

𝑜 𝐽𝑒𝑒,

which is invariant under rotations

Random Distribution

• Planted Distribution: We choose each entry of

the first 𝑒 − 1 columns of 𝑊 independently from 𝑂 0,

1 𝑜 . The last column of 𝑊 is our sparse unit

vector 𝑤.

• We then choose 𝑆 to be a random 𝑒 × 𝑒
• rthogonal/rotation matrix and take 𝑊𝑆 to be
• ur input matrix.

Planted Distribution

• We ask for an 𝑦 such that

1. 𝑊𝑆𝑦 = 1 2. 𝑊𝑆𝑦 is k-sparse (i.e. at most 𝑙 indices of 𝑊𝑆𝑦 are nonzero).

• Hard to search for 𝑦 such that 𝑊𝑆𝑦 is k-sparse,

so we’ll need to relax the problem.

Output

• Key idea: All unit vectors have the same 2-norm.

However, sparse vectors will have higher 4-norm

• 4-norm for a 𝑙-sparse unit vector in ℝ𝑜 is at

least

4 k ⋅ 1

𝑙2 = 1

4 𝑙 (obtained by setting 𝑙

coordinates to

±1 𝑙 and the rest to 0)

• Relaxation Attempt #1: Search for an 𝑦 such that

1. 𝑊𝑆𝑦 = 1 2. 𝑊𝑆𝑦 4 ≥

1

4 𝑙

Distinguishing Sparse Vectors

• This is the 2 to 4 Norm Problem: Given a matrix

𝐵, find the vector 𝑦 which maximizes

𝐵𝑦 4 𝐵𝑦

2 to 4 Norm Problem

Part II: SOS and 2 to 4 Norm on Random Subspaces

• Unfortunately, the 2 to 4 norm problem is hard

[BBH+12]:

– NP-hard to obtain an approximation ratio of 1 +

1 𝑜𝑞𝑝𝑚𝑧𝑚𝑝𝑕(𝑜)

– Assuming ETH (the exponential time hypothesis), it is hard to approximate to within a constant factor.

• Thus, we’ll need to relax our problem further.

2 to 4 Norm Hardness

• Relaxation: Find ෨

𝐹 which respects the following constraints:

1. 𝑊𝑆𝑦 2 = σ𝑗=1

𝑜

𝑊𝑆𝑦 𝑗

2 = 1

2. 𝑊𝑆𝑦 4

4 = σ𝑗=1 𝑜

𝑊𝑆𝑦 𝑗

4 ≥ 1 𝑙

SOS Relaxation

• Constraints:

1. 𝑊𝑆𝑦 2 = σ𝑗=1

𝑜

𝑊𝑆𝑦 𝑗

2 = 1

2. 𝑊𝑆𝑦 4

4 = σ𝑗=1 𝑜

𝑊𝑆𝑦 𝑗

4 ≥ 1 𝑙

• To show that SOS distinguishes between the

random and planted distribution, it is sufficient to show that there is no ෨ 𝐹 which respects these constraints and has a PSD moment matrix 𝑁.

• Remark: Although the 2 to 4 Norm problem is

hard in general, we just need to show that SOS can approximate it on random subspaces.

Showing a Distinguishing Algorithm

• Given a random subspace, what is the expected

value of the largest 4-norm of a unit vector in the subspace?

• Trivial strategy: Any unit vector’s 4-norm is at

least

1

4 𝑜.

• Can we do better?

2 to 4 Norm on Random Subspaces

• Another strategy: Take a basis for this space and

take a linear combination which maximizes one coordinate (subject to having length 1)

• If we add together 𝑒 random vectors with entries

≈ ±

1 𝑜, w.h.p. the result will have norm ෩

Θ 𝑒 . Diving the resulting vector by ෩ Θ 𝑒 , the maximized entry will have magnitude ෩ Θ

𝑒 𝑜 ,

• ther entries will have magnitude ෩

O

1 𝑜

2 to 4 Norm on Random Subspaces

• Calling our final result 𝑥, w.h.p. the maximized

entry of 𝑥 contributes ෩ Θ

𝑒2 𝑜2 to 𝑥 4 4 while the

• ther entries contribute ෩

Θ

1 𝑜 .

• It turns out that this strategy is essentially
• ptimal. Thus, with high probability the

maximum 4-norm of a unit vector in a d- dimensional random subspace will be ෩ Θ max

𝑒 𝑜 , 1

4 𝑜

.

2 to 4 Norm on Random Subspaces

• Planted dist: max 4-norm ≥

1

4 𝑙

• Random dist: max 4-norm is ෩

Θ max

𝑒 𝑜 , 1

4 𝑜

.

• IF SOS can certify the upper bound for a

random subspace, this gives a distinguishing algorithm when max

𝑒 𝑜 , 1

4 𝑜 ≪

1

4 𝑙 (which

happens when 𝑒 ≤ 𝑜 and 𝑙 ≪ 𝑜 or when 𝑒 ≥ 𝑜 and k ≪ 𝑜2

𝑒2)

Algorithm Boundary

Part III: Warmup: Showing 𝑦 ≈ 1

• Take 𝑥 = 𝑊𝑆𝑦.
• We expect that 𝑥

≈ 𝑦 . Since we require that 𝑥 = 1, this implies that we will have 𝑦 ≈ 1

• To check that 𝑥

≈ 𝑦 , observe that 𝑥 2

2 =

𝑦𝑈 RV T VR x. Thus, it is sufficient to show that RV T VR ≈ 𝐽𝑒.

Showing 𝑦 ≈ 1

• We have that RV T VR ≈ 𝐽𝑒 because the

columns of 𝑊𝑆 are 𝑒 random unit vectors (where 𝑒 ≪ 𝑜) and are thus approximately

• rthonormal.
• However, we will use graph matrices to analyze

the 4-norm, so as a warm-up, let’s check that RV T VR ≈ 𝐽𝑒 using graph matrices.

Checking RV T VR ≈ 𝐽𝑒

• So far we have worked over {−1, +1}𝑛.
• How can we use graph matrices over 𝑂 0,1 𝑛?
• Key idea: Look at the Fourier characters over

𝑂(0,1).

Graph Matrices Over 𝑂(0,1)

• Inner product on 𝑂 0,1 : 𝑔 ⋅ 𝑕 =

𝐹𝑦∼𝑂 0,1 𝑔 𝑦 𝑕(𝑦)

• Fourier characters: Hermite polynomials
• The first few Hermite polynomials (up to

normalization) are as follows:

1. ℎ0 = 1 2. ℎ1 = 𝑦 3. ℎ2 = 𝑦2 − 1 4. ℎ3 = 𝑦3 − 3𝑦

• To normalize, divide ℎ𝑘 by 𝑘!

Fourier Analysis Over 𝑂(0,1)

• Graph matrices over {−1,1}𝑛: 1 and 𝑦 are a

basis for functions over {−1,1}. We represent 𝑦 by an edge and 1 by the absence of an edge

• Graph matrices over 𝑂 0,1 𝑛: {ℎ𝑘} are a basis

for functions over 𝑂(0,1). We represent ℎ𝑘 by a multi-edge with multiplicity 𝑘.

Graph Matrices Over 𝑂(0,1)

• For convenience, take 𝐵 =

𝑜𝑆𝑊 and think of the entries of 𝐵 as the input. Now each entry of 𝐵 is chosen independently from 𝑂(0,1)

• 𝐵𝑗𝑘 is represented by an edge from node 𝑗 to

node 𝑘.

• In class challenge: What is RV T VR in terms
• f graph matrices?

Graph Matrices for RV T VR

𝑘1 𝑗 𝑗 𝑘2

×

1 𝑜 𝑜 𝑜 𝑒 𝑒

• In class challenge answer:

Graph Matrices for RV T VR

𝑘1 𝑗 𝑗 𝑘2

×

1 𝑜 𝑜 𝑜 𝑒 𝑒

=

d n d 𝑉 𝑊 𝑘1 𝑘2 𝑗 1 𝑜

+

d n 𝑉 = 𝑊 𝑘 𝑗 d n 𝑉 = 𝑊 𝑘 𝑗 1 𝑜

+

2 𝑜

• Here we have two different types of vertices,
• ne for the rows of 𝐵 (which has 𝑜 possibilities)

and one for the columns of 𝐵 (which has 𝑒 possibilities)

• Can generalize the rough norm bounds to handle

multiple types of vertices (writing this up is on my to-do list)

Generalizing Rough Norm Bounds

• Generalized rough norm bounds:
• Each isolated vertex outside of 𝑉 and 𝑊

contributes a factor equal to the number of possibilities for that vertex

• Each vertex in the minimum separator (which

minimizes the total number of possibilities for its vertices) contributes nothing

• Each other vertex contributes a factor equal to

the square root of the number of possibilities for that vertex

Generalizing Rough Norm Bounds

Norm Bounds for RV T VR

𝑘1 𝑗 𝑗 𝑘2

×

1 𝑜 𝑜 𝑜 𝑒 𝑒

=

d n d 𝑉 𝑊 𝑘1 𝑘2 𝑗 1 𝑜

+

d n 𝑉 = 𝑊 𝑘 𝑗 d n 𝑉 = 𝑊 𝑘 𝑗 1 𝑜

+

2 𝑜

෨ 𝑃

𝑒 𝑜

෨ 𝑃

1 𝑜

= 𝐽𝑒𝑒

Part IV: 4-Norm Analysis

• We want to bound

1 𝑜 𝐵𝑦 4 4

• Take 𝐶 to be the matrix with entries 𝐶𝑗,(𝑘1,𝑘2) =

𝐵𝑗𝑘1𝐵𝑗𝑘2

• 1

𝑜 𝐵𝑦 4 4

= 1

𝑜2 𝑦 ⊗ 𝑦 𝑈𝐶𝑈𝐶(𝑦 ⊗ 𝑦)

• Can try to bound 𝐶𝑈𝐶

4-Norm Analysis

• Picture for 𝐶𝑈𝐶:

Picture for 𝐶𝑈𝐶

𝑘1 𝑗 𝑜 𝑒 𝑘1 𝑗 𝑜 𝑒 𝑗 𝑜 𝑘1 𝑘2 2

+ +

𝑘3 𝑗 𝑜 𝑒 𝑘3 𝑗 𝑜 𝑒 𝑗 𝑜 𝑘3 𝑘4 2

+ + ×

• If 𝑒 ≤

𝑜, the target norm bound on 𝐶𝑈𝐶 is ෩ O(𝑜), giving a bound of ෩ O

1 𝑜 on 𝑊𝑆𝑦 4 4.

• If 𝑒 ≥

𝑜, the target norm bound on 𝐶𝑈𝐶 is ෩ O 𝑒2 , giving a bound of ෩ O

𝑒2 𝑜2 on 𝑊𝑆𝑦 4 4

Targets

Casework

d n 𝑉 𝑘1 𝑗 d 𝑘2 d 𝑊 𝑘3 d 𝑘4 𝑗 𝑜 𝑘1 𝑘2

×

𝑗 𝑜 𝑘3 𝑘4

Norm ෨ 𝑃 𝑒 𝑜 if 𝑒 ≤ 𝑜, norm ෨ 𝑃 𝑒2 if 𝑒 ≥ 𝑜

Casework

d n 𝑉 𝑘1 𝑗 d 𝑘2 𝑊 d 𝑘4 𝑗 𝑜 𝑘1 𝑘2

×

𝑗 𝑜 𝑘1 𝑘4

Norm ෨ 𝑃 𝑒𝑜 Note: 0 or 2 edges between 𝑗 and 𝑘1

Casework

d n 𝑉 = 𝑊 𝑘1 𝑗 d 𝑘2 𝑗 𝑜 𝑘1 𝑘2

×

𝑗 𝑜 𝑘1 𝑘2

= 𝑜𝐽𝑒 + Norm ෨ 𝑃 𝑜 Note: 0 or 2 edges between 𝑗 and 𝑘1, 0 or 2 edges between 𝑗 and 𝑘2

Casework

d n 𝑉 𝑘1 𝑗 d 𝑊 𝑘3 d 𝑘4 𝑗 𝑜 𝑘1

×

𝑗 𝑜 𝑘3 𝑘4

Norm ෨ 𝑃 𝑜𝑒3 Too large! Note: 0 or 2 edges between 𝑗 and 𝑘1 Note: 0 or 2 edges between 𝑗 and 𝑘1

Casework

n 𝑉 𝑗 d 𝑊 𝑘1 d 𝑘4 𝑗 𝑜 𝑘1

×

𝑗 𝑜 𝑘1 𝑘4

Norm ෨ 𝑃 𝑒𝑜 Note: 1 or 3 edges between 𝑗 and 𝑘1 Note: 0 or 2 edges between 𝑗 and 𝑘1

Casework

d n 𝑉 𝑘1 𝑗 d 𝑊 𝑘2 𝑗 𝑜 𝑘1

×

𝑗 𝑜 𝑘2

Norm ෨ 𝑃 𝑜𝑒 Too large! Note: 0 or 2 edges between 𝑗 and 𝑘1 and between 𝑗 and 𝑘2 Note: 0 or 2 edges between 𝑗 and 𝑘1 and between 𝑗 and 𝑘2

Casework

n 𝑗 d 𝑉 = 𝑊 𝑘1 𝑗 𝑜 𝑘1

×

𝑗 𝑜 𝑘1

Turns out to be 3𝐽𝑒 + Norm ෨ 𝑃( 𝑜) Note: 0 or 2 edges between 𝑗 and 𝑘1 on both ends Note: 0,2, or 4 edges between 𝑗 and 𝑘1

• Most cases have sufficiently small norm.
• Two cases have a norm which is too large, so

norm bounds alone are not enough…

Summary

Part V: SOS-Symmetry to the Rescue

• Instead of looking at max

𝑥: 𝑥 =1𝑥𝑈𝐶𝑈𝐶𝑥, we only

need to upper bound max

𝑦: 𝑦 =1 𝑦 ⊗ 𝑦 𝑈𝐶𝑈𝐶(𝑦 ⊗ 𝑦)

• As far as 𝑦 ⊗ 𝑦 𝑈𝐶𝑈𝐶(𝑦 ⊗ 𝑦) is concerned, we

can rearrange indices in pieces of 𝐶𝑈𝐶.

Key Idea: Rearranging Indices

Rearranging Indices Case #1

d n 𝑉 𝑘1 𝑗 d 𝑊 𝑘2 𝑗 𝑜 𝑘1

×

𝑗 𝑜 𝑘2 d n 𝑉 = 𝑊 𝑘1 𝑗 d 𝑘2 𝑗 𝑜 𝑘1 𝑘2

×

𝑗 𝑜 𝑘1 𝑘2 rearranging indices

Rearranging Indices Case #2

rearranging indices d n 𝑉 𝑘1 𝑗 d 𝑊 𝑘3 d 𝑘4 𝑗 𝑜 𝑘1

×

𝑗 𝑜 𝑘3 𝑘4 d n 𝑉 𝑘1 𝑗 d 𝑘2 𝑊 d 𝑘4 𝑗 𝑜 𝑘1 𝑘2

×

𝑗 𝑜 𝑘1 𝑘4

• For the two cases whose norm is too high, their

norm can be reduced by rearranging indices.

• This proves the upper bound on

max

𝑦: 𝑦 =1 𝑦 ⊗ 𝑦 𝑈𝐶𝑈𝐶(𝑦 ⊗ 𝑦)

Effect of Rearranging Indices

Part VI: Observations and Loose Ends

• Note: This 4-norm analysis roughly

corresponds to p.33-37 of [BBH+12]

• Remark: When 𝑒 ≪

𝑜, with a slightly more careful analysis we can show that 𝑦 ⊗ 𝑦 𝑈𝐶𝑈𝐶 𝑦 ⊗ 𝑦 = 3 ± 𝑝 1 𝑦 2

4,

matching the results in [BBH+12].

Observations: 4-Norm Analysis

• How can we handle arbitrary 𝑆 rather than a

random orthogonal 𝑆 (i.e. any span of the vectors)?

• SOS handles it automatically!
• Idea: The SOS-symmetry and 𝑁 ≽ 0 constraints

are invariant under linear transformations of the

• variables. Thus, having a different 𝑆 merely

applies a linear transformation to the pseudo- expectation values.

Loose Ends: Arbitrary 𝑆

• We have only shown a distinguishing algorithm

between the random and planted cases. How can we find the planted sparse vector 𝑤 exactly?

• Can be done in two steps:
• 1. The analysis shows that degree 4 SOS will output a

vector 𝑤′ which is highly correlated with 𝑤 (because the random part of the subspace has nothing with high 4-norm)

• 2. Using 𝑤′ as a guide, find 𝑤. This can be done by

minimizing then 𝑀1 norm of a vector 𝑤 in the subspace subject to 𝑤 ⋅ 𝑤′ = 1, see [BKS14] for details.

Loose Ends: Finding 𝑤 Exactly

Part VII: Open Problems

• What more can we say when 𝑒 ≫

𝑜?

• More specifically, can we find a better algorithm

using more than the 4-norm? Is there an SOS lower bound showing that 𝑙 =

𝑜2 𝑒2 is tight?

Open Problems

References

• [BBH+12] B. Barak, F. G. S. L. Brandão, A. W. Harrow, J. A. Kelner, D. Steurer, and Y.
• Zhou. Hypercontractivity, sum-of-squares proofs, and their applications. STOC p.

307–326, 2012.

• [BKS14] B. Barak, J. A. Kelner, and D. Steurer. Rounding Sum of Squares Relaxations.

STOC 2014.