PROMP PRe-projected Orthogonal Matching Pursuit Axel Flinth Winter - - PowerPoint PPT Presentation

PROMP PRe-projected Orthogonal Matching Pursuit

Axel Flinth Winter School on Compressed Sensing Technische Universit¨ at Berlin

• 5. 12. 2015

Axel Flinth PROMP WiCoS 2015 1 / 17

Orthogonal Matching Pursuit

Algorithm

OMP: Initalize r = b, S = ∅, then iteratively i∗ = argmaxi |ai, r| S ← S ∪ {i∗} , r ← Πran A⊥

S∪{i∗}r. Axel Flinth PROMP WiCoS 2015 2 / 17

Orthogonal Matching Pursuit

Algorithm

OMP: Initalize r = b, S = ∅, then iteratively i∗ = argmaxi |ai, r| S ← S ∪ {i∗} , r ← Πran A⊥

S∪{i∗}r.

Very easy to implement, very fast for small sparsities.

Axel Flinth PROMP WiCoS 2015 2 / 17

Orthogonal Matching Pursuit

Algorithm

OMP: Initalize r = b, S = ∅, then iteratively i∗ = argmaxi |ai, r| S ← S ∪ {i∗} , r ← Πran A⊥

S∪{i∗}r.

Very easy to implement, very fast for small sparsities. Lower recovery probabilities than e.g. Basis Pursuit. Also, for moderate sparsities, OMP gets slow.

Axel Flinth PROMP WiCoS 2015 2 / 17

Orthogonal Matching Pursuit

Algorithm

OMP: Initalize r = b, S = ∅, then iteratively i∗ = argmaxi |ai, r| S ← S ∪ {i∗} , r ← Πran A⊥

S∪{i∗}r.

Very easy to implement, very fast for small sparsities. Lower recovery probabilities than e.g. Basis Pursuit. Also, for moderate sparsities, OMP gets slow. Additionally x0 ∈ Zd ❀ Modification enables higher speed/ larger recovery probability?

Axel Flinth PROMP WiCoS 2015 2 / 17

Why is x0 ∈ Zd+ Sparse Interesting?

0011010011010

User transmits bit-sequence ❀ x0 ∈ Zd.

Axel Flinth PROMP WiCoS 2015 3 / 17

Why is x0 ∈ Zd+ Sparse Interesting?

0011010011010

User transmits bit-sequence ❀ x0 ∈ Zd. Random scattering

Axel Flinth PROMP WiCoS 2015 3 / 17

Why is x0 ∈ Zd+ Sparse Interesting?

0011010011010

User transmits bit-sequence ❀ x0 ∈ Zd. Random scattering ❀ b = Ax0, A Gaussian.

Axel Flinth PROMP WiCoS 2015 3 / 17

Why is x0 ∈ Zd+ Sparse Interesting?

0111010010101 0011010011010 1101001101100 1010111100101

User transmits bit-sequence ❀ x0 ∈ Zd. Random scattering ❀ b = Ax0, A Gaussian. At each moment, few users are transmitting ❀ x0 sparse.

Axel Flinth PROMP WiCoS 2015 3 / 17

Let us go Back to the Roots.

Classical solution to inverse problem Ax = b: ˆ x = A+b ( A+ Moore-Penrose Inverse) Corresponds to ℓ2-minimization min x2 subject to Ax = b.

Axel Flinth PROMP WiCoS 2015 4 / 17

Let us go Back to the Roots.

Classical solution to inverse problem Ax = b: ˆ x = A+b ( A+ Moore-Penrose Inverse) Corresponds to ℓ2-minimization min x2 subject to Ax = b. Fast, easy to implement and to analyze.

Axel Flinth PROMP WiCoS 2015 4 / 17

Let us go Back to the Roots.

Classical solution to inverse problem Ax = b: ˆ x = A+b ( A+ Moore-Penrose Inverse) Corresponds to ℓ2-minimization min x2 subject to Ax = b. Fast, easy to implement and to analyze. But: Bad for sparse recovery.

Axel Flinth PROMP WiCoS 2015 4 / 17

Let us go Back to the Roots.

Classical solution to inverse problem Ax = b: ˆ x = A+b ( A+ Moore-Penrose Inverse) Corresponds to ℓ2-minimization min x2 subject to Ax = b. Fast, easy to implement and to analyze. But: Bad for sparse recovery. If ˆ x(i) large, x0(i) large seems more probable . . .

Axel Flinth PROMP WiCoS 2015 4 / 17

PROMP

Sϑ =

• i | |x∗(i)| ≥ ϑm

d

• Last slide suggests Sϑ ≈ supp x0.

Axel Flinth PROMP WiCoS 2015 5 / 17

PROMP

Sϑ =

• i | |x∗(i)| ≥ ϑm

d

• Last slide suggests Sϑ ≈ supp x0.

Therefore, the following modification of OMP seems reasonable.

Algorithm

PROMP: Initalize r =Πran A⊥

Sϑ b, S =Sϑ, then iteratively

i∗ = argmax |ai, r| I ← I ∪ {i∗} , r ← Πran A⊥

S∪{i∗}r. Axel Flinth PROMP WiCoS 2015 5 / 17

Aspects That we Have to Analyze Is really Sϑ ≈ supp x0? How does initialization affect OMP?

Axel Flinth PROMP WiCoS 2015 6 / 17

Support Approximation

Is really Sϑ ≈ supp x0? Sϑ =

• i
• |ˆ

x(i)| ≥ m d ϑ

• .

Axel Flinth PROMP WiCoS 2015 7 / 17

Support Approximation

Is really Sϑ ≈ supp x0? Sϑ =

• i
• |ˆ

x(i)| ≥ m d ϑ

• .

Theorem (F., 2015)

Let x0 ∈ Zd, A ∈ Rm,d be Gaussian and η > 0 arbitrary. Then there exists a threshold m∗(x0, d, η) so that if m ≥ m∗(x0, d, η), we have i ∈ supp x0 ⇒ P (i ∈ Sϑ) ≥ 1 − η. i / ∈ supp x0 ⇒ P (i / ∈ Sϑ) ≥ 1 − η. I.e. m ≥ m∗, Sϑ is a good approximation of supp x0.

Axel Flinth PROMP WiCoS 2015 7 / 17

Support Approximation

Is really Sϑ ≈ supp x0? Sϑ =

• i
• |ˆ

x(i)| ≥ m d ϑ

• .

Theorem (F., 2015)

Let x0 ∈ Zd, A ∈ Rm,d be Gaussian and η > 0 arbitrary. Then there exists a threshold m∗(x0, d, η) so that if m ≥ m∗(x0, d, η), we have i ∈ supp x0 ⇒ P (i ∈ Sϑ) ≥ 1 − η. i / ∈ supp x0 ⇒ P (i / ∈ Sϑ) ≥ 1 − η. I.e. m ≥ m∗, Sϑ is a good approximation of supp x0. Proof sketch: Solution of ℓ2-minimization is given by ˆ x = Πker A⊥x0.

Axel Flinth PROMP WiCoS 2015 7 / 17

Proof Sketch

Since A is Gaussian, ker A⊥ ∼ U (G(d, m)). ˆ x(i) = ΠLx0, ei , L ∼ U (G(d, m)) .

Axel Flinth PROMP WiCoS 2015 8 / 17

Proof Sketch

Since A is Gaussian, ker A⊥ ∼ U (G(d, m)). ˆ x(i) = ΠLx0, ei , L ∼ U (G(d, m)) . L → ΠLx0, ei is Lipschitz.

Axel Flinth PROMP WiCoS 2015 8 / 17

Proof Sketch

Since A is Gaussian, ker A⊥ ∼ U (G(d, m)). ˆ x(i) = ΠLx0, ei , L ∼ U (G(d, m)) . L → ΠLx0, ei is Lipschitz. Measure Concentration: If X ∼ U (G(d, m)) and f is Lipschitz, then, with high probability, f (X) ≈ E (f (X)) . m larger ↔ Sharper concentration.

Axel Flinth PROMP WiCoS 2015 8 / 17

Proof Sketch

We need to calculate E (ΠLx0, ei).

Axel Flinth PROMP WiCoS 2015 9 / 17

Proof Sketch

We need to calculate E (ΠLx0, ei). Lemma: We have ΠLx0 ∼ R2x0 + R

• 1 − R2Qx0[θ, 0],

where R ∼

• ΠL
• x0

x02

• 2, θ ∼ U
• Sd−2

independent of R and Qx0 fixed with Qx0ed = x0.

Axel Flinth PROMP WiCoS 2015 9 / 17

Proof Sketch

We need to calculate E (ΠLx0, ei). Lemma: We have ΠLx0 ∼ R2x0 + R

• 1 − R2Qx0[θ, 0],

where R ∼

• ΠL
• x0

x02

• 2, θ ∼ U
• Sd−2

independent of R and Qx0 fixed with Qx0ed = x0. Therefore E (ΠLx0, ei) = E

• R2x0, ei
• +
• R
• 1 − R2Qx0[θ, 0], ei
• = E
• R2

x0(i) + E

• R
• 1 − R2
• E
• [θ, 0], Q∗

x0ei

• = m

d x0(i).

Axel Flinth PROMP WiCoS 2015 9 / 17

Proof Sketch

We need to calculate E (ΠLx0, ei). Lemma: We have ΠLx0 ∼ R2x0 + R

• 1 − R2Qx0[θ, 0],

where R ∼

• ΠL
• x0

x02

• 2, θ ∼ U
• Sd−2

independent of R and Qx0 fixed with Qx0ed = x0. Therefore E (ΠLx0, ei) = E

• R2x0, ei
• +
• R
• 1 − R2Qx0[θ, 0], ei
• = E
• R2

x0(i) + E

• R
• 1 − R2
• E
• [θ, 0], Q∗

x0ei

• = m

d x0(i). I.e. for i ∈ supp x0 (/ ∈ supp x0), |ˆ x(i)| will probably larger (smaller) than

m d ϑ |x0(i)| ϑ ≥ m d ϑ.

Axel Flinth PROMP WiCoS 2015 9 / 17

Proof Sketch

We need to calculate E (ΠLx0, ei). Lemma: We have ΠLx0 ∼ R2x0 + R

• 1 − R2Qx0[θ, 0],

where R ∼

• ΠL
• x0

x02

• 2, θ ∼ U
• Sd−2

independent of R and Qx0 fixed with Qx0ed = x0. Therefore E (ΠLx0, ei) = E

• R2x0, ei
• +
• R
• 1 − R2Qx0[θ, 0], ei
• = E
• R2

x0(i) + E

• R
• 1 − R2
• E
• [θ, 0], Q∗

x0ei

• = m

d x0(i). I.e. for i ∈ supp x0 (/ ∈ supp x0), |ˆ x(i)| will probably larger (smaller) than

m d ϑ |x0(i)| ϑ ≥ m d ϑ.

.

Axel Flinth PROMP WiCoS 2015 9 / 17

Aspects That we Have to Analyze Is really Sϑ ≈ supp x0?

• How does initialization affect OMP?

Axel Flinth PROMP WiCoS 2015 10 / 17

Initialization Effect

Classical condition for OMP to reconstruct s-sparse x0: Coherence of A: µ(A) = max

i=j |ai, aj| ,

(A has normalized columns). If µ(A)(2s − 1) < 1, OMP will recover s-sparse signals.

Axel Flinth PROMP WiCoS 2015 11 / 17

Initialization Effect

Classical condition for OMP to reconstruct s-sparse x0: Coherence of A: µ(A) = max

i=j |ai, aj| ,

(A has normalized columns). If µ(A)(2s − 1) < 1, OMP will recover s-sparse signals. Define S-coherence of A: µS(A) := max

i=j

• Πran A⊥

S ai, Πran A⊥ S aj

• Axel Flinth

PROMP WiCoS 2015 11 / 17

Initialization Effect

Classical condition for OMP to reconstruct s-sparse x0: Coherence of A: µ(A) = max

i=j |ai, aj| ,

(A has normalized columns). If µ(A)(2s − 1) < 1, OMP will recover s-sparse signals. Define S-coherence of A: µS(A) := max

i=j

• Πran A⊥

S ai, Πran A⊥ S aj

• Proposition (F., 2015)

Let A ∈ Rm,d. OMP warm-started with the set S will recover a signal x0 with support S0 provided max

i∈I

• Πran A⊥

Sϑ ai

• 2

2 ≥ µSϑ(2 |S0\Sϑ| − 1)

Axel Flinth PROMP WiCoS 2015 11 / 17

Is the new Condition More Probable?

max

i∈I

• Πran A⊥

Sϑ ai

• 2

2 ≥ µSϑ(2 |S0\Sϑ| − 1)

(1)

Axel Flinth PROMP WiCoS 2015 12 / 17

Is the new Condition More Probable?

max

i∈I

• Πran A⊥

Sϑ ai

• 2

2 ≥ µSϑ(2 |S0\Sϑ| − 1)

(1)

Theorem

Sϑ with |Sϑ| ≤ m given, A ∈ Rm,d Gaussian. A fulfills (1) with high probability if m ≥ |Sϑ| + C |S0\Sϑ|2 log(˜ d), where ˜ d = d − |S|

Axel Flinth PROMP WiCoS 2015 12 / 17

Is the new Condition More Probable?

max

i∈I

• Πran A⊥

Sϑ ai

• 2

2 ≥ µSϑ(2 |S0\Sϑ| − 1)

(1)

Theorem

Sϑ with |Sϑ| ≤ m given, A ∈ Rm,d Gaussian. A fulfills (1) with high probability if m ≥ |Sϑ| + C |S0\Sϑ|2 log(˜ d), where ˜ d = d − |S| Sϑ = ∅ (”cold start”): m ≥ C |S0|2 log(d). Sϑ = S0 (perfect warm start) m ≥ |S0| .

Axel Flinth PROMP WiCoS 2015 12 / 17

Idea of the Proof:

max

i∈I

• Πran A⊥

Sϑ ai

• 2

2 ≥ µSϑ(2 |S0\Sϑ| − 1)

Axel Flinth PROMP WiCoS 2015 13 / 17

Idea of the Proof:

max

i∈I

• Πran A⊥

Sϑ ai

• 2

2 ≥ µSϑ(2 |S0\Sϑ| − 1)

This is more or less the classical condition for the projected matrix (Πran A⊥

Sϑ ai)i /

∈Sϑ ∈ (ran A⊥ Sϑ)d−|Sϑ|

for recovering signal supported on S0\Sϑ!

Axel Flinth PROMP WiCoS 2015 13 / 17

Idea of the Proof:

max

i∈I

• Πran A⊥

Sϑ ai

• 2

2 ≥ µSϑ(2 |S0\Sϑ| − 1)

This is more or less the classical condition for the projected matrix (Πran A⊥

Sϑ ai)i /

∈Sϑ ∈ (ran A⊥ Sϑ)d−|Sϑ|

for recovering signal supported on S0\Sϑ! Since (ai)i∈Sϑ | = (ai)i /

∈Sϑ, this matrix is Gaussian in

(ran A⊥

Sϑ)d−|Sϑ| ≃ Rm−|Sϑ|,d−|Sϑ|.

Axel Flinth PROMP WiCoS 2015 13 / 17

Idea of the Proof:

max

i∈I

• Πran A⊥

Sϑ ai

• 2

2 ≥ µSϑ(2 |S0\Sϑ| − 1)

This is more or less the classical condition for the projected matrix (Πran A⊥

Sϑ ai)i /

∈Sϑ ∈ (ran A⊥ Sϑ)d−|Sϑ|

for recovering signal supported on S0\Sϑ! Since (ai)i∈Sϑ | = (ai)i /

∈Sϑ, this matrix is Gaussian in

(ran A⊥

Sϑ)d−|Sϑ| ≃ Rm−|Sϑ|,d−|Sϑ|.

Well known techniques ⇒ B ∈ Rm−|Sϑ|,d−|Sϑ| fulfills classical condition for recovery of signal supported on Sϑ\S0 if m − |Sϑ| ≥ C |S0\Sϑ|2 log(d − |Sϑ|).

• Axel Flinth

PROMP WiCoS 2015 13 / 17

Aspects That we Have to Analyze Is really Sϑ ≈ supp x0?

• How does initialization affect OMP?
• Axel Flinth

PROMP WiCoS 2015 14 / 17

Aspects That we Have to Analyze Is really Sϑ ≈ supp x0?

• How does initialization affect OMP?
• Let’s test it numerically!

Axel Flinth PROMP WiCoS 2015 14 / 17

Numerical Experiments

For m, s = 1, . . . , 100: Draw s-sparse signal x0 ∈ {0, ±1}d uniformly at random. Draw Gaussian A ∈ Rm,d Run OMP and PROMP, record recovery probabilities and computation times.

Axel Flinth PROMP WiCoS 2015 15 / 17

Numerical Experiments

For m, s = 1, . . . , 100: Draw s-sparse signal x0 ∈ {0, ±1}d uniformly at random. Draw Gaussian A ∈ Rm,d Run OMP and PROMP, record recovery probabilities and computation times.

Sparsity s

Number of measurements m

Sparsity s

Number of measurements m

Left: OMP. Right: PROMP.

Axel Flinth PROMP WiCoS 2015 15 / 17

Numerical Experiments

For m, s = 1, . . . , 100: Draw s-sparse signal x0 ∈ {0, ±1}d uniformly at random. Draw Gaussian A ∈ Rm,d Run OMP and PROMP, record recovery probabilities and computation times.

Sparsity s

Number of measurements m

Sparsity s

Number of measurements m

Left: OMP. Right: PROMP.

Axel Flinth PROMP WiCoS 2015 15 / 17

Conclusions

PROMP is a modification of OMP, designed for integer-valued signals.

Axel Flinth PROMP WiCoS 2015 16 / 17

Conclusions

PROMP is a modification of OMP, designed for integer-valued signals. Possible application: wireless communication networks.

Axel Flinth PROMP WiCoS 2015 16 / 17

Conclusions

PROMP is a modification of OMP, designed for integer-valued signals. Possible application: wireless communication networks. Main idea of the algorithm: ℓ2-minimization ❀ Sϑ, then warm start OMP.

Axel Flinth PROMP WiCoS 2015 16 / 17

Conclusions

PROMP is a modification of OMP, designed for integer-valued signals. Possible application: wireless communication networks. Main idea of the algorithm: ℓ2-minimization ❀ Sϑ, then warm start OMP. Theory + Numerics suggest that idea is not bad.

Axel Flinth PROMP WiCoS 2015 16 / 17

Thanks for listening!