Quantized Corrupted Sensing with Random Dithering Zhongxing Sun - - PowerPoint PPT Presentation
Quantized Corrupted Sensing with Random Dithering Zhongxing Sun Beijing Institute of Technology Joint work with Wei Cui and Yulong Liu ISIT 2020 Zhongxing Sun Beijing Institute of Technology ISIT 2020 1 / 22 Overview Introduction: From
Zhongxing Sun
Beijing Institute of Technology Joint work with Wei Cui and Yulong Liu
ISIT 2020
Zhongxing Sun Beijing Institute of Technology ISIT 2020 1 / 22
Introduction: From Corrupted Sensing to Quantized Corrupted Sensing Recovery Procedure Performance Guarantees Concrete Examples Numerical Simulations Summary
Zhongxing Sun Beijing Institute of Technology ISIT 2020 2 / 22
Corrupted sensing1: y = Φx⋆ + √mv ⋆ + n, (1) where: Φ ∈ Rm×n: the sensing matrix with m << n; x⋆ ∈ Rn : the unknown structured signal; v ⋆ ∈ Rm : the unknown structured corruption; n ∈ Rm : the random unstructured noise.
1The factor √m in (1) makes the columns of Φ and √mIm have the same scale,
which helps the theoretical results to be more interpretable.
Zhongxing Sun Beijing Institute of Technology ISIT 2020 3 / 22
Corrupted sensing1: y = Φx⋆ + √mv ⋆ + n, (1) where: Φ ∈ Rm×n: the sensing matrix with m << n; x⋆ ∈ Rn : the unknown structured signal; v ⋆ ∈ Rm : the unknown structured corruption; n ∈ Rm : the random unstructured noise.
Goal
To estimate x⋆ and v ⋆ given y and Φ. (See e.g. [FM14],[MT14],[CL19])
1The factor √m in (1) makes the columns of Φ and √mIm have the same scale,
which helps the theoretical results to be more interpretable.
Zhongxing Sun Beijing Institute of Technology ISIT 2020 3 / 22
Signal model: y = Φx⋆ + √mv ⋆ + n. Corrupted sensing model has seen a lot of interest in modern data-intensive science.
Zhongxing Sun Beijing Institute of Technology ISIT 2020 4 / 22
Signal model: y = Φx⋆ + √mv ⋆ + n. Corrupted sensing model has seen a lot of interest in modern data-intensive science. Examples of applications include: face recognition [WYGSM09]; subspace clustering [EV09]; video background subtraction [CLMW11]. ...
Zhongxing Sun Beijing Institute of Technology ISIT 2020 4 / 22
Signal model: y = Φx⋆ + √mv ⋆ + n. Corrupted sensing model has seen a lot of interest in modern data-intensive science. Examples of applications include: face recognition [WYGSM09]; subspace clustering [EV09]; video background subtraction [CLMW11]. ...
Figure: Examples from [WYGSM09].
Zhongxing Sun Beijing Institute of Technology ISIT 2020 4 / 22
Signal model: y = Φx⋆ + √mv ⋆ + n.
Zhongxing Sun Beijing Institute of Technology ISIT 2020 5 / 22
Signal model: y = Φx⋆ + √mv ⋆ + n.
Table: Theoretical Analyses of Corrupted Sensing
Measurement Matrix Recovery Proce- dure Noise Paper Random Orthogonal Constrained δ = 0 [MT14] Gaussian Constrained Partially Penalized Bounded [FM14] Sub-Gaussian Constrained Partially Penalized Fully Penalized Bounded Sub-Gaussian [CL19]
Zhongxing Sun Beijing Institute of Technology ISIT 2020 5 / 22
In many practical applications, one would like to quantize or digitize the measurements into bitstreams: y = Q(Φx⋆ + √mv ⋆ + n), where Q(·) stands for some quantization scheme.
Zhongxing Sun Beijing Institute of Technology ISIT 2020 6 / 22
In many practical applications, one would like to quantize or digitize the measurements into bitstreams: y = Q(Φx⋆ + √mv ⋆ + n), where Q(·) stands for some quantization scheme.
Question:
Can we recover a structured signal from the quantized corrupted measurements y = Q(Φx⋆ + √mv ⋆ + n)? Furthermore, what role does quantization play in the reconstruction error?
Zhongxing Sun Beijing Institute of Technology ISIT 2020 6 / 22
Signal model: y = Q(Φx⋆ + √mv ⋆ + n). (2) Good news: If v ⋆ = 0, model (2) reduces to Quantized Compressed Sensing (QCS). Some theoretical results exist: 1-bit measurements [BB08]; uniform quantization [XJ19], [TR20]; general non-linear function [PV16], [TAH15].
Zhongxing Sun Beijing Institute of Technology ISIT 2020 7 / 22
Signal model: y = Q(Φx⋆ + √mv ⋆ + n). (2) Good news: If v ⋆ = 0, model (2) reduces to Quantized Compressed Sensing (QCS). Some theoretical results exist: 1-bit measurements [BB08]; uniform quantization [XJ19], [TR20]; general non-linear function [PV16], [TAH15]. Bad news: If v ⋆ = 0, the existing theoretical analyses for QCS cannot be directly generalized to corrupted sensing model.
Zhongxing Sun Beijing Institute of Technology ISIT 2020 7 / 22
100 150 200 250 300 350 400 450500
Number of measurements (log)
10-1
Recover error (log)
hyperbolic tangent uniform quantizer sign theoretical scaling
(a) Corruption v ⋆ = 0
100 200 300 400 500
Number of measurements (log)
0.9 0.95 1 1.05 1.1
Recover error (log)
hyperbolic tangent uniform quantizer sign theoretical scaling
(b) Corruption v ⋆ = 0
Figure: Log-log error curves for different nonlinear quantization schemes.
Zhongxing Sun Beijing Institute of Technology ISIT 2020 8 / 22
100 150 200 250 300 350 400 450500
Number of measurements (log)
10-1
Recover error (log)
hyperbolic tangent uniform quantizer sign theoretical scaling
(a) Corruption v ⋆ = 0
100 200 300 400 500
Number of measurements (log)
0.9 0.95 1 1.05 1.1
Recover error (log)
hyperbolic tangent uniform quantizer sign theoretical scaling
(b) Corruption v ⋆ = 0
Figure: Log-log error curves for different nonlinear quantization schemes.
Solution: to consider some specific but more tractable nonlinear quantization schemes.
Zhongxing Sun Beijing Institute of Technology ISIT 2020 8 / 22
In this paper, we consider introducing a random noise before quantization (also known as dithering): ¯ y = Φx⋆ + √mv ⋆ + n, y = QU(¯ y + τ) = QU
(3) where QU(x) = ∆(⌊ x
∆⌋ + 1 2) is the uniform scalar quantizer with
resolution ∆ > 0, and τi ∼ Unif(− ∆
2 , ∆ 2 ] is the random uniform dithering.
Zhongxing Sun Beijing Institute of Technology ISIT 2020 9 / 22
In this paper, we consider introducing a random noise before quantization (also known as dithering): ¯ y = Φx⋆ + √mv ⋆ + n, y = QU(¯ y + τ) = QU
(3) where QU(x) = ∆(⌊ x
∆⌋ + 1 2) is the uniform scalar quantizer with
resolution ∆ > 0, and τi ∼ Unif(− ∆
2 , ∆ 2 ] is the random uniform dithering.
Goal
To disentangle signal x⋆ and corruption v ⋆ given Φ and the quantized samples {yi}m
i=1.
Zhongxing Sun Beijing Institute of Technology ISIT 2020 9 / 22
Key Observation [Sch65],[GS93]
Uniform dither can substantially result in independent distributed quantization error: z := QU(¯ y + τ) − (¯ y + τ) is independent from ¯ y and τ.
Zhongxing Sun Beijing Institute of Technology ISIT 2020 10 / 22
Key Observation [Sch65],[GS93]
Uniform dither can substantially result in independent distributed quantization error: z := QU(¯ y + τ) − (¯ y + τ) is independent from ¯ y and τ. Then, the problem can be reformulated as y = QU(¯ y + τ) = Φx⋆ + √mv ⋆ + τ + z + n.
Zhongxing Sun Beijing Institute of Technology ISIT 2020 10 / 22
Key Observation [Sch65],[GS93]
Uniform dither can substantially result in independent distributed quantization error: z := QU(¯ y + τ) − (¯ y + τ) is independent from ¯ y and τ. Then, the problem can be reformulated as y = QU(¯ y + τ) = Φx⋆ + √mv ⋆ + τ + z + n. It’s natural to use the generalized Lasso: min
x,v y − Φx − √mv2,
s.t. xsig ≤ x⋆sig vcor ≤ v ⋆cor. (4) e.g., · sig: sparse signals → ℓ1-norm; low-rank matrices → nuclear norm.
Zhongxing Sun Beijing Institute of Technology ISIT 2020 10 / 22
Under the linearized model: y =
x⋆ v ⋆
Zhongxing Sun Beijing Institute of Technology ISIT 2020 11 / 22
Under the linearized model: y =
x⋆ v ⋆
Extended matrix deviation inequality [CL19] implies a tight lower bound for the restricted singular value of the extended sensing matrix [Φ, √mIm]: inf
(a,b)∈T ∩Sn+m−1 Φa + √mb2 ≥ √m − C · γ(T ∩ Sn+m−1)
holds with probability at least 1 − exp{−γ(T ∩ Sn+m−1)2}.
Zhongxing Sun Beijing Institute of Technology ISIT 2020 11 / 22
Theorem (Recovery with Dithered Quantization) [S-Cui-Liu]
Suppose Φ ∈ Rm×n is sub-Gaussian and K = maxi Φiψ2. The independent mean-zero additive noise n satisfies n∞ ≤ ǫ, ∆ > 0 is the quantization step. If the number of measurement m ≥ CK 4 · γ2(Ce ∩ Sn+m−1), (5) then, with probability at least 1 − 2 exp{−γ2(Ce ∩ Sn+m−1)},
x − x⋆2
2 + ˆ
v − v ⋆2
2 ≤ CK(∆ + ǫ) · γ(Ce ∩ Sn+m−1)
√m .
Zhongxing Sun Beijing Institute of Technology ISIT 2020 12 / 22
Theorem (Recovery with Dithered Quantization) [S-Cui-Liu]
Suppose Φ ∈ Rm×n is sub-Gaussian and K = maxi Φiψ2. The independent mean-zero additive noise n satisfies n∞ ≤ ǫ, ∆ > 0 is the quantization step. If the number of measurement m ≥ CK 4 · γ2(Ce ∩ Sn+m−1), (5) then, with probability at least 1 − 2 exp{−γ2(Ce ∩ Sn+m−1)},
x − x⋆2
2 + ˆ
v − v ⋆2
2 ≤ CK(∆ + ǫ) · γ(Ce ∩ Sn+m−1)
√m . Error cone: Ce = {(a, b) ∈ Rn × Rm : a ∈ Tf and b ∈ Tg} = Tf × Tg. Gaussian complexity: γ(T ) := E supx∈T |g, x|, g ∼ N(0, In). Tf and Tg: tangent cone of f (x) = xsig and g(v) = vcor.
Zhongxing Sun Beijing Institute of Technology ISIT 2020 12 / 22
The influence of quantization Relation to quantized compressed sensing Distribution of noise Distribution of dither
Zhongxing Sun Beijing Institute of Technology ISIT 2020 13 / 22
Corollary (Sparse Recovery from Sparse Corruption)
Suppose that signal x⋆ ∈ Rn is s-sparse and corruption v ⋆ ∈ Rm is k-sparse, under settings of the main Theorem,
x − x⋆2
2 + ˆ
v − v ⋆2
2
≤ CK(∆ + ǫ) ·
√m provided m ≥ CK 4 s log n
s
m
k
Zhongxing Sun Beijing Institute of Technology ISIT 2020 14 / 22
Corollary (Low-rank Recovery from Sparse Corruption)
Suppose that signal x⋆ = vec(X ⋆) ∈ Rn, where X ⋆ ∈ Rn1×n2 is a matrix with rank ρ and n1n2 = n, and the corruption v ⋆ is p-sparse vector. Without loss of generality, we suppose that n1 ≥ n2, under settings of the main Theorem,
X − X ⋆2
F + ˆ
v − v ⋆2
2
≤ CK(∆ + ǫ) ·
√m provided m ≥ CK 4 ρ(n1 + n2) + p log
p
Zhongxing Sun Beijing Institute of Technology ISIT 2020 15 / 22
100 150 200 300 500
Number of measurements m
0.01 0.02 0.03 0.04 0.05
Gaussian measurements
(a) Gaussian measurements
100 150 200 300 500
Number of measurements m
0.01 0.02 0.03 0.04 0.05
Bernoulli measurements
(b) Bernoulli measurements
Figure: Sparse recovery from sparse corruption. The black dashed line illustrate the O(
1 √m) scaling predicted in the Corollary.
Zhongxing Sun Beijing Institute of Technology ISIT 2020 16 / 22
300 400 600 800 1200
Number of Measurements m
10-2 10-1
Gaussian Measurements
(a) Gaussian measurements
300 400 600 800 1200
Number of Measurements m
10-2 10-1
Bernoulli Measurements
(b) Bernoulli measurements
Figure: Low-rank recovery from sparse corruption. The black dashed line illustrate the O(
1 √m) scaling predicted in the Corollary.
Zhongxing Sun Beijing Institute of Technology ISIT 2020 17 / 22
100 150 200 300 500
Number of measurements m
10-1 100
Gaussian measurements
(a) Gaussian measurements
100 150 200 300 500
Number of measurements m
10-1 100
Bernoulli measurements
(b) Bernoulli measurements
Figure: Robustness to noise. The black dashed line illustrate the O(
1 √m) scaling
predicted in the Corollary.
Zhongxing Sun Beijing Institute of Technology ISIT 2020 18 / 22
Give a tractable quantization scheme for quantized corrupted sensing problem.
Zhongxing Sun Beijing Institute of Technology ISIT 2020 19 / 22
Give a tractable quantization scheme for quantized corrupted sensing problem. Suggest a convex recovery procedure.
Zhongxing Sun Beijing Institute of Technology ISIT 2020 19 / 22
Give a tractable quantization scheme for quantized corrupted sensing problem. Suggest a convex recovery procedure. Present performance guarantees for this approach under sub-Gaussian measurements.
Zhongxing Sun Beijing Institute of Technology ISIT 2020 19 / 22
Give a tractable quantization scheme for quantized corrupted sensing problem. Suggest a convex recovery procedure. Present performance guarantees for this approach under sub-Gaussian measurements. Concrete examples and simulations.
Zhongxing Sun Beijing Institute of Technology ISIT 2020 19 / 22
Give a tractable quantization scheme for quantized corrupted sensing problem. Suggest a convex recovery procedure. Present performance guarantees for this approach under sub-Gaussian measurements. Concrete examples and simulations. Future work: recovery from other quantization scheme under corrupted measurements.
Zhongxing Sun Beijing Institute of Technology ISIT 2020 19 / 22
[CL19] J. Chen and Y. Liu, Stable Recovery of Structured Signals from Corrupted Sub-Gaussian Measurements, IEEE Trans. Inf. Theory 65(5) 2976–2994. [FM14] R. Foygel and L. Mackey, Corrupted sensing: Novel guarantees for separating structured signals, IEEE Trans. Inf. Theory 60(2) 1223–1247. [MT14] M. B. Mccoy and J. A. Tropp, Sharp recovery bounds for convex demixing, with applications, Found. Comut. Math 14(3) 503–567. [WYGSM09] J. Wright, A. Y. Yang, A. Ganesh, S. S. Sastry, and Y. Ma, Robust face recognition via sparse representation, IEEE Trans. Pattern Anal. Mach. Intell. 21(2) 210–227. [EV09] E. Elhamifar and R. Vidal, Sparse subspace clustering, IEEE Conf. Comput. Vis. Pattern Recognit. Miami, FL, USA. 2790–2797. [HBRN08] J. Haupt, W. U. Bajwa, M. Rabbat, and R. Nowak, Compressed sensing for networked data, IEEE Signal Process. Mag. 25(2) 92–101. [CSPW08] V. Chandrasekaran, S. Sanghavi, P. A. Parrilo, and A. S. Willsky, Rank sparsity incoherence for matrix decomposition, J. Optim. 21(2) 572–596. [CLMW11] E. J. Cand` es, X. Li, Y. Ma, and J. Wright, Robust principal component analysis? J. ACM 58(3) 1–37. [BB08] P. T. Boufounos and R. G. Baraniuk, 1-bit compressive sensing, Annu. Conf.
[XJ19] C. Xu and L. Jacques, Quantized compressive sensing with rip matrices: The benefit of dithering, Inf. Inference 0(0) 1–44.
Zhongxing Sun Beijing Institute of Technology ISIT 2020 20 / 22
[TR20] C. Thrampoulidis and A. S. Rawat, The generalized lasso for subgaussian measurements with dithered quantization, IEEE Trans. Inf. Theory 66(4) 2487–2500. [PV16] Y. Plan and R. Vershynin, The generalized lasso with non-linear observations, IEEE Trans. Inf. Theory 62(3) 1528–1537. [TAH15] C. Thrampoulidis, E. Abbasi, and B. Hassibi, The LASSO with Non-linear Measurements is Equivalent to One With Linear Measurements, http://arxiv.org/abs/1506.02181. [Sch65] L. Schuchman, Dither signals and their effect on quantization noise, IEEE
[GS93] R. M. Gray and J. Stockham, Dithered quantizers, IEEE Trans. Inf. Theory 39(3) 805–812.
Zhongxing Sun Beijing Institute of Technology ISIT 2020 21 / 22
Zhongxing Sun Beijing Institute of Technology ISIT 2020 22 / 22