Upper Bound for Intermediate Singular Values of Random Sub-Gaussian - - PowerPoint PPT Presentation
Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices 1 Feng Wei 2 University of Michigan July 29, 2016 1 This presentation is based a project under the supervision of M. Rudelson. 2 Partially supported by M. Rudelsons
Feng Wei 2
University of Michigan
July 29, 2016
1This presentation is based a project under the supervision of M. Rudelson. 2Partially supported by M. Rudelson’s NSF Grant DMS-1464514, and USAF Grant
FA9550-14-1-0009.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 1 / 16
Motivation and Backgound
Let A be an n × n random matrix with i.i.d. entries of mean 0 and variance
µ(J) = 1 n#
A √n
By Quarter Circular Law, dµ(x) → 1
π
√ 4 − x21[0,2](x)dx as n → ∞.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 2 / 16
Motivation and Backgound
Let A be an n × n random matrix with i.i.d. entries of mean 0 and variance
µ(J) = 1 n#
A √n
By Quarter Circular Law, dµ(x) → 1
π
√ 4 − x21[0,2](x)dx as n → ∞. A simple computation using the limiting distribution shows the ℓth smallest singular value sn+1−ℓ(A) is in the order of
ℓ √n for ℓ = 1, 2, · · · , n.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 2 / 16
Motivation and Backgound
Let A be an n × n random matrix with i.i.d. entries of mean 0 and variance
µ(J) = 1 n#
A √n
By Quarter Circular Law, dµ(x) → 1
π
√ 4 − x21[0,2](x)dx as n → ∞. A simple computation using the limiting distribution shows the ℓth smallest singular value sn+1−ℓ(A) is in the order of
ℓ √n for ℓ = 1, 2, · · · , n.
Question What is the distribution of the singular values for a fixed large n?
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 2 / 16
Motivation and Backgound
Let A be an n × n random matrix with i.i.d. entries of mean 0 and variance
µ(J) = 1 n#
A √n
By Quarter Circular Law, dµ(x) → 1
π
√ 4 − x21[0,2](x)dx as n → ∞. A simple computation using the limiting distribution shows the ℓth smallest singular value sn+1−ℓ(A) is in the order of
ℓ √n for ℓ = 1, 2, · · · , n.
Question What is the distribution of the singular values for a fixed large n?
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 2 / 16
Motivation and Backgound
Definition Let θ > 0. Let Z be a random variable. Then the ψθ-norm of Z is defined as Zψθ := inf
|Z| λ θ ≤ 2
This condition is satisfied for broad classes of random variables. In particular, a bounded random variable is ψθ for any θ > 0, a normal random variable is ψ2, and a Poisson variable is ψ1. A ψ2 random variable is also called sub-gaussian.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 3 / 16
Motivation and Backgound
Definition Let θ > 0. Let Z be a random variable. Then the ψθ-norm of Z is defined as Zψθ := inf
|Z| λ θ ≤ 2
This condition is satisfied for broad classes of random variables. In particular, a bounded random variable is ψθ for any θ > 0, a normal random variable is ψ2, and a Poisson variable is ψ1. A ψ2 random variable is also called sub-gaussian. Moreover, Xψ2 = K ⇐ ⇒ P (|X| > t) ≤ exp(1 − ct2
K 2 ).
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 3 / 16
Motivation and Backgound
Definition Let θ > 0. Let Z be a random variable. Then the ψθ-norm of Z is defined as Zψθ := inf
|Z| λ θ ≤ 2
This condition is satisfied for broad classes of random variables. In particular, a bounded random variable is ψθ for any θ > 0, a normal random variable is ψ2, and a Poisson variable is ψ1. A ψ2 random variable is also called sub-gaussian. Moreover, Xψ2 = K ⇐ ⇒ P (|X| > t) ≤ exp(1 − ct2
K 2 ).
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 3 / 16
Motivation and Backgound
The extreme singular values are better studied: It is easy to show that the operator norm of N × n i.i.d. sub-gaussian matrix is in the order of √ N with high probability.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 4 / 16
Motivation and Backgound
The extreme singular values are better studied: It is easy to show that the operator norm of N × n i.i.d. sub-gaussian matrix is in the order of √ N with high probability. In 2008, M. Rudelson proved that the smallest singular value of a square i.i.d. sub-gaussian matrix is lower bounded by n−3/2 with high probability.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 4 / 16
Motivation and Backgound
The extreme singular values are better studied: It is easy to show that the operator norm of N × n i.i.d. sub-gaussian matrix is in the order of √ N with high probability. In 2008, M. Rudelson proved that the smallest singular value of a square i.i.d. sub-gaussian matrix is lower bounded by n−3/2 with high probability. This result was later extended and improved by A. Basak and M. Rudelson, M. Rudelson and R. Vershynin, T. Tao and V. Vu in square matrices case.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 4 / 16
Motivation and Backgound
The extreme singular values are better studied: It is easy to show that the operator norm of N × n i.i.d. sub-gaussian matrix is in the order of √ N with high probability. In 2008, M. Rudelson proved that the smallest singular value of a square i.i.d. sub-gaussian matrix is lower bounded by n−3/2 with high probability. This result was later extended and improved by A. Basak and M. Rudelson, M. Rudelson and R. Vershynin, T. Tao and V. Vu in square matrices case. In 2009, M. Rudelson and R. Vershynin proved a sharp bound for smallest singular value of all rectangular sub-gaussian matrices.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 4 / 16
Motivation and Backgound
The extreme singular values are better studied: It is easy to show that the operator norm of N × n i.i.d. sub-gaussian matrix is in the order of √ N with high probability. In 2008, M. Rudelson proved that the smallest singular value of a square i.i.d. sub-gaussian matrix is lower bounded by n−3/2 with high probability. This result was later extended and improved by A. Basak and M. Rudelson, M. Rudelson and R. Vershynin, T. Tao and V. Vu in square matrices case. In 2009, M. Rudelson and R. Vershynin proved a sharp bound for smallest singular value of all rectangular sub-gaussian matrices.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 4 / 16
Motivation and Backgound
Theorem.(M. Rudelson and R. Vershynin, 2009) Let G be an N × n random matrix, N ≥ n, whose elements are independent copies of a centered sub-gaussian random variable with unit
P
√ N − √ n − 1
where C, C ′ > 0 depend (polynomially) only on the sub-gaussian moment K. Consider an n × n i.i.d. sub-gaussian matrix A and let B be the first n + 1 − ℓ columns of A. Then with high probability, sn+1−ℓ(A) ≥ sn+1−ℓ(B) ≥ c √n − √ n − ℓ
√n.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 5 / 16
Motivation and Backgound
Theorem.(M. Rudelson and R. Vershynin, 2009) Let G be an N × n random matrix, N ≥ n, whose elements are independent copies of a centered sub-gaussian random variable with unit
P
√ N − √ n − 1
where C, C ′ > 0 depend (polynomially) only on the sub-gaussian moment K. Consider an n × n i.i.d. sub-gaussian matrix A and let B be the first n + 1 − ℓ columns of A. Then with high probability, sn+1−ℓ(A) ≥ sn+1−ℓ(B) ≥ c √n − √ n − ℓ
√n.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 5 / 16
Motivation and Backgound
Theorem.(M. Rudelson and R. Vershynin, 2008) Let A be an n × n i.i.d. sub-gaussian matrix whose entries have mean 0, variance 1 and sub-gaussian moment K. Then for any t ≥ 2, P
2
t + cn where C > 0 and c ∈ (0, 1) depend only on K. Theorem.(H. Nguyen and V. Vu., 2016) Let A be an n × n i.i.d. sub-gaussian matrix whose entries have mean 0, variance 1 and sub-gaussian moment K. Then for any t > 0, P
2
where C1, C2 > 0 depend only on K.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 6 / 16
Motivation and Backgound
Theorem.(M. Rudelson and R. Vershynin, 2008) Let A be an n × n i.i.d. sub-gaussian matrix whose entries have mean 0, variance 1 and sub-gaussian moment K. Then for any t ≥ 2, P
2
t + cn where C > 0 and c ∈ (0, 1) depend only on K. Theorem.(H. Nguyen and V. Vu., 2016) Let A be an n × n i.i.d. sub-gaussian matrix whose entries have mean 0, variance 1 and sub-gaussian moment K. Then for any t > 0, P
2
where C1, C2 > 0 depend only on K.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 6 / 16
Motivation and Backgound
Theorem.(S. Szarek, 1990) Let G be an n × n i.i.d. standard Gaussian matrix. Then there exist universal constants c1, c2, c, C, s.t. P c1ℓ √n ≤ sn+1−ℓ(G) ≤ c2ℓ √n
From a paper of T. Tao and V. Vu. in 2010, together with the result of Szarek, one can deduce some non-asymptotic bounds for random i.i.d. square matrix for ℓ ≤ nc where c is a small constant. However, the tail bound is not exponential type.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 7 / 16
Motivation and Backgound
Theorem.(S. Szarek, 1990) Let G be an n × n i.i.d. standard Gaussian matrix. Then there exist universal constants c1, c2, c, C, s.t. P c1ℓ √n ≤ sn+1−ℓ(G) ≤ c2ℓ √n
From a paper of T. Tao and V. Vu. in 2010, together with the result of Szarek, one can deduce some non-asymptotic bounds for random i.i.d. square matrix for ℓ ≤ nc where c is a small constant. However, the tail bound is not exponential type.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 7 / 16
Main Results
Definition Let Z be a random vector that takes values in Rn. The concentration function of Z is defined as L(Z, t) = sup
u∈Rn P{Z − u2 ≤ t}, t ≥ 0.
Assumption 1 Let p > 0. Let A be an n × m random matrix whose entries are i.i.d. random variables, with mean 0, variance 1 and ψ2-norm K. Assume also that there exists 0 < s ≤ s0(p, K) such that L(Ai,j, s) ≤ ps. Here, s0(p, K) is a given function depending only on p and K.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 8 / 16
Main Results
Definition Let Z be a random vector that takes values in Rn. The concentration function of Z is defined as L(Z, t) = sup
u∈Rn P{Z − u2 ≤ t}, t ≥ 0.
Assumption 1 Let p > 0. Let A be an n × m random matrix whose entries are i.i.d. random variables, with mean 0, variance 1 and ψ2-norm K. Assume also that there exists 0 < s ≤ s0(p, K) such that L(Ai,j, s) ≤ ps. Here, s0(p, K) is a given function depending only on p and K.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 8 / 16
Main Results
Theorem Let A be an n × n random matrix that satisfies Assumption 1. Then there exist constants C1, C2 > 0 such that for all t > 1 and ℓ = 1, 2, · · · , n, P
tℓ √n
where C1, C2 are constants that depend only on K, p.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 9 / 16
Main Results
Corollary Let A be an n × (n − k) random matrix that satisfies Assumption 1. Then there exist constants C1, C2 > 0 such that for all t > 1 and ℓ = 1, · · · , n, P
tℓ √n
where C1, C2 are constants that depend only on K, p. Corollary Let A be an n × n random matrix that satisfies Assumption 1. Then there exist 0 < C1 < C2 and C3 > 0, such that for all ℓ = 1, 2, · · · , n, P C1ℓ √n ≤ sn+1−ℓ(A) ≤ C2ℓ √n
where Cis are constants that depends only on K, p.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 10 / 16
Main Results
Corollary Let A be an n × (n − k) random matrix that satisfies Assumption 1. Then there exist constants C1, C2 > 0 such that for all t > 1 and ℓ = 1, · · · , n, P
tℓ √n
where C1, C2 are constants that depend only on K, p. Corollary Let A be an n × n random matrix that satisfies Assumption 1. Then there exist 0 < C1 < C2 and C3 > 0, such that for all ℓ = 1, 2, · · · , n, P C1ℓ √n ≤ sn+1−ℓ(A) ≤ C2ℓ √n
where Cis are constants that depends only on K, p.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 10 / 16
Outline of Proof
We want to prove sn+1−ℓ(A) ≤ cℓ
√n with high probability.
Instead of proving upper bound for sn+1−ℓ(A), we prove lower bound for sℓ(A−1). So we only need to find an ℓ-dimensional random subspace E such that with high probability, for all y ∈ E, A−1y2 y2 ≥ C√n ℓ
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 11 / 16
Outline of Proof
We want to prove sn+1−ℓ(A) ≤ cℓ
√n with high probability.
Instead of proving upper bound for sn+1−ℓ(A), we prove lower bound for sℓ(A−1). So we only need to find an ℓ-dimensional random subspace E such that with high probability, for all y ∈ E, A−1y2 y2 ≥ C√n ℓ Let Xi, i = 1, · · · , n denote the columns of A and Hℓ = span{Xℓ+1, · · · , Xn}. Then our aimed subspace is H⊥
ℓ and we want
to prove it using union bound argument.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 11 / 16
Outline of Proof
We want to prove sn+1−ℓ(A) ≤ cℓ
√n with high probability.
Instead of proving upper bound for sn+1−ℓ(A), we prove lower bound for sℓ(A−1). So we only need to find an ℓ-dimensional random subspace E such that with high probability, for all y ∈ E, A−1y2 y2 ≥ C√n ℓ Let Xi, i = 1, · · · , n denote the columns of A and Hℓ = span{Xℓ+1, · · · , Xn}. Then our aimed subspace is H⊥
ℓ and we want
to prove it using union bound argument.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 11 / 16
Outline of Proof
Two obstacles if we argue over the sphere on H⊥
ℓ :
H⊥
ℓ is random and it depends on A.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 12 / 16
Outline of Proof
Two obstacles if we argue over the sphere on H⊥
ℓ :
H⊥
ℓ is random and it depends on A.
Estimating A−1y2 may be as hard as the orginal problem.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 12 / 16
Outline of Proof
Two obstacles if we argue over the sphere on H⊥
ℓ :
H⊥
ℓ is random and it depends on A.
Estimating A−1y2 may be as hard as the orginal problem. Instead of applying union bound on unit sphere, we apply it on P⊥
ℓ AℓSℓ−1,
where Aℓ is the first ℓ columns of A and P⊥
ℓ is the orthogonal projection
ℓ .
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 12 / 16
Outline of Proof
Two obstacles if we argue over the sphere on H⊥
ℓ :
H⊥
ℓ is random and it depends on A.
Estimating A−1y2 may be as hard as the orginal problem. Instead of applying union bound on unit sphere, we apply it on P⊥
ℓ AℓSℓ−1,
where Aℓ is the first ℓ columns of A and P⊥
ℓ is the orthogonal projection
ℓ .
Let’s construct a net N on Sℓ−1, if we have control P⊥
ℓ Aℓ, then
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 12 / 16
Outline of Proof
Two obstacles if we argue over the sphere on H⊥
ℓ :
H⊥
ℓ is random and it depends on A.
Estimating A−1y2 may be as hard as the orginal problem. Instead of applying union bound on unit sphere, we apply it on P⊥
ℓ AℓSℓ−1,
where Aℓ is the first ℓ columns of A and P⊥
ℓ is the orthogonal projection
ℓ .
Let’s construct a net N on Sℓ−1, if we have control P⊥
ℓ Aℓ, then
For any y ∈ N, P⊥
ℓ Aℓy2 is bounded from above.
P⊥
ℓ AℓN is still a net on P⊥ ℓ AℓSℓ−1 but with a different scale.
A−1P⊥
ℓ Aℓy2 2 ≈ BAℓy2 2, where B is an (n − ℓ) × n random matrix
independent to Aℓ.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 12 / 16
Outline of Proof
Two obstacles if we argue over the sphere on H⊥
ℓ :
H⊥
ℓ is random and it depends on A.
Estimating A−1y2 may be as hard as the orginal problem. Instead of applying union bound on unit sphere, we apply it on P⊥
ℓ AℓSℓ−1,
where Aℓ is the first ℓ columns of A and P⊥
ℓ is the orthogonal projection
ℓ .
Let’s construct a net N on Sℓ−1, if we have control P⊥
ℓ Aℓ, then
For any y ∈ N, P⊥
ℓ Aℓy2 is bounded from above.
P⊥
ℓ AℓN is still a net on P⊥ ℓ AℓSℓ−1 but with a different scale.
A−1P⊥
ℓ Aℓy2 2 ≈ BAℓy2 2, where B is an (n − ℓ) × n random matrix
independent to Aℓ.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 12 / 16
Outline of Proof
There are several Quantities need to be estimated: Large deviation of P⊥
ℓ Aℓ.
Large deviation of
ℓ
Property of the matrix B. Small ball probability of BX2 given B is a fixed matrix and X is a sub-gaussian random vector.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 13 / 16
Remarks
Major measure concentration results we applied: Corollaries of Hanson-Wright inequality. Small ball probability for linear image of high dimension distribution. Small ball probability of distance from a sub-gaussian vector to a random subspace. Lower bound for smallest singular value of rectangular i.i.d. sub-gaussian matrix.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 14 / 16
Remarks
Theorem.(M. Rudelson and R. Vershynin, 2014) Consider a random vector Z where Zi are real-valued independent random
L(Zi, t) ≤ p for all i = 1, · · · , n Let D be an m × n matrix and ε ∈ (0, 1). Then L(DZ, tDHS) ≤ (cεp)(1−ε)r(D) where r(D) = D2
HS/D2 2 and cε depend only on ε.
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 15 / 16
Remarks
Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices July 29, 2016 16 / 16