Non asymptotic study of the singular values of some random - - PowerPoint PPT Presentation

Non asymptotic study of the singular values of some random covariance matrices.

Olivier Gu´ edon

Universit´ e Paris-Est Marne-la-Vall´ ee

High-dimensional problems and quantum physics. June 2015

The setting.

Let A be a random matrix defined as A = (X1 . . . XN) where X1, . . . , XN are independent random vectors in Rn. What can we say about the singular values of A ? Study of M = AAT = (X1 . . . XN)    XT

1

. . . XT

N

   =

N

• i=1

XiXT

i

The setting.

Let A be a random matrix defined as A = (X1 . . . XN) where X1, . . . , XN are independent random vectors in Rn. What can we say about the singular values of A ? Study of M = AAT = (X1 . . . XN)    XT

1

. . . XT

N

   =

N

• i=1

XiXT

i

           λmax(AAT) = sup

a∈Sn−1 N

• i=1

Xi, a2 λmin(AAT) = inf

a∈Sn−1 N

• i=1

Xi, a2

Random Matrix Theory.

All Xi’s have identically independent random entries. A = (aij)1≤i≤n,1≤j≤N, Mn = AAT Little n and N go to infinity and N/n → c > 1.

Random Matrix Theory.

All Xi’s have identically independent random entries. A = (aij)1≤i≤n,1≤j≤N, Mn = AAT Little n and N go to infinity and N/n → c > 1. Counting probability measure : νn,m = 1 n

n

• i=1

δλk(Mn/

√ N)

Random Matrix Theory.

Marchenko-Pastur ’67 (bulk of the spectrum) If Ea2

i,j = 1, then with probability one, for any continuous

bounded functon f : [0, +∞) → R, lim

n→+∞

• f dνn,m =
• f dν

where ν is the semi-circle law.

Random Matrix Theory.

Marchenko-Pastur ’67 (bulk of the spectrum) If Ea2

i,j = 1, then with probability one, for any continuous

bounded functon f : [0, +∞) → R, lim

n→+∞

• f dνn,m =
• f dν

where ν is the semi-circle law. Bai-Yin ’93 (edge of the spectrum) If Ea2

i,j = 1, Eai,j = 0, and Ea4 i,j < +∞, then with probability

• ne

       lim

n→∞ λmax( 1

N AAT) = 1 + n N lim

n→∞ λmin( 1

N AAT) = 1 − n N

Frame in Harmonic Analysis

Take an orthonormal basis from RM and project it on Rn (or on an n-dimensional subspace of RM). You get v1, . . . , vM : ∀x ∈ Rn, |x|2

2 = M

• j=1

cjx, uj2 where uj =

vj |vj|2 and cj = |vj|2 2.

Define a random vector X in Rn as X = √n uj with proba cj

n .

Then for every vector θ ∈ Rn EX, θ2 =

M

• j=1

cjuj, θ2 = |θ|2

2

Frame in Harmonic Analysis

Take an orthonormal basis from RM and project it on Rn (or on an n-dimensional subspace of RM. You get v1, . . . , vM : ∀x ∈ Rn, |x|2

2 = M

• j=1

cjx, uj2 where uj =

vj |vj|2 and cj = |vj|2 2.

Define a random vector X in Rn as X = √n uj with proba cj

n .

Hence Σ = EX ⊗ X = Id

Frame in Harmonic Analysis

Take an orthonormal basis from RM and project it on Rn (or on an n-dimensional subspace of RM. You get v1, . . . , vM : ∀x ∈ Rn, |x|2

2 = M

• j=1

cjx, uj2 where uj =

vj |vj|2 and cj = |vj|2 2.

Define a random vector X in Rn as X = √n uj with proba cj

n .

Hence Σ = EX ⊗ X = Id Question : Find the size N(ε) of a sample such that ∀θ ∈ Rn, (1 − ε)|θ|2

2 ≤ 1

N

N

• j=1

Xj, θ2 ≤ (1 + ε)|θ|2

2

Frame in Harmonic Analysis

Take an orthonormal basis from RM and project it on Rn (or on an n-dimensional subspace of RM. You get v1, . . . , vM : ∀x ∈ Rn, |x|2

2 = M

• j=1

cjx, uj2 where uj =

vj |vj|2 and cj = |vj|2 2.

Define a random vector X in Rn as X = √n uj with proba cj

n .

Hence Σ = EX ⊗ X = Id Question : Find the size N(ε) of a sample such that ∀θ ∈ Rn, (1 − ε)|θ|2

2 ≤ 1

N

N

• j=1

Xj, θ2 ≤ (1 + ε)|θ|2

2

This gives a subset with a very particular structure.

Frame in Harmonic Analysis

Theorem (Rudelson, ’97). If X is a random vector in Rn such that |X|2 ≤ K√n a.s. and ∀θ ∈ Rn, EX, θ2 = |θ|2

2

: isotropy then for N ≈ CK(ε)n log n, E

• 1

N

N

• j=1

XjXT

j − Id

• ≤ ε

Frame in Harmonic Analysis

Theorem (Rudelson, ’97). If X is a random vector in Rn such that |X|2 ≤ K√n a.s. and ∀θ ∈ Rn, EX, θ2 = |θ|2

2

: isotropy then for N ≈ CK(ε)n log n, E

• 1

N

N

• j=1

XjXT

j − Id

• ≤ ε

The main assumption is |X|2 ≤ K√n, otherwise, there is no moment assumption of the entries.

Frame in Harmonic Analysis

Theorem (Rudelson, ’97). If X is a random vector in Rn such that |X|2 ≤ K√n a.s. and ∀θ ∈ Rn, EX, θ2 = |θ|2

2

: isotropy then for N ≈ CK(ε)n log n, E

• 1

N

N

• j=1

XjXT

j − Id

• ≤ ε

The main assumption is |X|2 ≤ K√n, otherwise, there is no moment assumption of the entries. You can not do better ! Coupon collector.

Computing the volume of a convex body

K ⊂ Rn is given by a separation oracle

Computing the volume of a convex body

K ⊂ Rn is given by a separation oracle Elekes (’86), B´ ar´ any-F¨ uredi (’86) : it is not possible to compute with a deterministic algorithm in polynomial time the volume of a convex body (even approximately)

Computing the volume of a convex body

K ⊂ Rn is given by a separation oracle Elekes (’86), B´ ar´ any-F¨ uredi (’86) : it is not possible to compute with a deterministic algorithm in polynomial time the volume of a convex body (even approximately) Randomization - Given ε and η, Dyer-Frieze-Kannan(’89) established randomized algorithms returning a non-negative number ζ such that (1 − ε)ζ < Vol K < (1 + ε)ζ with probability at least 1 − η. The running time of the algorithm is polynomial in n, 1/ε and log(1/η).

Computing the volume of a convex body

K ⊂ Rn is given by a separation oracle Elekes (’86), B´ ar´ any-F¨ uredi (’86) : it is not possible to compute with a deterministic algorithm in polynomial time the volume of a convex body (even approximately) Randomization - Given ε and η, Dyer-Frieze-Kannan(’89) established randomized algorithms returning a non-negative number ζ such that (1 − ε)ζ < Vol K < (1 + ε)ζ with probability at least 1 − η. The running time of the algorithm is polynomial in n, 1/ε and log(1/η). The number of oracle calls is a random variable and the bound is for example on its expected value.

Computing the volume of a convex body

The randomized algorithm proposed by Kannan, Lov´ asz and Simonovits ’97 improves significantly the polynomial dependence.

Computing the volume of a convex body

The randomized algorithm proposed by Kannan, Lov´ asz and Simonovits ’97 improves significantly the polynomial dependence. Rounding - Put the convex body in a position where Bn

2 ⊂ K ⊂ d Bn 2

where d ≤ nconst.

Computing the volume of a convex body

The randomized algorithm proposed by Kannan, Lov´ asz and Simonovits ’97 improves significantly the polynomial dependence. Rounding - Put the convex body in a position where Bn

2 ⊂ K ⊂ d Bn 2

where d ≤ nconst.

• John (’48) : d ≤ n ( or d ≤ √n in the symmetric case).

How to find an algorithm to do so ?

Computing the volume of a convex body

The randomized algorithm proposed by Kannan, Lov´ asz and Simonovits ’97 improves significantly the polynomial dependence. Rounding - Put the convex body in a position where Bn

2 ⊂ K ⊂ d Bn 2

where d ≤ nconst.

• Idea : find an algorithm which produces in polynomial

time a matrix A such that AK is in an approximate isotropic position. Conjecture 2 of KLS (’97) : solved in 2010 by Adamczak, Litvak, Pajor, Tomczak-Jaegermann

Computing the volume of a convex body

The randomized algorithm proposed by Kannan, Lov´ asz and Simonovits ’97 improves significantly the polynomial dependence. Rounding - Put the convex body in a position where Bn

2 ⊂ K ⊂ d Bn 2

where d ≤ nconst.

• Idea : find an algorithm which produces in polynomial

time a matrix A such that AK is in an approximate isotropic position. Conjecture 2 of KLS (’97) : solved in 2010 by Adamczak, Litvak, Pajor, Tomczak-Jaegermann Computing the volume - Monte Carlo algorithm, estimates

• f local conductance.

Conjecture 1 of KLS (’95) : isoperimetric inequality -

• pen !

Approximation of the covariance matrix.

Question of KLS (’97) : let X be a vector uniformly distributed on a convex body K, X1, . . . , XN ind. copies of X, what is the smallest N such that

• 1

N

N

• j=1

Xj X⊤

j − EX X⊤

• ≤ ε
• EX X⊤
• · is the operator norm

Approximation of the covariance matrix.

Question of KLS (’97) : let X be a vector uniformly distributed on a convex body K, X1, . . . , XN ind. copies of X, what is the smallest N such that

• 1

N

N

• j=1

Xj X⊤

j − Id

• ≤ ε

Assume EX X⊤ = Id,

Approximation of the covariance matrix.

Question of KLS (’97) : let X be a vector uniformly distributed on a convex body K, X1, . . . , XN ind. copies of X, what is the smallest N such that

• 1

N

N

• j=1

Xj X⊤

j − Id

• ≤ ε

Assume EX X⊤ = Id, you want to control the smallest and the largest singular values. 1 − ε ≤ λmin

• 1

N

N

• j=1

Xj X⊤

j

• ≤ λmax
• 1

N

N

• j=1

Xj X⊤

j

• ≤ 1 + ε

KLS n2/ε2, Bourgain n log3 n/ε2, ... Rudelson, Gu´ edon, Paouris, Aubrun, Giannopoulos ALPT (’10) n/ε2 : for general log-concave vectors

Log-concave random vectors

X is a log-concave random vector ↔ log-concave density with respect to the Lebesgue measure on Rn

Log-concave random vectors

X is a log-concave random vector ↔ log-concave density with respect to the Lebesgue measure on Rn Main properties about linear functionals : ∃C > 1, for all log-concave X, for all p ≥ 2, ∀θ ∈ Rn, (E|X, θ|p)1/p ≤ C p

• E|X, θ|21/2

Log-concave random vectors

X is a log-concave random vector ↔ log-concave density with respect to the Lebesgue measure on Rn Main properties about linear functionals : ∃C > 1, for all log-concave X, for all p ≥ 2, ∀θ ∈ Rn, (E|X, θ|p)1/p ≤ C p

• E|X, θ|21/2

Or equivalently in isotropic position, ∀θ ∈ Rn, ∀t ≥ 1, P (|X, θ| ≥ t) ≤ exp(−c t).

Log-concave random vectors

Theorem (Paouris, ’06). If X is an isotropic log-concave random vector in Rn then ∀t ≥ 1, P

• |X|2 ≥ c t √n
• ≤ exp(−C t √n)

Log-concave random vectors

Theorem (Paouris, ’06). If X is an isotropic log-concave random vector in Rn then ∀t ≥ 1, P

• |X|2 ≥ c t √n
• ≤ exp(−C t √n)

Theorem (ALPT, ’10) If X is isotropic log-concave then with probability greater than 1 − 2 exp(−c√n), sup

a∈Sn−1

• 1

N

N

• j=1
• Xi, a2 − EXi, a2
• ≤ C

n N .

Log-concave random vectors

Theorem (Paouris, ’06). If X is an isotropic log-concave random vector in Rn then ∀t ≥ 1, P

• |X|2 ≥ c t √n
• ≤ exp(−C t √n)

Theorem (ALPT, ’10) If X is isotropic log-concave then with probability greater than 1 − 2 exp(−c√n), sup

a∈Sn−1

• 1

N

N

• j=1
• Xi, a2 − EXi, a2
• ≤ C

n N .

• 1

N

N

• j=1

XjXT

j − Id

• ≤ ε

Take N ≈ n

ε2.

Relaxation of the assumptions.

Several recent works in this direction : Srivastava-Vershynin, Vershynin, Mendelson-Paouris

Relaxation of the assumptions.

Several recent works in this direction : Srivastava-Vershynin, Vershynin, Mendelson-Paouris Theorem (Vershynin, ’11) If X is an isotropic random vector in Rn such that |X| ≤ K√n a.s. and ∀θ ∈ Rn, E|X, θ|q ≤ Lq for some q > 4 then with proba greater than 1 − δ sup

a∈Sn−1

• 1

N

N

• j=1
• Xi, a2 − 1
• ≤ Cq,L,δ(log log n)2 n

N 1

2− 1 q .

Relaxation of the assumptions.

Several recent works in this direction : Srivastava-Vershynin, Vershynin, Mendelson-Paouris Theorem (Mendelson-Paouris ’12) If X is an isotropic random vector in Rn such that |X|2 ≤ K(Nn)1/4 a.s. and ∀θ ∈ Rn, E|X, θ|q ≤ Lq for some q > 8 then with probability greater than 1 − ( 1 Nβ + exp(−cn)) sup

a∈Sn−1

• 1

N

N

• j=1
• Xi, a2 − 1
• ≤ Cq,L,K

n N .

Result

Theorem (GLPT ’14). If X1, . . . , XN are random vectors in Rn such that ∀j, ∀θ ∈ Rn, ∀t > 0 P (|Xj, θ| > t) ≤ 1 tq for some p ∈ (4, 8] Let ε ≤ min(1, p−4

4 ) and γ = p − 4 − 2ε. Then with

probability greater than 1 − 8 exp(−n) − 2ε−p/2 max(N−3/2, n− p−4

4 )

sup

a∈Sn−1

• 1

N

N

• j=1
• Xi, a2 − EXi, a2
• ≤

C 1 N max |Xi|2

2 + Cp,ε

n N γ/p .

Restricted Isometry Property

Let 1 ≤ m ≤ N, a vector z ∈ RN is said m-sparse if |{j : zj = 0}| ≤ m. And Um = SN−1 ∩ m − sparse. Let T be an n × N matrix, δm(T) is the smallest number such that for every m-sparse vector z (1 − δm(T))|z|2

2 ≤ |Tz|2 2 ≤ (1 + δm(T))|z|2 2

Restricted Isometry Property

Let 1 ≤ m ≤ N, a vector z ∈ RN is said m-sparse if |{j : zj = 0}| ≤ m. And Um = SN−1 ∩ m − sparse. Let T be an n × N matrix, δm(T) is the smallest number such that for every m-sparse vector z (1 − δm(T))|z|2

2 ≤ |Tz|2 2 ≤ (1 + δm(T))|z|2 2

Equivalently δm(T) = sup

z∈Um

• |Tz|2

2 − 1

Restricted Isometry Property

Let 1 ≤ m ≤ N, a vector z ∈ RN is said m-sparse if |{j : zj = 0}| ≤ m. And Um = SN−1 ∩ m − sparse. Let T be an n × N matrix, δm(T) is the smallest number such that for every m-sparse vector z (1 − δm(T))|z|2

2 ≤ |Tz|2 2 ≤ (1 + δm(T))|z|2 2

Equivalently δm(T) = sup

z∈Um

• |Tz|2

2 − 1

• Take T =

1 √nA. Then

δm A √n

• = sup

z∈Um

 1 n

• N
• j=1

zjXj

• 2

2

− 1  

Restricted Isometry Property

Take T =

1 √nA. Observe that δm is increasing in m and

δ1 A √n

• ≤ δm

A √n

• ≤ δ1

A √n

• + sup

z∈Um

 1 n

• N
• j=1

zjXj

• 2

2

−

N

• j=1

z2

j |Xj|2 2

 

Restricted Isometry Property

Take T =

1 √nA. Observe that δm is increasing in m and

δ1 A √n

• ≤ δm

A √n

• ≤ δ1

A √n

• + sup

z∈Um

 1 n

• N
• j=1

zjXj

• 2

2

−

N

• j=1

z2

j |Xj|2 2

  But δ1 A √n

• = max

1≤j≤N

• |Xj|2

2

n − 1

• .

Restricted Isometry Property

Take T =

1 √nA. Observe that δm is increasing in m and

δ1 A √n

• ≤ δm

A √n

• ≤ δ1

A √n

• + sup

z∈Um

 1 n

• N
• j=1

zjXj

• 2

2

−

N

• j=1

z2

j |Xj|2 2

  But δ1 A √n

• = max

1≤j≤N

• |Xj|2

2

n − 1

• .

Define P(δ) = P

• max

1≤j≤N

• |Xj|2

2

n − 1

• ≥ δ
• .

Result

Theorem (GLPT ’14). If X1, . . . , XN are random vectors in Rn such that ∀j, ∀θ ∈ Rn, ∀t > 0 P (|Xj, θ| > t) ≤ 1 tq for some p > 4. Let ε ≤ min(1, p−4

4 ) and γ = p − 4 − 2ε. Let δ ∈ (0, 1) and

m = Cp,ε,δ n N n − 2(2+ε)

p−4−2ε

Then with probability greater than 1 − Cε,p 1 N + N np/4

• − P

δ 2

• we have

δm A √n

• ≤ δ.

The problem.

On average. E sup

a∈Sn−1

• 1

N

N

• j=1
• Xj, a2 − EXj, a2

The problem.

On average. E sup

a∈Sn−1

• 1

N

N

• j=1
• Xj, a2 − E′X′

j, a2

The problem.

On average. EE′ sup

a∈Sn−1

• 1

N

N

• j=1
• Xj, a2 − X′

j, a2

The problem.

On average. EE′ sup

a∈Sn−1

• 1

N

N

• j=1
• Xj, a2 − X′

j, a2

• By symmetry

= EE′Eε sup

a∈Sn−1

• 1

N

N

• j=1

εj

• Xj, a2 − X′

j, a2

The problem.

On average. EE′ sup

a∈Sn−1

• 1

N

N

• j=1
• Xj, a2 − X′

j, a2

• By symmetry

= EE′Eε sup

a∈Sn−1

• 1

N

N

• j=1

εj

• Xj, a2 − X′

j, a2

• ≤ 2EEε sup

a∈Sn−1

• 1

N

N

• j=1

εjXj, a2

The very very first step.

Symmetrization. E sup

a∈Sn−1

• 1

N

N

• j=1
• Xj, a2 − EXj, a2
• ≤

2EEε sup

a∈Sn−1

• 1

N

N

• j=1

εjXj, a2

• How does
• N
• j=1

εjXj, a2

• concentrate ?

The very very first step.

Symmetrization. E sup

a∈Sn−1

• 1

N

N

• j=1
• Xj, a2 − EXj, a2
• ≤

2EEε sup

a∈Sn−1

• 1

N

N

• j=1

εjXj, a2

• How does
• N
• j=1

εjXj, a2

• concentrate ?

subgaussian or non-commutative Khintchine inequalities

The very very first step.

Observation : for any p ≥ 2

• E
• N
• j=1

εjtj

• p1/p

≤ C√p N

• j=1

t2

j

1/2 sub gaussian and

• E
• N
• j=1

εjtj

• p1/p

≤

N

• j=1

|tj| trivial

The very very first step.

Observation : for any p ≥ 2

• E
• N
• j=1

εjtj

• p1/p

≤ C√p N

• j=1

t2

j

1/2 sub gaussian and

• E
• N
• j=1

εjtj

• p1/p

≤

N

• j=1

|tj| trivial Combine

• E
• N
• j=1

εjtj

• p1/p

≤

k

• j=1

t∗

j + C√p

N

• j=k

(t∗

j )2

1/2

The very very first step.

EEε sup

a∈Sn−1

• N
• j=1

εjXj, a2

• ? → tj = Xj, a2

What is sup

a∈Sn−1 k

• j=1

t∗

j = sup a∈Sn−1 k

• j=1

(Xj, a∗)2 ?

The very very first step.

EEε sup

a∈Sn−1

• N
• j=1

εjXj, a2

• ? → tj = Xj, a2

Duality and sparsity

• sup

a∈Sn−1 k

• j=1

t∗

j

1/2 =

• sup

a∈Sn−1 k

• j=1

(Xj, a∗)2 1/2 = sup

a∈Sn−1 sup α∈Sk−1 k

• j=1

αjXj, a∗ = sup

a∈Sn−1 sup α∈Uk k

• j=1

αjXj, a = sup

α∈Uk

• k
• j=1

αj Xj

• 2

The key Lemma. Correct symmetrization

Define Ak by Ak = sup

α∈Uk

• k
• j=1

αj Xj

• 2
• Lemma. For every A, Z > 0,

sup

a∈Sn−1

• N
• j=1
• Xj, a2 − EXj, a2
• ≤ 2A2 + 6√nZ + 8N

2 min(p,4)

with probability larger than 1−4 exp(−n)−4P(Ak > A)−4·9n sup

a∈Sn−1 P

• i>k

(Xi, a∗)4 > Z2

The key Lemma. Correct symmetrization

Define Ak by Ak = sup

α∈Uk

• k
• j=1

αj Xj

• 2
• Lemma. For every A, Z > 0,

sup

a∈Sn−1

• N
• j=1
• Xj, a2 − EXj, a2
• ≤ 2A2 + 6√nZ + 8N

2 min(p,4)

with probability larger than 1−4 exp(−n)−4P(Ak > A)−4·9n sup

a∈Sn−1 P

• i>k

(Xi, a∗)4 > Z2

• Mixture of the trivail bound for the Rademacher

(combinatorics on the rearrangement) and subgaussian.

Second step.

Choose k = n. Then if p > 4 and for all t ≥ 1, for all a ∈ Sn−1, P (|Xi, a| > t) ≤ 1 tp then with proba greater than 1 − 10−n ;

• i>n

(Xi, a∗)4 ≤ Cp N → Z = Cp √ N

Third and main step. Evaluate Ak

A2

k = sup α∈Uk

• k
• j=1

αj Xj

• 2

2

= sup

α∈Uk

• i=j

αi Xi, αj Xj +

k

• j=1

α2

j |Xj|2 2

≤ sup

α∈Uk

• i=j

αi Xi, αj Xj + max

j≤N |Xj|2 2

Third and main step. Evaluate Ak

Call B2

k = sup α∈Uk

• i=j

αi Xi, αj Xj

Third and main step. Evaluate Ak

Call B2

k = sup α∈Uk

• i=j

αi Xi, αj Xj

• Combinatorial argument :
• i=j

αi Xi, αj Xj = 4 2N

• I⊂{1,...,N}
• i∈I

αiXi,

• j/

∈I

αjXj

Third and main step. Evaluate Ak

Call B2

k = sup α∈Uk

• i=j

αi Xi, αj Xj

• Combinatorial argument :
• i=j

αi Xi, αj Xj = 4 2N

• I⊂{1,...,N}
• i∈I

αiXi,

• j/

∈I

αjXj Call Qk(I) = sup

α∈Uk

• i∈I

αiXi,

• j/

∈I

αjXj

• Therefore

B2

k ≤ 4

2N

• I⊂{1,...,N}

Qk(I).

Third and main step. Evaluate Ak

Call B2

k = sup α∈Uk

• i=j

αi Xi, αj Xj

• Combinatorial argument :
• i=j

αi Xi, αj Xj = 4 2N

• I⊂{1,...,N}
• i∈I

αiXi,

• j/

∈I

αjXj Call Qk(I) = sup

α∈Uk

• i∈I

αiXi,

• j/

∈I

αjXj

• Therefore

B2

k ≤ 4

2N

• I⊂{1,...,N}

Qk(I). Study of quadratic forms.

Remarks.

Recent work about the smallest singular value : Srivastava-Vershynin, Oliveira (Roberto Imbuzeiro), Koltchinskii-Mendelson → It is ”easier” in the sense that you need weaker assumption : fourth moment assumption of the linear functional gives the good rate, like in Bai-Yin result. → Reconstruction

THANK YOU