Restricted Isometry Property of Low Dimensional Subspaces and Its - - PowerPoint PPT Presentation

▶
restricted isometry property of low dimensional subspaces
SMART_READER_LITE
LIVE PREVIEW

Restricted Isometry Property of Low Dimensional Subspaces and Its - - PowerPoint PPT Presentation

Jan 13, 2023 9 likes •933 views

Restricted Isometry Property of Low Dimensional Subspaces and Its Application in Compressed Subspace Clustering Co-work with: Gen Li, Yuchen Jiao, Linghang Meng, Qinghua Liu Dept. EE, Tsinghua University May 2018 Yuantao Gu (EE, Tsinghua) RIP

slide-1
SLIDE 1

Restricted Isometry Property of Low Dimensional Subspaces and Its Application in Compressed Subspace Clustering

Yuantao Gu (谷源涛) Co-work with: Gen Li, Yuchen Jiao, Linghang Meng, Qinghua Liu

  • Dept. EE, Tsinghua University

May 2018

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 1 / 92

slide-2
SLIDE 2

Outline

1

Background Dimensionality Reduction and Random Projection Application of Random Projection

2

Motivation Application Theory

3

Restricted Isometry Property (RIP) for Subspaces Preliminary Main Results Application and Usage Related Theories

4

Numerical Results

5

Conclusion

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 2 / 92

slide-3
SLIDE 3

Outline

1

Background Dimensionality Reduction and Random Projection Application of Random Projection

2

Motivation Application Theory

3

Restricted Isometry Property (RIP) for Subspaces Preliminary Main Results Application and Usage Related Theories

4

Numerical Results

5

Conclusion

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 3 / 92

slide-4
SLIDE 4

Outline

1

Background Dimensionality Reduction and Random Projection Application of Random Projection

2

Motivation Application Theory

3

Restricted Isometry Property (RIP) for Subspaces Preliminary Main Results Application and Usage Related Theories

4

Numerical Results

5

Conclusion

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 4 / 92

slide-5
SLIDE 5

High Dimensionality of Big Data

http://opticalnanofjlter.com/ http://www.thelancet.com/cms/

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 5 / 92

slide-6
SLIDE 6

The Curse of Dimensionality

Parsons, Haque, and Liu, Subspace Clustering for High Dimensional Data: A Review, 2004

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 6 / 92

slide-7
SLIDE 7

Principal Components Analysis (PCA)

PCA 1st dimension

Jollifge, Principal Component Analysis, 1986 Ghodsi, Dimensionality Reduction A Short Tutorial, 2006.

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 7 / 92

slide-8
SLIDE 8

Dimensionality Reduction Methods

Locally Linear Embedding

LLE 1st dimension

Laplacian Eigenmaps

LEM 1st dimension 2nd dimension

Isomap

Isomap 1st dimension 2nd dimension

Semidefjnite Embedding

SDE 1st dimension 2nd dimension

Ghodsi, Dimensionality Reduction A Short Tutorial, 2006.

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 8 / 92

slide-9
SLIDE 9

Random Projection

Original data as vector x ∈ RN Random matrix Φ ∈ Rn×N, n < N Dimension-reduced data y ∈ Rn

[ y ] = [ Φ ] [ x ]

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 9 / 92

slide-10
SLIDE 10

Johnson-Lindenstrauss Lemma (JL Lemma)

For any set V of L points in RN, there exists a map f : RN → Rn, n < N, such that for all x1, x2 ∈ V (1 − ε)∥x1 − x2∥2

2 ≤ ∥f(x1) − f(x2)∥2 2 ≤ (1 + ε)∥x1 − x2∥2 2

if n is a positive integer satisfying n ≥ 4lnL ε2/2 − ε3/3 where 0 < ε < 1 is a constant

Johnson and Lindenstrauss, Extensions of Lipschitz Maps into a Hilbert Space, 1984.

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 10 / 92

slide-11
SLIDE 11

Outline

1

Background Dimensionality Reduction and Random Projection Application of Random Projection

2

Motivation Application Theory

3

Restricted Isometry Property (RIP) for Subspaces Preliminary Main Results Application and Usage Related Theories

4

Numerical Results

5

Conclusion

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 11 / 92

slide-12
SLIDE 12

Compressed Sensing

single pixel camera Sparse MRI Modulated Wideband Converter

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 12 / 92

slide-13
SLIDE 13

Compressed Sensing

http://www.web.me.iastate.edu/sbhattac/research_cs.html

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 13 / 92

slide-14
SLIDE 14

Restricted Isometry Property (RIP) for Sparse Signals

The projection matrix Φ ∈ Rn×N, n < N satisfjes RIP with δ as the smallest nonnegative constant, such that (1 − δ)∥x1 − x2∥2

2 ≤ ∥Φx1 − Φx2∥2 2 ≤ (1 + δ)∥x1 − x2∥2 2

holds for any two k-sparse vectors x1, x2 ∈ RN A Gaussian random matrix Φ has the RIP for n ≥ c1kln (N k ) with probability 1 − e−c2n where c1, c2 > 0 are constants depending only on δ

Candès and Tao, Decoding by linear programming, 2005. Candès, The restricted isometry property and its implications for compressed sensing, 2008

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 14 / 92

slide-15
SLIDE 15

RIP for Low-Dimensional Signal Models

Baraniuk, Cevher, and Wakin, Low-Dimensional Models for Dimensionality Reduction and Signal Recovery: A Geometric Perspective, 2010

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 15 / 92

slide-16
SLIDE 16

Least Squares Approximation

Overconstrained least-squares approximation problem Find a vector x such that Ax ≈ b, A ∈ Rm×n, m ≫ n xopt = arg min

x ∥Ax − b∥2

Approximating least-squares approximation Randomly sample and rescale r = O(n log n/ε2) rows of A and b(denoted by S) Solve the induced subproblem ˜ xopt = arg min

x ∥SAx − Sb∥2

Drineas, Mahoney, and Muthukrishnan, Sampling algorithms for ℓ2 regression and applications, 2006

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 16 / 92

slide-17
SLIDE 17

Least Squares Approximation

The relative-error bounds of the form ∥b − A˜ xopt∥2 ≤ (1 + ε)∥b − Axopt∥2 ∥xopt − ˜ xopt∥2 ≤ c√εκ(A)∥xopt∥2 fail with a probability δ that is no greater than a constant, where κ(A) denotes the conditional number of A

Drineas, Mahoney, and Muthukrishnan. Relative-error CUR matrix decompositions, 2008 Drineas, Mahoney, Muthukrishnan, and Sarlós. Faster least squares approximation, 2010

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 17 / 92

slide-18
SLIDE 18

Support Vector Machine

Normalized Margin: A data set S is linearly separable by margin γ if there exists u ∈ Rd, such that for all (x, y) ∈ S, y ⟨u, x⟩ ∥u∥∥x∥ ≥ γ

https://www.safaribooksonline.com/library/view/python-machine- learning/9781783555130/graphics/

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 18 / 92

slide-19
SLIDE 19

Support Vector Machine

Given a random Gaussian matrix R ∈ Rn×N, if the data set S = {(xi ∈ RN, yi ∈ {−1, 1})}M

i=1

is linearly separable by margin γ ∈ (0, 1], then for any δ, ε ∈ (0, 1) and any n > 12 3ε2 − 2ε3 ln 6M δ , with probability at least 1 − δ, the data set S′ = {(Rxi ∈ Rn, yi ∈ {−1, 1})}M

i=1

is linearly separable by margin γ −

2ε 1−ε.

Shi, Shen, Hill, and van den Hengel, Is margin preserved after random projection? 2012

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 19 / 92

slide-20
SLIDE 20

Review of Background

To solve big data problems dimensionality reduction: random projection Theories for dimensionality reduction and its applications Johnson-Lindenstrauss (JL) Lemma Restricted Isometry Property (RIP) for sparse signals

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 20 / 92

slide-21
SLIDE 21

Outline

1

Background Dimensionality Reduction and Random Projection Application of Random Projection

2

Motivation Application Theory

3

Restricted Isometry Property (RIP) for Subspaces Preliminary Main Results Application and Usage Related Theories

4

Numerical Results

5

Conclusion

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 21 / 92

slide-22
SLIDE 22

Outline

1

Background Dimensionality Reduction and Random Projection Application of Random Projection

2

Motivation Application Theory

3

Restricted Isometry Property (RIP) for Subspaces Preliminary Main Results Application and Usage Related Theories

4

Numerical Results

5

Conclusion

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 22 / 92

slide-23
SLIDE 23

Large Volume of Big Data

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 23 / 92

slide-24
SLIDE 24

Subspace Clustering

Parsons, Haque, and Liu, Subspace Clustering for High Dimensional Data: A Review, 2004 René Vidal, Subspace Clustering, 2011

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 24 / 92

slide-25
SLIDE 25

Applications

Motion trajectories (Costeira and Kanade, 1998) Face images (Basri and Jacobs, 2003) Gene expression data (Jiang, Tang, and Zhang, 2004) Social graphs (Jalali, Chen, Sanghavi, et al., 2011) Network hop counts (Eriksson, Balzano, and Nowak, 2012) Movie ratings (Zhang, Fawaz, Ioannidis et al., 2012) Anomaly detection (Mazel, Casas, Fontugne et al, 2015)

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 25 / 92

slide-26
SLIDE 26

Applications

Face Images under Difgerent Illumination

http://vision.ucsd.edu/content/extended- yale-face-database-b-b

Gene Expression Data

  • Task: To classify novel samples

into known disease types (disease diagnosis)

  • Challenge: thousands of genes,

few samples

  • Solution: to apply dimensionality

reduction

Image Courtesy of Affymetrix Expression Microarray Expression Microarray Data Set

Yu, Ye, and Liu, Dimensionality Reduction for Data Mining - Techniques, Applications and Trends, 2007

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 26 / 92

slide-27
SLIDE 27

Algorithms for Subspace Clustering

Expectation-Maximization-style methods K-plane (Bradley and Mangasarian, 2000) Q-fmat (Tseng, 2000) Algebraic methods Matrix factorization methods (Costeira and Kanade, 1998) Generalized PCA (Vidal, Ma, and Sastry, 2005) Bottom-up local affjnity-based methods Local sampling and estimation (Yan and Pollefeys, 2006) Optimization-based approaches Sparse Subspace Clustering (Elhamifar and Vidal, 2009) Low Rank Representation (Liu, Lin, Yan et al., 2013)

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 27 / 92

slide-28
SLIDE 28

Sparse Subspace Clustering Algorithm

Input: data X = [x1, · · · , xM] ∈ RN×M, all x are normalized Step 1: to learn a sparse self-representation matrix C ∈ RM×M min

ci ∥ci∥1 + λ∥xi − Xci∥2 2,

subject to cii = 0 Step 2: to assign the M samples into disjoint clusters spectral clustering on the similarity matrix |C| + |C|T

Elhamifar and Vidal, Sparse Subspace Clustering, 2009

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 28 / 92

slide-29
SLIDE 29

Compressed Subspace Clustering

clustering in RN

− − − − − − − − − − →

(N is large)

↓ random projection (compression) ↓ more likely to fail?

clustering in Rn

− − − − − − − − − − →

(n≪N)

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 29 / 92

slide-30
SLIDE 30

Advantages

Effjcient for solving the large-scale problems Data privacy: clustering without seeing the raw data No access to raw data

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 30 / 92

slide-31
SLIDE 31

Compressed Subspace Clustering

clustering in RN

− − − − − − − − − − →

(N is large)

↓ random projection (compression) ↓ more likely to fail?

clustering in Rn

− − − − − − − − − − →

(n≪N)

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 31 / 92

slide-32
SLIDE 32

Sparse Subspace Clustering - Performance Analysis

introduce affjnity and sampling rate for feasible analysis provide geometric insights small affjnity + high sampling rate large affjnity + high sampling rate small affjnity + low sampling rate clustering success clustering failure clustering failure

Soltanolkotabi and Candès, A Geometric Analysis of Subspace Clustering with Outliers, 2012

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 32 / 92

slide-33
SLIDE 33

Compressed Subspace Clustering

clustering in RN

− − − − − − − − − − →

(N is large)

↓ random projection (compression) ↓ more likely to fail?

clustering in Rn

− − − − − − − − − − →

(n≪N)

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 33 / 92

slide-34
SLIDE 34

Literatures

Krishnamurthy, Azizyan, and Singh, Subspace Learning from Extremely Compressed Measurements, arXiv, 2014, 2016 Heckel, Tschannen, and Bölcskei, Subspace Clustering of Dimensionality-reduced Data, ISIT 2014 Mao and Gu, Compressed Subspace Clustering: A Case Study, GlobalSIP, 2014 Heckel, Tschannen, and Bölcskei, Dimensionality-reduced Subspace Clustering, arXiv, 2015 Mao, Wang, and Gu, Downsampling for Sparse Subspace Clustering, ICASSP, 2015 Wang, Wang, and Singh, A Deterministic Analysis of Noisy Sparse Subspace Clustering for Dimensionality-reduced Data, ICML, 2015 Arpit, Nwogu, and Govindaraju, Dimensionality Reduction with Subspace Structure Preservation, arXiv, 2016 Wang, Wang, and Singh, A Theoretical Analysis of Noisy Sparse Subspace Clustering on Dimensionality-Reduced Data, arXiv, 2016

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 34 / 92

slide-35
SLIDE 35

Literatures

(modifjed from) Baraniuk, Cevher, and Wakin, Low-Dimensional Models for Dimensionality Reduction and Signal Recovery: A Geometric Perspective, 2010

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 35 / 92

slide-36
SLIDE 36

Outline

1

Background Dimensionality Reduction and Random Projection Application of Random Projection

2

Motivation Application Theory

3

Restricted Isometry Property (RIP) for Subspaces Preliminary Main Results Application and Usage Related Theories

4

Numerical Results

5

Conclusion

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 36 / 92

slide-37
SLIDE 37

Johnson-Lindenstrauss Lemma

For any set V of L points in RN, there exists a map f : RN → Rn, n < N, such that for all x1, x2 ∈ V (1 − ε)∥x1 − x2∥2

2 ≤ ∥f(x1) − f(x2)∥2 2 ≤ (1 + ε)∥x1 − x2∥2 2

if n is a positive integer satisfying n ≥ 4lnL ε2/2 − ε3/3 where 0 < ε < 1 is a constant

Johnson and Lindenstrauss, Extensions of Lipschitz Maps into a Hilbert Space, 1984.

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 37 / 92

slide-38
SLIDE 38

Johnson-Lindenstrauss Lemma for Subspaces (?)

For any set V of L points in RN, there exists a map f : RN → Rn, n < N, such that for all x1, x2 ∈ V (1 − ε)∥x1 − x2∥2

2 ≤ ∥f(x1) − f(x2)∥2 2 ≤ (1 + ε)∥x1 − x2∥2 2

if n is a positive integer satisfying n ≥ 4lnL ε2/2 − ε3/3 where 0 < ε < 1 is a constant

Johnson and Lindenstrauss, Extensions of Lipschitz Maps into a Hilbert Space, 1984.

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 38 / 92

subspaces(?)

slide-39
SLIDE 39

Restricted Isometry Property

The projection matrix Φ ∈ Rn×N, n < N satisfjes RIP with δ as the smallest nonnegative constant, such that (1 − δ)∥x1 − x2∥2

2 ≤ ∥Φx1 − Φx2∥2 2 ≤ (1 + δ)∥x1 − x2∥2 2

holds for any two k-sparse vectors x1, x2 ∈ RN A Gaussian random matrix Φ has the RIP for n ≥ c1kln (N k ) with probability 1 − e−c2n where c1, c2 > 0 are constants depending only on δ

Candès and Tao, Decoding by linear programming, 2005. Candès, The restricted isometry property and its implications for compressed sensing, 2008

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 39 / 92

slide-40
SLIDE 40

Restricted Isometry Property for Subspaces (?)

The projection matrix Φ ∈ Rn×N, n < N satisfjes RIP with δ as the smallest nonnegative constant, such that (1 − δ)∥x1 − x2∥2

2 ≤ ∥Φx1 − Φx2∥2 2 ≤ (1 + δ)∥x1 − x2∥2 2

holds for any two k-sparse vectors x1, x2 ∈ RN A Gaussian random matrix Φ has the RIP for n ≥ c1kln (N k ) with probability 1 − e−c2n where c1, c2 > 0 are constants depending only on δ

Candès and Tao, Decoding by linear programming, 2005. Candès, The restricted isometry property and its implications for compressed sensing, 2008

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 40 / 92

low-dimensional subspaces(?)

slide-41
SLIDE 41

Outline

1

Background Dimensionality Reduction and Random Projection Application of Random Projection

2

Motivation Application Theory

3

Restricted Isometry Property (RIP) for Subspaces Preliminary Main Results Application and Usage Related Theories

4

Numerical Results

5

Conclusion

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 41 / 92

slide-42
SLIDE 42

Outline

1

Background Dimensionality Reduction and Random Projection Application of Random Projection

2

Motivation Application Theory

3

Restricted Isometry Property (RIP) for Subspaces Preliminary Main Results Application and Usage Related Theories

4

Numerical Results

5

Conclusion

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 42 / 92

slide-43
SLIDE 43

Principal Angles (Canonical Angles)

Two subspaces X1, X2 ⊂ RN with dimensions d1 ≤ d2 θi, i = 1, · · · , d1 between X1, X2 cos θi := max

u1∈X1 max u2∈X2

uT

1 u2

∥u1∥∥u2∥ =: uT

1,iu2,i

∥u1,i∥∥u2,i∥ with orthogonality constraints uT

k uk,j = 0,

j = 1, . . . , i − 1, k = 1, 2

  • C. Jordan, Essai sur la géométrieà n

dimensions, Bulletin de la Société mathématique de France, 1875

RN X1 X2 O u1,1 u2,1 θ1 u1,2 u2,2 u1,d1 u2,d1 θd1

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 43 / 92

slide-44
SLIDE 44

Principal Angles and Singular Values

Uk = [ uk,1 ∥uk,1∥, · · · , uk,dk uk,dk ] :

  • rthonormal bases for Xk, k = 1, 2

UT

2 U1 =

     cos θ1 ... cos θd1      λ1 ≥ λ2 ≥ · · · ≥ λd1 ≥ 0: singular values of UT

2 U1

cos θi = λi, i = 1, · · · , d1

  • C. Jordan, Essai sur la géométrieà n

dimensions, Bulletin de la Société mathématique de France, 1875

RN X1 X2 O u1,1 u2,1 θ1 u1,2 u2,2 u1,d1 u2,d1 θd1

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 44 / 92

slide-45
SLIDE 45

Affjnity

Affjnity between X1 and X2 with dimension d1 ≤ d2 aff(X1, X2) := ( d1 ∑

i=1

cos2 θi ) 1

2

= ( d1 ∑

i=1

λ2

i

) 1

2

Algebraic approach aff(X1, X2) = ∥UT

2 U1∥F

Soltanolkotabi and Candès, A Geometric Analysis of Subspace Clustering with Outliers, 2012

RN X1 X2 O u1,1 u2,1 θ1 u1,2 u2,2 u1,d1 u2,d1 θd1 cos θ1 cos θ2 cos θd1

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 45 / 92

slide-46
SLIDE 46

Metric Space for Subspaces

Subspaces X1 and X2 with dimensions d1 = d2 = d Projection F-norm distance (Chordal distance) D(X1, X2) := ( d ∑

i=1

sin2 θi ) 1

2

= 1 √ 2∥U1UT

1 − U2UT 2 ∥F

which does not hold for d1 ̸= d2

Edelman, Arias, and Smith, The geometry of algorithms with orthogonality constraints, SIAM Journal on Matrix Analysis and Applications, 1998

RN X1 X2 O u1,1 u2,1 θ1 u1,2 u2,2 u1,d1 u2,d1 θd1 sin θ1 sin θ2 sin θd1

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 46 / 92

slide-47
SLIDE 47

Metric Space for Subspaces

Subspaces X1 and X2 with dimensions d1 ≤ d2 Projection F-norm distance D(X1, X2) := ( d1 ∑

i=1

sin2 θi + d2 − d1 2 ) 1

2

:= 1 √ 2∥U1UT

1 − U2UT 2 ∥F

RN X1 X2 O u1,1 u2,1 θ1 u1,2 u2,2 u1,d1 u2,d1 θd1 sin θ1 sin θ2 sin θd1 u2,d2 1 √ 2

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 47 / 92

slide-48
SLIDE 48

Affjnity and Distance

D2(X1, X2) =d1 + d2 2 − aff2(X1, X2) RN X1 X2 O u1,1 u2,1 θ1 u1,2 u2,2 u1,d1 u2,d1 θd1 cos θ1 cos θ2 cos θd1 sin θ1 sin θ2 sin θd1 u2,d2 1 √ 2

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 48 / 92

slide-49
SLIDE 49

Projection of Subspaces

Φ = (φi,j) ∈ Rn×N, n < N, φi,j ∼ N(0, 1/n) Xk

Φ

− → Yk = {y|y = Φx, ∀x ∈ Xk} Assumption: Dimensions of subspaces d1 ≤ d2 < n remain unchanged after random projection (reasonable in statistics) For short, denote affX = aff(X1, X2), DX = D(X1, X2), and affY = aff(Y1, Y2), DY = D(Y1, Y2).

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 49 / 92

slide-50
SLIDE 50

Concentration Inequalities for Gaussian Distribution

Lemma Let A be an N × n matrix whose elements aij are independent Gaussian random variables. Then for every t ≥ 0, one has P ( smax(A) ≥ √ N + √n + t ) ≤ e− t2

2 ,

and P ( smin(A) ≤ √ N − √n − t ) ≤ e− t2

2 .

Davidson, Szarek, Local operator theory, random matrices and Banach spaces, Handbook in Banach Spaces, (2001) 317–366.

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 50 / 92

slide-51
SLIDE 51

Concentration Inequalities for Gaussian Distribution

Assume that a ∈ Rn is a standard Gaussian random vector. For any ε > 0, we have P ( ∥a∥2 − 1

  • > ε

) < e−cn hold for n > n0, where n0 and c are constants dependent on ε. Let a ∈ Rn be a standard Gaussian random vector. For any given

  • rthonormal matrix V = [v1, · · · , vd] ∈ Rn×d and ε > 0, we have

P (

  • ∥VTa∥2 − d

n

  • > ε

) < e−c2n hold for n > c1d, where c1, c2 are constants dependent on ε.

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 51 / 92

slide-52
SLIDE 52

Outline

1

Background Dimensionality Reduction and Random Projection Application of Random Projection

2

Motivation Application Theory

3

Restricted Isometry Property (RIP) for Subspaces Preliminary Main Results Application and Usage Related Theories

4

Numerical Results

5

Conclusion

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 52 / 92

slide-53
SLIDE 53

Main Results

Any set of L subspaces X1, · · · , XL ⊂ RN with dimension ≤ d Projected into Rn by a Gaussian random matrix Φ, n < N Xi

Φ

− → Yi := {y|y = Φx, ∀x ∈ Xi}, i = 1, · · · , L Projection F-norm distance between Xi and Xj defjned as D(Xi, Xj) := 1 √ 2∥UiUT

i − UjUT j ∥F

where Ui and Uj denotes the orthonormal basis of Xi and Xj There exist constants c1, c2 > 0 depending only on ε such that for any n > c1 max{d, ln L} (1 − ε) D2(Xi, Xj) < D2(Yi, Yj) < (1 + ε) D2(Xi, Xj) holds with probability at least 1 − e−c2n

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 53 / 92

slide-54
SLIDE 54

Proof Outline

Step 1. Concentration of affjnity between projected line and subspace Step 2. Concentration of affjnity between two projected subspaces Step 3. Restricted Isometry Property for subspaces

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 54 / 92

slide-55
SLIDE 55

Step 1. Projecting Line and Subspace

Concentration of affjnity between projected line and subspace Subspaces X1, X2 ⊂ RN with dimension 1, d Let λ = affX Projected to Y1, Y2 ⊂ Rn by a Gaussian matrix Φ The affjnity after projection can be estimated by aff

2 Y = λ2 + d

n ( 1 − λ2) There exist constants c1, c2 > 0 depending only on ε such that for any n > c1d

  • aff2

Y − aff 2 Y

  • <

( 1 − λ2) ε holds with probability at least 1 − e−c2n

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 55 / 92

slide-56
SLIDE 56

Step 1. Projecting Line and Subspace

Proof outline 1) Introduce variables 2) Intuition: why estimate aff2

Y by λ2 + d n

( 1 − λ2) ? 3) Derive the estimation error 4) Bound the error by concentration inequality

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 56 / 92

slide-57
SLIDE 57

1) Introduce variables

Before projection Denote u as basis of X1 Decompose u onto X2, X ⊥

2

u = λu1 + √ 1 − λ2u0, Choose {u2, · · · , ud} to build

  • rthonormal basis for X2

U = [u1, ..., ud] O RN X2 u X1 u0 u1 u2 u3 ud θ

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 57 / 92

slide-58
SLIDE 58

1) Introduce variables

After projection a = Φu = λΦu1 + √ 1 − λ2Φu0 = λa1 + √ 1 − λ2a0 ΦU no longer orthonormal Orthogonalization ΦU

Gram-Schmidt process

− − − − − − − − − − − − − − → V Aim to estimate aff2

Y =

  • VT a

∥a∥

  • 2

O Rn Y2 a = Φu Y1 a0 = Φu0 a1 = Φu1 v1 = a1/∥a1∥ v2 v3 vd

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 58 / 92

slide-59
SLIDE 59

2) why estimate aff2

Y by λ2 + d n

( 1 − λ2) ?

O RN X2 u X1 u0 u1 u2 u3 ud θ 1 − λ2 λ2 O Rn Y2 a = Φu Y1 a0 = Φu0 a1 = Φu1 v1 = a1/∥a1∥ v2 v3 vd 1 − λ2 λ2 d n(1 − λ2)

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 59 / 92

slide-60
SLIDE 60

3) Derive the estimation error

  • aff2

Y − aff 2 Y

  • =
  • VT a

∥a∥

  • 2

− ( λ2 + d n ( 1 − λ2))

  • = (1 − λ2)
  • (

1 − d n ) − (∥a0∥2 ∥a∥2 − ∥VTa0∥2 ∥a∥2 )

  • O

Rn Y2 a = Φu Y1 a0 = Φu0 a1 = Φu1 v1 = a1/∥a1∥ v2 v3 vd 1 − λ2 λ2 d n(1 − λ2)

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 60 / 92

slide-61
SLIDE 61

4) Bound the error by concentration inequality

  • aff2

Y − aff 2 Y

  • ≤

( 1 − λ2) (

  • ∥a0∥2

∥a∥2 − 1

  • +

1 ∥a∥2

  • ∥VTa0∥2 − d

n

  • + d

n

  • 1

∥a∥2 − 1

  • )

For n > c1,2(ε2)d, with probability at least 1 − 2e−c2,1(ε1)n − e−c2,2(ε2)n, we have

  • ∥a∥2 − 1
  • < ε1,
  • ∥a0∥2 − 1
  • < ε1,
  • VTa0
  • 2 − d

n

  • < ε2.

Complete the proof.

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 61 / 92

slide-62
SLIDE 62

4) Bound the error by concentration inequality

  • aff2

Y − aff 2 Y

  • ≤

( 1 − λ2) (

  • ∥a0∥2

∥a∥2 − 1

  • +

1 ∥a∥2

  • ∥VTa0∥2 − d

n

  • + d

n

  • 1

∥a∥2 − 1

  • )

For n > c1,2(ε2)d, with probability at least 1 − 2e−c2,1(ε1)n − e−c2,2(ε2)n, we have

  • ∥a∥2 − 1
  • < ε1,
  • ∥a0∥2 − 1
  • < ε1,
  • VTa0
  • 2 − d

n

  • < ε2.

Complete the proof.

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 62 / 92

slide-63
SLIDE 63

Step 2. Projecting Two Subspaces

Concentration of affjnity between two projected subspaces Subspaces X1, X2 ⊂ RN with dimension d1 ≤ d2 Projected to Y1, Y2 ⊂ Rn by a Gaussian matrix Φ The affjnity after projection can be estimated by aff

2 Y = aff2 X + d2

n (d1 − aff2

X )

There exist constants c1, c2 > 0 depending only on ε such that for any n > c1d2

  • aff2

Y − aff 2 Y

  • <

( d1 − aff2

X

) ε holds with probability at least 1 − e−c2n

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 63 / 92

slide-64
SLIDE 64

Step 2. Projecting Two Subspaces

Proof outline 1) Introduce variables 2) An important lemma

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 64 / 92

slide-65
SLIDE 65

1) Introduce variables

Before projection Choose orthonormal bases U1 = (u1,k), U2 = (u2,k) for X1, X2, resp. UT

2 U1 =

     λ1 ... λd1      After projection, for i = 1, 2 Original bases change to Ai = (ai,k) = ΦUi Normalize each column as ¯ Ai = (¯ ai,k) = ( ai,k ∥ai,k∥ ) Orthogonalize ¯ Ai

Gram-Schmidt process

− − − − − − − − − − − − − − → Vi Aim to estimate aff2

Y =

  • VT

2 V1

  • 2

F

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 65 / 92

slide-66
SLIDE 66

2) An important lemma

Lemma There exist constants c1,1, c2,1 > 0 depending only on ε, such that for any n > c1,1d2, with probability at least 1 − e−c2,1n, we have

  • ∥VT

2 v1,k∥2 −

  • VT

2 ¯

a1,k

  • 2
  • ≤

( 1 − λ2

k

) ε, ∀k = 1, · · · , d1. Proof Calculate and estimate

  • ∥VT

2 v1,k∥2 −

  • VT

2 ¯

a1,k

  • 2
  • ≤ 3

2 ( 1 − ˆ λ2

k + 1 − β2 k

) ( α2

k +

  • αk⟨¯

a⊥

1,k, b⊥ k ⟩

  • )

, where 1 − ˆ λ2

k ≤ (1 − λ2 k)(1 + ε1),

w.p. 1 − e−c2,1(ε1), ∀n > c1,1(ε1)d2 1 − β2

k ≤ (1 − λ2 k)(1 + ε2),

w.p. 1 − e−c2,2(ε1), ∀n > c1,2(ε1)d2 α2

k ≤ ε3,

w.p. 1 − e−c2,3(ε1), ∀n > c1,3(ε1)d2

  • ⟨¯

a⊥

1,k, b⊥ k ⟩

  • 2 ≤ ε4,

w.p. 1 − e−c2,4(ε1), ∀n > c1,4(ε1)d2 Complete the proof.

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 66 / 92

slide-67
SLIDE 67

2) An important lemma: Intuition

Rn Y1,1:k−1

αkbk ¯ a1,k

O Rn Y2 Y1,1:k−1

αkbk αkβk ¯ a1,k ˆ λk

O Rn Y2 Y1,1:k−1

αkbk αkβk

Y⊥

2

¯ a1,k ˆ λk

O Rn Y2 Y1,1:k−1

αkbk αkβk

Y⊥

2

√ 1 − ˆ λ2

k¯

a⊥

1,k

¯ a1,k ˆ λk αk √ 1 − β2

kb⊥ k

O αk = ∥VT

1,1:k−1¯

a1,k∥ bk = 1 αk PY1,1:k−1(¯ a1,k) αk= o(1) ˆ λk = ∥VT

2 ¯

a1,k∥ βk = ∥VT

2 bk∥

ˆ λk= λk(1 + o(1)) βk= λk(1 + o(1)) ¯ a⊥

1,k =

PY⊥

2 (¯

a1,k) ∥PY⊥

2 (¯

a1,k)∥ b⊥

k =

PY⊥

2 (bk)

∥PY⊥

2 (bk)∥

⟨¯ a⊥

1,k, b⊥ k ⟩= o(1)

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 67 / 92

slide-68
SLIDE 68

2) An important lemma

Lemma There exist constants c1,1, c2,1 > 0 depending only on ε, such that for any n > c1,1d2, with probability at least 1 − e−c2,1n, we have

  • ∥VT

2 v1,k∥2 −

  • VT

2 ¯

a1,k

  • 2
  • ≤

( 1 − λ2

k

) ε, ∀k = 1, · · · , d1. Proof Calculate and estimate

  • ∥VT

2 v1,k∥2 −

  • VT

2 ¯

a1,k

  • 2
  • ≤ 3

2 ( 1 − ˆ λ2

k + 1 − β2 k

) ( α2

k +

  • αk⟨¯

a⊥

1,k, b⊥ k ⟩

  • )

, where 1 − ˆ λ2

k ≤ (1 − λ2 k)(1 + ε1),

w.p. 1 − e−c2,1(ε1)n, ∀n > c1,1(ε1)d2 1 − β2

k ≤ (1 − λ2 k)(1 + ε2),

w.p. 1 − e−c2,2(ε1)n, ∀n > c1,2(ε1)d2 α2

k ≤ ε3,

w.p. 1 − e−c2,3(ε1)n, ∀n > c1,3(ε1)d2

  • ⟨¯

a⊥

1,k, b⊥ k ⟩

  • 2 ≤ ε4,

w.p. 1 − e−c2,4(ε1)n, ∀n > c1,4(ε1)d2 Complete the proof.

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 68 / 92

slide-69
SLIDE 69

Step 3. Restricted Isometry Property for Subspaces

Restricted Isometry Property for two subspaces Subspaces X1, X2 ⊂ RN with dimension d1 ≤ d2 Projected to subspaces Y1, Y2 ⊂ Rn The distance after compression can be estimated by D

2 Y = D2 X − d2

n ( D2

X − d2 − d1

2 ) There exist constants c1,1, c2,1 > 0 depending only on ε such that for any n > c1,1d2

  • D2

Y − D 2 Y

  • <

( D2

X − d2 − d1

2 ) ε holds with probability at least 1 − e−c2,1n

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 69 / 92

slide-70
SLIDE 70

Proof of Main Result

Fix i, j, assume di ≤ dj ≤ d, when n > max{c1,1dj, 2d/ε}, we have (1 − ε) D2(Xi, Xi) < D2(Yi, Yi) < (1 + ε) D2(Xi, Xi), hold with probability at least 1 − e−c2,1n. For any 1 ≤ i < j ≤ L, the probability is at least 1 − L(L−1)

2

e−c2,1n. If n >

1 c2,1 ln L(L−1) 2

, there exists constant c2 depending only on ε, such that L(L−1)

2

e−c2,1n < e−c2n. Take c1 := max{c1,1, 2

ε, 2 c2,1 }, then when n > c1 max{d, ln L}, conditions

n > c1,1d, n > 2d/ε, and n >

1 c2,1 ln L(L−1) 2

that are required above are all satisfjed, the probability is at least 1 − e−c2n with c2 > 0.

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 70 / 92

slide-71
SLIDE 71

Main Results

Any set of L subspaces X1, · · · , XL ⊂ RN with dimension ≤ d Projected into Rn by a Gaussian random matrix Φ, n < N Xi

Φ

− → Yi := {y|y = Φx, ∀x ∈ Xi}, i = 1, · · · , L Projection F-norm distance between Xi and Xj defjned as D(Xi, Xj) := 1 √ 2∥UiUT

i − UjUT j ∥F

where Ui and Uj denotes the orthonormal basis of Xi and Xj There exist constants c1, c2 > 0 depending only on ε such that for any n > c1 max{d, ln L} (1 − ε) D2(Xi, Xj) < D2(Yi, Yj) < (1 + ε) D2(Xi, Xj) holds with probability at least 1 − e−c2n

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 71 / 92

slide-72
SLIDE 72

Outline

1

Background Dimensionality Reduction and Random Projection Application of Random Projection

2

Motivation Application Theory

3

Restricted Isometry Property (RIP) for Subspaces Preliminary Main Results Application and Usage Related Theories

4

Numerical Results

5

Conclusion

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 72 / 92

slide-73
SLIDE 73

Application and Usage

Random Projection = ⇒ Increasing Affjnity + Disturbing Data Dimension of X1, . . . , XL ⊂ RN is no more than d. After compression by a Gaussian matrix Φ ∈ Rn×N, there exist constants c1, c2 > 0 depending only on ε such that when n > c1 max{d, ln L}, aff2

Y < aff2 X + dε

holds with probability at least 1 − e−c2n. the data matrix Y = ΦX can be factorized as Y = Y + E = Y = [y1, . . . , yM] + [e1, . . . , eM] , where yi, ∥yi∥2 = 1, distributed in subspaces Y1, . . . , YL ⊂ Rn uniformly, and maxi∥ei∥2 ≤ δ with probability at least 1 − 2Le−(√nδ−

√ d)2.

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 73 / 92

slide-74
SLIDE 74

Compressed SSC

Assume there are Ml pints in the lth subspace, and denote the dimension

  • f the lth subspace as dl.

Let Ml = ρldl + 1. There exist constants c1, c2 > 0 depending only on ε, such that when n > c1 max{d, ln L}, for any given t > 0, by selecting some proper parameter λ, the NFC property holds with a probability no less than 1 − e−c2n − 2Le−(√nδ−

√ d)2/2

− 1 L2 ∑

l̸=l′

e− t

4

(Ml + 1)Ml′ −

L

∑

l=1

Mle−√dlMl, where δ = r(r − µ)/(2 + 7r), with r = minl c(ρl) √

log(ρl) 2dl , c(ρl) is a

constant depending only on ρl, and µ = maxk,l t(log[(Ml + 1)Mk] + log L) √

aff2

X (l,k)+dε

dldk

.

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 74 / 92

slide-75
SLIDE 75

Compressed TSC

Denote the cardinality of the support set in TSC as q, Mmin = minl Ml, d = maxl dl. If q < Mmin/6, there exist constants c1, c2 > 0 depending only on ε, such that for n > c1 max{d, ln L}, when max

k̸=l

√ aff2

X (k, l) + dε

dk ∧ dl + d √n(log(M)−1 + √ 6d−1)2 ≤ 1 15 log M , the NFC property holds with a probability no less than 1 − 11 M −

L

∑

l=1

Mle−c(Ml+1) − e−c2n, where c > 1/20 is a constant.

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 75 / 92

slide-76
SLIDE 76

Outline

1

Background Dimensionality Reduction and Random Projection Application of Random Projection

2

Motivation Application Theory

3

Restricted Isometry Property (RIP) for Subspaces Preliminary Main Results Application and Usage Related Theories

4

Numerical Results

5

Conclusion

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 76 / 92

slide-77
SLIDE 77

Restricted Isometry Property (RIP) for Subspaces

JL Lemma RIP (for sparse sig.) RIP for subspaces

  • bjects

L points xi all k-sparse sig. xi L subspaces Xi metric ∥xi − xj∥2 ∥xi − xj∥2

1 √ 2∥Pi − Pj∥F

compress. map f Gaussian matrix Gaussian matrix

  • err. bound

(1 − ε, 1 + ε) (1 − δ, 1 + δ) (1−ε, 1+ε) condition n ≥

4lnL ε2/2−ε3/3

n ≥ c1kln ( N

k

) n>c1max{d, ln L}

  • suc. prob.

1 1 − e−c2n 1 − e−c2n

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 77 / 92

slide-78
SLIDE 78

Restricted Isometry Property (RIP) for Subspaces

(modifjed from) Baraniuk, Cevher, and Wakin, Low-Dimensional Models for Dimensionality Reduction and Signal Recovery: A Geometric Perspective, 2010

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 78 / 92

slide-79
SLIDE 79

Restricted Isometry Property (RIP) for Subspaces

Feature Deem subspace as a whole Limitation No subspace recovery Application Low-dimensional subspace-related problems Classifjcation, detection, · · ·

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 79 / 92

slide-80
SLIDE 80

Outline

1

Background Dimensionality Reduction and Random Projection Application of Random Projection

2

Motivation Application Theory

3

Restricted Isometry Property (RIP) for Subspaces Preliminary Main Results Application and Usage Related Theories

4

Numerical Results

5

Conclusion

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 80 / 92

slide-81
SLIDE 81

Experiment 1. Concentration of Compressed Affjnity

affY

2

0.5 1 1.5 2 2.5 3 3.5 4 4.5

frequency

500 1000 1500 2000 2500 3000 3500 4000 4500

affX

2 = 1

2 3 4

Figure: Simulated projected affjnity (curves) and theoretical estimate (bars). (N, n) = (500, 200), (d1, d2) = (5, 10), and aff2

X = 1, 2, 3, 4. The frequencies are

counted from 1E5 trials.

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 81 / 92

slide-82
SLIDE 82

Experiment 2. Various (d1, d2)

affX

2

1

affY

2

1

(d1,d2) = (1,5)

0.2 0.4 0.6 0.8 1

1 1

(1,20)

0.2 0.4 0.6 0.8 1

1 2 3 4 5 1 2 3 4 5

(5,5)

0.2 0.4 0.6 0.8 1

1 2 3 4 5 1 2 3 4 5

(5,10)

0.2 0.4 0.6 0.8 1

1 2 3 4 5 1 2 3 4 5

(5,20)

0.2 0.4 0.6 0.8 1

0 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10

(10,20)

0.2 0.4 0.6 0.8 1

Figure: Experimental projected affjnity (gray area) and theoretical estimate (blue line). (N, n) = (500, 200). 500 trials for each aff2

X .

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 82 / 92

slide-83
SLIDE 83

Experiment 3. Various (N, n)

affX

2

1 2 3 4 5

affY

2

1 2 3 4 5

(N,n) = (40,20)

0.2 0.4 0.6 0.8 1

1 2 3 4 5 1 2 3 4 5

(100,20)

0.2 0.4 0.6 0.8 1

1 2 3 4 5 1 2 3 4 5

(200,100)

0.2 0.4 0.6 0.8 1

1 2 3 4 5 1 2 3 4 5

(500,100)

0.2 0.4 0.6 0.8 1

1 2 3 4 5 1 2 3 4 5

(1000,500)

0.2 0.4 0.6 0.8 1

1 2 3 4 5 1 2 3 4 5

(2500,500)

0.2 0.4 0.6 0.8 1

Figure: Experimental projected affjnity (gray area) and the theoretical estimate (blue line). (d1, d2) = (5, 10). 500 trials for each aff2

X .

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 83 / 92

slide-84
SLIDE 84

Experiment 4. Novartis Multi-tissue Gene Expression Data

Broad-Institute, Cancer program data sets, http://www.broadinstitute.org/cgi-bin/cancer/datasets.cgi, 2013.

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 84 / 92

slide-85
SLIDE 85

Experiment 4. Clustering Error (Gene)

n=N

0.01 0.1 1

Clustering Error Rate

0.1 0.2 0.3 0.4

subject 1 vs subject 2 Distance

0.5 1 1.5 2 TSC SSC LRR OMP Distance

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 85 / 92

slide-86
SLIDE 86

Experiment 4. Clustering Error (Gene)

n=N

0.01 0.1 1

Clustering Error Rate

0.1 0.2 0.3 0.4 0.5 1 vs 2 0.5 1 1.5 2 TSC SSC LRR OMP Distance 0.01 0.1 1 0.1 0.2 0.3 0.4 0.5 1 vs 3 0.5 1 1.5 2 0.01 0.1 1 0.1 0.2 0.3 0.4 0.5 1 vs 4

Distance

0.5 1 1.5 2 0.01 0.1 1 0.1 0.2 0.3 0.4 0.5 2 vs 3 0.5 1 1.5 2 0.01 0.1 1 0.1 0.2 0.3 0.4 0.5 2 vs 4 0.5 1 1.5 2 0.01 0.1 1 0.1 0.2 0.3 0.4 0.5 3 vs 4 0.5 1 1.5 2

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 86 / 92

slide-87
SLIDE 87

Experiment 5. Extended YaleB Face Database

http://vision.ucsd.edu/content/extended-yale-face-database-b-b

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 87 / 92

slide-88
SLIDE 88

Experiment 5. Clustering Error (YaleB)

n=N

0.001 0.1 1

Clustering Error Rate

0.1 0.2 0.3 0.4 0.5

subject 1 vs subject 2 Distance

0.5 1 1.5 TSC SSC LRR OMP Distance

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 88 / 92

slide-89
SLIDE 89

Experiment 5. Clustering Error (YaleB)

n=N 0.0001 0.01 1 Clustering Error Rate 0.1 0.2 0.3 0.4 0.5 1 vs 2 0.5 1 1.5 2 2.5

TSC SSC OMP Distance

0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 1 vs 3 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 1 vs 4 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 1 vs 5 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 1 vs 6 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 1 vs 7 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 1 vs 8 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 1 vs 9 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 1 vs 10 Distance 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 2 vs 3 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 2 vs 4 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 2 vs 5 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 2 vs 6 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 2 vs 7 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 2 vs 8 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 2 vs 9 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 2 vs 10 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 3 vs 4 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 3 vs 5 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 3 vs 6 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 3 vs 7 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 3 vs 8 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 3 vs 9 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 3 vs 10 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 4 vs 5 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 4 vs 6 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 4 vs 7 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 4 vs 8 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 4 vs 9 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 4 vs 10 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 5 vs 6 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 5 vs 7 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 5 vs 8 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 5 vs 9 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 5 vs 10 0.5 1 1.5 2 2.5 0.0001 0.01 1 0.1 0.2 0.3 0.4 0.5 6 vs 7 0.5 1 1.5 2 2.5 6 vs 8 6 vs 9 6 vs 10 7 vs 8 7 vs 9 7 vs 10 8 vs 9 8 vs 10 9 vs 10 Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 89 / 92

slide-90
SLIDE 90

Outline

1

Background Dimensionality Reduction and Random Projection Application of Random Projection

2

Motivation Application Theory

3

Restricted Isometry Property (RIP) for Subspaces Preliminary Main Results Application and Usage Related Theories

4

Numerical Results

5

Conclusion

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 90 / 92

slide-91
SLIDE 91

Conclusion

A theory on dimensionality reduction by random projection Generalization of JL Lemma and RIP to the case of compressing subspaces A framework to analyze compressed subspace clustering algorithms [1] Li, Liu, and Gu, Rigorous Restricted Isometry Property of Low-Dimensional Subspaces, submitted, arXiv: 1801.10058 [2] Li and Gu, Restricted Isometry Property of Gaussian Random Projection for Finite Set of Subspaces, IEEE Transactions on Signal Processing, 66(7):1705-1720, 2018 [3] Meng, Li, Yan, and Gu, A General Framework for Understanding Compressed Subspace Clustering Algorithms, submitted, 2018 [4] Jiao, Shen, Li, and Gu, Subspace Principal Angle Preserving Property of Gaussian Random Projection, IEEE DSW, 2018

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 91 / 92

slide-92
SLIDE 92

Q & A

Thank You for Attention gyt@tsinghua.edu.cn http://gu.ee.tsinghua.edu.cn

Yuantao Gu (EE, Tsinghua) RIP for Subspaces May 2018 92 / 92