Dimensionality Reduction Jia-Bin Huang Virginia Tech Spring 2019 - - PowerPoint PPT Presentation

SLIDE 1

Dimensionality Reduction

Jia-Bin Huang Virginia Tech

Spring 2019

ECE-5424G / CS-5824

SLIDE 2

Administrative

• HW 3 due March 27.
• HW 4 out tonight

SLIDE 3

• J. Mark Sowers Distinguished Lecture
• Michael Jordan
• Pehong Chen Distinguished Professor

Department of Statistics and Electrical Engineering and Computer Sciences

• University of California, Berkeley
• 3/28/19
• 7:30 PM, McBryde 100

SLIDE 4

ECE Faculty Candidate Talk

• Siheng Chen
• Ph.D. Carnegie Mellon University
• Data science with graphs: From social network analysis to

autonomous driving

• Time: 10:00 AM - 11:00 AM March 28
• Location: 457B Whittemore

SLIDE 5

Expectation Maximization (EM) Algorithm

• Goal: Find 𝜄 that maximizes log-likelihood σ𝑗 log 𝑞(𝑦 𝑗 ; 𝜄)

σ𝑗 log 𝑞(𝑦 𝑗 ; 𝜄) = σ𝑗 log σ𝑨(𝑗) 𝑞(𝑦 𝑗 , 𝑨(𝑗); 𝜄) = σ𝑗 log σ𝑨(𝑗) 𝑅𝑗 𝑨 𝑗

𝑞 𝑦 𝑗 ,𝑨 𝑗 ;𝜄 𝑅𝑗(𝑨(𝑗))

≥ σ𝑗 σ𝑨(𝑗) 𝑅𝑗 𝑨 𝑗 log

𝑞 𝑦 𝑗 ,𝑨 𝑗 ;𝜄 𝑅𝑗(𝑨(𝑗))

Jensen’s inequality: 𝑔 𝐹 𝑌 ≥ 𝐹[𝑔 𝑌 ]

SLIDE 6

Expectation Maximization (EM) Algorithm

• Goal: Find 𝜄 that maximizes log-likelihood σ𝑗 log 𝑞(𝑦 𝑗 ; 𝜄)

σ𝑗 log 𝑞(𝑦 𝑗 ; 𝜄) ≥ σ𝑗 σ𝑨(𝑗) 𝑅𝑗 𝑨 𝑗 log

𝑞 𝑦 𝑗 ,𝑨 𝑗 ;𝜄 𝑅𝑗(𝑨(𝑗))

• The lower bound works for all possible set of distributions 𝑅𝑗
• We want tight lower-bound: 𝑔 𝐹 𝑌

= 𝐹[𝑔 𝑌 ]

• When will that happen? 𝑌 = 𝐹 𝑌 with probability 1 (𝑌 is a constant)

𝑞 𝑦 𝑗 , 𝑨 𝑗 ; 𝜄 𝑅𝑗(𝑨(𝑗)) = 𝑑

SLIDE 7

How should we choose 𝑅𝑗(𝑨(𝑗))?

• 𝑞 𝑦 𝑗 ,𝑨 𝑗 ;𝜄

𝑅𝑗(𝑨(𝑗))

= 𝑑

• 𝑅𝑗(𝑨(𝑗)) ∝ 𝑞 𝑦 𝑗 , 𝑨 𝑗 ; 𝜄
• σ𝑨 𝑅𝑗(𝑨(𝑗)) = 1 (because it is a distribution)
• 𝑅𝑗 𝑨 𝑗

=

𝑞 𝑦 𝑗 ,𝑨 𝑗 ;𝜄 σ𝑨 𝑞 𝑦 𝑗 ,𝑨 𝑗 ;𝜄 = 𝑞 𝑦 𝑗 ,𝑨 𝑗 ;𝜄 𝑞 𝑦 𝑗 ;𝜄

= 𝑞(𝑨 𝑗 |𝑦 𝑗 ; 𝜄)

SLIDE 8

EM algorithm

Repeat until convergence{ (E-step) For each 𝑗, set 𝑅𝑗 𝑨 𝑗 ≔ 𝑞(𝑨 𝑗 |𝑦 𝑗 ; 𝜄) (Probabilistic inference) (M-step) Set 𝜄 ≔ argmax𝜄 σ𝑗 σ𝑨(𝑗) 𝑅𝑗 𝑨 𝑗 log

𝑞 𝑦 𝑗 ,𝑨 𝑗 ;𝜄 𝑅𝑗(𝑨(𝑗))

}

SLIDE 9

Expectation Maximization (EM) Algorithm

 

     



z

z x  



| , log argmax ˆ p

Goal:

 

   

 

X f X f E E 

Jensen’s Inequality Log of sums is intractable

See here for proof: www.stanford.edu/class/cs229/notes/cs229-notes8.ps

for concave functions f(x) (so we maximize the lower bound!)

SLIDE 10

Expectation Maximization (EM) Algorithm

• 1. E-step: compute
• 2. M-step: solve

   

 

    



) ( , |

, | | , log | , log E

) (

t x z

p p p

t

  



x z z x z x

z





    



) ( ) 1 (

, | | , log argmax

t t

p p   



x z z x

z







 

     



z

z x  



| , log argmax ˆ p

Goal:

SLIDE 11

• 1. E-step: compute
• 2. M-step: solve

   

 

    



) ( , |

, | | , log | , log E

) (

t x z

p p p

t

  



x z z x z x

z





    



) ( ) 1 (

, | | , log argmax

t t

p p   



x z z x

z







 

     



z

z x  



| , log argmax ˆ p

Goal:

 

   

 

X f X f E E 

log of expectation of P(x|z) expectation of log of P(x|z)

SLIDE 12

EM for Mixture of Gaussians - derivation

 



           

m m m m n m

x    

2 2 2 exp

2 1

 

 



 

m m m m n n n

m z x p x p    , , | , , , |

2 2 π

σ μ

1. E-step: 2. M-step:

          



) ( , |

, | | , log | , log E

) (

t x z

p p p

t

  



x z z x z x

z





    



) ( ) 1 (

, | | , log argmax

t t

p p   



x z z x

z







SLIDE 13

EM for Mixture of Gaussians

 



           

m m m m n m

x    

2 2 2 exp

2 1

 

 



 

m m m m n n n

m z x p x p    , , | , , , |

2 2 π

σ μ

1. E-step: 2. M-step:

          



) ( , |

, | | , log | , log E

) (

t x z

p p p

t

  



x z z x z x

z





    



) ( ) 1 (

, | | , log argmax

t t

p p   



x z z x

z







) , , , | (

) ( ) ( 2 ) ( t t t n n nm

x m z p π σ μ     



 n n nm n nm t m

x    1 ˆ

) 1 (

 

 

 

 n m n nm n nm t m

x

2 ) 1 ( 2

ˆ 1 ˆ    

N

n nm t m







 

) 1 (

ˆ

SLIDE 14

EM algorithm - derivation

http://lasa.epfl.ch/teaching/lectures/ML_Phd/Notes/GP-GMM.pdf

SLIDE 15

EM algorithm – E-Step

SLIDE 16

EM algorithm – E-Step

SLIDE 17

EM algorithm – M-Step

SLIDE 18

EM algorithm – M-Step

Take derivative with respect to 𝜈𝑚

SLIDE 19

EM algorithm – M-Step

Take derivative with respect to σ𝑚

−1

SLIDE 20

EM Algorithm for GMM

SLIDE 21

EM Algorithm

• Maximizes a lower bound on the data likelihood at each iteration
• Each step increases the data likelihood
• Converges to local maximum
• Common tricks to derivation
• Find terms that sum or integrate to 1
• Lagrange multiplier to deal with constraints

SLIDE 22

Convergence of EM Algorithm

SLIDE 23

“Hard EM”

• Same as EM except compute z* as most likely values for hidden

variables

• K-means is an example
• Advantages
• Simpler: can be applied when cannot derive EM
• Sometimes works better if you want to make hard predictions at the end
• But
• Generally, pdf parameters are not as accurate as EM

SLIDE 24

Dimensionality Reduction

• Motivation
• Data compression
• Data visualization
• Principal component analysis
• Formulation
• Algorithm
• Reconstruction
• Choosing the number of principal components
• Applying PCA

SLIDE 25

Dimensionality Reduction

• Motivation
• Principal component analysis
• Formulation
• Algorithm
• Reconstruction
• Choosing the number of principal components
• Applying PCA

SLIDE 26

Data Compression

• Reduces the required time and storage space
• Removing multi-collinearity improves the interpretation of the

parameters of the machine learning model. 𝑦(1) ∈ 𝑆2 → 𝑨 1 ∈ 𝑆 𝑦(2) ∈ 𝑆2 → 𝑨 1 ∈ 𝑆 ⋮ 𝑦(𝑛) ∈ 𝑆2 → 𝑨 𝑛 ∈ 𝑆

𝑦2 𝑦1 𝑨1

SLIDE 27

Data Compression

• Reduces the required time and storage space
• Removing multi-collinearity improves the interpretation of the

parameters of the machine learning model. 𝑦(1) ∈ 𝑆2 → 𝑨 1 ∈ 𝑆 𝑦(2) ∈ 𝑆2 → 𝑨 1 ∈ 𝑆 ⋮ 𝑦(𝑛) ∈ 𝑆2 → 𝑨 𝑛 ∈ 𝑆

𝑦2 𝑦1 𝑨1

SLIDE 28

Data Compression

• Reduce data from 3D to 2D (in general 1000D -> 100D)

𝑦1 𝑦3 𝑦2 𝑦1 𝑦3 𝑦2 𝑨1 𝑨2 𝑨1 𝑨2

SLIDE 29

SLIDE 30

Dimensionality Reduction

• Motivation
• Principal component analysis
• Formulation
• Algorithm
• Reconstruction
• Choosing the number of principal components
• Applying PCA

SLIDE 31

Principal Component Analysis Formulation

𝑦2 𝑦1

SLIDE 32

Principal Component Analysis Formulation

• Reduce n-D to k-D: find 𝑣(1), 𝑣(2), ⋯ , 𝑣(𝑙) ∈ 𝑆𝑜 onto which

to project the data, so as to minimize the projection error

𝑦2 𝑦1 𝑣(1) 𝑣(1) 𝑣(2)

SLIDE 33

PCA vs. Linear regression

𝑧 𝑦1 𝑦2 𝑦1

SLIDE 34

Data pre-processing

• Training set: 𝑦(1), 𝑦(2), ⋯ , 𝑦(𝑛)
• Preprocessing (feature scaling/mean normalization)

𝜈𝑘 = 1 𝑛 ෍

𝑗

𝑦𝑘

(𝑗)

Replace each 𝑦𝑘

(𝑗) with 𝑦𝑘 − 𝜈𝑘

If different features on different scales, scale features to have comparable range of values 𝑦𝑘

(𝑗) ←

𝑦𝑘

(𝑗) − 𝜈𝑘

𝑡

𝑘

SLIDE 35

Principal Component Analysis Algorithm

• Goal: Reduce data from n-dimensions to k-dimensions
• Step 1: Compute “covariance matrix”

Σ = 1 𝑛 ෍

𝑗=1 𝑜

𝑦 𝑗 𝑦 𝑗

⊤

• Step 2: Compute “eigenvectors” of the covariance matrix

[U, S, V] = svd(Sigma); U = 𝑣(1), 𝑣(2), ⋯ , 𝑣(𝑜) ∈ 𝑆𝑜×𝑜 Principal components: 𝑣(1), 𝑣(2), ⋯ , 𝑣 𝑙 ∈ 𝑆𝑜

SLIDE 36

Principal Component Analysis Algorithm

• Goal: Reduce data from n-dimensions to k-dimensions
• Principal components: 𝑣(1), 𝑣(2), ⋯ , 𝑣 𝑙 ∈ 𝑆𝑜

𝑨 𝑗 = 𝑣 1 , 𝑣 2 , ⋯ , 𝑣 𝑙

⊤𝑦(𝑗) ∈ 𝑆𝑙

SLIDE 37

PCA algorithm summary

• After mean normalization (ensure every feature has

zero mean) and optionally feature scaling

• Simga =

1 𝑛 σ𝑗=1 𝑜

𝑦 𝑗 𝑦 𝑗

⊤

• [U, S, V] = svd(Sigma);
• Ureduce = U(:, 1:k);
• z = Ureduce’ * x;

SLIDE 38

Reconstruction from compressed representation

• Compression: 𝑨(𝑗) = 𝑉reduce

⊤

𝑦(𝑗)

• Reconstruction: 𝑦approx

(𝑗)

= 𝑉reduce𝑨(𝑗)

• 𝑦approx

(𝑗)

∈ 𝑆𝑜 𝑉reduce ∈ 𝑆𝑜×𝑙 𝑨(𝑗) ∈ 𝑆𝑙×1

𝑦2 𝑦1 𝑦2 𝑦1

SLIDE 39

3D face modeling

A morphable model for the synthesis of 3D faces, SIGGRAPH 1999

SLIDE 40

Shape modeling

SMPL: Skinned multi-person linear model, SIGGRAPH Asia 2015

SLIDE 41

Dimensionality Reduction

• Motivation
• Principal component analysis
• Formulation
• Algorithm
• Reconstruction
• Choosing the number of principal components
• Applying PCA

SLIDE 42

How do we choose k (number of principal components)

• Average squared projection error:

1 m σ𝑗

𝑦 𝑗 − 𝑦approx

𝑗 2

• Total variation in the data:

1 m σ𝑗 𝑦 𝑗 2

• Typically, choose 𝑙 to be the smallest value so that

1 m σ𝑗 𝑦 𝑗 −𝑦approx 𝑗 2 1 m σ𝑗 𝑦 𝑗 2

≤ 0.01 (1%) “99% of variance is retained”

SLIDE 43

How do we choose k (number of principal components)

• Try PCA with k = 1, 2, ⋯
• Compute Ureduce, z(1), z(2), ⋯ , z 𝑛 ,

𝑦approx

1

, 𝑦approx

2

, ⋯ , 𝑦approx

𝑛

• Check if

1 m σ𝑗 𝑦 𝑗 −𝑦approx 𝑗 2 1 m σ𝑗 𝑦 𝑗 2

≤ 0.01 ?

• [U, S, V] = svd(Sigma)
• 𝑇 =

𝑡11 ⋯ ⋮ ⋱ ⋮ ⋯ 𝑡𝑜𝑜

• For given 𝑙

1 − σ𝑗=1

𝑙

𝑡𝑗𝑗 σ𝑗=1

𝑜

𝑡𝑗𝑗 ≤ 0.01 σ𝑗=1

𝑙

𝑡𝑗𝑗 σ𝑗=1

𝑜

𝑡𝑗𝑗 ≥ 0.99

SLIDE 44

Dimensionality Reduction

• Motivation
• Principal component analysis
• Formulation
• Algorithm
• Reconstruction
• Choosing the number of principal components
• Applying PCA

SLIDE 45

Application of PCA

• Compression
• Reduce memory/disk needed to store data
• Speed up learning algorithm
• Visualization (k=2, k=3)
• Bad use of PCA
• Reduce the number of features -> less likely to overfit?
• Use regularization instead.

SLIDE 46

Taxonomy for dimensionality reduction

http://www.math.chalmers.se/Stat/Grundutb/GU/MSA220/S18/DimRed2.pdf

SLIDE 47

Things to remember

• Compression, visualization
• Principal component analysis
• Formulation
• Algorithm
• Reconstruction
• Choosing the number of principal components
• Applying PCA