linear dimensionality reduction
play

Linear Dimensionality Reduction Practical Machine Learning - PowerPoint PPT Presentation

Dec 08, 2023 •461 likes •2.33k views

Linear Dimensionality Reduction Practical Machine Learning (CS294-34) September 24, 2009 Percy Liang Lots of high-dimensional data... According to media reports, a pair of hackers said on Saturday that the Firefox Zambian President Levy

  1. PCA objective 2: projected variance Empirical distribution: uniform over x 1 , . . . , x n Expectation (think sum over data points): � n ˆ E [ f ( x )] = 1 i =1 f ( x i ) n Variance (think sum of squares if centered): � n E [ f ( x )]) 2 = ˆ var [ f ( x )] + (ˆ E [ f ( x ) 2 ] = 1 i =1 f ( x i ) 2 � n Assume data is centered: ˆ E [ x ] = 0 (what’s ˆ E [ U ⊤ x ] ?) Principal component analysis (PCA) / Basic principles 10

  2. PCA objective 2: projected variance Empirical distribution: uniform over x 1 , . . . , x n Expectation (think sum over data points): � n ˆ E [ f ( x )] = 1 i =1 f ( x i ) n Variance (think sum of squares if centered): � n E [ f ( x )]) 2 = ˆ var [ f ( x )] + (ˆ E [ f ( x ) 2 ] = 1 i =1 f ( x i ) 2 � n Assume data is centered: ˆ E [ x ] = 0 (what’s ˆ E [ U ⊤ x ] ?) Objective: maximize variance of projected data ˆ E [ � U ⊤ x � 2 ] max U ∈ R d × k , U ⊤ U = I Principal component analysis (PCA) / Basic principles 10

  3. Equivalence in two objectives Key intuition: variance of data = captured variance + reconstruction error � �� � � �� � � �� � fixed want small want large Principal component analysis (PCA) / Basic principles 11

  4. Equivalence in two objectives Key intuition: variance of data = captured variance + reconstruction error � �� � � �� � � �� � fixed want small want large Pythagorean decomposition: x = UU ⊤ x + ( I − UU ⊤ ) x Principal component analysis (PCA) / Basic principles 11

  5. Equivalence in two objectives Key intuition: variance of data = captured variance + reconstruction error � �� � � �� � � �� � fixed want small want large Pythagorean decomposition: x = UU ⊤ x + ( I − UU ⊤ ) x � x � � ( I − UU ⊤ ) x � � UU ⊤ x � Principal component analysis (PCA) / Basic principles 11

  6. Equivalence in two objectives Key intuition: variance of data = captured variance + reconstruction error � �� � � �� � � �� � fixed want small want large Pythagorean decomposition: x = UU ⊤ x + ( I − UU ⊤ ) x � x � � ( I − UU ⊤ ) x � � UU ⊤ x � Take expectations; note rotation U doesn’t affect length: E [ � x � 2 ] = ˆ ˆ E [ � U ⊤ x � 2 ] + ˆ E [ � x − UU ⊤ x � 2 ] Principal component analysis (PCA) / Basic principles 11

  7. Equivalence in two objectives Key intuition: variance of data = captured variance + reconstruction error � �� � � �� � � �� � fixed want small want large Pythagorean decomposition: x = UU ⊤ x + ( I − UU ⊤ ) x � x � � ( I − UU ⊤ ) x � � UU ⊤ x � Take expectations; note rotation U doesn’t affect length: E [ � x � 2 ] = ˆ ˆ E [ � U ⊤ x � 2 ] + ˆ E [ � x − UU ⊤ x � 2 ] Minimize reconstruction error ↔ Maximize captured variance Principal component analysis (PCA) / Basic principles 11

  8. Finding one principal component Input data: X = ( x 1 . . . x n ) Principal component analysis (PCA) / Basic principles 12

  9. Finding one principal component Objective: maximize variance of projected data Input data: X = ( x 1 . . . x n ) Principal component analysis (PCA) / Basic principles 12

  10. Finding one principal component Objective: maximize variance of projected data ˆ E [( u ⊤ x ) 2 ] = max � u � =1 Input data: X = ( x 1 . . . x n ) Principal component analysis (PCA) / Basic principles 12

  11. Finding one principal component Objective: maximize variance of projected data ˆ E [( u ⊤ x ) 2 ] = max � u � =1 n 1 � ( u ⊤ x i ) 2 = max � u � =1 n i =1 Input data: X = ( x 1 . . . x n ) Principal component analysis (PCA) / Basic principles 12

  12. Finding one principal component Objective: maximize variance of projected data ˆ E [( u ⊤ x ) 2 ] = max � u � =1 n 1 � ( u ⊤ x i ) 2 = max � u � =1 n i =1 1 Input data: n � u ⊤ X � 2 = max � u � =1 X = ( x 1 . . . x n ) Principal component analysis (PCA) / Basic principles 12

  13. Finding one principal component Objective: maximize variance of projected data ˆ E [( u ⊤ x ) 2 ] = max � u � =1 n 1 � ( u ⊤ x i ) 2 = max � u � =1 n i =1 1 Input data: n � u ⊤ X � 2 = max � u � =1 X = ( x 1 . . . x n ) � 1 � n XX ⊤ � u � =1 u ⊤ = max u Principal component analysis (PCA) / Basic principles 12

  14. Finding one principal component Objective: maximize variance of projected data ˆ E [( u ⊤ x ) 2 ] = max � u � =1 n 1 � ( u ⊤ x i ) 2 = max n � u � =1 i =1 1 Input data: n � u ⊤ X � 2 = max � u � =1 X = ( x 1 . . . x n ) � 1 � n XX ⊤ � u � =1 u ⊤ = max u = 1 def n XX ⊤ = largest eigenvalue of C ( C is covariance matrix of data) Principal component analysis (PCA) / Basic principles 12

  15. How many principal components? • Similar to question of “How many clusters?” • Magnitude of eigenvalues indicate fraction of variance captured. Principal component analysis (PCA) / Basic principles 15

  16. How many principal components? • Similar to question of “How many clusters?” • Magnitude of eigenvalues indicate fraction of variance captured. • Eigenvalues on a face image dataset: 1353.2 1086.7 820.1 λ i 553.6 287.1 2 3 4 5 6 7 8 9 10 11 i Principal component analysis (PCA) / Basic principles 15

  17. How many principal components? • Similar to question of “How many clusters?” • Magnitude of eigenvalues indicate fraction of variance captured. • Eigenvalues on a face image dataset: 1353.2 1086.7 820.1 λ i 553.6 287.1 2 3 4 5 6 7 8 9 10 11 i • Eigenvalues typically drop off sharply, so don’t need that many. • Of course variance isn’t everything... Principal component analysis (PCA) / Basic principles 15

  18. Computing PCA Method 1: eigendecomposition n XX ⊤ U are eigenvectors of covariance matrix C = 1 Computing C already takes O ( nd 2 ) time (very expensive) Principal component analysis (PCA) / Basic principles 16

  19. Computing PCA Method 1: eigendecomposition n XX ⊤ U are eigenvectors of covariance matrix C = 1 Computing C already takes O ( nd 2 ) time (very expensive) Method 2: singular value decomposition (SVD) Find X = U d × d Σ d × n V ⊤ n × n where U ⊤ U = I d × d , V ⊤ V = I n × n , Σ is diagonal Computing top k singular vectors takes only O ( ndk ) Principal component analysis (PCA) / Basic principles 16

  20. Computing PCA Method 1: eigendecomposition n XX ⊤ U are eigenvectors of covariance matrix C = 1 Computing C already takes O ( nd 2 ) time (very expensive) Method 2: singular value decomposition (SVD) Find X = U d × d Σ d × n V ⊤ n × n where U ⊤ U = I d × d , V ⊤ V = I n × n , Σ is diagonal Computing top k singular vectors takes only O ( ndk ) Relationship between eigendecomposition and SVD: Left singular vectors are principal components ( C = U Σ 2 U ⊤ ) Principal component analysis (PCA) / Basic principles 16

  21. Roadmap • Principal component analysis (PCA) – Basic principles – Case studies – Kernel PCA – Probabilistic PCA • Canonical correlation analysis (CCA) • Fisher discriminant analysis (FDA) • Summary Principal component analysis (PCA) / Case studies 17

  22. Eigen-faces [Turk and Pentland, 1991] • d = number of pixels • Each x i ∈ R d is a face image • x ji = intensity of the j -th pixel in image i Principal component analysis (PCA) / Case studies 18

  23. Eigen-faces [Turk and Pentland, 1991] • d = number of pixels • Each x i ∈ R d is a face image • x ji = intensity of the j -th pixel in image i X d × n ≅ U d × k Z k × n ) ( z 1 . . . z n ) ) ≅ ( ( . . . Principal component analysis (PCA) / Case studies 18

  24. Eigen-faces [Turk and Pentland, 1991] • d = number of pixels • Each x i ∈ R d is a face image • x ji = intensity of the j -th pixel in image i X d × n ≅ U d × k Z k × n ) ( z 1 . . . z n ) ) ≅ ( ( . . . Idea: z i more “meaningful” representation of i -th face than x i Can use z i for nearest-neighbor classification Much faster: O ( dk + nk ) time instead of O ( dn ) when n, d ≫ k Why no time savings for linear classifier? Principal component analysis (PCA) / Case studies 18

  25. Latent Semantic Analysis [Deerwater, 1990] • d = number of words in the vocabulary • Each x i ∈ R d is a vector of word counts • x ji = frequency of word j in document i X d × n ≅ U d × k Z k × n ( game: 1 · · · · · · · · · 3 ) ≅ ( 1.9 ) ( z 1 . . . z n ) stocks: 2 · · · · · · · · · 0 0.4 ·· -0.001 chairman: 4 · · · · · · · · · 1 0.8 ·· 0.03 the: 8 · · · · · · · · · 7 0.01 ·· 0.04 · · · . . · · · · · · · · · . . . . . . . . . ·· . wins: 0 · · · · · · · · · 2 0.002 ·· 2.3 0.003 ·· Principal component analysis (PCA) / Case studies 19

  26. Latent Semantic Analysis [Deerwater, 1990] • d = number of words in the vocabulary • Each x i ∈ R d is a vector of word counts • x ji = frequency of word j in document i X d × n ≅ U d × k Z k × n ( game: 1 · · · · · · · · · 3 ) ≅ ( 1.9 ) ( z 1 . . . z n ) stocks: 2 · · · · · · · · · 0 0.4 ·· -0.001 chairman: 4 · · · · · · · · · 1 0.8 ·· 0.03 the: 8 · · · · · · · · · 7 0.01 ·· 0.04 · · · . . · · · · · · · · · . . . . . . . . . ·· . wins: 0 · · · · · · · · · 2 0.002 ·· 2.3 0.003 ·· How to measure similarity between two documents? z ⊤ 1 z 2 is probably better than x ⊤ 1 x 2 Principal component analysis (PCA) / Case studies 19

  27. Latent Semantic Analysis [Deerwater, 1990] • d = number of words in the vocabulary • Each x i ∈ R d is a vector of word counts • x ji = frequency of word j in document i X d × n ≅ U d × k Z k × n ( game: 1 · · · · · · · · · 3 ) ≅ ( 1.9 ) ( z 1 . . . z n ) stocks: 2 · · · · · · · · · 0 0.4 ·· -0.001 chairman: 4 · · · · · · · · · 1 0.8 ·· 0.03 the: 8 · · · · · · · · · 7 0.01 ·· 0.04 · · · . . · · · · · · · · · . . . . . . . . . ·· . wins: 0 · · · · · · · · · 2 0.002 ·· 2.3 0.003 ·· How to measure similarity between two documents? z ⊤ 1 z 2 is probably better than x ⊤ 1 x 2 Applications: information retrieval Principal component analysis (PCA) / Case studies 19

  28. Latent Semantic Analysis [Deerwater, 1990] • d = number of words in the vocabulary • Each x i ∈ R d is a vector of word counts • x ji = frequency of word j in document i X d × n ≅ U d × k Z k × n ( game: 1 · · · · · · · · · 3 ) ≅ ( 1.9 ) ( z 1 . . . z n ) stocks: 2 · · · · · · · · · 0 0.4 ·· -0.001 chairman: 4 · · · · · · · · · 1 0.8 ·· 0.03 the: 8 · · · · · · · · · 7 0.01 ·· 0.04 · · · . . · · · · · · · · · . . . . . . . . . ·· . wins: 0 · · · · · · · · · 2 0.002 ·· 2.3 0.003 ·· How to measure similarity between two documents? z ⊤ 1 z 2 is probably better than x ⊤ 1 x 2 Applications: information retrieval Note: no computational savings; original x is already sparse Principal component analysis (PCA) / Case studies 19

  29. Network anomaly detection [Lakhina, ’05] x ji = amount of traffic on link j in the network during each time interval i Principal component analysis (PCA) / Case studies 20

  30. Network anomaly detection [Lakhina, ’05] x ji = amount of traffic on link j in the network during each time interval i Model assumption: total traffic is sum of flows along a few “paths” Principal component analysis (PCA) / Case studies 20

  31. Network anomaly detection [Lakhina, ’05] x ji = amount of traffic on link j in the network during each time interval i Model assumption: total traffic is sum of flows along a few “paths” Apply PCA: each principal component intuitively represents a “path” Principal component analysis (PCA) / Case studies 20

  32. Network anomaly detection [Lakhina, ’05] x ji = amount of traffic on link j in the network during each time interval i Model assumption: total traffic is sum of flows along a few “paths” Apply PCA: each principal component intuitively represents a “path” Anomaly when traffic deviates from first few principal components Principal component analysis (PCA) / Case studies 20

  33. Network anomaly detection [Lakhina, ’05] x ji = amount of traffic on link j in the network during each time interval i Model assumption: total traffic is sum of flows along a few “paths” Apply PCA: each principal component intuitively represents a “path” Anomaly when traffic deviates from first few principal components Principal component analysis (PCA) / Case studies 20

  34. Unsupervised POS tagging [Sch¨ utze, ’95] Part-of-speech (POS) tagging task: Input: I like reducing the dimensionality of data . Output: NOUN VERB VERB(-ING) DET NOUN PREP NOUN . Principal component analysis (PCA) / Case studies 21

  35. Unsupervised POS tagging [Sch¨ utze, ’95] Part-of-speech (POS) tagging task: Input: I like reducing the dimensionality of data . Output: NOUN VERB VERB(-ING) DET NOUN PREP NOUN . Each x i is (the context distribution of) a word. x ji is number of times word i appeared in context j Key idea: words appearing in similar contexts tend to have the same POS tags; so cluster using the contexts of each word type Problem: contexts are too sparse Principal component analysis (PCA) / Case studies 21

  36. Unsupervised POS tagging [Sch¨ utze, ’95] Part-of-speech (POS) tagging task: Input: I like reducing the dimensionality of data . Output: NOUN VERB VERB(-ING) DET NOUN PREP NOUN . Each x i is (the context distribution of) a word. x ji is number of times word i appeared in context j Key idea: words appearing in similar contexts tend to have the same POS tags; so cluster using the contexts of each word type Problem: contexts are too sparse Solution: run PCA first, then cluster using new representation Principal component analysis (PCA) / Case studies 21

  37. Multi-task learning [Ando & Zhang, ’05] • Have n related tasks (classify documents for various users) • Each task has a linear classifier with weights x i • Want to share structure between classifiers Principal component analysis (PCA) / Case studies 22

  38. Multi-task learning [Ando & Zhang, ’05] • Have n related tasks (classify documents for various users) • Each task has a linear classifier with weights x i • Want to share structure between classifiers One step of their procedure: given n linear classifiers x 1 , . . . , x n , run PCA to identify shared structure: Principal component analysis (PCA) / Case studies 22

  39. Multi-task learning [Ando & Zhang, ’05] • Have n related tasks (classify documents for various users) • Each task has a linear classifier with weights x i • Want to share structure between classifiers One step of their procedure: given n linear classifiers x 1 , . . . , x n , run PCA to identify shared structure: X = ( x 1 . . . x n ) ≅ UZ Principal component analysis (PCA) / Case studies 22

  40. Multi-task learning [Ando & Zhang, ’05] • Have n related tasks (classify documents for various users) • Each task has a linear classifier with weights x i • Want to share structure between classifiers One step of their procedure: given n linear classifiers x 1 , . . . , x n , run PCA to identify shared structure: X = ( x 1 . . . x n ) ≅ UZ Each principal component is a eigen-classifier Principal component analysis (PCA) / Case studies 22

  41. Multi-task learning [Ando & Zhang, ’05] • Have n related tasks (classify documents for various users) • Each task has a linear classifier with weights x i • Want to share structure between classifiers One step of their procedure: given n linear classifiers x 1 , . . . , x n , run PCA to identify shared structure: X = ( x 1 . . . x n ) ≅ UZ Each principal component is a eigen-classifier Other step of their procedure: Retrain classifiers, regularizing towards subspace U Principal component analysis (PCA) / Case studies 22

  42. PCA summary • Intuition: capture variance of data or minimize reconstruction error Principal component analysis (PCA) / Case studies 23

  43. PCA summary • Intuition: capture variance of data or minimize reconstruction error • Algorithm: find eigendecomposition of covariance matrix or SVD Principal component analysis (PCA) / Case studies 23

  44. PCA summary • Intuition: capture variance of data or minimize reconstruction error • Algorithm: find eigendecomposition of covariance matrix or SVD • Impact: reduce storage (from O ( nd ) to O ( nk ) ), reduce time complexity Principal component analysis (PCA) / Case studies 23

  45. PCA summary • Intuition: capture variance of data or minimize reconstruction error • Algorithm: find eigendecomposition of covariance matrix or SVD • Impact: reduce storage (from O ( nd ) to O ( nk ) ), reduce time complexity • Advantages: simple, fast Principal component analysis (PCA) / Case studies 23

  46. PCA summary • Intuition: capture variance of data or minimize reconstruction error • Algorithm: find eigendecomposition of covariance matrix or SVD • Impact: reduce storage (from O ( nd ) to O ( nk ) ), reduce time complexity • Advantages: simple, fast • Applications: eigen-faces, eigen-documents, network anomaly detection, etc. Principal component analysis (PCA) / Case studies 23

  47. Roadmap • Principal component analysis (PCA) – Basic principles – Case studies – Kernel PCA – Probabilistic PCA • Canonical correlation analysis (CCA) • Fisher discriminant analysis (FDA) • Summary Principal component analysis (PCA) / Kernel PCA 24

  48. Limitations of linearity Principal component analysis (PCA) / Kernel PCA 25

  49. Limitations of linearity PCA is effective Principal component analysis (PCA) / Kernel PCA 25

  50. Limitations of linearity PCA is effective Principal component analysis (PCA) / Kernel PCA 25

  51. Limitations of linearity PCA is effective PCA is ineffective Principal component analysis (PCA) / Kernel PCA 25

  52. Limitations of linearity PCA is effective PCA is ineffective Problem is that PCA subspace is linear: S = { x = Uz : z ∈ R k } Principal component analysis (PCA) / Kernel PCA 25

  53. Limitations of linearity PCA is effective PCA is ineffective Problem is that PCA subspace is linear: S = { x = Uz : z ∈ R k } In this example: S = { ( x 1 , x 2 ) : x 2 = u 2 u 1 x 1 } Principal component analysis (PCA) / Kernel PCA 25

  54. Going beyond linearity: quick solution Broken solution Principal component analysis (PCA) / Kernel PCA 26

  55. Going beyond linearity: quick solution Broken solution Desired solution u 1 x 2 We want desired solution: S = { ( x 1 , x 2 ) : x 2 = u 2 1 } Principal component analysis (PCA) / Kernel PCA 26

  56. Going beyond linearity: quick solution Broken solution Desired solution u 1 x 2 We want desired solution: S = { ( x 1 , x 2 ) : x 2 = u 2 1 } 1 , x 2 ) ⊤ We can get this: S = { φ ( x ) = Uz } with φ ( x ) = ( x 2 Principal component analysis (PCA) / Kernel PCA 26

  57. Going beyond linearity: quick solution Broken solution Desired solution u 1 x 2 We want desired solution: S = { ( x 1 , x 2 ) : x 2 = u 2 1 } 1 , x 2 ) ⊤ We can get this: S = { φ ( x ) = Uz } with φ ( x ) = ( x 2 Linear dimensionality reduction in φ ( x ) space ⇔ Nonlinear dimensionality reduction in x space Principal component analysis (PCA) / Kernel PCA 26

  58. Going beyond linearity: quick solution Broken solution Desired solution u 1 x 2 We want desired solution: S = { ( x 1 , x 2 ) : x 2 = u 2 1 } 1 , x 2 ) ⊤ We can get this: S = { φ ( x ) = Uz } with φ ( x ) = ( x 2 Linear dimensionality reduction in φ ( x ) space ⇔ Nonlinear dimensionality reduction in x space 1 , x 1 x 2 , sin( x 1 ) , . . . ) ⊤ In general, can set φ ( x ) = ( x 1 , x 2 Principal component analysis (PCA) / Kernel PCA 26

  59. Going beyond linearity: quick solution Broken solution Desired solution u 1 x 2 We want desired solution: S = { ( x 1 , x 2 ) : x 2 = u 2 1 } 1 , x 2 ) ⊤ We can get this: S = { φ ( x ) = Uz } with φ ( x ) = ( x 2 Linear dimensionality reduction in φ ( x ) space ⇔ Nonlinear dimensionality reduction in x space 1 , x 1 x 2 , sin( x 1 ) , . . . ) ⊤ In general, can set φ ( x ) = ( x 1 , x 2 Problems: (1) ad-hoc and tedious (2) φ ( x ) large, computationally expensive Principal component analysis (PCA) / Kernel PCA 26

  60. Towards kernels Representer theorem: PCA solution is linear combination of x i s Principal component analysis (PCA) / Kernel PCA 27

  61. Towards kernels Representer theorem: PCA solution is linear combination of x i s Why? Recall PCA eigenvalue problem: XX ⊤ u = λ u Principal component analysis (PCA) / Kernel PCA 27

  62. Towards kernels Representer theorem: PCA solution is linear combination of x i s Why? Recall PCA eigenvalue problem: XX ⊤ u = λ u Notice that u = X α = � n i =1 α i x i for some weights α Principal component analysis (PCA) / Kernel PCA 27

  63. Towards kernels Representer theorem: PCA solution is linear combination of x i s Why? Recall PCA eigenvalue problem: XX ⊤ u = λ u Notice that u = X α = � n i =1 α i x i for some weights α Analogy with SVMs: weight vector w = X α Principal component analysis (PCA) / Kernel PCA 27

  64. Towards kernels Representer theorem: PCA solution is linear combination of x i s Why? Recall PCA eigenvalue problem: XX ⊤ u = λ u Notice that u = X α = � n i =1 α i x i for some weights α Analogy with SVMs: weight vector w = X α Key fact: PCA only needs inner products K = X ⊤ X Principal component analysis (PCA) / Kernel PCA 27

  65. Towards kernels Representer theorem: PCA solution is linear combination of x i s Why? Recall PCA eigenvalue problem: XX ⊤ u = λ u Notice that u = X α = � n i =1 α i x i for some weights α Analogy with SVMs: weight vector w = X α Key fact: PCA only needs inner products K = X ⊤ X Why? Use representer theorem on PCA objective: � u � =1 u ⊤ XX ⊤ u = α ⊤ ( X ⊤ X )( X ⊤ X ) α max max α ⊤ X ⊤ X α =1 Principal component analysis (PCA) / Kernel PCA 27

  66. Kernel PCA Kernel function: k ( x 1 , x 2 ) such that K , the kernel matrix formed by K ij = k ( x i , x j ) , is positive semi-definite Principal component analysis (PCA) / Kernel PCA 28

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Probabilistic Dimensionality Reduction Neil D. Lawrence University of Sheffield Facebook, London

Probabilistic Dimensionality Reduction Neil D. Lawrence University of Sheffield Facebook, London

Probabilistic Dimensionality Reduction Neil D. Lawrence University of Sheffield Facebook, London 14th April 2016 Outline Probabilistic Linear Dimensionality Reduction Non Linear Probabilistic Dimensionality Reduction Examples Conclusions

1.68k views • 148 slides
STAT 209 Dimensionality Reduction November 26, 2019 Colin Reimer Dawson 1 / 24 Dimensionality

STAT 209 Dimensionality Reduction November 26, 2019 Colin Reimer Dawson 1 / 24 Dimensionality

Dimensionality Reduction STAT 209 Dimensionality Reduction November 26, 2019 Colin Reimer Dawson 1 / 24 Dimensionality Reduction Outline Dimensionality Reduction 2 / 24 Dimensionality Reduction High Dimensional Data Modern datasets

470 views • 24 slides
Dimensionality Reduction: Linear Discriminant Analysis and Principal Component Analysis CMSC 678

Dimensionality Reduction: Linear Discriminant Analysis and Principal Component Analysis CMSC 678

Dimensionality Reduction: Linear Discriminant Analysis and Principal Component Analysis CMSC 678 UMBC Outline Linear Algebra/Math Review Two Methods of Dimensionality Reduction Linear Discriminant Analysis (LDA, LDiscA) Principal Component

884 views • 70 slides
Applied Machine Learning Dimensionality reduction using PCA Siamak Ravanbakhsh COMP 551 (Fall

Applied Machine Learning Dimensionality reduction using PCA Siamak Ravanbakhsh COMP 551 (Fall

Applied Machine Learning Dimensionality reduction using PCA Siamak Ravanbakhsh COMP 551 (Fall 2020) Learning objectives What is dimensionality reduction? What is it good for? Linear dimensionality reduction: Principal Component Analysis

661 views • 26 slides
Spatial Data: Dimensionality Reduction CS444 Techniques, Lecture 3 In this subfield, we think

Spatial Data: Dimensionality Reduction CS444 Techniques, Lecture 3 In this subfield, we think

Spatial Data: Dimensionality Reduction CS444 Techniques, Lecture 3 In this subfield, we think of a data point as a vector in R^n (what could possibly go wrong?) Linear dimensionality reduction: Reduction is achieved by is a single

1.02k views • 20 slides
Dimensionality Reduction Based on Geodesic Distance Hao Li,515030910494 Yifan Shen,515030910491

Dimensionality Reduction Based on Geodesic Distance Hao Li,515030910494 Yifan Shen,515030910491

Dimensionality Reduction Based on Geodesic Distance Hao Li,515030910494 Yifan Shen,515030910491 2018.5.28 Background Dimensionality reduction method: Principle Component Analysis PCA Isometric Mapping ISOMAP Locally Linear Embedding LLE

341 views • 18 slides
Spatial Data: Dimensionality Reduction CSC444 Techniques In this subfield, we think of a data

Spatial Data: Dimensionality Reduction CSC444 Techniques In this subfield, we think of a data

Spatial Data: Dimensionality Reduction CSC444 Techniques In this subfield, we think of a data point as a vector in R^n (what could possibly go wrong?) Linear dimensionality reduction: Reduction is achieved by multiplying a point by

354 views • 22 slides
Dimensionality Reduction Algorithms (and how to interpret their output) Dalya Baron (Tel Aviv

Dimensionality Reduction Algorithms (and how to interpret their output) Dalya Baron (Tel Aviv

Dimensionality Reduction Algorithms (and how to interpret their output) Dalya Baron (Tel Aviv University) XXX Winter School, November 2018 What is Dimensionality Reduction? Dimensionality Reduction algorithm 28 x 28 features per object 2

437 views • 33 slides
APPLIED MACHINE LEARNING Methods for Reduction of Dimensionality through Linear Projection

APPLIED MACHINE LEARNING Methods for Reduction of Dimensionality through Linear Projection

APPLIED MACHINE LEARNING APPLIED MACHINE LEARNING Methods for Reduction of Dimensionality through Linear Projection Principal Component Analysis (PCA) 1 APPLIED MACHINE LEARNING Curse of Dimensionality Computational Costs O(N 2 ) O(N) N:

848 views • 69 slides
Machine Learning 2 DS 4420 - Spring 2020 Dimensionality reduction 2 Byron C Wallace Today A

Machine Learning 2 DS 4420 - Spring 2020 Dimensionality reduction 2 Byron C Wallace Today A

Machine Learning 2 DS 4420 - Spring 2020 Dimensionality reduction 2 Byron C Wallace Today A bit of wrap up on PCA Then : Non-linear dimensionality reduction! (SNE/t-SNE) In Sum: Principal Component Analysis X = ( x 1 x

660 views • 52 slides
Dimensionality Reduction Alexandros Tantos Assistant Professor Aristotle University of

Dimensionality Reduction Alexandros Tantos Assistant Professor Aristotle University of

DataCamp Dimensionality Reduction in R DIMENSIONALITY REDUCTION IN R Dimensionality Reduction Alexandros Tantos Assistant Professor Aristotle University of Thessaloniki DataCamp Dimensionality Reduction in R Curse of Dimensionality

550 views • 35 slides
6.1 Dimensionality reduction Previously in the course, we have discussed algorithms suited for a

6.1 Dimensionality reduction Previously in the course, we have discussed algorithms suited for a

CS/CNS/EE 253: Advanced Topics in Machine Learning Topic: Dimensionality Reduction Lecturer: Andreas Krause Scribe: Matt Faulkner Date: 25 January 2010 6.1 Dimensionality reduction Previously in the course, we have discussed algorithms suited for

442 views • 6 slides
DIMENSIONALITY REDUCTION DIMENSIONALITY REDUCTION MATTHIEU BLOCH April 21, 2020 1 / 26

DIMENSIONALITY REDUCTION DIMENSIONALITY REDUCTION MATTHIEU BLOCH April 21, 2020 1 / 26

DIMENSIONALITY REDUCTION DIMENSIONALITY REDUCTION MATTHIEU BLOCH April 21, 2020 1 / 26 MULTIDIMENSIONAL SCALING MULTIDIMENSIONAL SCALING There are situations for which Euclidean distance is not appropriate Suppose we have access to a

500 views • 26 slides
Nonlinear Dimensionality Reduction Donovan Parks Overview Direct visualization vs.

Nonlinear Dimensionality Reduction Donovan Parks Overview Direct visualization vs.

Nonlinear Dimensionality Reduction Donovan Parks Overview Direct visualization vs. dimensionality reduction Nonlinear dimensionality reduction techniques: ISOMAP, LLE, Charting A fun example that uses non- metric, replicated

882 views • 38 slides
Preprocessing and Dimensionality Reduction J er emy Fix CentraleSup elec

Preprocessing and Dimensionality Reduction J er emy Fix CentraleSup elec

Datasets Preprocessing Dimensionality reduction Preprocessing and Dimensionality Reduction J er emy Fix CentraleSup elec jeremy.fix@centralesupelec.fr 2017 1 / 73 Datasets Preprocessing Dimensionality reduction Where to get data

1.22k views • 73 slides
E9 205 Machine Learning for Signal Processing Supervised-Dimensionality-Reduction. Decision

E9 205 Machine Learning for Signal Processing Supervised-Dimensionality-Reduction. Decision

E9 205 Machine Learning for Signal Processing Supervised-Dimensionality-Reduction. Decision Theory 26-08-2019 Probability Distributions Advantages and Disadvantages of PCA Simple linear transform Eigen decomposition of Data Covariance

487 views • 9 slides
Data Mining and Machine Learning: Fundamental Concepts and Algorithms dataminingbook.info

Data Mining and Machine Learning: Fundamental Concepts and Algorithms dataminingbook.info

Data Mining and Machine Learning: Fundamental Concepts and Algorithms dataminingbook.info Mohammed J. Zaki 1 Wagner Meira Jr. 2 1 Department of Computer Science Rensselaer Polytechnic Institute, Troy, NY, USA 2 Department of Computer Science

601 views • 34 slides
Principal component analysis DS-GA 1013 / MATH-GA 2824 Mathematical Tools for Data Science

Principal component analysis DS-GA 1013 / MATH-GA 2824 Mathematical Tools for Data Science

Principal component analysis DS-GA 1013 / MATH-GA 2824 Mathematical Tools for Data Science https://cims.nyu.edu/~cfgranda/pages/MTDS_spring20/index.html Carlos Fernandez-Granda Discussion Covariance matrix The spectral theorem Principal

1.38k views • 107 slides
Principal Component Analysis Eric Eager Data Scientist at Pro Football Focus DataCamp Linear

Principal Component Analysis Eric Eager Data Scientist at Pro Football Focus DataCamp Linear

DataCamp Linear Algebra for Data Science in R LINEAR ALGEBRA FOR DATA SCIENCE IN R Principal Component Analysis Eric Eager Data Scientist at Pro Football Focus DataCamp Linear Algebra for Data Science in R Big Data > head(combine) >

967 views • 27 slides
1 Principal Components Analysis (PCA) Review of basic setup: N vectors, { x 1 , . . .

1 Principal Components Analysis (PCA) Review of basic setup: N vectors, { x 1 , . . .

Statistical Modeling and Analysis of Neural Data (NEU 560) Princeton University, Spring 2018 Jonathan Pillow Lecture 5 notes: PCA, part II Tues, 2.20 1 Principal Components Analysis (PCA) Review of basic setup: N vectors, { x 1 , . . .

83 views • 4 slides
A Cluster Target Similarity Based g y Principal Component Analysis for Interval Valued

A Cluster Target Similarity Based g y Principal Component Analysis for Interval Valued

A Cluster Target Similarity Based g y Principal Component Analysis for Interval Valued Data University of Tsukuba School of Systems and Information Engineering Mika Sato Ilic Energy Evaluation Data Energy JP1 JP2 UK1 UK2

575 views • 41 slides
PRINCIPAL COMPONENT ANALYSIS(PCA) By Deepen naorem Latent(hidden) representation Method A

PRINCIPAL COMPONENT ANALYSIS(PCA) By Deepen naorem Latent(hidden) representation Method A

PRINCIPAL COMPONENT ANALYSIS(PCA) By Deepen naorem Latent(hidden) representation Method A method or mechanism to see or view data(matrix) in different ways. Data matrix 2 4 5 7 Change of Basis Let us consider a scenario System

985 views • 16 slides
Continuous Latent Variables Oliver Schulte - CMPT 419/726 Bishop PRML Ch. 12 Principal Component

Continuous Latent Variables Oliver Schulte - CMPT 419/726 Bishop PRML Ch. 12 Principal Component

Principal Component Analysis Continuous Latent Variables Oliver Schulte - CMPT 419/726 Bishop PRML Ch. 12 Principal Component Analysis Outline Principal Component Analysis Principal Component Analysis Outline Principal Component Analysis

621 views • 24 slides
Topological Data Analysis A Framework for Machine Learning Samarth Bansal (11630) Deepak

Topological Data Analysis A Framework for Machine Learning Samarth Bansal (11630) Deepak

Topological Data Analysis A Framework for Machine Learning Samarth Bansal (11630) Deepak Choudhary (11234) Motivation Ayasdi was started in 2008 to bring a groundbreaking new approach to solving the worlds most complex problems after a

420 views • 17 slides

More recommend