e m method for latent variable models
play

E-M method for latent variable models Define augmented likelihood n - PowerPoint PPT Presentation

Feb 13, 2023 •181 likes •1.73k views

E-M method for latent variable models Define augmented likelihood n k R ij ln p ( x i , y i = j ) L ( ; R ) := , R ij i =1 j =1 with responsibility matrix R R n,k := { R [0 , 1] n k : R 1 k = 1 n } . Alternate two

  1. Parameter constraints. E-M for GMMs still works if we freeze or constrain some parameters. Examples: ◮ No weights: initialize π = ( 1 / k , . . . , 1 / k ) and never update it. ◮ Diagonal covariance matrices: update everything as before, except Σ j := diag(( σ j ) 2 1 , . . . , ( σ j ) 2 d ) where � n i =1 R ij ( x i − µ j ) 2 ( σ j ) 2 l l := ; nπ j that is: we use coordinate-wise sample variances weighted by R . Why is this a good idea? 38 / 70

  2. Parameter constraints. E-M for GMMs still works if we freeze or constrain some parameters. Examples: ◮ No weights: initialize π = ( 1 / k , . . . , 1 / k ) and never update it. ◮ Diagonal covariance matrices: update everything as before, except Σ j := diag(( σ j ) 2 1 , . . . , ( σ j ) 2 d ) where � n i =1 R ij ( x i − µ j ) 2 ( σ j ) 2 l l := ; nπ j that is: we use coordinate-wise sample variances weighted by R . Why is this a good idea? Computation (of inverse), sample complexity, . . . 38 / 70

  3. Gaussian Mixture Model with diagonal covariances. 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 5 0 5 10 15 39 / 70

  4. Gaussian Mixture Model with diagonal covariances. 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 5 0 5 10 15 39 / 70

  5. Gaussian Mixture Model with diagonal covariances. 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 5 0 5 10 15 39 / 70

  6. Gaussian Mixture Model with diagonal covariances. 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 5 0 5 10 15 39 / 70

  7. Gaussian Mixture Model with diagonal covariances. 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 5 0 5 10 15 39 / 70

  8. Gaussian Mixture Model with diagonal covariances. 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 5 0 5 10 15 39 / 70

  9. Gaussian Mixture Model with diagonal covariances. 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 5 0 5 10 15 39 / 70

  10. Gaussian Mixture Model with diagonal covariances. 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 5 0 5 10 15 39 / 70

  11. Gaussian Mixture Model with diagonal covariances. 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 5 0 5 10 15 39 / 70

  12. Gaussian Mixture Model with diagonal covariances. 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 5 0 5 10 15 39 / 70

  13. Gaussian Mixture Model with diagonal covariances. 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 5 0 5 10 15 39 / 70

  14. Gaussian Mixture Model with diagonal covariances. 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 5 0 5 10 15 39 / 70

  15. Gaussian Mixture Model with diagonal covariances. 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 10 5 0 5 10 15 39 / 70

  16. Gaussian Mixture Model with diagonal covariances. 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 10 5 0 5 10 15 39 / 70

  17. Gaussian Mixture Model with diagonal covariances. 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 10 5 0 5 10 15 39 / 70

  18. Gaussian Mixture Model with diagonal covariances. 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 10 5 0 5 10 15 39 / 70

  19. Gaussian Mixture Model with diagonal covariances. 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 10 5 0 5 10 15 39 / 70

  20. Gaussian Mixture Model with diagonal covariances. 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 10 5 0 5 10 15 39 / 70

  21. Gaussian Mixture Model with diagonal covariances. 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 10 5 0 5 10 15 39 / 70

  22. Gaussian Mixture Model with diagonal covariances. 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 10 5 0 5 10 15 39 / 70

  23. Gaussian Mixture Model with diagonal covariances. 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 10 5 0 5 10 15 39 / 70

  24. Gaussian Mixture Model with diagonal covariances. 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 10 5 0 5 10 15 39 / 70

  25. Gaussian Mixture Model with diagonal covariances. 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 10 5 0 5 10 15 39 / 70

  26. Gaussian Mixture Model with diagonal covariances. 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 10 5 0 5 10 15 39 / 70

  27. Gaussian Mixture Model with diagonal covariances. 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 10 5 0 5 10 15 39 / 70

  28. Gaussian Mixture Model with diagonal covariances. 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 10 5 0 5 10 15 39 / 70

  29. Gaussian Mixture Model with diagonal covariances. 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 10 5 0 5 10 15 39 / 70

  30. Gaussian Mixture Model with diagonal covariances. 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 10 5 0 5 10 15 39 / 70

  31. Gaussian Mixture Model with diagonal covariances. 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 10 5 0 5 10 15 39 / 70

  32. Gaussian Mixture Model with diagonal covariances. 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 10 5 0 5 10 15 39 / 70

  33. Gaussian Mixture Model with diagonal covariances. 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 10 5 0 5 10 15 39 / 70

  34. Gaussian Mixture Model with diagonal covariances. 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 10 5 0 5 10 15 39 / 70

  35. Singularities E-M with GMMs suffers from singularities : trivial situations where the likelihood goes to ∞ but the solution is bad. ◮ Suppose: d = 1 , k = 2 , π j = 1 / 2 , n = 3 with x 1 = − 1 and x 2 = +1 and x 3 = +3 . Initialize with µ 1 = 0 and σ 1 = 1 , but µ 2 = +3 = x 3 and σ 2 = 1 / 100 . Then σ 2 → 0 and L ↑ ∞ . 40 / 70

  36. Interpolating between k -means and GMM E-M Same M-step: fix π = ( 1 / k , . . . , 1 / k ) and Σ j = c I for a fixed c > 0 . 41 / 70

  37. Interpolating between k -means and GMM E-M Same M-step: fix π = ( 1 / k , . . . , 1 / k ) and Σ j = c I for a fixed c > 0 . Same E-step: define q ij := 1 2 � x i − µ j � 2 ; the E-step chooses R ij := p θ ( y i = j | x i ) = p θ ( y i = j, x i ) p θ ( y i = j, x i ) = p θ ( x i ) � k l =1 p θ ( y i = l, x i ) π j p µ j , Σ j ( x i ) exp( − q ij /c ) = = � k � k l =1 exp( − q il /c ) l =1 π l p µ l , Σ l ( x i ) Fix i ∈ { 1 , . . . , n } and suppose minimum q i := min j q ij is unique: 41 / 70

  38. Interpolating between k -means and GMM E-M Same M-step: fix π = ( 1 / k , . . . , 1 / k ) and Σ j = c I for a fixed c > 0 . Same E-step: define q ij := 1 2 � x i − µ j � 2 ; the E-step chooses R ij := p θ ( y i = j | x i ) = p θ ( y i = j, x i ) p θ ( y i = j, x i ) = p θ ( x i ) � k l =1 p θ ( y i = l, x i ) π j p µ j , Σ j ( x i ) exp( − q ij /c ) = = � k � k l =1 exp( − q il /c ) l =1 π l p µ l , Σ l ( x i ) Fix i ∈ { 1 , . . . , n } and suppose minimum q i := min j q ij is unique: exp( − q ij /c ) exp( q i − q ij /c ) lim c ↓ 0 R ij = lim = lim � k � k c ↓ 0 l =1 exp( − q il /c ) c ↓ 0 l =1 exp( q i − q il /c ) � 1 q ij = q i , = q ij � = q i . 0 That is, R becomes hard assignment A as c ↓ 0 . 41 / 70

  39. Interpolating between k -means and GMM E-M (part 2) We can interpolate algorithmically, meaning we can create algorithms that have elements of both. Here’s something like k -means but with weights and covariances. 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 5 0 5 10 15 42 / 70

  40. Interpolating between k -means and GMM E-M (part 2) We can interpolate algorithmically, meaning we can create algorithms that have elements of both. Here’s something like k -means but with weights and covariances. 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 5 0 5 10 15 42 / 70

  41. Interpolating between k -means and GMM E-M (part 2) We can interpolate algorithmically, meaning we can create algorithms that have elements of both. Here’s something like k -means but with weights and covariances. 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 5 0 5 10 15 42 / 70

  42. Interpolating between k -means and GMM E-M (part 2) We can interpolate algorithmically, meaning we can create algorithms that have elements of both. Here’s something like k -means but with weights and covariances. 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 5 0 5 10 15 42 / 70

  43. Interpolating between k -means and GMM E-M (part 2) We can interpolate algorithmically, meaning we can create algorithms that have elements of both. Here’s something like k -means but with weights and covariances. 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 5 0 5 10 15 42 / 70

  44. Interpolating between k -means and GMM E-M (part 2) We can interpolate algorithmically, meaning we can create algorithms that have elements of both. Here’s something like k -means but with weights and covariances. 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 5 0 5 10 15 42 / 70

  45. Interpolating between k -means and GMM E-M (part 2) We can interpolate algorithmically, meaning we can create algorithms that have elements of both. Here’s something like k -means but with weights and covariances. 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 5 0 5 10 15 42 / 70

  46. Interpolating between k -means and GMM E-M (part 2) We can interpolate algorithmically, meaning we can create algorithms that have elements of both. Here’s something like k -means but with weights and covariances. 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 5 0 5 10 15 42 / 70

  47. Interpolating between k -means and GMM E-M (part 2) We can interpolate algorithmically, meaning we can create algorithms that have elements of both. Here’s something like k -means but with weights and covariances. 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 5 0 5 10 15 42 / 70

  48. Interpolating between k -means and GMM E-M (part 2) We can interpolate algorithmically, meaning we can create algorithms that have elements of both. Here’s something like k -means but with weights and covariances. 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 5 0 5 10 15 42 / 70

  49. Interpolating between k -means and GMM E-M (part 2) We can interpolate algorithmically, meaning we can create algorithms that have elements of both. Here’s something like k -means but with weights and covariances. 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 5 0 5 10 15 42 / 70

  50. Interpolating between k -means and GMM E-M (part 2) We can interpolate algorithmically, meaning we can create algorithms that have elements of both. Here’s something like k -means but with weights and covariances. 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 5 0 5 10 15 42 / 70

  51. Interpolating between k -means and GMM E-M (part 2) We can interpolate algorithmically, meaning we can create algorithms that have elements of both. Here’s something like k -means but with weights and covariances. 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 10 5 0 5 10 15 42 / 70

  52. Interpolating between k -means and GMM E-M (part 2) We can interpolate algorithmically, meaning we can create algorithms that have elements of both. Here’s something like k -means but with weights and covariances. 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 10 5 0 5 10 15 42 / 70

  53. Interpolating between k -means and GMM E-M (part 2) We can interpolate algorithmically, meaning we can create algorithms that have elements of both. Here’s something like k -means but with weights and covariances. 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 10 5 0 5 10 15 42 / 70

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

1 Latent variable models In the next section we will discuss latent variable models for

1 Latent variable models In the next section we will discuss latent variable models for

Statistical Modeling and Analysis of Neural Data (NEU 560) Princeton University, Spring 2018 Jonathan Pillow Lecture 16 notes: Latent variable models and EM Tues, 4.10 1 Latent variable models In the next section we will discuss latent

271 views • 4 slides
Latent Variable Models CS3750 Xiaoting Li 1 Out utli line Latent Variable Models

Latent Variable Models CS3750 Xiaoting Li 1 Out utli line Latent Variable Models

2/4/2020 Latent Variable Models CS3750 Xiaoting Li 1 Out utli line Latent Variable Models Expectation Maximization Algorithm (EM) Factor Analysis Probabilistic Principal Component Analysis Model Formulation Maximum

654 views • 18 slides
Learning Overcomplete Latent Variable Models through Tensor Methods Anima Anandkumar UC Irvine

Learning Overcomplete Latent Variable Models through Tensor Methods Anima Anandkumar UC Irvine

Learning Overcomplete Latent Variable Models through Tensor Methods Anima Anandkumar UC Irvine Joint work with Majid Janzamin Rong Ge UC Irvine Microsoft Research Latent Variable Probabilistic Models Latent (hidden) variable h R k ,

769 views • 47 slides
A New Method of Moments for Latent Variable Models Matteo Ruffini, Marta Casanellas, Ricard

A New Method of Moments for Latent Variable Models Matteo Ruffini, Marta Casanellas, Ricard

A New Method of Moments for Latent Variable Models Matteo Ruffini, Marta Casanellas, Ricard Gavald` a Universitat Polit` ecnica de Catalunya, Barcelona, Spain Ruffini, Casanellas, Gavald` a (UPC) Methods of Moments for Topic Models 1 / 30

857 views • 68 slides
Outline Latent Variable Generative Models Cooperative Vector Quantizer Model Model

Outline Latent Variable Generative Models Cooperative Vector Quantizer Model Model

2/4/2020 CS 3750 Advanced Machine Learning Latent Variable Generative Models II Ahmad Diab AHD23@cs.pitt.edu Feb 4, 2020 Based on slides of Professor Milos Hauskrecht Outline Latent Variable Generative Models Cooperative Vector

435 views • 20 slides
Part III: Latent Tree Models Le Song ICML 2012 Tutorial on Spectral Algorithms for Latent

Part III: Latent Tree Models Le Song ICML 2012 Tutorial on Spectral Algorithms for Latent

Spectral Algorithms for Latent Variable Models Part III: Latent Tree Models Le Song ICML 2012 Tutorial on Spectral Algorithms for Latent Variable Models, Edinburgh, UK Joint work with Mariya Ishteva, Ankur Parikh, Eric Xing, Byron Boots , Geoff

548 views • 40 slides
Learning Latent Variable Models through Tensor Methods Anima Anandkumar U.C. Irvine Challenges

Learning Latent Variable Models through Tensor Methods Anima Anandkumar U.C. Irvine Challenges

Learning Latent Variable Models through Tensor Methods Anima Anandkumar U.C. Irvine Challenges in Unsupervised Learning Learn a latent variable model without labeled examples. E.g. topic models, hidden Markov models, Gaussian mixtures,

763 views • 61 slides
Learning Overcomplete Latent Variable Models through Tensor Methods Majid Janzamin UC Irvine

Learning Overcomplete Latent Variable Models through Tensor Methods Majid Janzamin UC Irvine

Learning Overcomplete Latent Variable Models through Tensor Methods Majid Janzamin UC Irvine Joint work with Anima Anandkumar Rong Ge UC Irvine Microsoft Research Latent Variable Modeling Goal: Discover hidden effects from observed

1.27k views • 115 slides
Pengtao Xie Joint work with Yuntian Deng and Eric Xing Carnegie Mellon University 1 Latent

Pengtao Xie Joint work with Yuntian Deng and Eric Xing Carnegie Mellon University 1 Latent

Mutual Angular Regularization of Latent Variable Models: Theory, Algorithm and Applications Pengtao Xie Joint work with Yuntian Deng and Eric Xing Carnegie Mellon University 1 Latent Variable Models (LVMs) Machine Learning Latent Variable

1.04k views • 47 slides
Maximum Reconstruction Estimation for Generative Latent-Variable Models Yong Cheng joint work

Maximum Reconstruction Estimation for Generative Latent-Variable Models Yong Cheng joint work

Maximum Reconstruction Estimation for Generative Latent-Variable Models Yong Cheng joint work with Yang Liu, Wei Xu 1 Problem Generative latent-variable models are important for natural language processing due to their capability of providing

494 views • 26 slides
Guaranteed Learning of Latent Variable Models through Tensor Methods Furong Huang University of

Guaranteed Learning of Latent Variable Models through Tensor Methods Furong Huang University of

Guaranteed Learning of Latent Variable Models through Tensor Methods Furong Huang University of Maryland furongh@cs.umd.edu ACM SIGMETRICS Tutorial 2018 1 / 75 Tutorial Topic Learning algorithms for latent variable models based on

1.96k views • 180 slides
Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models

Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models

Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models Yarin Gal Mark van der Wilk Carl E. Rasmussen yg279@cam.ac.uk June 25th, 2014 Outline Gaussian process regression and latent variable

1.18k views • 54 slides
Latent Variable models for GWAs Oliver Stegle Machine Learning and Computational Biology Research

Latent Variable models for GWAs Oliver Stegle Machine Learning and Computational Biology Research

Latent Variable models for GWAs Oliver Stegle Machine Learning and Computational Biology Research Group Max-Planck-Institutes T ubingen, Germany September 2011 O. Stegle Latent variable models for GWAs T ubingen 1 Motivation Why

1.07k views • 84 slides
Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model CS330

Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model CS330

Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model CS330 Student Presentation Table of Contents Motivation & problem Method overview Experiments Takeaways Discussion (strengths

653 views • 24 slides
Discriminative Regularization for Latent Variable Models with Applications to Electrocardiography

Discriminative Regularization for Latent Variable Models with Applications to Electrocardiography

Discriminative Regularization for Latent Variable Models with Applications to Electrocardiography Poster #53 Andrew C. Miller with Ziad Obermeyer, John P . Cunningham, and Sendhil Mullainathan <latexit

674 views • 9 slides
Language and Document Analysis: Motivating Latent variable Models Wray Buntine National ICT

Language and Document Analysis: Motivating Latent variable Models Wray Buntine National ICT

Language and Document Analysis: Motivating Latent variable Models Wray Buntine National ICT Australia (NICTA) MLSS, ANU, Jan., 2009 Buntine Document Models Part-of-Speech with Hidden Markov Models Topics in Text with Discrete Component

1.25k views • 59 slides
C u stomer and prod u ct segmentation basics MAC H IN E L E AR N IN G FOR MAR K E TIN G IN P

C u stomer and prod u ct segmentation basics MAC H IN E L E AR N IN G FOR MAR K E TIN G IN P

C u stomer and prod u ct segmentation basics MAC H IN E L E AR N IN G FOR MAR K E TIN G IN P YTH ON Karolis Urbonas Head of Anal y tics & Science , Ama z on Data format # Customer by product/service matrix wholesale.head() MACHINE

510 views • 37 slides
Joint Parameter Estimation of the Ornstein-Uhlenbeck SDE driven by Fractional Brownian Motion

Joint Parameter Estimation of the Ornstein-Uhlenbeck SDE driven by Fractional Brownian Motion

Joint Parameter Estimation of the Ornstein-Uhlenbeck SDE driven by Fractional Brownian Motion Luis Barboza October 23, 2012 Department of Statistics, Purdue University () Probability Seminar 1 / 59 Introduction Main Objective: To study the

725 views • 59 slides
Expectation maximization Subhransu Maji CMPSCI 689: Machine Learning 14 April 2015 Motivation

Expectation maximization Subhransu Maji CMPSCI 689: Machine Learning 14 April 2015 Motivation

Expectation maximization Subhransu Maji CMPSCI 689: Machine Learning 14 April 2015 Motivation Suppose you are building a naive Bayes spam classifier. After your are done your boss tells you that there is no money to label the data. You

332 views • 19 slides
Lecture on Parameter Estimation for Stochastic Differential Equations Erik Lindstrm

Lecture on Parameter Estimation for Stochastic Differential Equations Erik Lindstrm

Lecture on Parameter Estimation for Stochastic Differential Equations Erik Lindstrm FMS161/MASM18 Financial Statistics Erik Lindstrm Lecture on Parameter Estimation for Stochastic Differential Equations Recap We are interested in the

747 views • 42 slides
Bag-of-features models for category classification for category classification Cordelia Schmid

Bag-of-features models for category classification for category classification Cordelia Schmid

Bag-of-features models for category classification for category classification Cordelia Schmid Category recognition Category recognition Image classification: assigning a class label to the image Image classification: assigning a class

998 views • 75 slides
tabula rasa Exploring sound/gesture typo-morphology for enactive computer music performance

tabula rasa Exploring sound/gesture typo-morphology for enactive computer music performance

tabula rasa Exploring sound/gesture typo-morphology for enactive computer music performance IRCAM Musical Research Residency 2011 Thomas Grill http://grrrr.org Contents Introduction concepts Implementation current status

520 views • 37 slides
trrt tt s

trrt tt s

trrt tt s trts t t rts r

860 views • 55 slides
Histogram-based matching of GMM encoded features for online signature verification Vivek

Histogram-based matching of GMM encoded features for online signature verification Vivek

Histogram-based matching of GMM encoded features for online signature verification Vivek Venugopal On behalf of Abhishek Sharma,Dr. Suresh Sundaram Multimedia Analytics Laboratory, Electronics and Electrical Engineering Department IIT Guwahati

187 views • 15 slides

More recommend