EM Algorithm 09-09-2019 For Mixture Gaussian Models Instructor - - - PowerPoint PPT Presentation

▶
em algorithm
SMART_READER_LITE
LIVE PREVIEW

EM Algorithm 09-09-2019 For Mixture Gaussian Models Instructor - - - PowerPoint PPT Presentation

Jul 17, 2023 367 likes •608 views

E9 205 Machine Learning for Signal Processing EM Algorithm 09-09-2019 For Mixture Gaussian Models Instructor - Sriram Ganapathy (sriramg@iisc.ac.in) Teaching Assistant - Prachi Singh (prachisingh@iisc.ac.in). Basics of Information Theory

slide-1
SLIDE 1

E9 205 Machine Learning for Signal Processing

09-09-2019

EM Algorithm

For Mixture Gaussian Models

Instructor - Sriram Ganapathy (sriramg@iisc.ac.in) Teaching Assistant - Prachi Singh (prachisingh@iisc.ac.in).

slide-2
SLIDE 2

Basics of Information Theory

  • Entropy of distribution
  • KL divergence
  • Jensen’s inequality
  • Expectation Maximization Algorithm for MLE
slide-3
SLIDE 3

Gaussian Mixture Models

A Gaussian Mixture Model (GMM) is defined as The weighting coefficients have the property

slide-4
SLIDE 4

The number of parameters is

Gaussian Mixture Models

  • Properties of GMM
  • Can model multi-modal

data.

  • Identify data clusters.
  • Can model arbitrarily

complex data distributions

The set of parameters for the model are

slide-5
SLIDE 5

MLE for GMM

  • The log-likelihood function over the entire data in this

case will have a logarithm of a summation

  • Solving for the optimal parameters using MLE for

GMM is not straight forward.

  • Resort to the Expectation Maximization (EM) algorithm
slide-6
SLIDE 6

Basics of Information Theory

  • Entropy of distribution
  • KL divergence
  • Jensen’s inequality
  • Expectation Maximization Algorithm for MLE
slide-7
SLIDE 7

Expectation Maximization Algorithm

  • Iterative procedure.
  • Assume the existence of hidden variable

associated with each data sample

  • Let the current estimate (at iteration n) be

Define the Q function as

slide-8
SLIDE 8

Expectation Maximization Algorithm

  • It can be proven that if we choose

then

  • In many cases, finding the maximum for the Q

function may be easier than likelihood function w.r.t. the parameters.

  • Solution is dependent on finding a good choice of

the hidden variables which eases the computation

  • Iteratively improve the log-likelihood function.
slide-9
SLIDE 9

EM Algorithm Summary

  • Initialize with a set of model parameters (n=1)
  • Compute the conditional expectation (E-step)
  • Maximize the conditional expectation w.r.t.
  • parameter. (M-step) (n = n+1)
  • Check for convergence
  • Go back to E-step if model has not converged.
slide-10
SLIDE 10
  • The hidden variables will be the index of the

mixture component which generated

  • Re-estimation formulae

EM Algorithm for GMM

slide-11
SLIDE 11

EM Algorithm for GMM

E-step M-step

slide-12
SLIDE 12

EM Algorithm for GMM

3.3 3.4 3.5 3.6 3.7 3.8 3.9 4 3.7 3.8 3.9 4 4.1 4.2 4.3 4.4 ANEMIA PATIENTS AND CONTROLS Red Blood Cell Volume Red Blood Cell Hemoglobin Concentration

Cadez, Igor V., et al. "Hierarchical models for screening of iron deficiency anemia." ICML. 1999.

slide-13
SLIDE 13

EM Algorithm for GMM

3.3 3.4 3.5 3.6 3.7 3.8 3.9 4 3.7 3.8 3.9 4 4.1 4.2 4.3 4.4

Red Blood Cell Volume Red Blood Cell Hemoglobin Concentration

EM ITERATION 1

Cadez, Igor V., et al. "Hierarchical models for screening of iron deficiency anemia." ICML. 1999.

slide-14
SLIDE 14

EM Algorithm for GMM

3.3 3.4 3.5 3.6 3.7 3.8 3.9 4 3.7 3.8 3.9 4 4.1 4.2 4.3 4.4

Red Blood Cell Volume Red Blood Cell Hemoglobin Concentration

EM ITERATION 3

Cadez, Igor V., et al. "Hierarchical models for screening of iron deficiency anemia." ICML. 1999.

slide-15
SLIDE 15

EM Algorithm for GMM

3.3 3.4 3.5 3.6 3.7 3.8 3.9 4 3.7 3.8 3.9 4 4.1 4.2 4.3 4.4

Red Blood Cell Volume Red Blood Cell Hemoglobin Concentration

EM ITERATION 5

Cadez, Igor V., et al. "Hierarchical models for screening of iron deficiency anemia." ICML. 1999.

slide-16
SLIDE 16

EM Algorithm for GMM

3.3 3.4 3.5 3.6 3.7 3.8 3.9 4 3.7 3.8 3.9 4 4.1 4.2 4.3 4.4

Red Blood Cell Volume Red Blood Cell Hemoglobin Concentration

EM ITERATION 10

Cadez, Igor V., et al. "Hierarchical models for screening of iron deficiency anemia." ICML. 1999.

slide-17
SLIDE 17

EM Algorithm for GMM

3.3 3.4 3.5 3.6 3.7 3.8 3.9 4 3.7 3.8 3.9 4 4.1 4.2 4.3 4.4

Red Blood Cell Volume Red Blood Cell Hemoglobin Concentration

EM ITERATION 15

Cadez, Igor V., et al. "Hierarchical models for screening of iron deficiency anemia." ICML. 1999.

slide-18
SLIDE 18

EM Algorithm for GMM

3.3 3.4 3.5 3.6 3.7 3.8 3.9 4 3.7 3.8 3.9 4 4.1 4.2 4.3 4.4

Red Blood Cell Volume Red Blood Cell Hemoglobin Concentration

EM ITERATION 25

Cadez, Igor V., et al. "Hierarchical models for screening of iron deficiency anemia." ICML. 1999.

slide-19
SLIDE 19

EM Algorithm for GMM

3.3 3.4 3.5 3.6 3.7 3.8 3.9 4 3.7 3.8 3.9 4 4.1 4.2 4.3 4.4

Red Blood Cell Volume Red Blood Cell Hemoglobin Concentration

ANEMIA DATA WITH LABELS

Cadez, Igor V., et al. "Hierarchical models for screening of iron deficiency anemia." ICML. 1999.

slide-20
SLIDE 20

K-means Algorithm for Initialization

slide-21
SLIDE 21

Other Considerations

  • Initialization - random or k-means
  • Number of Gaussians
  • Type of Covariance matrix
  • Spherical covariance
slide-22
SLIDE 22

Other Considerations

  • Initialization - random or k-means
  • Number of Gaussians
  • Type of Covariance matrix
  • Diagonal covariance
slide-23
SLIDE 23

Other Considerations

  • Initialization - random or k-means
  • Number of Gaussians
  • Type of Covariance matrix
  • Full covariance