Estimating Gaussian Mixture Models from Data with Missing Features - - PowerPoint PPT Presentation

▶
estimating gaussian mixture models from data with missing
SMART_READER_LITE
LIVE PREVIEW

Estimating Gaussian Mixture Models from Data with Missing Features - - PowerPoint PPT Presentation

Oct 05, 2023 375 likes •478 views

Estimating Gaussian Mixture Models from Data with Missing Features by Daniel McMichael CSSIP Missing Data In classification we frequently see to classify objects using vectors of measured features. Sometimes these features are missing: [ 0

slide-1
SLIDE 1

Estimating Gaussian Mixture Models from Data with Missing Features

by

Daniel McMichael CSSIP

slide-2
SLIDE 2

Missing Data

In classification we frequently see to classify objects using vectors of measured features. Sometimes these features are missing:

x = [0 :5 0 :7 0 :4 X 0 :2 X 0 :8 ]
slide-3
SLIDE 3

Gaussian Mixture Models (GMMs)

A probability density model (a weighted sum of Gaussians):

p(y jf i ;
  • i
;
  • i
g n i=1 ) = n X i=1
  • i exp
f(y
  • i
) T
  • 1
(y
  • i
)=2 g p j2
  • i
j

(1) : Qualities of GMMs:

Can model any density (given enough components) Can be applied to classification Widely used “Easy” to analyse
slide-4
SLIDE 4

Heteroscedastic GMMs (HGMMs)

Conventionally, GMMs are homoscedastic: all data are modelled with the same Gaussian distribution:

p(y j j i ;
  • i
;
  • i
) = n X i=1
  • i
p(y j i ;
  • i
)

(2) Introduce a heteroscedastic variant, where the response to each datum is different:

p(y j jy j ; M j ;
  • i
;
  • i
;
  • i
) = n X i=1
  • i
p(y jy j + M j
  • i
; M j
  • i
M T j +
  • y
j )

(3)

slide-5
SLIDE 5

Uses for HGMMs

estimation of GMM parameters from data with missing features; estimation and prediction of indirectly observed mixture processes; modelling heteroscedastic data.

Need only a simplified HGMM:

p(y j j) = n X i=1
  • i
p(y j jM j
  • i
; M j
  • i
M T j )

(4) The gain matrices of each of the

N data, fM j g N j =1, contain only 1s and 0s,

and are formed by deleting the rows from an identity matrix corresponding to the missing features of each datum. This is the marginal distribution of the remaining features.

slide-6
SLIDE 6

The EM Algorithm

[Refer: Dempster, Laird and Rubin, 1977] Aim: to maximise a likelihood or posterior, over the parameters

, whilst

integrating out the nuisance parameters

Z.

To maximise the likelihood

p(Y j; Z ), start with the guess,
  • =
  • :
E-step: Q(j
  • )
= R p(Y j; Z )p(Z jY ;
  • )
d Z M-step:
  • = arg max
  • Q(j
  • )
slide-7
SLIDE 7

The E-step for HGMMs

Assume conditionally independent data

Y = fy j g N j =1, and group the

heteroscedastic parameters

fM j g N j =1 together into a set M.

The E-step is the calculation of

P (ijy j ;
  • ;
M ) for all the pairs of data y j

and HGMM components:

P (ijy j ;
  • ;
M ) = p(y j ji;
  • ;
M ) P (ij
  • ;
M ) P n l=1 p(y j jl ;
  • ;
M ) P (l j
  • ;
M ) :

i.e.

P (ijy j ;
  • ;
M ) = p(y j ji;
  • ;
M )
  • i
P n l=1 p(y j jl ;
  • ;
M )
  • l
slide-8
SLIDE 8

The M-step for HGMMs

Maximise

Q(j
  • ) with respect to
:
  • i
=

1

N P (ijy j ;
  • ;
M )
  • i
=
  • P
N j =1 P (ijy j ;
  • ))H
j M j
  • 1
  • P
N j =1 P (ijy j ;
  • )H
j y j
  • iterate to find
  • i :
  • i
!
  • i
+
  • i
;
  • i
=
  • i

2

N X j =1 P (ijy j ;
  • )H
j [(y j
  • M
j
  • i
)(y j
  • M
j
  • i
) T H T j
  • I
]
  • i

If

  • < 2 then
  • i
8i will never be non-positive definite.
slide-9
SLIDE 9

Results

Figure 1: Left: each feature missing 10% of the time. Figure 2: Right: each feature missing 60% of the time.

slide-10
SLIDE 10

Conclusions

Method for ML or MAP estimation of GMMs for data with missing

features.

An EM algorithm: fast. Able to stand very high levels of missing data. Other applications