Statistical Machine Learning Lecture 06 Extra: Expectation - - PowerPoint PPT Presentation

Statistical Machine Learning

Lecture 06 Extra: Expectation Maximization

Kristian Kersting TU Darmstadt

Summer Term 2020

• K. Kersting based on Slides from J. Peters· Statistical Machine Learning· Summer Term 2020

1 / 9

Expectation Maximization

Basic Idea

t, qt

log-Likelihood Desired lower bound (q, )

• K. Kersting based on Slides from J. Peters· Statistical Machine Learning· Summer Term 2020

2 / 9

Expectation Maximization

t

( ) (q,

t)

Choose q *

Requirements

• 1. Guarantee a lower bound (aka surrogate function )

F (q, θ) ≤ L (θ) ∀q, θ where q is the “guessed” distribution and θ are the parameters

• 2. Choose q∗ such that they touch

F (q∗, θt) = L (θt)

• K. Kersting based on Slides from J. Peters· Statistical Machine Learning· Summer Term 2020

3 / 9

Expectation Maximization

Expectation-Step (E-Step)

t

( ) Choose q *

t + 1

(q, ) (q *

t + 1,

)

• K. Kersting based on Slides from J. Peters· Statistical Machine Learning· Summer Term 2020

4 / 9

Expectation Maximization

Maximization-Step (M-Step)

*

( ) (q *

t + 1, * )

Find θ∗ by maximization θ∗ = arg max

θ

F

• q∗

t+1, θ

• K. Kersting based on Slides from J. Peters· Statistical Machine Learning· Summer Term 2020

5 / 9

Expectation Maximization

Find a lower bound on L (θ) L (θ) =

• i

log pθ (xi) =

• i

log

• pθ (xi, zi) dzi

=

• i

log

• q (zi) pθ (xi, zi)

q (zi) dzi (by Jensen’s inequality) ≥

• i
• q (zi) log pθ (xi, zi)

q (zi) dzi ≡ F (q, θ) s.t.

• q (zi) dzi = 1

∀i

• K. Kersting based on Slides from J. Peters· Statistical Machine Learning· Summer Term 2020

6 / 9

Expectation Maximization

Constrained Optimization Problem

max

• i
• q (zi) log pθ (xi, zi)

q (zi) dzi s.t.

• q (zi) dzi = 1

∀i

• K. Kersting based on Slides from J. Peters· Statistical Machine Learning· Summer Term 2020

7 / 9

Expectation Maximization

L =

• i
• q (zi) log pθ (xi, zi)

q (zi) dzi

• + λi
• qi (zi) dzi − 1
• ∀i

∇q(zi)L =

• log pθ (xi, zi)

q (zi) − 1

• + λi

!

= 0 = ⇒ q (zi) = exp (λi − 1) pθ (xi, zi) ∇λiL =

• q (zi) dzi − 1

!

= 0 exp (λi − 1)

• pθ (xi, zi) dzi = 1

= ⇒ λi = 1 − log

• pθ (xi, zi) dzi

q (zi) = pθ (zi|xi) ≡ E-step

• K. Kersting based on Slides from J. Peters· Statistical Machine Learning· Summer Term 2020

8 / 9

Expectation Maximization

• 1. We have a lower bound for the likelihood
• 2. We guaranteed

F (q∗, θt) = L (θt)

• 3. We want to guarantee

L (θt+1) ≥ L (θt) thus L (θt+1) ≥ F

• q∗

t+1, θt+1

• = max

θ

F

• q∗

t+1, θ

• ≥ L (θt)
• 4. Choose θt+1 as

θt+1 = arg max

θ

F

• q∗

t+1, θ

• K. Kersting based on Slides from J. Peters· Statistical Machine Learning· Summer Term 2020

9 / 9