Gaussian Mixture Models Instructor: John Thickstun Discussion Board: - - PowerPoint PPT Presentation

Gaussian Mixture Models

Discussion Board: Available on Canvas Zoom Link: Available on Canvas Instructor Contact: thickstn@cs.washington.edu Course Webpage: https://courses.cs.washington.edu/courses/cse599i/20au/ CSE 599I: Generative Models University of Washington Lecture 2 Instructor: John Thickstun

Logistics

CSE 599I: Generative Models University of Washington Lecture 2

• Lectures will be recorded and posted internally starting today.
• A better discussion board (Ed) will be up in the next couple days.
• TA: Sami Davies will be helping out with course infrastructure and

moderating questions during lecture

Recap: Overfitting

CSE 599I: Generative Models University of Washington Lecture 2

• Given finite samples from a continuous distribution
• Estimate the probability of each element of using the MLE?
• Regularization: restrict our estimator to a parametric family

x1, . . . , xn ∼ p

X

ˆ p(x) = (

1 n

if x ∈ {x1, . . . , xn},

• therwise.

p(x).

Maximum Likelihood Estimation

CSE 599I: Generative Models University of Washington Lecture 2

• Given a parametric family of probability distributions
• Choose to maximize the likelihood of observations :
• KL-divergence minimization:

sup

θ

E

x∼p log pθ(x) ≈ sup θ

1 n

n

X

i=1

log pθ(xi).

{pθ : θ ∈ Θ}

θ ∈ Θ

x1, . . . , xn

E

x∼p log pθ(x) = E x∼p log pθ(x)

p(x) p(x) = H(p) + D(p k pθ) H(p).

Generalization

CSE 599I: Generative Models University of Washington Lecture 2

• Measure performance via likelihood of a test set :
• Deep learning for classification: send training error to zero
• Deep learning for generative modeling: send log-likelihood to zero?

xtest

1

, . . . , xtest

m

1 m

m

X

i=1

log pθ(xtest

i

).

E

x∼p − log pθ(x) ≥ H(p),

pθ(x) ≈ (

1 n

if x ∈ {x1, . . . , xn},

• therwise.

Gaussian Mixture Models

CSE 599I: Generative Models University of Washington Lecture 2

• K-Gaussian mixture model over data .
• Each data point belongs to a latent cluster .
• Each cluster is drawn from a Gaussian with mean and variance :

x ∈ Rd

z ∈ {1, . . . , K}

x

µk

Σk

1. z ∼ Categoricalπ(K), π ∈ ∆K−1, 2. x ∼ N(µz, Σz), µ ∈ RK×d, Σ ∈ RK×d×d.

Gaussian Mixture Models

CSE 599I: Generative Models University of Washington Lecture 2

• Generative model:
• Likelihood:

p(x) = Z

Z

p(x, z) dz = Z

Z

p(x|z)p(z) dz =

K

X

k=1

πkp(x|z = k) =

K

X

k=1

πkN(x; µk, Σk).

1. z ∼ Categoricalπ(K), π ∈ ∆K−1, 2. x ∼ N(µz, Σz), µ ∈ RK×d, Σ ∈ RK×d×d.

• Generative model:
• The maximum likelihood estimator (parameters in red):
• No analytical solution; non-convex optimization problem

Maximize the Likelihood

CSE 599I: Generative Models University of Washington Lecture 2

1. z ∼ Categoricalπ(K), π ∈ ∆K−1, 2. x ∼ N(µz, Σz), µ ∈ RK×d, Σ ∈ RK×d×d.

sup

θ

1 n

n

X

i=1

log pθ(xi) = sup

θ

1 n

n

X

i=1

log

K

X

k=1

πkN(xi; µk, Σk).

• The maximum likelihood estimator (parameters in red):
• Gradient ascent: initialize with random parameters and iteratively apply
• Very important to start with a random initialization

Gradient Ascent

CSE 599I: Generative Models University of Washington Lecture 2

θ(i) = θ(i−1) + ηrθ

n

X

i=1

log pθ(xi).

sup

θ

1 n

n

X

i=1

log pθ(xi) = sup

θ

1 n

n

X

i=1

log

K

X

k=1

πkN(xi; µk, Σk).

• Gradient ascent:
• Stochastic gradient ascent (SGD):
• Gradient ascent update: O(n)
• SGD update: O(1)

Stochastic Gradient Ascent

CSE 599I: Generative Models University of Washington Lecture 2

θ(i) = θ(i−1) + ηrθ log pθ(xi (mod n)).

θ(i) = θ(i−1) + ηrθ

n

X

i=1

log pθ(xi).

E

x∼p log pθ(x) ≈ log pθ(xi).

E

x∼p log pθ(x) ≈ 1

n

n

X

i=1

log pθ(xi).

5-Minute Break

CSE 599I: Generative Models University of Washington Lecture 2

Recap: Evaluating the Likelihood

CSE 599I: Generative Models University of Washington Lecture 2

• To run SGD, we needed to evaluate the marginal probability :
• For GMM’s, this is tractable (the integral is a simple analytical sum).
• What would we do if the integral weren’t tractable?

pθ(x)

pθ(x) = Z

Z

pθ(x, z) dz = Z

Z

pθ(x|z)pθ(z) dz.

The Evidence Lower-Bound

CSE 599I: Generative Models University of Washington Lecture 2

• Approximate the marginal with importance sampling:
• The distribution is called a proposal distribution
• Bound is tight when (when the proposal is the posterior)

q(z|x)

log pθ(x) = log E

z∼q(·|x)

pθ(x, z) q(z|x)

• =

E

z∼q(·|x)

 log pθ(x, z) q(z|x)

• + D(q(z|x) k pθ(z|x))
• E

z∼q(·|x)

 log pθ(x, z) q(z|x)

• .

q(z|x) = p(z|x)

The Evidence Lower-Bound

CSE 599I: Generative Models University of Washington Lecture 2

• We can also derive the Evidence Lower-Bound by Jensen’s inequality:
• Machine Learning community calls this lower bound the ELBO
• Another way to look at the ELBO:

log pθ(x) = log E

z∼q(·|x)

pθ(x, z) q(z|x)

• ≥

E

z∼q(·|x)

 log pθ(x, z) q(z|x)

• .

E

z∼q(·|x)

 log pθ(x, z) q(z|x)

• =

E

z∼q(·|x)

⇥ log pθ(x|z) ⇤ D(q(z|x) k p(z)).

Expectation Maximization

CSE 599I: Generative Models University of Washington Lecture 2

• Jointly optimize the ELBO over and :
• Alternating Optimization (Expectation Maximization):

ˆ θmle = arg max

θ

max

q

E

x∼p z∼q(·|x)

 log pθ(x, z) q(z|x)

• .

θ

q

• 1. Fix and optimize the proposal distribution (E-step).
• 2. Fix the proposal distribution and optimize (M-step).

θ

q

q

θ

EM for GMMs

CSE 599I: Generative Models University of Washington Lecture 2

• Sometimes the maximizer of the E-step has an analytic solution.
• For GMM’s, the E-step is:
• GMM’s also have an analytical M-step:

pθ(z|x) = pθ(x|z)pθ(z) pθ(x) = πzN(x; µz, Σz) PK

k=1 πkN(x; µk, Σk)

.

π0

k = 1

n

n

X

i=1

pθ(zk|xi), µ0

k =

1 nπ0

k n

X

i=1

pθ(zk|xi)xi, Σ0

k =

1 nπ0

k n

X

i=1

pθ(zk|xi)(xi − µ0

k)⌦2.

Preview of Upcoming Topics

CSE 599I: Generative Models University of Washington Lecture 2

• On Wednesday we’ll start talking about sequence models
• Next week: deep dive into neural parameterization of sequence models
• Homework 1 will be posted by the end of the week (due October 26th)
• The language modeling on HW1 will rely on next week’s lectures