Variational Autoencoders Instructor: John Thickstun Discussion - - PowerPoint PPT Presentation

Discussion Board: Available on Ed Zoom Link: Available on Canvas Instructor Contact: thickstn@cs.washington.edu Course Webpage: https://courses.cs.washington.edu/courses/cse599i/20au/ CSE 599I: Generative Models University of Washington Lecture 6 Instructor: John Thickstun

Variational Autoencoders

Latent Variable Models

CSE 599I: Generative Models University of Washington Lecture 6

• Given finite samples , and unlimited samples
• Generative latent variable model:
• Want to learn the marginal defined by

x1, . . . , xn ∼ p

1. z ∼ r, 2. x ∼ pθ(·|z)

z ∼ r.

pθ(x) ≈ p(x)

pθ(x) = Z

Z

pθ(x|z)r(z) dz.

Motivation for Latent Variables

CSE 599I: Generative Models University of Washington Lecture 6

• Conditioning on a latent code could give

samples more global coherence.

• Learned latent codes might reveal structure in

the data distribution.

• The latent codes could be useful for

downstream tasks or interpretability.

z

x

Kingma and Welling, ICLR 2013

Motivation for Latent Variables

CSE 599I: Generative Models University of Washington Lecture 6

• Conditioning on a latent code could give

samples more global coherence.

• Learned latent codes might reveal structure in

the data distribution.

• The latent codes could be useful for

downstream tasks or interpretability.

z

x

Klys, Snell, and Zemel, Neurips 2018

The MLE with Latent Variables

CSE 599I: Generative Models University of Washington Lecture 6

• Want to learn the marginal defined by
• Fit the maximum likelihood estimator?
• This doesn’t look promising…

pθ(x) ≈ p(x)

pθ(x) = Z

Z

pθ(x|z)r(z) dz.

ˆ θmle ≡ arg max

θ

E

x∼p log pθ(x) = arg max θ

E

x∼p log

Z

Z

pθ(x|z)r(z) dz.

Gaussian Mixture Models

CSE 599I: Generative Models University of Washington Lecture 6

• Generative model:
• Likelihood:
• But what if is a continuous distribution over, e.g. ?

1. z ∼ Categoricalπ(K), π ∈ ∆K−1, 2. x ∼ N(µz, Σz), µ ∈ RK×d, Σ ∈ RK×d×d.

pθ(x) = Z

Z

pθ(x|z)r(z) dz =

K

X

k=1

πkN(x; µk, Σk).

r(z)

z ∈ Rk

The Evidence Lower Bound

CSE 599I: Generative Models University of Washington Lecture 6

• Fit the maximum likelihood estimator?
• Use importance sampling to estimate the integral.
• Construct a lower-bound on the marginal log-likelihood (the ELBO):

ˆ θmle ≡ arg max

θ

E

x∼p log pθ(x) = arg max θ

E

x∼p log

Z

Z

pθ(x|z)r(z) dz.

log pθ(x) = log E

z∼q(·|x)

pθ(x, z) q(z|x)

• ≥

E

z∼q(·|x)

 log pθ(x, z) q(z|x)

• .

Monte Carlo ELBO Estimation

CSE 599I: Generative Models University of Washington Lecture 6

• Importance-sampling estimator:
• Cannot directly estimate the log-likelihood with samples.
• Let . Evidence lower-bound (often use ; like “hard” EM):

log pθ(x) = log E

z∼q(·|x)

pθ(x, z) q(z|x)

• .

zi ∼ q(·|x)

m = 1

log pθ(x) ≥ E

z∼q(·|x)

 log pθ(x, z) q(z|x)

• ≈ 1

m

m

X

i=1

 log pθ(x, zi) q(zi|x)

• .

Joint Maximization

CSE 599I: Generative Models University of Washington Lecture 6

• Define the ELBO to be
• Estimate the marginal log-likelihood with by
• Equality holds when
• Jointly optimize over :

q(z|x) = pθ(z|x) = pθ(x|z)r(z) pθ(x) .

θ, q

L(x, z; θ, q) ≡ log pθ(x, z) q(z|x) .

log pθ(x) ≥ E

z∼q(·|x) L(x, z; θ, q).

ˆ θmle ≡ arg max

θ

E

x∼p log pθ(x) = arg max θ

sup

q

E

x∼p z∼q(·|x)

L(x, z; θ, q).

Posterior Inference

CSE 599I: Generative Models University of Washington Lecture 6

• How to estimate the posterior ?
• For GMM’s this was easy. We can compute the posterior exactly:
• What if the model isn’t so simple?

q(z|x) ≈ pθ(z|x)

q(z|x) = pθ(z|x) = pθ(x|z)pθ(z) pθ(x) = πzN(x; µz, Σz) PK

k=1 πkN(x; µk, Σk)

.

• What if the likelihood isn’t just a Gaussian?
• What if the prior is a continuous distribution on ?

pθ(x|z)

r(z)

z ∈ Rk

Approximate Posterior Inference

CSE 599I: Generative Models University of Washington Lecture 6

• How to estimate the posterior ?
• Learn a model that approximates the posterior!
• Let be a family of density estimators with parameters
• People sometimes call this amortized inference.
• Wait a minute… are we begging the question here?

q(z|x) ≈ pθ(z|x)

qφ(z|x)

φ.

5-Minute Break

CSE 599I: Generative Models University of Washington Lecture 6

Stochastic Backpropagation

CSE 599I: Generative Models University of Washington Lecture 6

• Jointly optimize over :
• Let’s use SGD, given a sample .
• Estimate the gradient w.r.t :
• But we’re in trouble computing

θ, φ

xi ∼ p, zi ∼ qφ(·|xi)

θ

ˆ θmle ≡ arg max

θ

E

x∼p log pθ(x) = arg max θ

sup

φ

E

x∼p z∼qφ(·|x)

L(x, z; θ, φ).

rθ E

x∼p z∼qφ(·|x)

L(x, z; θ, φ) ⇡ rθ log pθ(xi, zi) qφ(zi|xi) .

rφ E

x∼p z∼qφ(·|x)

L(x, z; θ, φ).

The Reparameterization Trick

CSE 599I: Generative Models University of Washington Lecture 6

• Need to construct a Monte Carlo estimate of
• Suppose is defined by a pushforward distribution, e.g.
• Then
• This is an example of Monte Carlo gradient estimation.

qφ(z|x)

z = fφ(x, ε), where ε ∼ N(0, I).

rφ E

x∼p z∼qφ(·|x)

L(x, z; θ, φ).

rφ E

x∼p z∼qφ(·|x)

L(x, z; ✓, ) = E

x∼p ε∼N (0,I)

rφL(x, fφ(x, ✏); ✓, ).

The Gaussian VAE

CSE 599I: Generative Models University of Washington Lecture 6

• Use a prior where ( is a hyper-parameter).
• With a Gaussian likelihood
• Where (the “decoder”) and are neural nets.
• Use a posterior approximation
• Where (the “encoder") and are neural nets.
• Think of it like a Gaussian mixture model with infinitely many components!

r(z) = N(0, I)

z ∈ Rk

k

pθ(x|z) = N(x; gθ(z), σ2

θ(z)I).

gθ : Z → X

σ2

θ : Z → R

qφ(z|x) = N(z; fφ(x), Σφ(x))

fφ : X → Z

Σφ : X → Z ⊗ Z

Reconstruction and Divergence

CSE 599I: Generative Models University of Washington Lecture 6

• The ELBO of the Gaussian VAE is:
• If is held constant, then
• People refer to these to terms as “reconstruction” and “divergence.”

σ2

dim(X) 2 log(2πσ2) 1 2σ2 E

z∼qφ(·|x) kx gθ(z)k2 D(qφ(z|x) k r(z)).

ˆ θmle = arg min

θ

inf

ϕ

E

x∼p z∼qϕ(·|x)

 1 2σ2 kx gθ(z)k2 + D(qϕ(z|x) k r(z))

• .