Advanced Machine Learning CS 7140 - Spring 2018 Lecture 20: - - PowerPoint PPT Presentation

Advanced Machine Learning

CS 7140 - Spring 2018

Lecture 20: Generative Adversarial Networks

Jan-Willem van de Meent Slide credits: Ian Goodfellow

Variational Autoencoders

784 (28 x 28) 256 Input Images Hidden Units 2-50 Encoding (random) Mean
 Std Dev 256 784 (28 x 28) Hidden Units Reconstructed
 Images

Variational Autoencoders

Assume prior:

Objective: Evidence Lower Bound (ELBO)

Reconstruction
 log-likelihood KL between approximate 
 posterior and prior Log marginal likelihood KL between approximate posterior and posterior

Generative Adversarial Networks

x sampled from data Differentiable function D D(x) tries to be near 1 Input noise z Differentiable function G x sampled from model D D tries to make D(G(z)) near 0, G tries to make D(G(z)) near 1

Implicit Generative Models

z x

• Can generate samples x ~ pG(x)
• Cannot evaluate density pG(x)
• Requirement: G(z ; θ) is differentiable

Maximum Likelihood

θ∗ = arg max

θ

Ex∼pdata log pmodel(x | θ)

Objective: Minimax Game

• Generator G(z) generates fake images.
• Discriminator D(x) classifies real images as 1 


and fake images as 0.

• Objective for G: Maximize probability that G fools D.
• Objective for D: Minimize probability that D is fooled.

Exercise

• Suppose that we could evaluate pDATA(x) and pG(x)
• Can you express D(x) in terms of pDATA(x) and pG(x)?

SOLUTION:

Learning Implicit Densities

z x

(Goodfellow et al, 2014)

Data: Discriminator: Implicit Density:

Generator:

Key Idea: Use classifier to 
 estimate ratio 


• f densities

Understanding the Loss

= JSD(pDATA(x) || pG(x)) log4

' 1 2Ex⇠pDATA(x) ï log pDATA(x) pDATA(x) + pG(x) ò ï ò

L(G, D) = 1 2Ex⇠pDATA(x) [log D(z)] 1 2Ez⇠p(z) [log(1 G(D(z)))] ï ò

GAN loss minimizes the Jensen-Shannon divergence

ï ò 1 2Ex⇠pG(x) ï log pG(x) pDATA(x) + pG(x) ò

Problem: Saturation

Question: What happens to the gradient w.r.t. θ 
 when the discriminator D(x) = 0

Problem: Saturation

Question: What happens to the gradient w.r.t. θ 
 when the discriminator D(x) = 0

Non-Saturating GAN Loss

• Use different loss for generator parameters θ
• Αvoid 0 gradient issue when D(x) = 0, 


(but loss no longer takes form of JS divergence)

DCGAN Architecture

(Radford et al 2015) Most “deconvs” are batch normalized

LSUN Bedrooms

(Radford et al 2015)

Vector Space Arithmetic

• +

=

Man with Glasses Man Woman Woman with Glasses (Radford et al, 2015)

GANs vs VAEs

• VAEs learn a generative model and an inference model
• GANs learn a generator and a discriminator
• VAEs assume an explicit likelihood
• GANs assume an implicit likelihood
• VAEs do inference, but yield blurry images
• GANs don’t do inference, but yield crisper images

Next Frame Video Prediction

Ground Truth MSE Adversarial

(Lotter et al 2016)

Next Frame Video Prediction

(Mathieu et al. 2015)

Mean Squared Error Mean Absolute Error Adversarial

Is the Divergence Important?

x Probability Density

q∗ = argminqDKL(pq) p(x) q∗(x)

x Probability Density

q∗ = argminqDKL(qp) p(x) q∗(x)

(Goodfellow et al 2016) Variational Inference (and VAEs) Expectation Propagation (and GANs)

Using GANs for ML estimation

Generator loss minimizes reconstruction loss

Conclusion: GANs and VAEs not different due divergence

Using GANs for ML estimation

Generator loss minimizes reconstruction loss

Conclusion: GANs and VAEs not different due divergence

Applications of GANs

What is in this Image?

(Yeh et al., 2016)

Generative Modeling Reveals a Face

(Yeh et al., 2016)

Image-to-Image Translation

(Liu et al., 2017) Day to Night

CycleGAN

(Zhu et al., 2017) Horse to Zebra

CycleGAN

(Zhu et al., 2017)

Zebras Horses

horse zebra zebra horse

Summer Winter

summer winter winter summer Photograph Van Gogh Cezanne Monet Ukiyo-e

Monet Photos

Monet photo photo Monet

CycleGAN

(Zhu et al., 2017)

Zebras Horses

horse zebra zebra horse

Summer Winter

summer winter winter summer Photograph Van Gogh Cezanne Monet Ukiyo-e

Monet Photos

Monet photo photo Monet

CycleGAN

(Zhu et al., 2017)

X Y G F DY DX

G

F ˆ Y X Y

• X

Y

• G

F ˆ X

(a) (b) (c)

cycle-consistency

loss

cycle-consistency loss

DY DX

ˆ y ˆ x

x y

Which of These are Real?

vue.ai

✓ ✓

Celebrities Who Have Never Existed

(Karras et al., 2017) “never before wholly perceived in reality”