Generative Adversarial Networks MIX+GAN Ian Goodfellow, Sta ff - - PowerPoint PPT Presentation

▶
generative adversarial networks
SMART_READER_LITE
LIVE PREVIEW

Generative Adversarial Networks MIX+GAN Ian Goodfellow, Sta ff - - PowerPoint PPT Presentation

Nov 02, 2022 210 likes •632 views

CoGAN ID-CGAN LR-GAN MedGAN CGAN IcGAN A ff GAN DiscoGAN LS-GAN b-GAN LAPGAN MPM-GAN AdaGAN AMGAN iGAN LSGAN InfoGAN IAN CatGAN Generative Adversarial Networks MIX+GAN Ian Goodfellow, Sta ff Research Scientist, Google Brain McGAN

slide-1
SLIDE 1

Ian Goodfellow, Staff Research Scientist, Google Brain NVIDIA GPU Technology Conference San Jose, California 2017-05-09

Generative Adversarial Networks

3D-GAN AC-GAN AdaGAN AffGAN AL-CGAN ALI AMGAN AnoGAN ArtGAN b-GAN Bayesian GAN BEGAN BiGAN BS-GAN CGAN CCGAN CatGAN CoGAN Context-RNN-GAN C-RNN-GAN C-VAE-GAN CycleGAN DTN DCGAN DiscoGAN DR-GAN DualGAN EBGAN f-GAN FF-GAN GAWWN GoGAN GP-GAN IAN iGAN IcGAN ID-CGAN InfoGAN LAPGAN LR-GAN LS-GAN LSGAN MGAN MAGAN MAD-GAN MalGAN MARTA-GAN McGAN MedGAN MIX+GAN MPM-GAN

slide-2
SLIDE 2

(Goodfellow 2017)

Generative Modeling

  • Density estimation
  • Sample generation

Training examples Model samples

slide-3
SLIDE 3

(Goodfellow 2017)

Maximum Likelihood

θ∗ = arg max

θ

Ex∼pdata log pmodel(x | θ)

slide-4
SLIDE 4

(Goodfellow 2017)

Adversarial Nets Framework

x sampled from data Differentiable function D D(x) tries to be near 1 Input noise z Differentiable function G x sampled from model D D tries to make D(G(z)) near 0, G tries to make D(G(z)) near 1

(Goodfellow et al., 2014)

slide-5
SLIDE 5

(Goodfellow 2017)

What can you do with GANs?

  • Simulated environments and training data
  • Missing data
  • Semi-supervised learning
  • Multiple correct answers
  • Realistic generation tasks
  • Simulation by prediction
  • Solve inference problems
  • Learn useful embeddings
slide-6
SLIDE 6

(Goodfellow 2017)

slide-7
SLIDE 7

(Goodfellow 2017)

GANs for simulated training data

(Shrivastava et al., 2016)

slide-8
SLIDE 8

(Goodfellow 2017)

What can you do with GANs?

  • Simulated environments and training data
  • Missing data
  • Semi-supervised learning
  • Multiple correct answers
  • Realistic generation tasks
  • Simulation by prediction
  • Solve inference problems
  • Learn useful embeddings
slide-9
SLIDE 9

(Goodfellow 2017)

What is in this image?

(Yeh et al., 2016)

slide-10
SLIDE 10

(Goodfellow 2017)

Generative modeling reveals a face

(Yeh et al., 2016)

slide-11
SLIDE 11

(Goodfellow 2017)

What can you do with GANs?

  • Simulated environments and training data
  • Missing data
  • Semi-supervised learning
  • Multiple correct answers
  • Realistic generation tasks
  • Simulation by prediction
  • Solve inference problems
  • Learn useful embeddings
slide-12
SLIDE 12

(Goodfellow 2017)

Supervised Discriminator

Input Real Hidden units Fake Input Real dog Hidden units Fake Real cat

(Odena 2016, Salimans et al 2016)

slide-13
SLIDE 13

(Goodfellow 2017)

Semi-Supervised Classification

Model Number of incorrectly predicted test examples for a given number of labeled samples 20 50 100 200 DGN [21] 333 ± 14 Virtual Adversarial [22] 212 CatGAN [14] 191 ± 10 Skip Deep Generative Model [23] 132 ± 7 Ladder network [24] 106 ± 37 Auxiliary Deep Generative Model [23] 96 ± 2 Our model 1677 ± 452 221 ± 136 93 ± 6.5 90 ± 4.2 Ensemble of 10 of our models 1134 ± 445 142 ± 96 86 ± 5.6 81 ± 4.3

(Salimans et al 2016) MNIST (Permutation Invariant)

slide-14
SLIDE 14

(Goodfellow 2017)

Semi-Supervised Classification

(Salimans et al 2016)

Model Test error rate for a given number of labeled samples 1000 2000 4000 8000 Ladder network [24] 20.40±0.47 CatGAN [14] 19.58±0.46 Our model 21.83±2.01 19.61±2.09 18.63±2.32 17.72±1.82 Ensemble of 10 of our models 19.22±0.54 17.25±0.66 15.59±0.47 14.87±0.89

Model Percentage of incorrectly predicted test examples for a given number of labeled samples 500 1000 2000 DGN [21] 36.02±0.10 Virtual Adversarial [22] 24.63 Auxiliary Deep Generative Model [23] 22.86 Skip Deep Generative Model [23] 16.61±0.24 Our model 18.44 ± 4.8 8.11 ± 1.3 6.16 ± 0.58 Ensemble of 10 of our models 5.88 ± 1.0

CIFAR-10 SVHN

slide-15
SLIDE 15

(Goodfellow 2017)

What can you do with GANs?

  • Simulated environments and training data
  • Missing data
  • Semi-supervised learning
  • Multiple correct answers
  • Realistic generation tasks
  • Simulation by prediction
  • Solve inference problems
  • Learn useful embeddings
slide-16
SLIDE 16

(Goodfellow 2017)

Next Video Frame Prediction

Ground Truth MSE Adversarial

(Lotter et al 2016) What happens next?

slide-17
SLIDE 17

(Goodfellow 2017)

Ground Truth MSE Adversarial

Next Video Frame Prediction

(Lotter et al 2016)

slide-18
SLIDE 18

(Goodfellow 2017)

What can you do with GANs?

  • Simulated environments and training data
  • Missing data
  • Semi-supervised learning
  • Multiple correct answers
  • Realistic generation tasks
  • Simulation by prediction
  • Solve inference problems
  • Learn useful embeddings
slide-19
SLIDE 19

(Goodfellow 2017)

iGAN

youtube (Zhu et al., 2016)

slide-20
SLIDE 20

(Goodfellow 2017)

Introspective Adversarial Networks

youtube (Brock et al., 2016)

slide-21
SLIDE 21

(Goodfellow 2017)

Image to Image Translation

Input Ground truth Output

(Isola et al., 2016)

Aerial to Map Labels to Street Scene

input

  • utput

input

  • utput
slide-22
SLIDE 22

(Goodfellow 2017)

Unsupervised Image-to-Image Translation

(Liu et al., 2017) Day to night

slide-23
SLIDE 23

(Goodfellow 2017)

CycleGAN

(Zhu et al., 2017)

slide-24
SLIDE 24

(Goodfellow 2017)

Text-to-Image Synthesis

(Zhang et al., 2016)

This bird has a yellow belly and tarsus, grey back, wings, and brown throat, nape with a black face

slide-25
SLIDE 25

(Goodfellow 2017)

What can you do with GANs?

  • Simulated environments and training data
  • Missing data
  • Semi-supervised learning
  • Multiple correct answers
  • Realistic generation tasks
  • Simulation by prediction
  • Solve inference problems
  • Learn useful embeddings
slide-26
SLIDE 26

(Goodfellow 2017)

Simulating particle physics

(de Oliveira et al., 2017) Save millions of dollars of CPU time by predicting

  • utcomes of explicit

simulations

slide-27
SLIDE 27

(Goodfellow 2017)

What can you do with GANs?

  • Simulated environments and training data
  • Missing data
  • Semi-supervised learning
  • Multiple correct answers
  • Realistic generation tasks
  • Simulation by prediction
  • Solve inference problems
  • Learn useful embeddings
slide-28
SLIDE 28

(Goodfellow 2017)

Adversarial Variational Bayes

(Mescheder et al, 2017)

slide-29
SLIDE 29

(Goodfellow 2017)

What can you do with GANs?

  • Simulated environments and training data
  • Missing data
  • Semi-supervised learning
  • Multiple correct answers
  • Realistic generation tasks
  • Simulation by prediction
  • Solve inference problems
  • Learn useful embeddings
slide-30
SLIDE 30

(Goodfellow 2017)

Vector Space Arithmetic

  • +

=

Man with glasses Man Woman Woman with Glasses (Radford et al, 2015)

slide-31
SLIDE 31

(Goodfellow 2017)

Learning interpretable latent codes / controlling the generation process

InfoGAN (Chen et al 2016)

slide-32
SLIDE 32

(Goodfellow 2017)

How long until GANs can do this?

Training examples Model samples

slide-33
SLIDE 33

(Goodfellow 2017)

AC-GANs

(Odena et al., 2016)

slide-34
SLIDE 34

(Goodfellow 2017)

Minibatch GAN on ImageNet

(Salimans et al., 2016)

slide-35
SLIDE 35

(Goodfellow 2017)

Cherry-Picked Results

slide-36
SLIDE 36

(Goodfellow 2017)

Problems with Counting

slide-37
SLIDE 37

(Goodfellow 2017)

Problems with Perspective

slide-38
SLIDE 38

(Goodfellow 2017)

Problems with Global Structure

slide-39
SLIDE 39

(Goodfellow 2017)

This one is real

slide-40
SLIDE 40

(Goodfellow 2017)

Conclusion

  • GANs are generative models based on game theory
  • GANs open the door to a wide range of engineering

tasks

  • There are still important research challenges to solve

before GANs can generate arbitrary data