Generative Adversarial Networks (GANs) M. Soleymani Sharif - PowerPoint PPT Presentation

Generative Adversarial Networks (GANs) M. Soleymani Sharif University of Technology Spring 2020 Most slides are based on Fei Fei Li and colleagues lectures, cs231n, Stanford 2018 and some slides from Raymond Yeh et al., CS598LAZ, Illinois, 2017. 1

Generative Models • Explicitly working with the distribution (likelihood): – Fully visible belief networks – Variational Autoencoder (variational inference) • GAN as an implicit generative model – trained without even needing to explicitly define a density functions. 2 Ian Goodfellow, Generative Adversarial Networks, NIPS 2016 Tutorial.

Generative Adversarial Networks • Problem: Want to sample from complex, high- dimensional training distribution. – No direct way to do this! • Solution: – Sample from a simple distribution, e.g. random noise. – Then, learn transformation to training distribution. • Q: What can we use to represent this complex transformation? – A neural network! 3 Ian Goodfellow et al., “Generative Adversarial Nets”, NIPS 2014.

Training GANs: Two-player game • Generator network: try to fool the discriminator by generating real- looking images • Discriminator network: try to distinguish between real and fake images 4

GAN architecture Discriminator real/fake Sample Network Generated Output sample Generator Network Input noise 5

GAN architecture D Sample from x training data Discriminator real/fake Network Generated G(z) sample Generator G Network z noise 6

Training GANs • Generator network: intend to generate real-looking samples • Discriminator network: intend to distinguish between real and fake samples Discriminator output Discriminator output for generated samples for real samples 𝐾 (#) = 𝔽 𝒚~) *+,+ log 𝐸 𝒚 + 𝔽 𝒜~)(𝒜) log 1 − 𝐸 𝐻(𝒜) 𝐾 (6) = 𝔽 𝒜~)(𝒜) log 𝐸 𝐻(𝒜) ∗ = max 𝜄 # < = 𝔽 𝒚~) *+,+ log 𝐸 < = (𝒚) + 𝔽 𝒜~)(𝒜) log 1 − 𝐸 < = 𝐻 < > (𝒜) ∗ = max 𝜄 6 < > 𝔽 𝒜~)(𝒜) log 𝐸 < = 𝐻 < > (𝒜) 7 Ian Goodfellow et al., Generative Adversarial Nets, NIPS 2014.

Training GANs: Two-player game • Generator network: intend to generate real-looking samples – try to fool the discriminator by generate real-looking samples • Discriminator network: intend to distinguish between real and fake samples – to evaluate the generated samples by generator network Discriminator output Discriminator output for generated samples for real samples min < > max < = 𝔽 𝒚~) *+,+ log 𝐸 < = (𝒚) + 𝔽 𝒜~)(𝒜) log 1 − 𝐸 < = 𝐻 < > (𝒜) Discriminator (θd ) wants to maximize objective such that D(x) is close to 1 (real) and D(G(z)) is close to 0 (fake) Generator (θg ) wants to minimize objective such that D(G(z)) is close to 1 (discriminator is fooled into thinking generated G(z) is real) 8 Ian Goodfellow et al., Generative Adversarial Nets, NIPS 2014.

Training GANs: Two-player game • Train jointly in a minimax game: min < > max < = 𝔽 𝒚~) *+,+ log 𝐸 < = (𝒚) + 𝔽 𝒜~)(𝒜) log 1 − 𝐸 < = 𝐻 < > (𝒜) • Objectives of the generators and discriminators: 𝐾 (#) = 𝔽 𝒚~) *+,+ log 𝐸 𝒚 + 𝔽 𝒜~)(𝒜) log 1 − 𝐸 𝐻(𝒜) 𝐾 (6) = −𝐾 (#) The training procedure for G is to maximize the probability of D making a mistake. 9 Ian Goodfellow et al., Generative Adversarial Nets, NIPS 2014.

GAN: Adversarial • Two networks: – Generator G: creates (fake) samples that the discriminator cannot distinguish • trained to capture the data distribution – Discriminator D: distinguish fake and real samples • estimate the probability that a sample came from the training data rather than G. • Equilibrium is a saddle point of the discriminator loss 10 Ian J. Goodfellow et al. Generative Adversarial Networks, 2014.

Training GANs: Two-player game Gradient is relatively flat for likely fake samples while we intend to improve generator from them. 11 Ian Goodfellow et al., Generative Adversarial Nets, NIPS 2014.

Training GANs: Two-player game 12 Ian Goodfellow et al., “Generative Adversarial Nets”, NIPS 2014.

Training GANs: Two-player game Higher gradient for likely fake samples as we intend. In generator, instead of minimizing likelihood of discriminator being correct, maximize likelihood of discriminator being wrong. 13 Ian Goodfellow et al., Generative Adversarial Nets, NIPS 2014.

Training GANs: Two-player game Aside: Jointly training two networks is challenging, can be unstable. Choosing objectives with better loss landscapes helps training, is an active area of research. 14 Ian Goodfellow et al., “Generative Adversarial Nets”, NIPS 2014.

GAN Training Sample from D x Each player’s cost depends on the training data Discriminator parameters of other player. However, each real/fake Network player can only optimize its own parameters. Generated 𝜄 # updates by fixing 𝜄 6 G(z) sample Generator G 𝜄 6 updates Network for fixed 𝜄 6 z noise 15

GAN architecture Sample from D x training data Discriminator real/fake Network Generated G(z) sample After training, this generator network is used for sample Generator G generation Network z noise 16

Training GAN 17 Ian Goodfellow et al., “Generative Adversarial Nets”, NIPS 2014.

Putting it together: GAN training algorithm 18 Ian Goodfellow et al., “Generative Adversarial Nets”, NIPS 2014.

Generate samples by GAN Nearest training sample to the generated sample in the second rightmost column MNIST TFD CIFAR-10 CIFAR-10 19 Ian Goodfellow et al., Generative Adversarial Nets, NIPS 2014.

Deep Convolutional GANs (DCGANs) • Generator: an upsampling network with fractionally-strided convolutions • Discriminator is a convolutional network • Conditioning on a class label improves the generated samples All convolutional net 20 Radford et al., Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, ICLR 2016.

DCGAN: Convolutional Architecture 21 Radford et al., Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, ICLR 2016.

Conditional GAN Learning a conditional model p(y|x) often gives much better samples 22 M. Mirza, S. Osindero, Conditional Generative Adversarial Nets, 2014.

Conditional GAN: MNIST 23 M. Mirza, S. Osindero, Conditional Generative Adversarial Nets, 2014.

DCGAN 24 Radford et al., Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, ICLR 2016.

Disentangling the representation • GANs learn a representation z of the image x. – this representation can capture useful high-level abstract semantic properties of data – However, it is difficult to make use of it. 25

DCGAN: Latent space Interpolation between a series of 9 random points in Z 26 Radford et al., Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, ICLR 2016.

Vector arithmetic for visual concepts Samples generated by model 27 Radford et al., Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, ICLR 2016.

Vector arithmetic for visual concepts Samples generated by model Sample corresponding to average Z of the above images Radford et al., Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, ICLR 2016. 28

Vector arithmetic for visual concepts 29 Radford et al., Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, ICLR 2016.

Optimal Discriminator and Generator • For a fix generator 𝐻 , given enough capacity the optimal discriminator is: 𝑞 CDED 𝐸 = 𝑞 CDED + 𝑞 F • If we insert the above discriminator in the objective: – when both models have sufficient capacity, Nash equilibrium of the game corresponds to: 𝑞 F ≈ 𝑞 CDED 𝐻(𝑨) being drawn from the same distribution as the training data D(x) will be 1/2 for all x. 30 Ian Goodfellow et al., Generative Adversarial Nets, NIPS 2014.

Global Optimality 31 Ian Goodfellow et al., “Generative Adversarial Nets”, NIPS 2014.

Training 32 Ian Goodfellow et al., Generative Adversarial Nets, NIPS 2014.

GAN Applications • Conditional image generation tasks – Image to image tasks • style transfer, super-resolution, in-painting, and etc – Text-to-Image Generation • Other generative models – Speech synthesis: Text to Speech – Text generation 33

SRGAN: weighted combination of several loss components C. Ledig et al, Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network, CVPR 2017. 34

SRGAN: weighted combination of several loss components C. Ledig et al, Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network, CVPR 2017. 35

Image-to-Image Translation (pix2pix) 36 Phillip Isola et al., Image-to-Image Translation with Conditional Adversarial Networks, CVPR 2017.