PixelGAN Autoencoders Alireza Makhzani, Brendan Frey Machine - PowerPoint PPT Presentation

PixelGAN Autoencoders Alireza Makhzani, Brendan Frey Machine learning Group University of Toronto CIFAR Deep Learning Summer School Montreal, Canada June 29 th , 2017 Alireza Makhzani PixelGAN Autoencoders 1 / 27

Outline 1. Background • PixelCNNs • Variational Autoencoders • Adversarial Autoencoders 2. PixelGAN Autoencoders • Gaussian Priors • Categorical Priors ✦ Clustering ✦ Semi-supervised Learning Alireza Makhzani PixelGAN Autoencoders 2 / 27

PixelCNNs ✦ Learn the image statistics directly at the pixel level. ✦ Good at modelling low-level pixel statistics. ✦ Samples lack global structure. ✦ Lacking latent representation. ✦ Conditional PixelCNNs can learn conditional densities. van den Oord et al., 2016 Alireza Makhzani PixelGAN Autoencoders 3 / 27

Variational Autoencoders log p ( x ) > � E q ( z | x ) [ � log( p ( x | z )] � KL( q ( z | x ) k p ( z )) ✦ Good at capturing the global structure, but samples are blurry. ✦ Learn hierarchical representations useful for down-stream tasks. ✦ Attempts at combining PixelCNN with VAEs: • PixelVAE ( Gulrajani et al., 2016 ) • VLAE ( Chen et al., 2017 ) Kingma et al., 2013 Alireza Makhzani PixelGAN Autoencoders 4 4 / 27

Adversarial Autoencoders B D Code Space of MNIST: Gaussian Prior Mixture of Gaussians Makhzani et al., 2015 Alireza Makhzani PixelGAN Autoencoders 5 / 27

Outline 1. Background • PixelCNNs • Variational Autoencoders • Adversarial Autoencoders 2. PixelGAN Autoencoders • Gaussian Priors • Categorical Priors ✦ Clustering ✦ Semi-supervised Learning Alireza Makhzani PixelGAN Autoencoders 6 / 27

Limitations of Variational/Adversarial Autoencoders ✦ All the image statistics are captured by the single latent vector. VAE label, style p ( z ) Latent Variable z global and local Deterministic p ( x | z ) None (factorized Gaussians) x Alireza Makhzani PixelGAN Autoencoders 7 / 27

PixelGAN Autoencoders ✦ The image statistics are captured jointly by the latent vector and the autoregressive decoder. p ( z ) Latent Variable z p ( x | z ) PixelCNN x Alireza Makhzani PixelGAN Autoencoders 8 / 27

PixelGAN Autoencoders ✦ The image statistics are captured jointly by the latent vector and the autoregressive decoder. p ( z ) Latent Variable z GAN p ( x | z ) PixelCNN x Alireza Makhzani PixelGAN Autoencoders 9 / 27

PixelGAN Autoencoders ✦ The image statistics are captured jointly by the latent vector and the autoregressive decoder. PixelGAN PixelGAN (Gaussian) (Categorical) Global Discrete p ( z ) Latent Variable z (low-frequency) (label) GAN Local Continuous p ( x | z ) PixelCNN (high-frequency) (Style) x Alireza Makhzani PixelGAN Autoencoders 10 / 27

PixelGAN Autoencoders ✦ The image statistics are captured jointly by the latent vector and the autoregressive decoder. PixelGAN PixelGAN (Gaussian) (Categorical) Global Discrete p ( z ) Latent Variable z (low-frequency) (label) GAN Local Continuous p ( x | z ) PixelCNN (high-frequency) (Style) x Semi-supervised Learning Alireza Makhzani PixelGAN Autoencoders 11 / 27

PixelGAN Autoencoders Cost function of PixelGAN = Reconstruction + Adversarial Cost Alireza Makhzani PixelGAN Autoencoders 12 / 27

Outline 1. Background • PixelCNNs • Variational Autoencoders • Adversarial Autoencoders 2. PixelGAN Autoencoders • Gaussian Priors • Categorical Priors Clustering Semi-supervised Learning Alireza Makhzani PixelGAN Autoencoders 13 / 27

Global vs. Local Decomposition Alireza Makhzani PixelGAN Autoencoders 14 / 27

Code Space Code Space of MNIST: Alireza Makhzani PixelGAN Autoencoders 15 / 27

Outline 1. Background • PixelCNNs • Variational Autoencoders • Adversarial Autoencoders 2. PixelGAN Autoencoders • Gaussian Priors • Categorical Priors ✦ Clustering ✦ Semi-supervised Learning Alireza Makhzani PixelGAN Autoencoders 16 / 27

PixelGAN Autoencoders with Categorical Priors Alireza Makhzani PixelGAN Autoencoders 17 / 27

Discrete vs. Continuous Decomposition (Clustering) Alireza Makhzani PixelGAN Autoencoders 18 / 27

Discrete vs. Continuous Decomposition (Clustering) 0.3% Error rate Alireza Makhzani PixelGAN Autoencoders 19 / 27

Unsupervised Clustering Alireza Makhzani PixelGAN Autoencoders 20 / 27

Unsupervised Clustering Alireza Makhzani PixelGAN Autoencoders 21 / 27

Unsupervised Clustering 5% Error rate Alireza Makhzani PixelGAN Autoencoders 22 / 27

Outline 1. Background • PixelCNNs • Variational Autoencoders • Adversarial Autoencoders 2. PixelGAN Autoencoders • Gaussian Priors • Categorical Priors ✦ Clustering ✦ Semi-supervised Learning Alireza Makhzani PixelGAN Autoencoders 23 / 27

Semi-supervised Learning Semi-supervised Learning Alireza Makhzani PixelGAN Autoencoders 24 / 27

Semi-supervised Learning Semi-supervised Learning Alireza Makhzani PixelGAN Autoencoders 25 / 27

Semi-supervised Classification Alireza Makhzani PixelGAN Autoencoders 26 / 27

Thank you! Alireza Makhzani PixelGAN Autoencoders 27 / 27