Addressing GAN limitations: resolution, lack of novelty and control - - PowerPoint PPT Presentation

Addressing GAN limitations: resolution, lack of novelty and control on generations

Camille Couprie Facebook AI Research Joint works with O. Sbai, M. Aubry, A, Bordes, M. Elhoseiny, M. Riviere, Y. LeCun, M. Mathieu, P. Luc, N. Neverova, J. Verbeek.

2019

1

Why do we care about generative models

2

• Scene Understanding can be assessed by checking the ability to

generate plausible new scenes.

• Generative models are interesting if they can be used to go beyond

training data: data of higher resolution, data augmentation to help train better classifiers, use the learned representations in other tasks, or make prediction about uncertain events...

• 1/ Design inspiration from adversarial generative networks
• 2/ High resolution, decoupled generation
• 3/ Vector image generation by learning parametric layer

decomposition

• 4/ Future frame prediction

Outline

3

4

Sbai, Elhoseiny, Bordes, LeCun, Couprie, ECCV workshop 17

Design inspiration from generative networks

Novelty Hedonic Value 1 2 3 4

RANDOM NUMBERS 0 . 3 0 . 7 0 . 1 0 . 8 AdVERSARIAL NETWORK Generator Fake

GENERATED IMAGE

Generative Adversarial Networks

Goodfellow et al, 2014

RANDOM NUMBERS

Real INPUT

Generator Real AdVERSARIAL NETWORK

GENERATED IMAGE

0 . 3 0 . 7 0 . 1 0 . 8

Generative Adversarial Networks

Goodfellow et al, 2014

Deep convolutional GANs

RADFORD ET AL : ICLR 2015

Training with pictures of about 2000 Clothing items

Floral Striped Tiled Uniform Dotted Animal Print Graphical

Texture and shape labels

Dress Skirt Jacket Pullover T-Shirt Coat Top

RANDOM NUMBERS

Real INPUT

Generator AdVERSARIAL NETWORK

GENERATED IMAGE

0 . 3 0 . 7 0 . 1 0 . 8

S h a p e C L A S S 0 / 1 ( R E A L / FA K E ) T E X T U R E C L A S S

Class conditioned GAN

• Generator’s

loss

• Discriminator’s

loss

• Auxiliary

classifier discriminator:

• Additional loss

for the generator:

GAN Optimization objectives

Without conditioning With class conditioning

RANDOM NUMBERS Generator

0 . 3 0 . 7 0 . 1 0 . 8

AdVERSARIAL NETWORK Generated Image

Dotted Floral graphical uniform tiled striped Animal print

1 0 0 %

Introduction of a Style Deviation criterion

8 % 6 % 7 % 5 7 % 6 % 1 1 % 9 %

RANDOM NUMBERS Generator

0 . 3 0 . 7 0 . 1 0 . 8

AdVERSARIAL NETWORK Generated Image

Dotted Floral graphical uniform tiled striped Animal print

Introduction of a Style Deviation criterion

RANDOM NUMBERS Generator

0 . 3 0 . 7 0 . 1 0 . 8

AdVERSARIAL NETWORK Generated Image

D o t t e d F l o r a l g r a p h i c a l u n i f o r m t i l e d s t r i p e d A n i m a l p r i n t

With the Style Deviation criterion (CAN H)

Multi-class cross entropy loss: Binary cross entropy loss :

Tested deviation objectives

O v e r a l L L i k a b i l i t y ( % )

6 5 7 0 7 5 6 0 8

CAN: GAN with Creativity loss, (H) stands for the use of a holistic loss.

CAN texTURE GAN Style CAN(h) texTURE CAN(h) texTURE

r e a l i s t i c A p p e a r a n c e

4 8 5 3 . 5 5 9 6 4 . 5 7

Human Evaluation Study

Models with texture deviation are Most Popular

N o v e l t y

1

L i k e a b i l i t y

6 2 6 6 7 0 7 4 7 8

judged by humans and measured as a distance to similar training images

Can texture Can (H) Shape Can (H) texture style can (h) GAN

2

• M. Riviere, C. Couprie, Y. LeCun

Decoupled adversarial image generation

Motivation:

• Take advantage of white background clothing datasets
• Potentially avoid defaults in generated shapes
• Better enforce shape conditioning of generations

1024x1024 generations on the RTW dataset

2 1

Using Morgane’s pytorch “progressive growing of GANs” available online, Karras et al., ICLR’18

Decoupled architecture

2 2

Shape Generator Texture Generator Generation Real image

Shape and pose classes Texture, color, shape and pose classes

Discriminator Real / Fake Real image Discriminator Real / Fake

Texture, color, shape and pose classes

×

Dg

Ds

Gs

Gt

z

Random generations

2 3

Progressive growing Progressive growing with decoupled architecture

Better class conditioning

2 4

Accuracy of classifiers trained on FashionGen Clothing and FashionGen Shoes on our different models results (GAN-test metric) = +12% +14%

• 12%

Overall average improvement: 4.7%

2 5

Sbai, Couprie, Aubry, arxiv dec18

Vector Image Generation

by learning parametric layer decomposition

Current deep generative models are great but… … are limited in resolution, and control in generations

Kanan et al: Layered GANs (LR-GANs), ICLR’17 GANIN et al. SPIRAL, ICML’18

Related work

2 6

Our approach

2 7

Spoiler alert: yes, we can generate sharper images, this is just an example.

Iterative generation : !" = $(!, !"'() Vectorized mask generation *"(+, ,) = $(+, ,, -") Alpha-blending !"(+, ,) = !"'( +, , . (1-*"(+, ,))+ /"*"(+, ,)

Our iterative pipeline for image reconstruction

2 8

x x

×

×

+

ct

ctMt

2 9

Our iterative pipeline for image generation

3

Training criteria

Adversarial net criterion: Wasserstein loss with Gradient Penalty (WGAN-GP), Gulrajani et al. NIPS’17 Our generator loss in the GAN setting: Our generator loss in the image reconstruction setting:

Results using a l1 reconstruction loss

3 1

Results using a l1 reconstruction loss

3 2

Target Our iterative reconstruction

Editing the original image using extracted masks, by performing local modifications of luminosity (top), or color modification using a blending

• f masks (bottom).

Applications

3 3

Image editing using masks: Using chosen extracted mask(s) from image reconstruction, we apply object removal (top) and color modifications with

• bject removal (bottom).

Applications

3 4

Image vectorization: Reconstruction results on MNIST images. Our model learns a vectorized mask representation of digits that can be generated at any resolution without interpolation artifacts.

Applications

3 5

RGB image

Baselines

3 6

RGB image z 1/ MLP-baseline MLP Resnet

RGB image

Baselines

3 8

RGB image z 1/ MLP-baseline 2/ ResNet baseline Resnet Resnet

RGB image

Baselines

3 9

RGB image z 1/ MLP-baseline 2/ ResNet baseline 3/ MLP-xy baseline MLP (x,y) Resnet

Comparative results

4

• Using a l1 reconstruction loss
• Parameters for our approach:

20 masks, p of size 10, c of size 3: 260 parameters

• Baselines: size of the latent

code z = 20x13=260

CelebA generations trained on 64x64 images, sampled at 256x256

GAN results

4 1

CIFAR10 generations trained on 32x32 images, sampled at 256x256

GAN Results on ImageNet

4 2

trained on 64x64 images, sampled at 1024x1024

b

Results

4 4

Result with perceptual loss Perceptual loss Target

Conclusion and Future work

4 5

Faster training Use class conditioning Texture image generation

Predicting next frames in videos

Michael Mathieu, Camille Couprie, Yann LeCun, ICLR16

4 input images Our 2 predictions

Deep multiscale video prediction beyond Mean square error

• Result with a simple convolutional network

trained minimizing an l2 loss

• Our result using
• A multiscale architecture
• an image gradient different loss
• Use adversarial training

4 8

• P. Luc, N. Neverova, C. Couprie, J. Verbeek, Y. LeCun ECCV18

Predicting deeper into the future of semantic segmentations

4 input images Our 2 predictions

• Predictions in the RGB space quickly become blurry

despites previous attempts

• Idea: predict in the space of semantic segmentation

Approach – predicting deeper into the future

Single time-step

Autoregressive model

Batch model

St−3 St−2 St−1 St St+1 Lt+1 St−3 St−2 St−1 St St+1 St+2 St+3 Lt+1 Lt+2 Lt+3 St−3 St−2 St−1 St St+1 St+2 St+3 Lt+1 Lt+2 Lt+3

Autoregressive model is either :

• used for inference without

additional training (w.r.t. to single time step model) AR

• Fine-tuned using BPTT

AR fine-tune Autoregressive mode is only possible for X2X, S2S, XS2XS

Same color = shared weights

4 9

Some results

5

Baselines :

• Copy the last input frame to the
• utput
• Estimate flow between the two last

inputs, and project the last input forward using the flow

Performance measure (mean IoU) of our approach and baselines on the Cityscapes dataset

Flow baseline Our AutoRegressive fine-tune result

Instance level segmentation: Mask RCNN

• Extends Faster RCNN [Ren et

al.’15] by adding a branch for predicting an object mask in parallel with the existing branch for bounding box recognition

• K. He G. Gkioxari P. Dollar R. Girshick’17

1

5 2

• P. Luc, C. Couprie, Y. LeCun, J. Verbeek, ECCV18

Predicting future segmentation instances by forecasting convolutional features

Luc, Neverova et al. ICCV17 F2F predictions

Conclusions

5 3

Some open problems:

• Automatic metrics to evaluate generative models performances
• Non deterministic training losses for future prediction

Torch code online :

• For future video prediction:
• Vector image generation: available soon on Othman Sbai’s github
• f RGBs : on Michael Mathieu’s github
• f semantic segmentations : on Pauline’s Luc github
• f instance segmentation : on Pauline’s Luc github