Addressing GAN limitations: resolution, lack of novelty and control - PowerPoint PPT Presentation

1 2019 Addressing GAN limitations: resolution, lack of novelty and control on generations Camille Couprie Facebook AI Research Joint works with O. Sbai, M. Aubry, A, Bordes, M. Elhoseiny, M. Riviere, Y. LeCun, M. Mathieu, P. Luc, N. Neverova, J. Verbeek.

2 Why do we care about generative models • Scene Understanding can be assessed by checking the ability to generate plausible new scenes. • Generative models are interesting if they can be used to go beyond training data: data of higher resolution, data augmentation to help train better classifiers, use the learned representations in other tasks, or make prediction about uncertain events...

3 Outline • 1/ Design inspiration from adversarial generative networks • 2/ High resolution, decoupled generation • 3/ Vector image generation by learning parametric layer decomposition • 4/ Future frame prediction

4 Design inspiration from generative networks Sbai, Elhoseiny, Bordes, LeCun, Couprie, ECCV workshop 17 Hedonic Value 0 1 2 3 4 Novelty

Generative Adversarial Networks Goodfellow et al, 2014 0 . 3 0 . 7 0 . 1 0 . 8 Fake GENERATED IMAGE AdVERSARIAL RANDOM NUMBERS Generator NETWORK

Generative Adversarial Networks Goodfellow et al, 2014 Real INPUT 0 . 3 0 . 7 0 . 1 0 . 8 Real GENERATED IMAGE AdVERSARIAL RANDOM NUMBERS Generator NETWORK

Deep convolutional GANs RADFORD ET AL : ICLR 2015

Training with pictures of about 2000 Clothing items

Texture and shape labels Floral Tiled Uniform Dotted Animal Print Graphical Striped Skirt Pullover T-Shirt Coat Top Jacket Dress

Class conditioned GAN Real INPUT S h a p e C L A S S 0 . 3 0 . 7 0 / 1 0 . 1 ( R E A L / FA K E ) 0 . 8 GENERATED IMAGE T E X T U R E C L A S S AdVERSARIAL RANDOM NUMBERS Generator NETWORK

GAN Optimization objectives • Generator’s loss • Discriminator’s loss • Auxiliary classifier discriminator: • Additional loss for the generator:

Without conditioning With class conditioning

Introduction of a Style Deviation criterion Dotted Floral graphical 0 . 3 1 0 0 % 0 . 7 0 . 1 uniform 0 . 8 tiled RANDOM Generated AdVERSARIAL Generator NUMBERS Image NETWORK striped Animal print

Introduction of a Style Deviation criterion Dotted 6 % Floral 7 % 5 7 % graphical 0 . 3 0 . 7 0 . 1 0 . 8 6 % uniform 8 % RANDOM Generated AdVERSARIAL tiled Generator NUMBERS Image NETWORK 1 1 % striped 9 % Animal print

With the Style Deviation criterion (CAN H) D o t t e d 0 . 3 F l o r a l 0 . 7 g r a p h i c a l 0 . 1 0 . 8 u n i f o r m t i l e d s t r i p e d A n i m a l p r i n t RANDOM AdVERSARIAL Generator Generated Image NUMBERS NETWORK

Tested deviation objectives Binary cross entropy loss : Multi-class cross entropy loss:

Human Evaluation Study 7 0 6 4 . 5 CAN(h) CAN r e a l i s t i c GAN texTURE texTURE A p p e a r a n c e 5 9 Style 5 3 . 5 CAN(h) texTURE 4 8 8 6 5 7 0 7 5 6 0 0 O v e r a l L L i k a b i l i t y ( % ) CAN: GAN with Creativity loss, (H) stands for the use of a holistic loss.

Models with texture deviation are Most Popular Can (H) 7 Can (H) Shape 8 texture style 7 4 can (h) L i k e a b i l i t y 7 0 Can texture 6 6 GAN 6 2 N o v e l t y 0 1 judged by humans and measured as a distance to similar training images

2 0 Decoupled adversarial image generation M. Riviere, C. Couprie, Y. LeCun Motivation: - Take advantage of white background clothing datasets - Potentially avoid defaults in generated shapes - Better enforce shape conditioning of generations

2 1 1024x1024 generations on the RTW dataset Using Morgane’s pytorch “progressive growing of GANs” available online, Karras et al., ICLR’18

2 2 Decoupled architecture Real image Real image Texture, color, shape and Discriminator Discriminator pose classes D s D g Real / Fake Real / Fake Shape and Shape × pose classes G s Generator Generation z Texture Texture, color, shape and pose classes G t Generator

2 3 Random generations Progressive growing with decoupled Progressive growing architecture

2 4 Better class conditioning = +12% +14% -12% Accuracy of classifiers trained on FashionGen Clothing and FashionGen Shoes on our different models results (GAN-test metric) Overall average improvement: 4.7%

2 5 Vector Image Generation by learning parametric layer decomposition Sbai, Couprie, Aubry, arxiv dec18 Current deep generative models are great but… … are limited in resolution, and control in generations

2 6 Related work Kanan et al: Layered GANs (LR-GANs), ICLR’17 GANIN et al. SPIRAL, ICML’18

2 7 Our approach Spoiler alert: yes, we can generate sharper images, this is just an example.