Conditio ditional al Generati ative Adversa sarial al - PowerPoint PPT Presentation

Conditio ditional al Generati ative Adversa sarial al Networks works (cGANs Ns) conti tinued! nued! Prof. Leal-Taixé and Prof. Niessner 1

Paired vs Unpair ired Settin ing Prof. Leal-Taixé and Prof. Niessner 2

pix2pix ix: : Image-to to-Ima mage Transla latio ion slides credit: Isola / Zhu

z G(z) G D real or fake? Generator Discriminator min 𝐻 max 𝔽 𝑨,𝑦 log 𝐸(𝐻 𝑨 ) + log(1 − 𝐸 𝑦 ) 𝐸 [Goodfellow et al. 2014] slides credit: Isola / Zhu

x G(x) G D real or fake? Generator Discriminator min 𝐻 max 𝔽 𝑦,𝑧 log 𝐸(𝐻 𝑦 ) + log(1 − 𝐸 𝑧 ) 𝐸 slides credit: Isola / Zhu

x G(x) Real! G D Generator Discriminator min 𝐻 max 𝔽 𝑦,𝑧 log 𝐸(𝐻 𝑦 ) + log(1 − 𝐸 𝑧 ) 𝐸 slides credit: Isola / Zhu

x G(x) Real too! G D Generator Discriminator min 𝐻 max 𝔽 𝑦,𝑧 log 𝐸(𝐻 𝑦 ) + log(1 − 𝐸 𝑧 ) 𝐸 slides credit: Isola / Zhu

x G(x) G D real or fake pair ? min 𝐻 max 𝔽 𝑦,𝑧 log 𝐸(𝑦, 𝐻 𝑦 ) + log(1 − 𝐸 𝑦, 𝑧 ) 𝐸 fake pair real pair match joint distribution p G x , y ∼ p(x, y) slides credit: Isola / Zhu

pix2pix ix 9

pix2pix: ix: Paired Settin ing • Great when we have ‘free’ training data • Often called self-supervised • Think about these settings  10

Edges → Images Input Output Input Output Input Output Edges from [Xie & Tu, 2015] slides credit: Isola / Zhu

Sketches → Images Input Output Input Output Input Output Trained on Edges → Images Data from [Eitz, Hays, Alexa, 2012] slides credit: Isola / Zhu

#edges2cats [Christopher Hesse] @gods_tail @matthematician Vitaly Vidmirov @vvid Ivy Tasi @ivymyt https://affinelayer.com/pixsrv/ slides credit: Isola / Zhu

Input Output Groundtruth Data from [maps.google.com] slides credit: Isola / Zhu

BW → Color Input Output Input Output Input Output Data from [Russakovsky et al. 2015] slides credit: Isola / Zhu

Ideas behind Pix2Pi Pix • 𝑀 = 𝑀 𝐻𝐵𝑂 + 𝜇𝑀 1 (makes it more constrained) • Unet / skip connections for preserving structure • Noise only through dropout – cGANs tend to learn to ignore the random vector z – Still want probabilistic model Prof. Leal-Taixé and Prof. Niessner 16

Ideas behind Pix2Pi Pix • L1 or L2 loss for low frequency details • GAN discriminator for high frequency details -> PatchGAN – GAN discriminator applied only to local patches – It’s fully -convolutional; i.e., can run on arbitrary image sizes Prof. Leal-Taixé and Prof. Niessner 17

Pix2Pi PixHD • Expand the pix2pix idea to multi-scale • Coarse-to-fine generator + discriminator • G’s and D’s are the same but since they operate on different resolutions, they have effectively a larger receptive field Prof. Leal-Taixé and Prof. Niessner 18 [Wang et al. 18]

Pix2Pi PixHD Prof. Leal-Taixé and Prof. Niessner 19 [Wang et al. 18]

Pix2Pi PixHD • Use of multi-scale discriminators • min 𝑙=1,2,3 𝑀 𝐻𝐵𝑂 (𝐻, 𝐸 𝑙 ) max 𝐻 𝐸 1 ,𝐸 2 ,𝐸 3 • Can make various combinations of stacking discriminator and generator – E.g., have a single G and downsample generated and real images – or have intermediate real images (cf. ProGAN) Prof. Leal-Taixé and Prof. Niessner 20 [Wang et al. 18]

Pix2Pi PixHD Prof. Leal-Taixé and Prof. Niessner 21 [Wang et al. 18]

Pix2Pi PixHD Prof. Leal-Taixé and Prof. Niessner 22 [Wang et al. 18]

Pix2Pi PixH xHD D (interactive ive result lts) Prof. Leal-Taixé and Prof. Niessner 23 [Wang et al. 18]

Paired Label ↔ photo: per-pixel labeling Horse ↔ zebra: how to get zebras? … - Expensive to collect pairs. - Impossible in many scenarios. slides credit: Isola / Zhu

Paired Unpaired … … … slides credit: Isola / Zhu

x G(x) G D Generator No input-output pairs! slides credit: Isola / Zhu

x G(x) Real! G D Generator Discriminator slides credit: Isola / Zhu

x G(x) Real too! G D Generator Discriminator GANs doesn’t force output to correspond to input slides credit: Isola / Zhu

mode collapse! slides credit: Isola / Zhu

Cycle-Consistent Adversarial Networks ⋯ ⋯ [Zhu*, Park*, Isola, and Efros, ICCV 2017] slides credit: Isola / Zhu

Cycle-Consistent Adversarial Networks ⋯ ⋯ [Mark Twain, 1903] [Zhu*, Park*, Isola, and Efros, ICCV 2017] slides credit: Isola / Zhu

Cycle Consistency Loss x G(x) F(G x ) D Y (G x ) Reconstruction error F G x − x 1 [Zhu*, Park*, Isola, and Efros, ICCV 2017] slides credit: Isola / Zhu

Cycle Consistency Loss Large cycle loss Small cycle loss x G(x) F(G x ) D Y (G x ) Reconstruction error F G x − x 1 [Zhu*, Park*, Isola, and Efros, ICCV 2017] slides credit: Isola / Zhu

Cycle Consistency Loss x G(x) F(G x ) 𝑧 F(y) G(F x ) D Y (G x ) D G (F x ) Reconstruction Reconstruction error error F G x − x 1 G F y − 𝑧 1 [Zhu*, Park*, Isola, and Efros, ICCV 2017] slides credit: Isola / Zhu

Cycle GAN - Overvie view Prof. Leal-Taixé and Prof. Niessner 35 https://junyanz.github.io/CycleGAN/ [Zhu et al. 17.]

Cycle GAN: Objective ive Domain X Domain Y Cycle consistency Full Loss: Prof. Leal-Taixé and Prof. Niessner 36 https://junyanz.github.io/CycleGAN/ [Zhu et al. 17.]

Monet’s paintin ings → photos https://junyanz.github.io/CycleGAN/ [Zhu et al. 17.] slides credit: Isola / Zhu

https://junyanz.github.io/CycleGAN/ [Zhu et al. 17.] slides credit: Isola / Zhu

https://junyanz.github.io/CycleGAN/ [Zhu et al. 17.] slides credit: Isola / Zhu

Adminis istrative ive Prof. Leal-Taixé and Prof. Niessner 40

Adminis istrative ive • Deadline for final projects – Wed Feb 6 th th , 11:59pm – Submission via moodle – Submission must contain • Code (results must be replicable) • 2-3 pages of final report (at most 1 page of text, rest results; i.e., images and tables) • Use CVPR templates: http://cvpr2019.thecvf.com/submission/main_conference/au thor_guidelines Prof. Leal-Taixé and Prof. Niessner 41

Adminis istrative ive • Poster presentation – Friday Feb 8 th th , 1pm-3pm – Location: • Magistrale (preliminary – will update if it changes) • In the area next to the back entrance (parking lot direction) – Poster stands will be provided – You need to print posters yourself (poster@in.tum.de) – Hang posters 15 mins before presentation session starts Prof. Leal-Taixé and Prof. Niessner 42

Guest Speakers • Oriol Vinyals: – https://ai.google/research/people/OriolVinyals – Time: Ja January 31 st st , 6pm – 8pm – Location: HS-1 (CS building – the big one) Prof. Leal-Taixé and Prof. Niessner 43

Next Lectures • Next Lecture -> Jan 21 st • Keep working on the projects! Prof. Leal-Taixé and Prof. Niessner 44

Conditio ditional al Generati ative Adversa sarial al Networks works (cGANs Ns) conti tinued! nued! Prof. Leal-Taixé and Prof. Niessner 45