Conditio ditional al Generati ative Adversa sarial al - - PowerPoint PPT Presentation

▶
conditio ditional al generati ative adversa sarial al
SMART_READER_LITE
LIVE PREVIEW

Conditio ditional al Generati ative Adversa sarial al - - PowerPoint PPT Presentation

Dec 09, 2023 245 likes •709 views

Conditio ditional al Generati ative Adversa sarial al Networks works (cGANs Ns) conti tinued! nued! Prof. Leal-Taix and Prof. Niessner 1 Paired vs Unpair ired Settin ing Prof. Leal-Taix and Prof. Niessner 2 pix2pix ix: :

slide-1
SLIDE 1

Conditio ditional al Generati ative Adversa sarial al Networks works (cGANs Ns) conti tinued! nued!

  • Prof. Leal-Taixé and Prof. Niessner

1

slide-2
SLIDE 2

Paired vs Unpair ired Settin ing

  • Prof. Leal-Taixé and Prof. Niessner

2

slide-3
SLIDE 3

pix2pix ix: : Image-to to-Ima mage Transla latio ion

slides credit: Isola / Zhu

slide-4
SLIDE 4

real or fake?

[Goodfellow et al. 2014] Discriminator

z G(z)

D

Generator

G

slides credit: Isola / Zhu

min

𝐻 max 𝐸

𝔽𝑨,𝑦 log 𝐸(𝐻 𝑨 ) + log(1 − 𝐸 𝑦 )

slide-5
SLIDE 5

real or fake?

Discriminator

x G(x)

D

Generator

G min

𝐻 max 𝐸

𝔽𝑦,𝑧 log 𝐸(𝐻 𝑦 ) + log(1 − 𝐸 𝑧 )

slides credit: Isola / Zhu

slide-6
SLIDE 6

Real!

Discriminator

x G(x)

D

Generator

G min

𝐻 max 𝐸

𝔽𝑦,𝑧 log 𝐸(𝐻 𝑦 ) + log(1 − 𝐸 𝑧 )

slides credit: Isola / Zhu

slide-7
SLIDE 7

Discriminator

x G(x)

D

Generator

G

Real too!

min

𝐻 max 𝐸

𝔽𝑦,𝑧 log 𝐸(𝐻 𝑦 ) + log(1 − 𝐸 𝑧 )

slides credit: Isola / Zhu

slide-8
SLIDE 8

min

𝐻 max 𝐸

𝔽𝑦,𝑧 log 𝐸(𝑦, 𝐻 𝑦 ) + log(1 − 𝐸 𝑦, 𝑧 )

real or fake pair ? x G(x)

G D

match joint distribution p G x , y ∼ p(x, y)

fake pair real pair

slides credit: Isola / Zhu

slide-9
SLIDE 9

pix2pix ix

9

slide-10
SLIDE 10

pix2pix: ix: Paired Settin ing

  • Great when we have ‘free’ training data
  • Often called self-supervised
  • Think about these settings 

10

slide-11
SLIDE 11

Edges → Images

Input Output Input Output Input Output

Edges from [Xie & Tu, 2015]

slides credit: Isola / Zhu

slide-12
SLIDE 12

Sketches → Images

Input Output Input Output Input Output

Trained on Edges → Images

Data from [Eitz, Hays, Alexa, 2012]

slides credit: Isola / Zhu

slide-13
SLIDE 13

#edges2cats

[Christopher Hesse] Ivy Tasi @ivymyt @gods_tail @matthematician

https://affinelayer.com/pixsrv/

Vitaly Vidmirov @vvid

slides credit: Isola / Zhu

slide-14
SLIDE 14

Input Output Groundtruth Data from [maps.google.com]

slides credit: Isola / Zhu

slide-15
SLIDE 15

BW → Color

Input Output Input Output Input Output

Data from [Russakovsky et al. 2015]

slides credit: Isola / Zhu

slide-16
SLIDE 16

Ideas behind Pix2Pi Pix

  • 𝑀 = 𝑀𝐻𝐵𝑂 + 𝜇𝑀1 (makes it more constrained)
  • Unet / skip connections for preserving structure
  • Noise only through dropout

– cGANs tend to learn to ignore the random vector z – Still want probabilistic model

  • Prof. Leal-Taixé and Prof. Niessner

16

slide-17
SLIDE 17

Ideas behind Pix2Pi Pix

  • L1 or L2 loss for low frequency details
  • GAN discriminator for high frequency details
  • > PatchGAN

– GAN discriminator applied only to local patches – It’s fully-convolutional; i.e., can run on arbitrary image sizes

  • Prof. Leal-Taixé and Prof. Niessner

17

slide-18
SLIDE 18

Pix2Pi PixHD

  • Expand the pix2pix idea to multi-scale
  • Coarse-to-fine generator + discriminator
  • G’s and D’s are the same but since they operate
  • n different resolutions, they have effectively a larger

receptive field

  • Prof. Leal-Taixé and Prof. Niessner

18

[Wang et al. 18]

slide-19
SLIDE 19

Pix2Pi PixHD

  • Prof. Leal-Taixé and Prof. Niessner

19

[Wang et al. 18]

slide-20
SLIDE 20

Pix2Pi PixHD

  • Use of multi-scale discriminators
  • min

𝐻

max

𝐸1,𝐸2,𝐸3

𝑙=1,2,3𝑀𝐻𝐵𝑂 (𝐻, 𝐸𝑙)

  • Can make various combinations of stacking

discriminator and generator

– E.g., have a single G and downsample generated and real images – or have intermediate real images (cf. ProGAN)

  • Prof. Leal-Taixé and Prof. Niessner

20

[Wang et al. 18]

slide-21
SLIDE 21

Pix2Pi PixHD

  • Prof. Leal-Taixé and Prof. Niessner

21

[Wang et al. 18]

slide-22
SLIDE 22

Pix2Pi PixHD

  • Prof. Leal-Taixé and Prof. Niessner

22

[Wang et al. 18]

slide-23
SLIDE 23

Pix2Pi PixH xHD D (interactive ive result lts)

  • Prof. Leal-Taixé and Prof. Niessner

23

[Wang et al. 18]

slide-24
SLIDE 24
  • Expensive to collect pairs.
  • Impossible in many

scenarios. Label ↔ photo: per-pixel labeling

…

Paired

Horse ↔ zebra: how to get zebras?

slides credit: Isola / Zhu

slide-25
SLIDE 25

… … …

Paired Unpaired

slides credit: Isola / Zhu

slide-26
SLIDE 26

x G(x)

Generator

G D

No input-output pairs!

slides credit: Isola / Zhu

slide-27
SLIDE 27

Discriminator

x G(x)

D

Generator

G

Real!

slides credit: Isola / Zhu

slide-28
SLIDE 28

Discriminator

x G(x)

D

Generator

G

Real too!

GANs doesn’t force output to correspond to input

slides credit: Isola / Zhu

slide-29
SLIDE 29

mode collapse!

slides credit: Isola / Zhu

slide-30
SLIDE 30

⋯

Cycle-Consistent Adversarial Networks

⋯

[Zhu*, Park*, Isola, and Efros, ICCV 2017]

slides credit: Isola / Zhu

slide-31
SLIDE 31

Cycle-Consistent Adversarial Networks

[Mark Twain, 1903]

⋯ ⋯

[Zhu*, Park*, Isola, and Efros, ICCV 2017]

slides credit: Isola / Zhu

slide-32
SLIDE 32

Cycle Consistency Loss

G(x) F(G x ) x

F G x − x 1

[Zhu*, Park*, Isola, and Efros, ICCV 2017]

DY(G x )

Reconstruction error

slides credit: Isola / Zhu

slide-33
SLIDE 33

Cycle Consistency Loss

G(x) F(G x ) x

F G x − x 1

Large cycle loss Small cycle loss

[Zhu*, Park*, Isola, and Efros, ICCV 2017]

DY(G x )

Reconstruction error

slides credit: Isola / Zhu

slide-34
SLIDE 34

G(x) F(G x ) x F(y) G(F x ) 𝑧

Cycle Consistency Loss

F G x − x 1 G F y − 𝑧 1

[Zhu*, Park*, Isola, and Efros, ICCV 2017]

DY(G x )

Reconstruction error Reconstruction error

DG(F x )

slides credit: Isola / Zhu

slide-35
SLIDE 35

Cycle GAN - Overvie view

https://junyanz.github.io/CycleGAN/ [Zhu et al. 17.]

  • Prof. Leal-Taixé and Prof. Niessner

35

slide-36
SLIDE 36

Cycle GAN: Objective ive

  • Prof. Leal-Taixé and Prof. Niessner

36

Domain X Domain Y Cycle consistency Full Loss:

https://junyanz.github.io/CycleGAN/ [Zhu et al. 17.]

slide-37
SLIDE 37

Monet’s paintin ings → photos

slides credit: Isola / Zhu

https://junyanz.github.io/CycleGAN/ [Zhu et al. 17.]

slide-38
SLIDE 38

slides credit: Isola / Zhu

https://junyanz.github.io/CycleGAN/ [Zhu et al. 17.]

slide-39
SLIDE 39

slides credit: Isola / Zhu

https://junyanz.github.io/CycleGAN/ [Zhu et al. 17.]

slide-40
SLIDE 40

Adminis istrative ive

  • Prof. Leal-Taixé and Prof. Niessner

40

slide-41
SLIDE 41

Adminis istrative ive

  • Deadline for final projects

– Wed Feb 6th

th, 11:59pm

– Submission via moodle – Submission must contain

  • Code (results must be replicable)
  • 2-3 pages of final report (at most 1 page of text, rest results;

i.e., images and tables)

  • Use CVPR templates:

http://cvpr2019.thecvf.com/submission/main_conference/au thor_guidelines

  • Prof. Leal-Taixé and Prof. Niessner

41

slide-42
SLIDE 42

Adminis istrative ive

  • Poster presentation

– Friday Feb 8th

th, 1pm-3pm

– Location:

  • Magistrale (preliminary – will update if it changes)
  • In the area next to the back entrance (parking lot direction)

– Poster stands will be provided – You need to print posters yourself (poster@in.tum.de) – Hang posters 15 mins before presentation session starts

  • Prof. Leal-Taixé and Prof. Niessner

42

slide-43
SLIDE 43

Guest Speakers

  • Oriol Vinyals:

– https://ai.google/research/people/OriolVinyals – Time: Ja January 31st

st, 6pm – 8pm

– Location: HS-1 (CS building – the big one)

  • Prof. Leal-Taixé and Prof. Niessner

43

slide-44
SLIDE 44

Next Lectures

  • Next Lecture -> Jan 21st
  • Keep working on the projects!
  • Prof. Leal-Taixé and Prof. Niessner

44

slide-45
SLIDE 45

Conditio ditional al Generati ative Adversa sarial al Networks works (cGANs Ns) conti tinued! nued!

  • Prof. Leal-Taixé and Prof. Niessner

45