Condit itio ional Generativ ive Adversaria ial Networks - - PowerPoint PPT Presentation
Condit itio ional Generativ ive Adversaria ial Networks (cGANs) Prof. Leal-Taix and Prof. Niessner 1 Conditional GANs (cGANs) Gain control of output Modeling (e.g., sketch-based modeling, etc.) Add semantic meaning to latent
1
– Add semantic meaning to latent space manifold
– Labels on A -> transfer to B, train network on ‘B’, test on B – More later
2
3
[Radford et al. 15]
Train Data Sampled Data -> G(z)
4
[Bojanowski et al 17]
5
6
[Radford et al. 15]
𝐻(𝑨0) 𝐻(𝑨1) Linear interpolation in z space: 𝐻(𝑨0 + 𝑢 ⋅ 𝑨1 − 𝑨0 )
7
Slide credit Zhu
projection on manifold Project Edit Transfer transition between the original and edited projection different degree of image manipulation
Editing UI
Slide credit Zhu / [Zhu et al. 16]
projection on manifold Project Edit Transfer transition between the original and edited projection different degree of image manipulation
Editing UI
Slide credit Zhu / [Zhu et al. 16]
0.196 0.238 0.332
Optimization
Input: real image 𝑦𝑆 Output: latent vector z
Generative model 𝐻(𝑨) Reconstruction loss 𝑀 Slide credit Zhu / [Zhu et al. 16]
0.196 0.238 0.332
Optimization
Inverting Network z = 𝑄 𝑦
0.218 0.242 0.336
Auto-encoder with a fixed decoder G Input: real image 𝑦𝑆 Output: latent vector z
Slide credit Zhu / [Zhu et al. 16]
0.196 0.238 0.332
Optimization
Inverting Network z = 𝑄 𝑦
0.218 0.242 0.336 0.153 0.167
Hybrid Method Use the network as initialization for the optimization problem
0.268
Input: real image 𝑦𝑆 Output: latent vector z
Slide credit Zhu / [Zhu et al. 16]
projection on manifold Project Edit Transfer transition between the original and edited projection different degree of image manipulation
Editing UI
Slide credit Zhu / [Zhu et al. 16]
Objective:
𝐻(𝑨) Guidance 𝑤 𝑨0
user guidance image constraint violation loss 𝑀 Slide credit Zhu / [Zhu et al. 16]
projection on manifold Project Edit Transfer transition between the original and edited projection different degree of image manipulation
Editing UI
Slide credit Zhu / [Zhu et al. 16]
𝐻(𝑨1) 𝐻(𝑨0) Input
Motion (u, v)+ Color (𝑩𝟒×𝟓): estimate per-pixel geometric and color variation
Linear Interpolation in 𝑨 space
Slide credit Zhu / [Zhu et al. 16]
𝐻(𝑨1) 𝐻(𝑨0) Input Linear Interpolation in 𝑨 space
Motion (u, v)+ Color (𝑩𝟒×𝟓): estimate per-pixel geometric and color variation
Slide credit Zhu / [Zhu et al. 16]
Result
𝐻(𝑨1) 𝐻(𝑨0) Input
Motion (u, v)+ Color (𝑩𝟒×𝟓): estimate per-pixel geometric and color variation
Linear Interpolation in 𝑨 space
https://github.com/junyanz/iGAN [Zhu et al. 16.]
Interactive GANs: projection to GAN embedding
19
20
https://github.com/junyanz/iGAN [Zhu et al. 16.]
21
https://github.com/junyanz/iGAN [Zhu et al. 16.]
22
InfoGAN [Chen et al. 16]
23
slides credit: Isola / Zhu
real or fake?
[Goodfellow et al. 2014] Discriminator
z G(z)
D
Generator
G
slides credit: Isola / Zhu
min
𝐻 max 𝐸
𝔽𝑨,𝑦 log 𝐸(𝐻 𝑨 ) + log(1 − 𝐸 𝑦 )
real or fake?
Discriminator
x G(x)
D
Generator
G min
𝐻 max 𝐸
𝔽𝑦,𝑧 log 𝐸(𝐻 𝑦 ) + log(1 − 𝐸 𝑧 )
slides credit: Isola / Zhu
Discriminator
x G(x)
D
Generator
G min
𝐻 max 𝐸
𝔽𝑦,𝑧 log 𝐸(𝐻 𝑦 ) + log(1 − 𝐸 𝑧 )
slides credit: Isola / Zhu
Discriminator
x G(x)
D
Generator
G
min
𝐻 max 𝐸
𝔽𝑦,𝑧 log 𝐸(𝐻 𝑦 ) + log(1 − 𝐸 𝑧 )
slides credit: Isola / Zhu
min
𝐻 max 𝐸
𝔽𝑦,𝑧 log 𝐸(𝑦, 𝐻 𝑦 ) + log(1 − 𝐸 𝑦, 𝑧 )
real or fake pair ? x G(x)
G D
match joint distribution p G x , y ∼ p(x, y)
fake pair real pair
slides credit: Isola / Zhu
Edges → Images
Input Output Input Output Input Output
Edges from [Xie & Tu, 2015]
slides credit: Isola / Zhu
31
Sketches → Images
Input Output Input Output Input Output
Trained on Edges → Images
Data from [Eitz, Hays, Alexa, 2012]
slides credit: Isola / Zhu
#edges2cats
[Christopher Hesse] Ivy Tasi @ivymyt @gods_tail @matthematician
https://affinelayer.com/pixsrv/
Vitaly Vidmirov @vvid
slides credit: Isola / Zhu
Input Output Groundtruth Data from [maps.google.com]
slides credit: Isola / Zhu
BW → Color
Input Output Input Output Input Output
Data from [Russakovsky et al. 2015]
slides credit: Isola / Zhu
scenarios. Label ↔ photo: per-pixel labeling
Horse ↔ zebra: how to get zebras?
slides credit: Isola / Zhu
slides credit: Isola / Zhu
x G(x)
Generator
G D
slides credit: Isola / Zhu
Discriminator
x G(x)
D
Generator
G
slides credit: Isola / Zhu
Discriminator
x G(x)
D
Generator
G
slides credit: Isola / Zhu
slides credit: Isola / Zhu
Cycle-Consistent Adversarial Networks
[Zhu*, Park*, Isola, and Efros, ICCV 2017]
slides credit: Isola / Zhu
Cycle-Consistent Adversarial Networks
[Mark Twain, 1903]
[Zhu*, Park*, Isola, and Efros, ICCV 2017]
slides credit: Isola / Zhu
G(x) F(G x ) x
F G x − x 1
[Zhu*, Park*, Isola, and Efros, ICCV 2017]
DY(G x )
Reconstruction error
slides credit: Isola / Zhu
G(x) F(G x ) x
F G x − x 1
Large cycle loss Small cycle loss
[Zhu*, Park*, Isola, and Efros, ICCV 2017]
DY(G x )
Reconstruction error
slides credit: Isola / Zhu
G(x) F(G x ) x F(y) G(F x ) 𝑧
F G x − x 1 G F y − 𝑧 1
[Zhu*, Park*, Isola, and Efros, ICCV 2017]
DY(G x )
Reconstruction error Reconstruction error
DG(F x )
slides credit: Isola / Zhu
https://junyanz.github.io/CycleGAN/ [Zhu et al. 17.]
48
slides credit: Isola / Zhu
slides credit: Isola / Zhu
slides credit: Isola / Zhu
– Xmas s GANs – No Lecture re
around asap…
52
53