[PPT] - Genera&ve Adversarial Networks NTT PowerPoint Presentation

SLIDE 1

Genera&ve Adversarial Networks
NTT

2020610@JSAI2020 KS-02 AI – MIRU2020

SLIDE 2

2014. 4

NTT

2017. 42020.3

2

CFGAN

[CVPR 2017] DTLC-GAN [CVPR 2018]

S

rGAN† [CVPR 2019] CP-GAN† [BMVC 2019]

I

NR-GAN† [CVPR 2020]

CycleGAN-VC

GAN-PF [ICASSP 2017] GAN-VC [Interspeech 2017] CycleGAN-VC [EUSIPCO 2018] GAN-PF for STFT [Interspeech 2017] CycleGAN-VC2 [ICASSP 2019] StarGAN-VC2 [Interspeech 2019]

SLIDE 3

2014. 4

NTT

2017. 42020.3

3

S

CFGAN [CVPR 2017] DTLC-GAN [CVPR 2018]

IN

rGAN† [CVPR 2019] CP-GAN† [BMVC 2019]

GN

NR-GAN† [CVPR 2020]

3 CycleGAN-VC

GAN-PF [ICASSP 2017] GAN-VC [Interspeech 2017] CycleGAN-VC [EUSIPCO 2018] GAN-PF for STFT [Interspeech 2017] CycleGAN-VC2 [ICASSP 2019] StarGAN-VC2 [Interspeech 2019]

A

SLIDE 4

Q1.

– = 4

GAN
T. Karras et al., “Analyzing and Improving the Image Quality of StyleGAN,” arXiv 2019.

4

→

SLIDE 5

Q1. 5

– = 5

GAN
T. Karras et al., “Analyzing and Improving the Image Quality of StyleGAN,” arXiv 2019.

2

→

SLIDE 6

Q2. 6?
GAN
by Monet
JY. Zhu et al., “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks,” ICCV 2017.

SLIDE 7

Q2. 7
GAN
by Monet
JY. Zhu et al., “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks,” ICCV 2017.

SLIDE 8

Q3.
GAN
T. Kaneko et al., “Generative Attribute Controller with Conditional Filtered Generative Adversarial Networks,” CVPR 2017.

SLIDE 9

Q3.
GAN
T. Kaneko et al., “Genera2ve A5ribute Controller with Condi2onal Filtered Genera2ve Adversarial Networks,” CVPR 2017.

9http://www.kecl.ntt.co.jp/people/kaneko.takuhiro/projects/gac/index.html

SLIDE 10

GAN ()
CycleGAN [Zhu+ICCV2017]

CFGAN [Kaneko+CVPR2017] StyleGAN2 [Karras+arXiv2019]

SLIDE 11

GAN ()
CycleGAN [Zhu+ICCV2017]

CFGAN [Kaneko+CVPR2017] StyleGAN2 [Karras+arXiv2019]

SLIDE 12

(1

()2†

† h$p://www.kecl.n$.co.jp/people/kaneko.takuhiro/#talks

SLIDE 13

SLIDE 14

1 (Generator)
4
4
Pictures: A. Brock et al., “Large Scale GAN Training for High Fidelity Natural Image Synthesis,” ICLR 2019.

SLIDE 15

15
Pictures: A. Brock et al., “Large Scale GAN Training for High Fidelity Natural Image Synthesis,” ICLR 2019.

1 1 5 1

SLIDE 16

MVNAAR, Flow, VAE, GAN
MG

M M

6----1
616--
-6--1-6

(Chain Rule)

F

1--1

N z = f(x)

<latexit sha1_base64="QLlkCjr28Ua3oZ3AMcbE39dZsPg=">AB73icbVBNSwMxEJ31s9avqkcvwSLUS9mtgl6EohePFewHtEvJptk2NJusSVasS/+EFw+KePXvePfmLZ70NYHA4/3ZpiZF8ScaeO6387S8srq2npuI7+5tb2zW9jb2iZKELrRHKpWgHWlDNB64YZTluxojgKOG0Gw+uJ3ygSjMp7swopn6E+4KFjGBjpdYTukRh6fGkWyi6ZXcKtEi8jBQhQ61b+Or0JEkiKgzhWOu258bGT7EyjHA6zncSTWNMhrhP25YKHFHtp9N7x+jYKj0USmVLGDRVf0+kONJ6FAW2M8JmoOe9ifif105MeOGnTMSJoYLMFoUJR0aiyfOoxQlho8swUQxeysiA6wMTaivA3Bm395kTQqZe+0XLk9K1avsjhycAhHUAIPzqEKN1CDOhDg8Ayv8ObcOy/Ou/Mxa1yspkD+APn8wd9o70</latexit>

x = f −1(z)

<latexit sha1_base64="Xnt9lX2cNEUMxaHM7bxRUcRtMvU=">AB9HicbVBNS8NAEJ3Ur1q/qh69LBahHixJFfQiFL14rGA/oI1ls920SzebuLsp1tDf4cWDIl79Md78N27bHLT1wcDjvRlm5nkRZ0rb9reVWVpeWV3Lruc2Nre2d/K7e3UVxpLQGgl5KJseVpQzQWuaU6bkaQ48DhteIPrid8YUqlYKO70KJugHuC+YxgbST3EV0i/z45cbFp+NOvmCX7CnQInFSUoAU1U7+q90NSRxQoQnHSrUcO9JugqVmhNxrh0rGmEywD3aMlTgCo3mR49RkdG6SI/lKaERlP190SCA6VGgWc6A6z7at6biP95rVj7F27CRBRrKshskR9zpEM0SQB1maRE85EhmEhmbkWkjyUm2uSUMyE48y8vknq5JyWyrdnhcpVGkcWDuAQiuDAOVTgBqpQAwIP8Ayv8GYNrRfr3fqYtWasdGYf/sD6/AHauJDa</latexit>

Reference: I. Goodfellow, “NIPS 2016 Tutorial: GeneraBve Aversarial Networks,” NIPS 2016.

SLIDE 17

cMAR, Flow, VAE, GAN
e
iG

i

7----1

17--
---1-

(Chain Rule) Ni gVa

1--1

b

Reference: I. Goodfellow, “NIPS 2016 Tutorial: GeneraBve Aversarial Networks,” NIPS 2016.

ANd

F

SLIDE 18

Generative Adversarial Networks [Goodfellow+2014]

– (Generator) (Discriminator) – Min-MaxGenerator vs. Discriminator

GAN 1/4

Latent code z Generator Fake image G(z) Discriminator Real/Fake D(x) Real image x Random

G D

I. Goodfellow et al., “Generative Adversarial Nets,” NIPS 2014.

SLIDE 19

Genera&ve Adversarial Networks [Goodfellow+2014]

– (Generator) (Discriminator) – Min-MaxGenerator vs. Discriminator

GAN 2/4

Latent code z Generator Fake image G(z) Discriminator Real/Fake D(x) Real image x Random

G D

I. Goodfellow et al., “Generative Adversarial Nets,” NIPS 2014.

19

Fake

Discriminator

SLIDE 20

Genera&ve Adversarial Networks [Goodfellow+2014]

– (Generator) (Discriminator) – Min-Max0Generator vs. Discriminator

GAN 3/4

Latent code z Generator Fake image G(z) Discriminator Real/Fake D(x) Real image x Random

G D

I. Goodfellow et al., “Generative Adversarial Nets,” NIPS 2014.

Real/Fake

Generator0 Real/Fake

SLIDE 21

Genera&ve Adversarial Networks [Goodfellow+2014]

– (Generator) (Discriminator) – Min-MaxGenerator vs. Discriminator

GAN 4/4

Latent code z Generator Fake image G(z) Discriminator Real/Fake D(x) Real image x Random

G D

I. Goodfellow et al., “Generative Adversarial Nets,” NIPS 2014.
Real/Fake

Generator 21

GeneratorDiscriminatorMin-Max21

SLIDE 22

SLIDE 23

43
GAN
DCGAN, LAPGAN, StackGAN,

SAGAN, ProGAN, StyleGAN, MSG-GAN, StyleGAN2

LSGAN, WGAN, WGAN-GP,

SN-GAN, R1 GP, Hinge GAN, Relativistic GAN, (BigGAN)

GAN

rGAN, CP-GAN, S3GAN,

AmbientGAN, NR-GAN

2

cGAN, AC-GAN, InfoGAN, CFGAN, DTLC-GAN Latent code z Generator Fake image G(z) Discriminator Real/Fake D(x) Real image x Random

G D

SLIDE 24

4
GAN
4

DCGAN, LAPGAN, StackGAN, SAGAN, ProGAN, StyleGAN, MSG-GAN, StyleGAN2

LSGAN, WGAN, WGAN-GP,

SN-GAN, R1 GP, Hinge GAN, Relativistic GAN, (BigGAN)

GAN

rGAN, CP-GAN, S3GAN,

AmbientGAN, NR-GAN

2

cGAN, AC-GAN, InfoGAN, CFGAN, DTLC-GAN Latent code z Generator Fake image G(z) Discriminator Real/Fake D(x) Real image x Random

G D

SLIDE 25

Min-Max

– Jensen-Shannon Divergence (JSD)

GeneratorDiscriminator
5

–

GeneratorDiscriminator
5
GAN

5 5

Mode Collapse

(2)

I. Goodfellow et al., “Generative Adversarial Nets,” NIPS 2014.

SLIDE 26

Mode Collapse [Metz+2017]

– A

Mode Collapse
NG
6
6
RealFake2
L. Metz et al., “Unrolled Genera?ve Adversarial Networks,” ICLR 2017.

SLIDE 27

Least Squares GAN [Mao+2017]

– Sigmoid Cross Entropy02Least Squares Loss

LSGAN

Discriminator Generator

GAN LSGAN GAN LSGAN

LSGAN1

0Chi-square Distance7

(1 )(1 )( (

X. Mao et al., “Least Squares Generative Adversarial Networks,” ICCV 2017.

SLIDE 28

8
LSGAN ()
10
8
6
4
2

2 4 6 8 10

10
8
6
4
2

2 4 6 8 10 Fake samples Real samples Fake samples for updating G Sigmoid decision boundary

10
8
6
4
2

2 4 6 8 10

10
8
6
4
2

2 4 6 8 10 Fake samples Real samples Fake samples for updating G Least squares decision boundary

GAN: Sigmoid Cross Entropy LSGAN: Least Squares Loss 2 Loss0 → 2 Loss →

Ns

Mode Collapse
X. Mao et al., “Least Squares Generative Adversarial Networks,” ICCV 2017.

SLIDE 29

Wasserstein GAN [Arjovsky+2017]

– ClassifierCritic

WGAN

Discriminator Generator

GAN WGAN GAN WGAN

M. Arjovsky et al., “Wasserstein Generative Adversarial Networks,” ICML 2017.

WGAN

Earth Mover Distance

F22 F DLipschitz9

→ Weight Clipping

SLIDE 30

WGAN ()
M. Arjovsky et al., “Wasserstein Generative Adversarial Networks,” ICML 2017.

GAN (): WGAN (): Wasserstein0 3

Mode Collapse

SLIDE 31

WGAN

– DLipschitz3Weight Clipping () →

WGAN Gradient Penalty [Gulrajani+2017]

– f → f1-Lipschitz – Gradient Penalty (GP) () – (FC, Conv, ResNet)

WGAN-GP
I. Gulrajani et al., “Improved Training of Wasserstein GANs,” NIPS 2017.

1-Lipschitz

Weight Clipping1f3

krfk  1 everywhere

<latexit sha1_base64="H8ouAUDzcB4a+9qs43ypquye53k=">ACKHicbZC9TsMwFIUd/il/BUYWiwqJqUoACcYKFkaQaIvURJXj3lALxwn2DRCFvgIPwswKz8CGurLxFrg/A9AeydKnc+6VdU+YSmHQdfvOzOzc/MLi0nJpZXVtfaO8udUwSaY51HkiE30dMgNSKijQAnXqQYWhxKa4e3ZIG/egzYiUVeYpxDE7EaJSHCG1mqX9/0n6isWSkYjOmAJd9SjPsIjFhTsav7QBQ29drniVt2h6CR4Y6iQsS7a5W+/k/AsBoVcMmNanptiUDCNgkvolfzMQMr4LbuBlkXFYjBMbyoR/es06FRou1TSIfu742CxcbkcWgnY4Zd8z8bmFOzMJ5mtzKMToJCqDRDUHz0f5RJigkdtEY7QgNHmVtgXAt7AuVdphlH23JduP9b2ISGgdV7B6cHlUqZ2OW1oiO2SX7BOPHJMaOScXpE4eSav5I28Oy/Oh/Pp9EejM854Z5v8kfP1A5Rhpe8=</latexit>

kf(x) f(y)k  kx yk for all x, y

<latexit sha1_base64="jtPeQUvUiRvSOnL+KBJuWkFpJXc=">ACVnicbVHLTsMwEHTDu7wCHLlYVEgQZUJDgiuHAsEi1ITVU57gYs7CTYG0QV+md8CFy5IfgGhJvmAIW5eDQzK3vHYSqFQc97rThT0zOzc/ML1cWl5ZVd29bZJMc2jxRCb6OmQGpIihQIlXKcamAolXIV3ZyP/6gG0EUl8iYMUuordxCISnKGVem47eKLRThCq/HG4S/dLPrDcGoGE+IsbOuOvUJCeMScRomTEo6LDN7ZaTn1ry6V4D+JX5JaqREs+e+B/2EZwpi5JIZ0/G9FLs50yi4hGE1yAykjN+xG+hYGjMFpsX+w/ptlX6xVuiJEZaqD8ncqaMGajQJhXDWzPpjcR/vVD9J3cyjI67uYjTDCHm4/ujTFJM6Khj2hcaOMqBJYxrYVeg/JZpxtH+RNV240828Ze0G3X/oN64OKydnJYtzZNskV2iE+OyAk5J03SIpw8kzfyQT4rL5UvZ8aZG0edSjmzQX7Bcb8BF0q0cA=</latexit>

SLIDE 32

Spectral Normalization GAN [Miyato+2018]

– Wa Lipschitz → fLipschitz – Spectral Normalization (SN) – (ImageNet)

SN-GAN

f(x) = W L+1(aL(W L(aL−1(W L−1(. . . a1(W 1x) . . . )))))

<latexit sha1_base64="0D/mU9VabMTn/8Uh0op7OoGXcA=">ACa3icbZHLS+QwHMfT+tz6GvWmHoKD0CIOjQruZUHWyx7moOBYR4lzaRjmKQtSo7lP6Re/RP8LBnr6a14vMHCZ98f4+Qb6KM6V9/8Gy5+YXFpeWfzgrq2vrG63NrRuV5pLQHkl5Km8jrChnCe1pjm9zSTFIuI0iKYXVT64p1KxNLnWs4wOBZ4kLGYEayOFrWnsFoNIwL+lB385wajoHqLSxWHXdSAMRl2DRfcIlbA6V0p9cgfjVKtagRCHyG0wGKHXaW8Vntk9z/G8sNX2O34d8CugBtqgicuw9WimkFzQRBOleojP9PDAkvNCKelM8gVzTCZ4gntG0ywoGpY1KaU8MAoYxin0qxEw1p931FgodRMRKZSYH2nPucq8dtcJL6T+7mOfw4LlmS5pgl5uT/OdQprIyHYyYp0XxmABPJzBMgucMSE2+xzHeoM9OfIWb4w46RxfnbPfzcuLYNdsA9cgMAZOAd/wCXoAQL+gScLWJb13962d+y9l1Lbanq2wYewD54BtOSvEw=</latexit>

Lipschitz W3Lipschitz2 σ(W)

T. Miyato et al., “Spectral Normalization for Generative Adversarial Networks,” ICLR 2018.

SLIDE 33

Large Scale GAN [Brock+2019]

–

3
3
→
BigGAN
A. Brock et al., “Large Scale GAN Training for High Fidelity Natural Image Synthesis,” ICLR 2019.

SLIDE 34

GAN

– Min-Max

GAN

Jensen-Shannon Divergence

LSGAN

Chi-Square Divergence

WGAN

Earth Mover Distance + Weight Clipping

WGAN-GP

Gradient Penalty

SN-GAN

Spectral Normalization

Lipschitz
BigGAN

Scale Up

R1 GP, Hinge GAN, Relativistic GAN

34

SLIDE 35

43
GAN
DCGAN, LAPGAN, StackGAN,

SAGAN, ProGAN, StyleGAN, MSG-GAN, StyleGAN2

LSGAN, WGAN, WGAN-GP,

SN-GAN, R1 GP, Hinge GAN, Relativistic GAN, (BigGAN)

GAN

5

rGAN, CP-GAN, S3GAN, AmbientGAN, NR-GAN

cGAN, AC-GAN, InfoGAN,

CFGAN, DTLC-GAN Latent code z Generator Fake image G(z) Discriminator Real/Fake D(x) Real image x Random

G D

SLIDE 36

1

– 0

0623
GAN
1
40

Picgures: T. Karras et al., “Analyzing and Improving the Image Quality of StyleGAN,” arXiv 2019.

SLIDE 37

Laplacian GAN [Denton+2015]

– 84

LAPGAN
G
G4

G3 G2 G1 G0

9GAN

G: 12

LAPGAN (367)

G0, …, G4: 1

E. Denton et al., “Deep Generative Image Modeling using a Laplacian Pyramid of Adversarial Networks,” NIPS 2015.

SLIDE 38

Progressive Growing of GANs [Kerras+2018]

–

ProGAN

4x4

G D

4x4 8x8 Reals 4x4 4x4 Reals 8x8 4x4 Latent Reals 4x4

…

Training progresses Latent Latent 1024x1024 1024x1024

Latent

4x4 8x8 4x4 8x8 16x16

G1 G2 G3

LAPGAN

GD → 3/8

ProGAN

GD → 3/8

…

Real/ Fake Real/ Fake Real/ Fake

D1 D2 D3

T. Karras et al., “Progressive Growing of GANs for Improved Quality, Stability, and Variation,” ICLR 2018.

SLIDE 39

ProGAN ()
T. Karras et al., “Progressive Growing of GANs for Improved Quality, Stability, and VariaBon,” ICLR 2018.

1024x1024

SLIDE 40

Style-Based Generator for GANs [Kerras+2019]

– Generator

StyleGAN

Normalize Fully-connected PixelNorm PixelNorm Conv 3×3 Conv 3×3 Conv 3×3 PixelNorm PixelNorm Upsample 4×4 8×8

Latent

Normalize FC FC FC FC FC FC FC FC A A A A B B B B Const 4×4×512 AdaIN AdaIN AdaIN AdaIN Upsample Conv 3×3 Conv 3×3 Conv 3×3 4×4 8×8 style style style style

Noise Latent Mapping network Synthesis network

04
04
T. Karras et al., “A Style-Based Generator Architecture for GeneraCve Adversarial Networks,” CVPR 2019.

SLIDE 41

StyleGAN ()
T. Karras et al., “A Style-Based Generator Architecture for Generative Adversarial Networks,” CVPR 2019.

SLIDE 42

GAN

– 1

42
LAPGAN
ProGAN
StyleGAN

StyleGAN2

SLIDE 43

43
GAN
4

DCGAN, LAPGAN, StackGAN, SAGAN, ProGAN, StyleGAN, MSG-GAN, StyleGAN2

LSGAN, WGAN, WGAN-GP,

SN-GAN, R1 GP, Hinge GAN, Relativistic GAN, (BigGAN)

GAN

rGAN, CP-GAN, S3GAN,

AmbientGAN, NR-GAN

cGAN, AC-GAN, InfoGAN,

CFGAN, DTLC-GAN Latent code z Generator Fake image G(z) Discriminator Real/Fake D(x) Real image x Random

G D

SLIDE 44

GAN

Latent code z Generator Discriminator Real/Fake D(x) G D

4

?

NO control

?

Pictures: T. Kaneko et al., “Genera6ve Adversarial Image Synthesis with Decision Tree Latent Controller,” CVPR 2018.

T. Kaneko et al., “Genera6ve AKribute Controller with Condi6onal Filtered Genera6ve Adversarial Networks,” CVPR 2017.
Random

SLIDE 45

Conditional GAN [Mirza+2014]

– (5) y → ON/OFF4

cGAN

Latent code z Generator Discriminator Real/Fake D(x) G D

z:

y:

Supervision y (0, 1) Supervision y (0, 1)

M. Mirza & S. Osindero, “Conditional Generative Adversarial Nets,” arXiv 2014.

SLIDE 46

cGAN ()
)
)
)

64 (ON/OFF

M. Mirza & S. Osindero, “Conditional Generative Adversarial Nets,” arXiv 2014.

Pictures: T. Kaneko et al., “Generative Attribute Controller with Conditional Filtered Generative Adversarial Networks,” CVPR 2017.

SLIDE 47

Conditional Filtered GAN [Kaneko+2017]
1. 1 za +4
2. 3 y
3. +1 za’ + →

CFGAN

Latent code z Generator Discriminator Real/Fake D(x) G D Supervision y (0, 1) Supervision y (0, 1) Latent za Latent za'

Condi>onal Filter

+4
+
za’

+7

. 2

T. Kaneko et al., “Generative Attribute Controller with Conditional Filtered Generative Adversarial Networks,” CVPR 2017.

SLIDE 48

CFGAN ()

: h$p://www.kecl.n$.co.jp/people/kaneko.takuhiro/projects/gac/index.html

T. Kaneko et al., “Generative Attribute Controller with Conditional Filtered Generative Adversarial Networks,” CVPR 2017.

SLIDE 49

Decision Tree Latent Controller GAN [Kaneko+2018]

– DCDL → 9 – C → 44:

DTLC-GAN

Latent code z Generator Discriminator Real/Fake D(x) G D

c
+4
Code c
T. Kaneko et al., “Genera@ve Adversarial Image Synthesis with Decision Tree Latent Controller,” CVPR 2018.

SLIDE 50

DTLC-GAN ()
5
5

( ) 5(0

T. Kaneko et al., “Generative Adversarial Image Synthesis with Decision Tree Latent Controller,” CVPR 2018.

SLIDE 51

GAN

– 15

教師なし

（弱）教師あり

G G

Supervision

G

Latent

G

Latent Supervision

GAN [Goodfellow+2014] cGAN [Mirza+2014] AC-GAN [Odena+2017]

分離なし教師情報に制約

InfoGAN [Chen+2016] CFGAN [Kaneko+2017]

一層の潜在変数

層数

1 2, 3, ...

DTLC-GAN [Kaneko+2018]

多階層の潜在変数

G

Latent code z Generator Discriminator Real/Fake D(x) G D Random

SLIDE 52

4
GAN
DCGAN, LAPGAN, StackGAN,

SAGAN, ProGAN, StyleGAN, MSG-GAN, StyleGAN2

LSGAN, WGAN, WGAN-GP,

SN-GAN, R1 GP, Hinge GAN, Relativistic GAN, (BigGAN)

GAN

5

rGAN, CP-GAN, S3GAN, AmbientGAN, NR-GAN

2

cGAN, AC-GAN, InfoGAN, CFGAN, DTLC-GAN Latent code z Generator Fake image G(z) Discriminator Real/Fake D(x) Real image x Random

G D

SLIDE 53

3

– Clean3

GAN 1/4

(Clean) (Noisy)

G

Generator

Generator

5

T. Kaneko et al., “Label-Noise Robust Generative Adversarial Networks,” CVPR 2019.

SLIDE 54

Label-Noise Robust GAN [Kaneko+2019]

– Noise Transition ModelAC-GAN [Odena+2017] cGAN [Mirza+2014, Miyato+2018] → )4

rGAN

T. Kaneko et al., “Label-Noise Robust GeneraKve Adversarial Networks,” CVPR 2019.
A. Odena et al., “CondiKonal Image Synthesis With Auxiliary Classifier GANs,” ICML 2017.
M. Mirza & S. Osindero, “CondiKonal GeneraKve Adversarial Nets,” arXiv 2014.
T. Miyato & M. Koyama, “cGANs with ProjecKon Discriminator,” ICLR 2018.

p(˜ y|ˆ y)

<latexit sha1_base64="U76tA9Mz7kpaJIgkixOqENDvr3w=">ACDnicbZDJSgNBEIZr4hbjFhW8eGkShIgQZrzoMejFYwSzQCaEnk4nadKz0F0jDJO8g48gXvXsTbz6Ch59EzvLwST+UPDxVxV/F4khUb/rYya+sbm1vZ7dzO7t7+Qf7wqK7DWDFeY6EMVdOjmksR8BoKlLwZKU59T/KGN7yd9BuPXGkRBg+YRLzt034geoJRNFYnfxKVXBSy9NkTEbEHVA0dN7JF+2yPRVZBWcOxUrBvXgGgGon/+N2Qxb7PEAmqdYtx46wnVKFgk+zrmx5hFlQ9rnLYMB9blup9P/x+TMOF3SC5WpAMnU/buRUl/rxPfMpE9xoJd7E/O/XivG3nU7FUEUIw/Y7FAvlgRDMgmDdIXiDGVigDIlzK+EDaiDE1kC1c8f2wycZYTWIX6ZdkxfG/CuYGZsnAKBSiBA1dQgTuoQg0YjOAFXuHNerLerQ/rczaseY7x7Ag6+sXDAKeCA=</latexit><latexit sha1_base64="bljVtG6foLbmWotaiB3KIQEcSo=">ACDnicbZC7SgNBFIZn4y3G26pgYzMkCBEh7NpoGbSxjGASIbuE2dlJMmT2wsxZYdnsO1j4BLZa24mtr5DSN3FyKUziDwc+/nMO5/B7seAKLGtsFNbWNza3itulnd29/QPz8KilokRS1qSRiOSjRxQTPGRN4CDYywZCTzB2t7wdtJvPzGpeBQ+QBozNyD9kPc4JaCtrnkSVx3gwmdZmuMRdgYENJ13zYpVs6bCq2DPoVIvOxcv43ra6Jo/jh/RJGAhUEGU6thWDG5GJHAqWF5yEsViQoekzoaQxIw5WbT/3N8ph0f9yKpKwQ8df9uZCRQKg08PRkQGKjl3sT8r9dJoHftZjyME2AhnR3qJQJDhCdhYJ9LRkGkGgiVXP+K6YBIQkFHtnDFC3Kdib2cwCq0Lmu25nsdzg2aqYhOURlVkY2uUB3doQZqIopG6BW9oXfj2fgwPo2v2WjBmO8cowUZ378sRp+O</latexit><latexit sha1_base64="rdvUqbcmdb3mFWLiBIwQdy5QAc=">ACDnicbZDLSsNAFIYnXmu9RQU3bgaLUDclcaPLohuXFewFmlAmk0k7dHJh5kQIad7BZ3Cra3fi1ldw6Zs4bOwrT8c+PjPOZzD7yWCK7Csb2NtfWNza7uyU93d2z84NI+OypOJWVtGotY9jyimOARawMHwXqJZCT0BOt647tpv/vEpOJx9AhZwtyQDCMecEpAWwPzNKk7wIXP8qzAE+yMCGi6HJg1q2HNhFfBLqGSrUG5o/jxzQNWQRUEKX6tpWAmxMJnApWVJ1UsYTQMRmyvsaIhEy5+ez/Al9ox8dBLHVFgGfu342chEploacnQwIjtdybmv/1+ikEN27OoyQFtH5oSAVGI8DQP7XDIKItNAqOT6V0xHRBIKOrKFK15Y6Ezs5QRWoXPVsDU/WLXmbZlOBZ2hc1RHNrpGTXSPWqiNKJqgF/SK3oxn4934MD7no2tGuXOCFmR8/QLibZx/</latexit>

Clean Noisy(

D D D/C D/C

AC-GAN

5

rAC-GAN

cGAN

5

rcGAN

Classifier

4 Discriminator 4

SLIDE 55

rGAN ()
Noisy

cGAN

rcGAN

5

Noisy

Clean
(: https://github.com/takuhirok/rGAN/
T. Kaneko et al., “Label-Noise Robust Generative Adversarial Networks,” CVPR 2019.

)

SLIDE 56

65

– Class-Separate5

GAN 2/4

(Class-Separate) (Class-Overlapping)

T. Kaneko et al., “Class-Distinct and Class-Mutual Image Generation with GANs,” BMVC 2019.

G

Generator

Generator
Clothing1M

Xiao et al., CVPR 2015.

Knitwear Sweater

?

SLIDE 57

Classifier’s Posterior GAN [Kaneko+2019]

– Classifier’s PosteriorAC-GAN [Odena+2017] )57

CP-GAN

D/C Real/Fake G D/C Real/Fake D/C Real/Fake G D/C Real/Fake

AC-GAN CP-GAN

T. Kaneko et al., “Class-Distinct and Class-Mutual Image Generation with GANs,” BMVC 2019.
A. Odena et al., “Conditional Image Synthesis With Auxiliary Classifier GANs,” ICML 2017.
(

) Classifier’s Posterior(

SLIDE 58

CP-GAN ()

Smaller than 5 Even A B A ∩ B B A (i) Discrete label A B

Probability density

(ii) Discretely controllable Generator B A A∩¬B A ∩B ¬A∩ B

Probability density

(ii) Class-specificity controllable (i) Classifie s posterior r’ Generator

(Class-Overlapping)

AC-GAN

CP-GAN

5

(

(
): https://github.com/takuhirok/CP-GAN/

(8

T. Kaneko et al., “Class-Distinct and Class-Mutual Image Generation with GANs,” BMVC 2019.

SLIDE 59

9

–

GAN 3/4

()

G

Generator

Generator

?DB5

M. Lucic et al., “High-Fidelity Image Generation With Fewer Labels,” ICML 2019.

SLIDE 60

Semi-Supervised GAN with Self-Supervision [Lucic+2019]

– G6ASG6AGAN3N

B
S3GAN
M. Lucic et al., “High-Fidelity Image GeneraJon With Fewer Labels,” ICML 2019.

%(

)%(
0163B

SLIDE 61

61

– Clean1

GAN 4/4

(Clean) (Noisy) Generator 6 6

T. Kaneko et al., “Noise Robust Generative Adversarial Networks,” CVPR 2020 (to appear).

Image Generator Image Generator

SLIDE 62

AmbientGAN [Bora+2018]

– 26

AmbientGAN

x̃

Discriminator

D(x)

Real image Real/fake Noisy Noisy Image Generator

z

Latent vector Noise (known)

+

Clean
A. Bora et al., “AmbientGAN: Generative models from lossy measurements,” ICLR 2018.

SLIDE 63

Noise Robust GAN [Kaneko+2020]

– Generator3

NR-GAN

Discriminator Image Generator Discriminator Image Generator Noise Generator

+

GAN () NR-GAN ()

6
T. Kaneko et al., “Noise Robust Generative Adversarial Networks,” CVPR 2020.

SLIDE 64

46
NR-GAN ()

Discriminator Image Generator Noise Generator

+

Discriminator Image Generator Noise Generator

+

· ✏ ∼ N(0, diag()2)

<latexit sha1_base64="c2O7N5QJ5Nd+LBL7N96FIWEwWqk=">AHicbZFNaxRBEIZrJ2rixo9Vj14aFyEBWZCMB4DXjxJAm4S2FmXmp7aTZP+GLprxGWY3+UP8eQhV8Wf4M3e3TlokoKGl+et6qbeLiqtAqfpj16yde/+g+2dh/3dR4+fPB08e34WXO0ljaXTzl8UGEgrS2NWrOmi8oSm0HReXL1f+edfyAfl7CdeVjQ1uLBqriRyRLPBaV6YJg9qYbAVuSwdizWhKijtbGRBGdHkErX42O6tvLR9I3Kmr9yUChcb1t2w/lgfzYpqN0XeK2yDoxhK5OZoPfelkbciy1BjCJEsrnjboWUlNbT+vA1Uor3BkygtGgrTZr16K15HUoq58/FYFmv670SDJoSlKWKnQb4MN70VvNMrzF14UvP83bRtqZrNy8P6+1YCdW8YpSeZKsl1Gg9CquIOQlepQcP6Efs8luJnFbnB2MsNReno4PH7bpbQDL+EV7EGR3AMH+AExiDhG1zDT/jV+97k0CSbFqTXjfzAv6rZPcvUGm0Ng=</latexit>

✏ ∼ N(0, I)

<latexit sha1_base64="SwLUa9BbLuQRZ1HhDKVjp69LBvM=">AHicbVDLSsNAFJ3UV62vqEs3g0WoKCWRoi4LbnQjFewDmlAm02k7dGYSZiZCfkIP8S1W/0GV4pbF/6FkzYLrT1wuYdz7uVyTxAxqrTjvFuFpeWV1bXiemljc2t7x97da6kwlpg0chC2QmQIowK0tRUM9KJE8YKQdjK8yv/1ApKhuNeTiPgcDQUdUIy0kXr2iRfwxCORoiwUKfQU5TDxMGLwNq1knpOewqzfpMc9u+xUnSngf+LmpAxyNHr2t9cPcyJ0JghpbquE2k/QVJTzEha8mJFIoTHaEi6hgrEifKT6VMpPDJKHw5CaUpoOFV/bySIKzXhgZnkSI/UvJeJC72AL5K7sR5c+gkVUayJwLP7g5hBHcIsONinkmDNJoYgLKl5AeIRkghrE2/JZOPOJ/GftM6qbq3q3NXK9fM8pSI4AIegAlxwAergGjRAE2DwCJ7BC3i1nqw368P6nI0WrHxnH/yB9fUDMiKnLg=</latexit>

Reparameterization

trick [Kingma+2014]

T. Kaneko et al., “Noise Robust Generative Adversarial Networks,” CVPR 2020.

Kingma & Welling, “Auto-Encoding Variational Bayes,” ICLR 2014.

ˆ

n

<latexit sha1_base64="oEpOuEPSd+N9V4VarZOPucVW/30=">AHicbZDLSgMxGIUz9VbrerSTbAIrsqMFHVZcOygr3AzFAyaYNTJD8o9Qhj6Ba7f6DO7ErU/hI/gWpu0stO2BwMc5/0+SE6WCG3Ddb6e0sbm1vVPereztHxweVY9POibJNGVtmohE9yJimOCKtYGDYL1UMyIjwbrR+G6Wd5+YNjxRjzBJWSjJUPGYUwLW8oMRgTyIJFbTfrXm1t258Cp4BdRQoVa/+hMEpJpoAKYozvuSmEOdHAqWDTSpAZlhI6JkPmW1REMhPm8ydP8YV1BjhOtD0K8Nz9u5ETacxERnZSEhiZ5Wxmrs0iuc72M4hvw5yrNAOm6OL+OBMYEjyrBQ+4ZhTExAKhmtsvYDoimlCw5VsN95yE6vQuap7jbr70Kg1r4uWyugMnaNL5KEb1ET3qIXaiKIEvaBX9OY8O+/Oh/O5GC05xc4p+ifn6xeEv5pb</latexit>

n = T(ˆ n)

<latexit sha1_base64="3dN6KIyS9cq4ajWrtlQg7C5q2o=">AHicbVDLSsNAFJ3UV62vqBvBzWAR6qYkUtSNUHDjskJf0IQymU7aoTOTMDMRSogf4tqtfoM7cevOT/AvnLZaNsDF849514u9wQxo0o7zrdVWFvf2Nwqbpd2dvf2D+zDo7aKEolJC0cskt0AKcKoIC1NSPdWBLEA0Y6wfhu6nceiVQ0Ek09iYnP0VDQkGKkjdS3T7yApyKDt7BZ8UZIp/M+u+jbZafqzACXiZuTMsjR6Ns/3iDCSdCY4aU6rlOrP0USU0xI1nJSxSJER6jIekZKhAnyk9nH2Tw3CgDGEbSlNBwpv7dSBFXasIDM8mRHqlFbyqu9AK+Su4lOrzxUyriRBOB5/fDhEdwWlKcEAlwZpNDEFYUvMCxCMkEdYmy5LJxl1MYpm0L6tureo81Mr1qzylIjgFZ6ACXHAN6uAeNEALYPAEXsAreLOerXfrw/qcjxasfOcY/IP19Qt7kJ+Z</latexit>

σ

<latexit sha1_base64="1T0UWHIuVKpCYh7xwLHQ+wn5dWI=">AHicbZDLSgMxGIX/qbdab1WXboJFcFVmpKjLghuXFewFZoaSTNtaDIZkoxQhj6Ba7f6DO7ErU/hI/gWpu0stO2BwMc5/0+SE6WcaeO6305pY3Nre6e8W9nbPzg8qh6fdLTMFKFtIrlUvQhrylC24YZTnupolhEnHaj8d0s7z5RpZlMHs0kpaHAw4TFjGBjLT+IRB5oNhR42q/W3Lo7F1oFr4AaFGr1qz/BQJM0MQjrX2PTc1Y6VYTaSXINE0xGeMh9S0mWFAd5vMnT9GFdQYolsqexKC5+3cjx0LriYjspMBmpJezmbk2i8Q6289MfBvmLEkzQxOyuD/ODISzWpBA6YoMXxiARPF7BcQGWGFibHlVWw3nITq9C5qnuNuvQqDWvi5bKcAbncAke3EAT7qEFbSAg4QVe4c15dt6dD+dzMVpyip1T+Cfn6xf1zJqh</latexit>

Rotation
Channel shuffle
Color inversion

T

6

SLIDE 65

NR-GAN ()
Addi+ve

Gaussian noise Brown Gaussian noise

GAN

() NR-GAN (65)

GAN

() NR-GAN (65) Mul+plica+ve Gaussian noise Poisson noise

T. Kaneko et al., “Noise Robust Generative Adversarial Networks,” CVPR 2020 (to appear).
: https://github.com/takuhirok/NR-GAN/

SLIDE 66

GAN

– v

s

3

vn

6
Clean

Class-Separate

v
(
)/-)(

e

a

iN

l

p

yClean

,-

C

Cr
e
l

p

Cg

SLIDE 67

4
GAN ()
7

DCGAN, LAPGAN, StackGAN, SAGAN, ProGAN, StyleGAN, MSG-GAN, StyleGAN2

6

LSGAN, WGAN, WGAN-GP, SN-GAN, R1 GP, Hinge GAN, Rela?vis?c GAN, (BigGAN)

GAN

rGAN, CP-GAN, S3GAN,

AmbientGAN, NR-GAN

cGAN, AC-GAN, InfoGAN,

CFGAN, DTLC-GAN Latent code z Generator Fake image G(z) Discriminator Real/Fake D(x) Real image x Random

G D

SLIDE 68

(

()†6

† http://www.kecl.ntt.co.jp/people/kaneko.takuhiro/#talks

SLIDE 69

GAN9

– G vs. 6D

6
GANB
6BA [Kaneko+2017]

6 [Kaneko+2017]

6B

6
Latent code

z Generator Fake image G(z) Discriminator Real/Fake D(x) Real image x Random G D

SLIDE 70

7
https://www.jstage.jst.go.jp/article/jasj/74/4/74_208/_pdf

h:ps://www.jstage.jst.go.jp/ar;cle/ast/39/3/39_E183003/_pdf/-char/en http://www.kecl.ntt.co.jp/people/kaneko.takuhiro/index.html

WebTakuhiro Kaneko NTT

7

0GAN

E-mail: takuhiro.kaneko.tb at hco.ntt.co.jp