Self-Supervised Model Training and Selection for Disentangling GANs - - PowerPoint PPT Presentation

β–Ά
self supervised model training and selection for
SMART_READER_LITE
LIVE PREVIEW

Self-Supervised Model Training and Selection for Disentangling GANs - - PowerPoint PPT Presentation

Sep 28, 2023 249 likes β€’420 views

Paper ID: 4410 Self-Supervised Model Training and Selection for Disentangling GANs Previous title: InfoGAN-CR: Disentangling Generative Adversarial Networks with Contrastive Regularizers Zinan Lin Kiran Thekumparampil Giulia Fanti Sewoong

slide-1
SLIDE 1

Previous title:

Zinan Lin

Self-Supervised Model Training and Selection for Disentangling GANs

Kiran Thekumparampil Giulia Fanti Sewoong Oh CMU UIUC CMU UW

InfoGAN-CR: Disentangling Generative Adversarial Networks with Contrastive Regularizers

Paper ID: 4410

slide-2
SLIDE 2
  • How do 𝑨!s control the factors?

Generative Adversarial Networks (GANs)

Generator 𝑨! 𝑨" 𝑨#

…

𝑙 factors

  • Hair color
  • Rotation
  • Background
  • Bangs

Vanilla GANs 𝑨! 𝑨" 𝑨#

…

Factor! Factor" Factor$

…

2 ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR

𝑒 input noise Disentangled GANs 𝑑! 𝑑" 𝑑$

…

Factor! Factor" Factor$

…

𝑨%s

slide-3
SLIDE 3

Examples of Disentanglement

Generator 𝑑! 𝑑" 𝑑#

…

Changing only: 𝑑! 𝑑" 𝑑& 𝑑' 𝑑( hair color rotation lighting background bangs shape scale rotation x-position y-position

(CelebA dataset) (dSprites dataset)

3 ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR

𝑨%s

* CelebA example is generated by InfoGAN-CR. Dsprites example is synthetic for illustration.

Latent codes The remaining noise dimensions

slide-4
SLIDE 4

Challenges in Learning Disentangled GANs

  • 1. How to train disentangled GANs?
  • GANs are good at generating high fidelity images,

but are reported to be bad at disentanglement (e.g. compared with VAEs)

  • 2. How to do unsupervised model selection?
  • In practice, we don’t have ground-truth factor labels

for selecting the best disentangled models.

4 ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR

slide-5
SLIDE 5

Our Solution: Self-Supervision

  • 1. How to train disentangled GANs?
  • InfoGAN-CR
  • 2. How to do unsupervised model selection?
  • ModelCentrality

Generator

Pairs of images with self-supervised labels

Contrastive Regularizer (CR)

Assessing the disentanglement quality

A pair of trained models: Self-supervised distance metric

…

5 ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR

All trained models: Pick the most central model

slide-6
SLIDE 6

Benchmark dSprites Dataset

Model FactorVAE Metric ↑ VAE VAE 0.63 𝛾-TCVAE 0.62 HFVAE 0.63 𝛾-VAE 0.63 CHyVAE 0.77 FactorVAE 0.82 GAN InfoGAN 0.59 InfoGAN (modified) 0.83 IB-GAN 0.80 InfoGAN-CR 0.90±𝟏. 𝟏𝟐 InfoGAN-CR model selected with ModelCentrality 0.92±𝟏. 𝟏𝟏

Supervised hyper-parameter selection Unsupervised model selection Code & Paper & More results: https://github.com/fjxmlzn/InfoGAN-CR

6 ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR

slide-7
SLIDE 7

Details

  • 1. InfoGAN-CR (model training)
  • 2. ModelCentrality (model selection)

7 ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR

slide-8
SLIDE 8

Intuition of Contrastive Regularizer (CR)

Generator (𝐻)

𝑗 = 1 𝑗 = 2 𝑗 = 3 Same shape Same x-position Same y-position

8 ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR

Same i-th latent code

  • Use two latent codes (𝑑",…,𝑑!,…,𝑑#), (𝑑"

$,…,𝑑! $,…,𝑑# $ ) to generate a pair of images

Equal

slide-9
SLIDE 9

Intuition of Contrastive Regularizer (CR)

Generator (𝐻)

𝑗 = 1 𝑗 = 2 𝑗 = 3

Contrastive Regularizer (CR)

1 2 3

Classification task!

9 ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR

Same i-th latent code

  • Use two latent codes (𝑑",…,𝑑!,…,𝑑#), (𝑑"

$,…,𝑑! $,…,𝑑# $ ) to generate a pair of images

Equal

slide-10
SLIDE 10

InfoGAN-CR

InfoGAN-CR loss: min

%,',( max )

𝑀*+, 𝐻, 𝐸 βˆ’ πœ‡π½ 𝐻, 𝑅 βˆ’ 𝛽𝑀-(𝐻, 𝐼)

classification accuracy of CR

𝐼

[1] InfoGAN. Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., & Abbeel, P. (NeurIPS 2016)

! ∈ ℝ! $ ∈ ℝ" % ∈ ℝ#

!

GAN Discriminator InfoGAN Encoder

" #

ℝ" CR ℝ Μ‚ $ ∈ ℝ" Input Noise Latent Factors %β€² ∈ ℝ# %β€²β€² ∈ ℝ#

10 ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR

InfoGAN [1] GAN’s adversarial loss Mutual info loss

slide-11
SLIDE 11

CR Achieves a Further Gain that Cannot be Gotten with InfoGAN Alone

#training iterations Disentanglement

(dSprites dataset)

11 ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR

Split training

slide-12
SLIDE 12

Details

  • 1. InfoGAN-CR (model training)
  • 2. ModelCentrality (model selection)

12 ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR

slide-13
SLIDE 13

Intuition of ModelCentrality

  • Well-disentangled models are close to the (unknown) ground-truth model

Model space

Ground-truth model (unknown) Good model Bad model

  • Idea: pick the model at the β€œcenter”

13 ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR

slide-14
SLIDE 14

ModelCentrality

  • How to find the model at the β€œcenter”?

[1] Disentangling by Factorising. Kim, H., & Mnih, A. (ICML 2018) 14 ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR

Model 𝑗 Model π‘˜ Similarity 𝑒!"? Any supervised disentanglement metric (e.g. FactorVAE metric [1]) Treat as the ground-truth factors Asymmetric similarity 𝑑!" (The higher, the more similar) Similarity 𝑒!" = 𝑒"! = (𝑑!" + 𝑑"!)/2

(2) ModelCentrality score 𝑁% =

! ./! βˆ‘01% 𝑒%0

(3) Selection the model with the highest 𝑁% (1)

Two models are close Γ³ Their latent codes control similar factors

High similarity Low similarity

slide-15
SLIDE 15

Results of ModelCentrality

  • Works well for GANs. Take InfoGAN-CR for example:
  • Also works well for VAEs! Take FactorVAE for example:

Method FactorVAE Metric ↑ Supervised hyper-parameter selection 0.90 Unsupervised model selection with UDR Lasso [1] 0.86 Unsupervised model selection with UDR Spearman [1] 0.84 Unsupervised model selection with ModelCentrality 0.92 Method FactorVAE Metric ↑ Supervised hyper-parameter selection 0.83 Unsupervised model selection with UDR Lasso [1] 0.81 Unsupervised model selection with UDR Spearman [1] 0.79 Unsupervised model selection with ModelCentrality 0.84

15 ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR [1] Unsupervised Model Selection for Variational Disentangled Representation Learning. Duan, S., Matthey, L., Saraiva, A., Watters, N., Burgess, C. P., Lerchner, A., & Higgins, I. (ICLR 2020).

slide-16
SLIDE 16

Conclusion

  • InfoGAN-CR
  • New disentangled GANs, achieving state-of-the-art results
  • ModelCentrality
  • Unsupervised model selection approach for disentangled GANs and VAEs
  • InfoGAN-CR + ModelCentrality
  • Unsupervised disentangled generative model training and selection package,

achieving state-of-the-art results

Code & Paper & More results: https://github.com/fjxmlzn/InfoGAN-CR

Including:

  • More theoretical & empirical analysis of InfoGAN & InfoGAN-CR
  • Analysis of total correlation for GANs and CR for VAEs
  • New challenging disentanglement datasets

16 ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR