Bias and Generalization in Deep Generative Models Shengjia Zhao*, - - PowerPoint PPT Presentation

▶
bias and generalization in deep generative models
SMART_READER_LITE
LIVE PREVIEW

Bias and Generalization in Deep Generative Models Shengjia Zhao*, - - PowerPoint PPT Presentation

Feb 22, 2023 143 likes •352 views

Bias and Generalization in Deep Generative Models Shengjia Zhao*, Hongyu Ren*, Arianna Yuan, Jiaming Song, Noah Goodman and Stefano Ermon *equal contribution Success in Generative Modeling of Images Brock A, et al. "Large scale gan

slide-1
SLIDE 1

Bias and Generalization in Deep Generative Models

Shengjia Zhao*, Hongyu Ren*, Arianna Yuan, Jiaming Song, Noah Goodman and Stefano Ermon

*equal contribution

slide-2
SLIDE 2

Success in Generative Modeling of Images

Brock A, et al. "Large scale gan training for high fidelity natural image synthesis."

slide-3
SLIDE 3

Goal: Understanding Generalization

How do generative models generalize?

slide-4
SLIDE 4

Generalization Example: Object Count

slide-5
SLIDE 5

Empirical Study of Generalization: Method

  • Design datasets
  • Train generative models (VAE, GAN, PixelCNN)
  • Observe generalization behavior
  • Find common patterns
slide-6
SLIDE 6

Generalization Example: Object Count

slide-7
SLIDE 7

Generalization in Feature Space: Numerosity

# Objects # Objects 2 2 3 4 1 Frequency Frequency Training Distribution Generated Distribution (Observed) Generates a log-normal shaped distribution

slide-8
SLIDE 8

Multiple Numerosities

# Objects 2 7 Frequency Training Distribution

?

slide-9
SLIDE 9

Multiple Numerosities: Only 2

# Objects # Objects 2 3 4 1 2 7 Frequency Frequency Training Distribution Generated Distribution

slide-10
SLIDE 10

Multiple Numerosities: Only 7

# Objects # Objects 2 7 7 8 9 6 Frequency Frequency Training Distribution Generated Distribution

slide-11
SLIDE 11

Multiple Numerosities: Additive Hypothesis

# Objects # Objects 2 3 4 1 2 7 7 8 9 6 Frequency Frequency Training Distribution Generated Distribution (Observed)

slide-12
SLIDE 12

Additive Hypothesis with 2 and 4 Objects

# Objects # Objects 2 3 4 1 5 6 4 2 Frequency Frequency Training Distribution Generated Distribution (Hypothesized)

slide-13
SLIDE 13

Actual Result: Prototype Enhancement

# Objects # Objects 2 3 4 1 5 6 4 2 Frequency Frequency Training Distribution Generated Distribution (Observed) 3 objects most likely, even though no training image contains 3 objects!

slide-14
SLIDE 14

Prototype Enhancement

2 3 4 1 5 6 4 2 Frequency Frequency Training Distribution Similar pattern for

  • ther features:

color, size, location Generated Distribution (Observed) # Objects # Objects

slide-15
SLIDE 15

Multiple Features

slide-16
SLIDE 16

Memorization vs. Generalization

slide-17
SLIDE 17

Memorization vs. Generalization

slide-18
SLIDE 18

Different Setups, Similar Results

  • Different features (shape, color, size, numerosity, etc.)
  • Different models: (VAE, GAN, PixelCNN, etc.)
  • Different architectures (fully connected, convolutional, etc.)
  • Different hyper-parameters (network size, learning rate, etc.)
slide-19
SLIDE 19

Conclusion

  • New methodology: design datasets to probe generative models
  • Observed common patterns across different setups

Welcome to our poster session for further discussions!

Tuesday 5-7pm @ Room 210 & 230 AB #6

Code available at github.com/ermongroup/BiasAndGeneralization