Residual Flows for Invertible Generative Modeling Ricky T. Q. Chen, - - PowerPoint PPT Presentation

▶
residual flows
SMART_READER_LITE
LIVE PREVIEW

Residual Flows for Invertible Generative Modeling Ricky T. Q. Chen, - - PowerPoint PPT Presentation

Mar 17, 2024 274 likes •445 views

Residual Flows for Invertible Generative Modeling Ricky T. Q. Chen, Jens Behrmann, David Duvenaud, Jrn-Henrik Jacobsen Invertible Residual Networks (i-ResNet) It can be shown that residual blocks can be inverted by fixed-point iteration and

slide-1
SLIDE 1

Residual Flows

for Invertible Generative Modeling

Ricky T. Q. Chen, Jens Behrmann, David Duvenaud, Jörn-Henrik Jacobsen

slide-2
SLIDE 2

Invertible Residual Networks (i-ResNet)

It can be shown that residual blocks

(Behrmann et al. 2019)

can be inverted by fixed-point iteration and has a unique inverse (ie. invertible) if i.e. Lipschitz. Enforced with spectral normalization.

slide-3
SLIDE 3

Applying Change of Variables to i-ResNets

If

(Behrmann et al. 2019)

Then

slide-4
SLIDE 4

Unbiased Estimation of Log Probability Density

Enter the “Russian roulette” estimator (Kahn, 1955). Suppose we want to estimate

(Require )

slide-5
SLIDE 5

Unbiased Estimation of Log Probability Density

Enter the “Russian roulette” estimator (Kahn, 1955). Suppose we want to estimate Flip a coin b with probability q.

(Require )

slide-6
SLIDE 6

Unbiased Estimation of Log Probability Density

Enter the “Russian roulette” estimator (Kahn, 1955). Suppose we want to estimate Flip a coin b with probability q.

(Require )

slide-7
SLIDE 7

Unbiased Estimation of Log Probability Density

Enter the “Russian roulette” estimator (Kahn, 1955). Suppose we want to estimate Flip a coin b with probability q.

(Require )

slide-8
SLIDE 8

Unbiased Estimation of Log Probability Density

Enter the “Russian roulette” estimator (Kahn, 1955). Suppose we want to estimate Flip a coin b with probability q. Has probability q of being evaluated in finite time.

(Require )

slide-9
SLIDE 9

Unbiased Estimation of Log Probability Density

If we repeatedly apply the same procedure infinitely many times, we obtain an unbiased estimator of the infinite series. Directly sample the first successful coin toss. k-th term is weighted by

  • prob. of seeing >= k tosses.

Residual Flow: Computed in finite time with prob. 1!!

slide-10
SLIDE 10

Decoupled Training Objective & Estimation Bias

Unbiased but... variable compute and memory!

slide-11
SLIDE 11

Constant-Memory Backpropagation

Naive gradient computation: Alternative (Neumann series) gradient formulation:

  • 1. Estimate
  • 2. Differentiate

Don’t need to store random number of terms in memory!!

  • 1. Analytically

Differentiate

  • 2. Estimate
slide-12
SLIDE 12

Density Estimation Experiments

Contribution Summary:

  • Unbiased estimator of log-likelihood.
  • Memory-efficient computation of log-likelihood.
  • LipSwish activation function [not discussed in talk].

(LipSwish)

slide-13
SLIDE 13

Density Estimation Experiments

Contribution Summary:

  • Unbiased estimator of log-likelihood.
  • Memory-efficient computation of log-likelihood.
  • LipSwish activation function [not discussed in talk].

(LipSwish)

slide-14
SLIDE 14

Qualitative Samples

CelebA: Data PixelCNN CIFAR10: Data Residual Flow Residual Flow Flow++

slide-15
SLIDE 15

Qualitative Samples

CelebA: CelebA-HQ 256x256: Data Residual Flow

slide-16
SLIDE 16

Thanks for Listening!

Jens Behrmann David Duvenaud Jörn-Henrik Jacobsen Code and pretrained models: https://github.com/rtqichen/residual-flows Co-authors: