Convolutional Networks Instructor: John Thickstun Discussion Board: - - PowerPoint PPT Presentation

Discussion Board: Available on Ed Zoom Link: Available on Canvas Instructor Contact: thickstn@cs.washington.edu Course Webpage: https://courses.cs.washington.edu/courses/cse599i/20au/ CSE 599I: Generative Models University of Washington Lecture 7 Instructor: John Thickstun

Convolutional Networks

Recap: Variational Autoencoders

CSE 599I: Generative Models University of Washington Lecture 7

• Generative model . Learn , where
• Estimate the MLE using the ELBO:
• Modeling choices: prior , likelihood , and proposal .

ˆ θmle ≡ arg max

θ

E

x∼p log pθ(x) = arg max θ

sup

qϕ

E

x∼p z∼qϕ(·|x)

log pθ(x, z) qϕ(z|x) .

pθ(x, z) = pθ(x|z)r(z)

pθ(x) ≈ p(x)

pθ(x) = Ez∼r[pθ(x|z)] = Z

Z

pθ(x|z)r(z) dz.

r(z)

pθ(x|z)

qϕ(z|x)

Recap: The Gaussian VAE

CSE 599I: Generative Models University of Washington Lecture 7

• Use a prior
• Gaussian likelihood
• Decoder
• Gaussian posterior approximation
• Encoder
• How to parameterize the neural networks? What is the structure of the data?

pθ(x|z) = N(x; gθ(z), σ2

θ(z)I).

x = gθ(z) + σθ(z)εx, where εx ∼ N(0, IX ).

r(z) = N(0, IZ).

qϕ(z|x) = N(z; fϕ(x), Σϕ(x)).

z = fϕ(x) + Σ1/2

ϕ (x)εz, where εz ∼ N(0, IZ).

Image Modeling

CSE 599I: Generative Models University of Washington Lecture 7

• Represent an image as a tensor
• For R/G/B images, C = 3; for grayscale C = 1.
• Normalize color intensities
• Color intensities at position (i,j) constitute a pixel.
• Can discretize intensity values, e.g. 8-bit. color.

x

x ∈ RC×w×h.

➡ Color channels C, height h, width w.

xc,i,j ∈ [0, 1].

Vahdat and Kautz (Preprint 2020)

Convolutional Networks

CSE 599I: Generative Models University of Washington

• A convolutional layer is a family of functions
• A convolutional neural network (convnet, CNN) is a stack of blocks:
• Compare this spatial transformation to the sequential transformer model.
• 2-d spatial structure isn’t so important: generalizes to 1-d or 3-d convnet.

Lecture 7

b : RC×d×d → RC×d×d.

f = b(L) · · · b(1) : RC×d×d ! RC×d×d.

Convolutional Layer

CSE 599I: Generative Models University of Washington

• A convolutional layer is a family of functions
• If then where
• And the Pad function is defined by

Lecture 7

x ∈ RCin×d×d

fθ(x) = z

zc,i,j = ReLU Cin X

c0=1

hWc,c0, Pad(x)i:i+k,j:j+ki ! , W 2 RCout×Cin×k×k.

fθ : RCin×d×d → RCout×d×d.

Pad(x)d,i,j = ( xd,ibk/2c,jbk/2c if i bk/2c and j bk/2c,

• therwise.

Convolutional Layer

CSE 599I: Generative Models University of Washington Lecture 7

Dumoulin and Visin, 2019

• Visualizing application of a filter to a

single input channel

• Zero-padding ensures that the output

has the same dimensions as the input .

• Compare this spatial transformation to the

sequential transformer model.

• Easy generalization to 1-d or 3-d convnet.

Wc

xc.

z

x

Deep Residual Convnets

CSE 599I: Generative Models University of Washington

• A residual convolutional block is a family
• If then where

Lecture 7

fθ(x) = z

fθ : RCin×d×d → RCout×d×d.

x ∈ RC×d×d

He et. al., CVPR 2016 (C = 64, k = 3)

Transformation Weights

u0

c,i,j = C

X

c0=1

hW 1

c,c0, Pad(x)i:i+k,j:j+ki,

W 1 2 RC⇥C⇥k⇥k, u = ReLU (BatchNorm(u0; γ1, β1)) , γ1, β1 2 RC, z0

c,i,j = C

X

c0=1

hW 2

c,c0, Pad(u)i:i+k,j:j+ki,

W 2 2 RC⇥C⇥k⇥k. z = ReLU (x + BatchNorm(z0); γ2, β2) , γ2, β2 2 RC.

Batch Normalization

CSE 599I: Generative Models University of Washington Transformation Weights Lecture 7

• Normalize activations of a batch to have learned mean and variance .
• Applied per-batch. Compute mean, variance of full dataset after training.

β

γ2

BatchNorm(z; γ, β)i,c = γc (zi,c − µz,c) σz,c + βc, γ, β ∈ RC. µz = 1 B

B

X

i=1

zi, σz = v u u t1 k

B

X

i=1

(zi − µz)2.

A Modern Convnet Block

CSE 599I: Generative Models University of Washington

• If then where

Lecture 7

fθ(x) = z

x ∈ RC×d×d

He et. al., CVPR 2016 (C = 256, D = 64, k = 3)

Transformation Weights

u0

c,i,j = C

X

c0=1

hW 1

c,c0, xi:i+1,j:j+1i,

W 1 2 RC⇥D⇥1⇥1, u = ReLU (BatchNorm(u0; γ1, β1)) , γ1, β1 2 RD, v0

c,i,j = D

X

c0=1

hW 2

c,c0, Pad(u)i:i+k,j:j+ki,

W 2 2 RD⇥D⇥k⇥k, v = ReLU (BatchNorm(v0; γ1, β1)) , γ2, β2 2 RD, z0

c,i,j = D

X

c0=1

hW 3

c,c0, Pad(v)i:i+1,j:j+1i,

W 3 2 RD⇥C⇥1⇥1. z = ReLU (x + BatchNorm(z0); γ3, β3) , γ2, β2 2 RC.

5-Minute Break

CSE 599I: Generative Models University of Washington Lecture 7

Convnet Hyper-Parameters

CSE 599I: Generative Models University of Washington Lecture 7

• Capacity of the network C.
• Inner convolutional capacity D.
• Convolution kernel size k.
• Depth of the convolution stack L.
• Size of a minibatch B (possible interaction with BatchNorm).
• All the usual hyper-parameters: dropout, l2, learning rates, initialization…

Parameterizing the VAE: Decoder

CSE 599I: Generative Models University of Washington Lecture 7

• Gaussian likelihood
• Let be a linear function.
• Let be a convnet.
• Let be a 1x1 convolutional layer.
• Let be a linear function (or just a constant).
• Define and

pθ(x|z) = N(x; gθ(z), σ2

θ(z)I).

fin : Z → RC×d×d

hθ : RC×d×d → RC×d×d

fmean : RC×d×d → RCout×d×d

fvar : RC×d×d → R

σθ(z) = fvar hθ fin(z).

gθ(z) = fmean hθ fin(z)

Parameterizing the VAE: Encoder

CSE 599I: Generative Models University of Washington Lecture 7

• Gaussian posterior approximation
• Let be a 1x1 convolution layer.
• Let be a convnet.
• Let be a linear function.
• Let be a linear function.
• Define and

fin : RCin×d×d → RC×d×d

hϕ : RC×d×d → RC×d×d

fmean : RC×d×d → Z

fvar : RC×d×d → Z × Z

qϕ(z|x) = N(z; fϕ(x), Σϕ(x)).

fϕ(x) = fmean hϕ fin(x)

Σ1/2

ϕ (x) = fvar hθ fin(x).

VAE Training

CSE 599I: Generative Models University of Washington Lecture 7