Autoencoders Lecture slides for Chapter 14 of Deep Learning - - PowerPoint PPT Presentation

Autoencoders

Lecture slides for Chapter 14 of Deep Learning www.deeplearningbook.org Ian Goodfellow 2016-09-30

(Goodfellow 2016)

Structure of an Autoencoder

x r h f g

Figure 14.1 Input Hidden layer (code) Reconstruction

(Goodfellow 2016)

Stochastic Autoencoders

x r h pencoder(h | x) pdecoder(x | h)

Figure 14.2

(Goodfellow 2016)

Avoiding Trivial Identity

• Undercomplete autoencoders
• h has lower dimension than x
• f or g has low capacity (e.g., linear g)
• Must discard some information in h
• Overcomplete autoencoders
• h has higher dimension than x
• Must be regularized

(Goodfellow 2016)

Regularized Autoencoders

• Sparse autoencoders
• Denoising autoencoders
• Autoencoders with dropout on the hidden layer
• Contractive autoencoders

(Goodfellow 2016)

Sparse Autoencoders

• Limit capacity of autoencoder by adding a term to the cost

function penalizing the code for being larger

• Special case of variational autoencoder
• Probabilistic model
• Laplace prior corresponds to L1 sparsity penalty
• Dirac variational posterior

(Goodfellow 2016)

Denoising Autoencoder

˜ x ˜ x L h f g x C(˜ x | x)

Figure 14.3 C: corruption process (introduce noise)

ss L = − log pdecoder(x | h = f(˜ x)), xample x, obtained through a given corr

(Goodfellow 2016)

Denoising Autoencoders Learn a Manifold

x ˜ x g f ˜ x C(˜ x | x) x

Figure 14.4

(Goodfellow 2016)

Score Matching

• Score:
• Fit a density model by matching score of model to

score of data

• Some denoising autoencoders are equivalent to score

matching applied to some density models

rx log p(x). (14.15)

(Goodfellow 2016)

Vector Field Learned by a Denoising Autoencoder

Figure 14.5

(Goodfellow 2016)

Tangent Hyperplane of a Manifold

Figure 14.6

(Goodfellow 2016)

Learning a Collection of 0-D Manifolds by Resisting Perturbation

x0 x1 x2 x 0.0 0.2 0.4 0.6 0.8 1.0 r(x)

Identity Optimal reconstruction

Figure 14.7

(Goodfellow 2016)

Non-Parametric Manifold Learning with Nearest-Neighbor Graphs

Figure 14.8: Non-parametric manifold learning procedures build a nearest neighbor grap

Figure 14.8

(Goodfellow 2016)

Tiling a Manifold with Local Coordinate Systems

Figure 14.9

(Goodfellow 2016)

Contractive Autoencoders

Ω(h) = λ

• ∂f(x)

∂x

• 2

F

. (14.18)

Input point Tangent vectors Local PCA (no sharing across regions) Contractive autoencoder

Figure 14.10