Deep-Learning: Unsupervised Generative models Deep Belief Networks - - PDF document

▶
deep learning
SMART_READER_LITE
LIVE PREVIEW

Deep-Learning: Unsupervised Generative models Deep Belief Networks - - PDF document

Jun 29, 2023 529 likes •782 views

Deep-Learning: Unsupervised Generative models Deep Belief Networks Deep Stacked AutoEncoders Generative Adversarial Networks Pr. Fabien MOUTARDE Center for Robotics MINES ParisTech PSL Universit Paris Fabien.Moutarde@mines-paristech.fr

slide-1
SLIDE 1

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 1

Deep-Learning:

Unsupervised Generative models

Deep Belief Networks Deep Stacked AutoEncoders Generative Adversarial Networks

  • Pr. Fabien MOUTARDE

Center for Robotics MINES ParisTech PSL Université Paris

Fabien.Moutarde@mines-paristech.fr http://people.mines-paristech.fr/fabien.moutarde

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 2

Acknowledgements

During preparation of these slides, I got inspiration and borrowed some slide content from several sources, in particular:

  • Fei-Fei Li & J. Johnson & S. Yeung: course on Generative Models

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture13.pdf

  • I. Kokkinos: slides of a CentraleParis course on Deep Belief Networks

http://cvn.ecp.fr/personnel/iasonas/course/DL5.pdf

  • I. Goodfellow: NIPS’2016 tutorial on Generative Adversarial Networks (GANs)

https://media.nips.cc/Conferences/2016/Slides/6202-Slides.pdf

  • Binglin, Shashank & Bhargav: A short tutorial on Generative Adversarial

Networks (GANs) http://slazebni.cs.illinois.edu/spring17/lec11_gan.pdf

slide-2
SLIDE 2

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 3

Outline

  • Unsupervised Learning and Generative Models
  • Deep Belief Networks (DBN)

and Deep Boltzman Machine (DBM)

  • Autoencoders
  • Generative Adversarial Networks (GAN)

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 4

Deep vs Shallow Learning techniques overview

DEEP

SHALLOW

GAN

slide-3
SLIDE 3

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 5

Supervised vs Unsupervised

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 6

Unsupervised Learning

Examples: General framework:

slide-4
SLIDE 4

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 7

Generative models

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 8

Why Generative?

slide-5
SLIDE 5

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 9

Why generative?

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 10

Taxonomy of Generative Models

slide-6
SLIDE 6

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 11

Outline

  • Unsupervised Learning and Generative Models
  • Deep Belief Networks (DBN)

and Deep Boltzman Machine (DBM)

  • Autoencoders
  • Generative Adversarial Networks (GAN)

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 12

Deep Belief Networks (DBN)

  • One of first Deep-Learning models
  • Proposed by G. Hinton in 2006
  • Generative probabilistic model (mostly UNSUPERVISED)

For capturing high-order correlations of

  • bserved/visible data (à pattern analysis,
  • r synthesis); and/or characterizing

joint statistical distributions of visible data

Greedy successive UNSUPERVISED learning of layers

  • f Restricted Boltzmann Machine (RBM)
slide-7
SLIDE 7

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 13

Restricted Boltzmann Machine (RBM)

h, hidden (~ latent variables) v, observed

Modelling probability distribution as: with « Energy » E given by

NB: connections are BI-DIRECTIONAL (with same weight)

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 14

Training RBM

Finding q=(W,a,b) maximizing likelihood !"#$ %&(v) of dataset S

ó minimize NegLogLikelihood ' *+#, log %&(-)

So objective = find

./ = argMin

&

' 0

+#, 1

log %&(-2)

Algo: Contrastive Divergence

» Gibbs sampling used inside a gradient descent procedure In binary input case: with % -3 = 4 5) = 6 73 8 9

:;35

6 < = >? >? 8 4

% 52 = 4 -) = 6 @

2 8 9 2;:-

Independance within layers è % - 5) = A

3

% -3 5 % 5 -) = A

2

% 52 -

and

slide-8
SLIDE 8

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 15

Repeat:

  • 1. Take a training sample v, compute B C1 = D +) = E F1 8 G1;:+

and sample a vector h from this probability distribution

  • 2. Compute positive gradient as outer product HI = +JC = +CK
  • 3. From h, compute B +LN = D C) = E ON 8 G:;NC and sample reconstructed v',

then resample h' using B CL1 = D +L) = E F1 8 G1;:+L

[Gibbs sampling single step; should theoretically be repeated until convergence]

  • 4. Compute negative gradient as outer product HP = +LJCL = +LCLK
  • 5. Update weight matrix by QG = R HI ' HP = R +CK ' +SCLK
  • 6. Update biases a and b analogously: QO = R + ' +L and QF = R C ' CL

Contrastive Divergence algo

Gibbs sampling

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 16

Use of trained RBM

  • Input data "completion" : set some vi then

compute h, and generate compatible full samples

  • Generating representative samples
  • Classification if trained

with inputs=data+label

slide-9
SLIDE 9

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 17

Modeling of input data distribution from trained RBM

Initial data is in blue, reconstructed in red

(and green line connects each data point with reconstructed one).

Learnt energy function: minima created where data points are

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 18

Interpretation of trained RBM hidden layer

  • Look at weights of hidden nodes à low-level features
slide-10
SLIDE 10

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 19

Why go deeper with DBN ?

DBN: upper layers à more « abstract » features

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 20

Learning of DBN

Greedy learning of successive layers

slide-11
SLIDE 11

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 21

Using low-dim final features for clustering

Much better results than clustering in input space

  • r using other dimension reduction (PCA, etc…)

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 22

Example application of DBN: Clustering of documents in database

slide-12
SLIDE 12

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 23

Image Retrieval application example of DBN

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 24

DBN supervised tuning

UNSUPERVISED SUPERVISED

slide-13
SLIDE 13

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 25

Outline

  • Unsupervised Learning and Generative Models
  • Deep Belief Networks (DBN)

and Deep Boltzman Machine (DBM)

  • Autoencoders
  • Generative Adversarial Networks (GAN)

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 26

Autoencoders

Learn qq and pF in order to minimize reconstruction cost:

à unsupervised learning of latent variables, and of a generative model

T = 0

U

V WU 'WU X = 0

U

BY Z[ WU 'WU

X

slide-14
SLIDE 14

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 27

Variants of autoencoders

  • Denoising autoencoders
  • Sparse autoencoders
  • Stochastic autoencoders
  • Contractive autoencoders
  • VARIATIONAL autoencoders
  • …

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 28

Deep Stacked Autoencoders

Proposed by Yoshua Bengio in 2007

slide-15
SLIDE 15

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 29

Training of Stacked Autoencoers

Greedy layerwise training:

for each layer k, use backpropagation to minimize || Ak(h(k))-h(k) ||2

(+ regularization cost l Sij |Wij|2) possibly + additional term for "sparsity"

etc…

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 30

Variational AutoEncoders (VAE)

KL = Kullback-Leibler divergence (a.k.a. ‘relative entropy’)

KL(Q || P) measures how different are distributions

slide-16
SLIDE 16

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 31

Outline

  • Unsupervised Learning and Generative Models
  • Deep Belief Networks (DBN)
  • Autoencoders
  • Generative Adversarial Networks (GAN)

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 32

Generative Adversarial Network

Goal: generate « artificial » but credible examples

credible = sampled from same probability distribution p(x)

Idea: instead of trying to explicitly estimate p(x),

  • 1. LEARN a transformation G from a simple and known

distribution (e.g. random) into X,

  • 2. then sampling z à generate realistic samples G(z)

[Introduced in 2014 by Ian Goodfellow et al. (incl. Yoshua Bengio) from University of Montreal]

slide-17
SLIDE 17

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 33

GAN’s architecture

(Gaussian/Uniform). Z ~ latent representation of the image.

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 34

GAN training: minimax two-player game!

Joint training of D and G

slide-18
SLIDE 18

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 35

GAN training detail

In practice, alternate Discriminator training (gradient ascent) and Generator training:

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 36

Training the Discriminator

slide-19
SLIDE 19

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 37

Training the Generator

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 38

Convolutional Generator for GAN

slide-20
SLIDE 20

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 39

Example of fake samples generated by GAN

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 40

Trajectory in latent space à continous image transform

slide-21
SLIDE 21

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 41

« Arithmetic »

  • f latent vectors

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 42

Image-to-Image translation

Link to an interactive demo of this paper

slide-22
SLIDE 22

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 43

GAN for synthesis of realistic images

"Video-to-Video Synthesis", NeurIPS’2018 [Nvidia+MIT] Using Generative Adversarial Network (GAN)

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 44

Domain transfer!

slide-23
SLIDE 23

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 45

Summary and perspectives

  • n DBN/DBM/DSA/VAE/GAN
  • Intrinsicly UNSUPERVISED

è can be used on UNLABELLED DATA

  • Impressive results in Image Retrieval
  • DBN/DBM/VAE = Generative probabilistic models
  • GAN = most promising generative model, with

already many remarkable & exciting applications

  • Strong potential for enhancement of datasets and

for ultra-realistic synthetic data

  • Interest for "creative« /artistic computing?

Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 46

Any QUESTIONS ?