Representation Learning UCA Deep Learning School - Deep in France - - PowerPoint PPT Presentation

▶
representation learning
SMART_READER_LITE
LIVE PREVIEW

Representation Learning UCA Deep Learning School - Deep in France - - PowerPoint PPT Presentation

Aug 16, 2023 431 likes •1.31k views

Representation Learning UCA Deep Learning School - Deep in France Nice 2017 Soufiane Belharbi Romain Hrault Clment Chatelain Sbastien Adam soufiane.belharbi@insa-rouen.fr LITIS lab., Apprentissage team - INSA de Rouen, France 13 June,

slide-1
SLIDE 1

Representation Learning

UCA Deep Learning School - Deep in France Nice 2017

Soufiane Belharbi Romain Hérault Clément Chatelain Sébastien Adam

soufiane.belharbi@insa-rouen.fr

LITIS lab., Apprentissage team - INSA de Rouen, France 13 June, 2017 LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning

slide-2
SLIDE 2

My PhD work

3rd year PhD student at LITIS lab. Deep learning, structured output prediction, learning representations.

1

  • S. Belharbi, C. Chatelain, R.Hérault, S. Adam, Learning Structured

Output Dependencies Using Deep Neural Networks. 2015. in: Deep Learning Workshop in the 32nd International Conference on Machine Learning (ICML), 2015.

2

  • S. Belharbi, R.Hérault, C. Chatelain, S. Adam, Deep multi-task

learning with evolving weights, in: European Symposium on Artificial Neural 445 Networks (ESANN), 2016

3

  • S. Belharbi, C. Chatelain, R.Hérault, S. Adam, Multi-task Learning for

Structured Output Prediction. 2017. submitted to Neurocomputing. ArXiv: arxiv.org/abs/1504.07550.

4

  • S. Belharbi, R.Hérault, C. Chatelain, R. Modzelewski, S. Adam, M.

Chastan, S. Thureau, Spotting L3 slice in CT scans using deep convolutional network and transfer learning, in: Computers in Biology and Medicine, 2017. Current work: Learning invariance within neural networks.

  • S. Belharbi, C. Chatelain, R.Hérault, S. Adam, Class-invariance hint: a

regularization framework for training neural networks. Coming up soon.

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 1/76

slide-3
SLIDE 3

Roadmap

1 Representation Learning 2 Sparse Coding 3 Auto-encoders 4 Restricted Boltzmann Machines (RBMs) 5 Conclusion

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 2/76

slide-4
SLIDE 4

Representation Learning

Representation Learning

Representation Learning is fundamental in Machine Learning How to represent the class “dog”? (input variations) Conference: ICLR www.iclr.cc (since 2013).

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 3/76

slide-5
SLIDE 5

Representation Learning

Representation Learning

Stanford, CS331B. LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 4/76

slide-6
SLIDE 6

Representation Learning

Representation Learning

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 5/76

slide-7
SLIDE 7

Representation Learning

Representation Learning

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 6/76

slide-8
SLIDE 8

Representation Learning

Features representation: Handcrafting

Let us build a cat detector . . .

Stanford, A.Zamir LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 7/76

slide-9
SLIDE 9

Representation Learning

Features representation: Handcrafting

Let us build a cat detector . . .

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 8/76

slide-10
SLIDE 10

Representation Learning

Features representation: Handcrafting

Let us build a cat detector . . .

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 9/76

slide-11
SLIDE 11

Representation Learning

Features representation: Handcrafting

Let us build a cat detector . . .

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 10/76

slide-12
SLIDE 12

Representation Learning

Features representation: Handcrafting

Handcrafted features . . . Pros: Was the only way for a long time. Works quite good. Sometimes you need to combine many features. Generic.

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 11/76

slide-13
SLIDE 13

Representation Learning

Features representation: Handcrafting

Handcrafted features . . . Cons: Generic. Time consuming. What you will do if nothing works? In many cases, it is difficult to build discriminative features.

Figure 2: Classifier: Happy vs Sad

Ideal: Application-dependent features ⇒ Deep Learning

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 12/76

slide-14
SLIDE 14

Representation Learning

Representation Learning Approaches

Two main approaches Supervised Unsupervised: Representation constrained on reconstruction

Stanford, A.Zamir

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 13/76

slide-15
SLIDE 15

Representation Learning

Representation Learning Approaches

Inverting a representation

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 14/76

slide-16
SLIDE 16

Representation Learning

Representation Learning Approaches

Inverting a representation

Dosovitskiy and Brox, 2015.

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 15/76

slide-17
SLIDE 17

Representation Learning

Representation Learning Approaches

Two main approaches Supervised

LeCun et al. 1998.

Unsupervised: Representation constrained on reconstruction

Hinton et al. 2006.

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 16/76

slide-18
SLIDE 18

Representation Learning

Representation Learning Approaches

Pros: Exploit millions of unlabeled data from the internet: Images. Text (Wikipedia, . . . ) Records and videos. Unsupervised: Representation constrained on reconstruction

Hinton et al. 2006.

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 17/76

slide-19
SLIDE 19

Representation Learning

Representation Learning Approaches

The reconstruction loss: how to reconstruct? L2 pixel loss. Applications: Data compression. Dimenstionality reduction. Pre-train neuranl networks (initialization). Unsupervised: Representation constrained on reconstruction

Hinton et al. 2006.

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 18/76

slide-20
SLIDE 20

Representation Learning

Unsupervised Representation Learning Methods

Sparse coding. Auto-encoders (AEs). Restricted Boltzmann Machines (RBMs).

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 19/76

slide-21
SLIDE 21

Sparse Coding

Sparse Coding

Objective: x =

k

  • i=1

aiφi, where φi is a set of basis (dictionnary). Cost function on a set of m input vectors: min

a(j)

i ,φi

m

  • j=1
  • x(j) −

k

  • i=1

a(j)

i φi

  • 2
  • reconstruction term

+ λ

k

  • i=1

S(a(j)

i )

  • sparsity term

. Similar to: min

a(j)

i ,φi

m

  • j=1
  • x(j) − H(j)W
  • 2
  • reconstruction term

+ λ||H(j)||1

  • sparsity term

.

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 20/76

slide-22
SLIDE 22

Sparse Coding

Sparse Coding

Andrew Ng.

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 21/76

slide-23
SLIDE 23

Sparse Coding

Sparse Coding

Andrew Ng.

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 22/76

slide-24
SLIDE 24

Auto-encoders

Auto-encoders

Q.V. Le LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 23/76

slide-25
SLIDE 25

Auto-encoders

Auto-encoders

Encoder: f(x) = s(Wx + b) = z. Decoder: g(z) = s(W ′ + b′) = x, W ′ = W T (tied weight). Objective over a set of n examples x: J(x; W, b, b′) = 1 n

n

  • i=1

||x − x||2. Similar to PCA.

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 24/76

slide-26
SLIDE 26

Auto-encoders

Auto-encoders

Keras blog.

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 25/76

slide-27
SLIDE 27

Auto-encoders

Auto-encoders

Example:

Keras blog.

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 25/76

slide-28
SLIDE 28

Auto-encoders

Deep Auto-encoders

wikidocs.

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 26/76

slide-29
SLIDE 29

Auto-encoders

Denoising Auto-encoders

Basic auto-encoder: J(x, W, b, b′) = 1 n

n

  • i=1

||x − s(W T(s(Wx + b))) + b′

  • x

||2 Denoising auto-encoder: build good representations by recovering a corrupted input. J(x, W, b, b′) = 1 n

n

  • i=1

||x − s(W T(s(Wφ(x) + b))) + b′

  • x

||2

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 27/76

slide-30
SLIDE 30

Auto-encoders

Denoising Auto-encoders

P .Vincent, 2010. LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 28/76

slide-31
SLIDE 31

Auto-encoders

Denoising Auto-encoders

Unsupervised Manifold hypothesis: Data in high dimensional spaces concentrate in sub-manifolds of much lower dimensionality.

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 29/76

slide-32
SLIDE 32

Auto-encoders

Denoising Auto-encoders

  • Manifolds. (G.Mesnil.)

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 30/76

slide-33
SLIDE 33

Auto-encoders

Denoising Auto-encoders

Manifold learning perspective. (P

.Vincent, 2010.) LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 31/76

slide-34
SLIDE 34

Auto-encoders

Denoising Auto-encoders

Left: filters of basic AE. Right: filters of DAE (Gaussian noise). (trained on natural images) (P

.Vincent, 2010.) LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 32/76

slide-35
SLIDE 35

Auto-encoders

Denoising Auto-encoders

filters of DAE (zero-masking noise). (trained on MNIST) (P

.Vincent, 2010.) LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 33/76

slide-36
SLIDE 36

Auto-encoders

Stacked Denoising Auto-encoders

Stacked denoising AEs. (pre-training) (P

.Vincent, 2010.) LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 34/76

slide-37
SLIDE 37

Auto-encoders

Stacked Denoising Auto-encoders

Stacked denoising AEs. (fine-tunning) (P

.Vincent, 2010.) LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 35/76

slide-38
SLIDE 38

Auto-encoders

Layer-wise pre-training: auto-encoders

x1 x2 x3 x4 x5 x6 ˆ y1 ˆ y2 A DNN to train

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 36/76

slide-39
SLIDE 39

Auto-encoders

Layer-wise pre-training: auto-encoders

x1 x2 x3 x4 x5 x6 ˆ x1 ˆ x2 ˆ x3 ˆ x4 ˆ x5 ˆ x6

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 37/76

1) Step 1: Unsupervised layer-wise pre-training

Train layer by layer sequentially using only x (labeled or unlabeled)

slide-40
SLIDE 40

Auto-encoders

Layer-wise pre-training: auto-encoders

x1 x2 x3 x4 x5 x6 h1,1 h1,2 h1,3 h1,4 h1,5

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 37/76

1) Step 1: Unsupervised layer-wise pre-training

Train layer by layer sequentially using only x (labeled or unlabeled)

slide-41
SLIDE 41

Auto-encoders

Layer-wise pre-training: auto-encoders

x1 x2 x3 x4 x5 x6 h1,1 h1,2 h1,3 h1,4 h1,5 ˆ h1,1 ˆ h1,2 ˆ h1,3 ˆ h1,4 ˆ h1,5

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 37/76

1) Step 1: Unsupervised layer-wise pre-training

Train layer by layer sequentially using only x (labeled or unlabeled)

slide-42
SLIDE 42

Auto-encoders

Layer-wise pre-training: auto-encoders

x1 x2 x3 x4 x5 x6 h2,1 h2,2 h2,3

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 37/76

1) Step 1: Unsupervised layer-wise pre-training

Train layer by layer sequentially using only x (labeled or unlabeled)

slide-43
SLIDE 43

Auto-encoders

Layer-wise pre-training: auto-encoders

x1 x2 x3 x4 x5 x6 h2,1 h2,2 h2,3 ˆ h2,1 ˆ h2,2 ˆ h2,3

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 37/76

1) Step 1: Unsupervised layer-wise pre-training

Train layer by layer sequentially using only x (labeled or unlabeled)

slide-44
SLIDE 44

Auto-encoders

Layer-wise pre-training: auto-encoders

x1 x2 x3 x4 x5 x6 h3,1 h3,2 h3,3

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 37/76

1) Step 1: Unsupervised layer-wise pre-training

Train layer by layer sequentially using only x (labeled or unlabeled)

slide-45
SLIDE 45

Auto-encoders

Layer-wise pre-training: auto-encoders

x1 x2 x3 x4 x5 x6

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 37/76

1) Step 1: Unsupervised layer-wise pre-training

Train layer by layer sequentially using only x (labeled or unlabeled)

slide-46
SLIDE 46

Auto-encoders

Layer-wise pre-training: auto-encoders

x1 x2 x3 x4 x5 x6 ˆ y1 ˆ y2

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 38/76

2) Step 2: Supervised training

Train the whole network using (x, y)

Back-propagation

slide-47
SLIDE 47

Auto-encoders

Generating Samples with SAE, SDAE

P .Vincent, 2010. LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 39/76

slide-48
SLIDE 48

Auto-encoders

Generating Samples with SAE, SDAE

P .Vincent, 2010. LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 40/76

slide-49
SLIDE 49

Auto-encoders

Sparse Auto-encoders

Cost = reconstruction + sparsity Objective: build sparse features (most components are zeros). Example: K-sparse AEs. (Makhzani et al. 13) ˆ zi = argmin

z

xi − Wz2

2 s.t. z0 < k

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 41/76

slide-50
SLIDE 50

Auto-encoders

Sparse Auto-encoders

Filters of the k -sparse auto-encoder for different sparsity levels k, learnt from MNIST. (A. Makhzani, 2013.)

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 42/76

slide-51
SLIDE 51

Auto-encoders

Variational Auto-encoders

How to generate samples?

kvfrans. LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 43/76

slide-52
SLIDE 52

Auto-encoders

Variational Auto-encoders

Sample AE. (no constraints on the latent code)

kvfrans. LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 44/76

slide-53
SLIDE 53

Auto-encoders

Variational Auto-encoders

VAE: AE with constraints on the latent code. (unit Gaussian distribution)

kvfrans. LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 45/76

slide-54
SLIDE 54

Auto-encoders

Variational Auto-encoders

Sampling from VAE.

C.Doersch, 2016. LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 46/76

slide-55
SLIDE 55

Auto-encoders

Variational Auto-encoders

Fictional celebrity faces generated by a variational autoencoder.

Link to video: https://www.youtube.com/watch?v=XNZIN7Jh3Sg. (J.Altosaar.)

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 47/76

slide-56
SLIDE 56

Auto-encoders

Contractive Auto-encoders

Regularize the AE so the features will be robust to slight in the input. (local invariance) cost = reconstruction + contraction      h = f(x) = sf(Wx + bh) (code) ||Jf(x)||2

F

=

ij

∂hj(x)

∂xi

2 (contraction) JCAE(x; θ) = 1

n

  • x∈Dn(||x −

x||2 + λ||Jf(x)||2

F)

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 48/76

slide-57
SLIDE 57

Auto-encoders

Contractive Auto-encoders

Regularize the AE so the features will be robust to slight in the input. (local invariance) cost = reconstruction + contraction      h = f(x) = sf(Wx + bh) (code) ||Jf(x)||2

F

=

ij

∂hj(x)

∂xi

2 (contraction) JCAE(x; θ) = 1

n

  • x∈Dn(||x −

x||2 + λ||Jf(x)||2

F)

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 48/76

slide-58
SLIDE 58

Auto-encoders

Contractive Auto-encoders

Contraction ration: MNIST (top) and CIFAR-bw (bottom). (S.Rifai,

2011.) LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 49/76

slide-59
SLIDE 59

Auto-encoders

Contractive Auto-encoders

Contraction ration with respect to depth. (S.Rifai, 2011.)

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 50/76

slide-60
SLIDE 60

Auto-encoders

Higher Order Contractive Auto-encoders

cost = reconstruction+contraction

  • 1st derivative

+ curvature of the contraction

  • 2nd derivative

Objective: extract local charts (local directions of variations)

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 51/76

slide-61
SLIDE 61

Auto-encoders

Higher Order Contractive Auto-encoders

Unsupervised invariance to local transformations. (C.H.Martin.)

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 52/76

slide-62
SLIDE 62

Auto-encoders

Higher Order Contractive Auto-encoders

Use High CAE to extract local tangents directions of variations to ensure invariance of classifier to these directions. (S.Rifai, 2011.)

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 53/76

slide-63
SLIDE 63

Auto-encoders

Higher Order Contractive Auto-encoders

Learned tangents over MNIST and CIFAR-10. (S.Rifai, 2011.)

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 54/76

slide-64
SLIDE 64

Auto-encoders

Higher Order Contractive Auto-encoders

1

Train stack of CAE layers.

2

Compute the tangents of each example by SVD of

∂h1 ∂x .

     D = {x(1), . . . , x(n)} ⇓ D′ = {(x(1), ∂h1

∂x (x(1))), . . . , (x(1), ∂h1 ∂x (x(1)))}

(S.Rifai, 2011.)

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 55/76

slide-65
SLIDE 65

Auto-encoders

Higher Order Contractive Auto-encoders

1

Train stack of CAE layers.

2

Compute the tangents of each example by SVD of

∂h1 ∂x .

     D = {x(1), . . . , x(n)} ⇓ D′ = {(x(1), ∂h1

∂x (x(1))), . . . , (x(1), ∂h1 ∂x (x(1)))}

3

Train a logistic regression

  • n top of the CAE stack

with the tangent propagation penalty. (S.Rifai, 2011.)

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 56/76

slide-66
SLIDE 66

Auto-encoders

Next generation AEs

Toward: Adversarial Autoencoders, GANs+VAE, CNN+AEs.

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 57/76

slide-67
SLIDE 67

Restricted Boltzmann Machines (RBMs)

Restricted Boltzmann Machines

Boltzmann Machines: undirected graphical model. Fully connected graph. v: visible units. h: hidden units. G.Hinton et al. 85

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 58/76

slide-68
SLIDE 68

Restricted Boltzmann Machines (RBMs)

Restricted Boltzmann Machines

Restricted Boltzmann Machines The input x is not fully

  • bserved ⇒

x = v

  • visible part

+ h

  • hidden part

.

Figure 3: visible ⇐ ⇒ hidden.

Binary variables. (possible extension to continuous variables). (Wikipedia.)

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 59/76

slide-69
SLIDE 69

Restricted Boltzmann Machines (RBMs)

Restricted Boltzmann Machines

Restricted Boltzmann Machines

Figure 4: x ⇐ ⇒ h.

(H.Larochelle.)

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 60/76

slide-70
SLIDE 70

Restricted Boltzmann Machines (RBMs)

Restricted Boltzmann Machines

Restricted Boltzmann Machines

Figure 5: x ⇐ ⇒ h.

(H.Larochelle.)

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 61/76

slide-71
SLIDE 71

Restricted Boltzmann Machines (RBMs)

Restricted Boltzmann Machines

Restricted Boltzmann Machines RBMs are: Generative models. Model the density P(v) (i.e. P(input)). Energy based model P(v) = f(E(v, h)).

Figure 6: visible ⇐ ⇒ hidden.

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 62/76

slide-72
SLIDE 72

Restricted Boltzmann Machines (RBMs)

Restricted Boltzmann Machines

RBMs are energy based models. It associates an energy E(v, h) with every possible configuration of the system. Learning consists in modifying the energy shape. Physics: lower energy = more stability. (Y.LeCun)

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 63/76

slide-73
SLIDE 73

Restricted Boltzmann Machines (RBMs)

Restricted Boltzmann Machines

Learning: Modify the energy shape so that the desirable configurations have lower energy. (Y.LeCun)

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 64/76

slide-74
SLIDE 74

Restricted Boltzmann Machines (RBMs)

Restricted Boltzmann Machines

Training of RBMs: k-steps Contrastive Divergence (CD-k). Ingredients: Markov chain using Gibbs sampling. Gradient approximation (gradient descent). (H.Larochelle.)

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 65/76

slide-75
SLIDE 75

Restricted Boltzmann Machines (RBMs)

Restricted Boltzmann Machines

Training of RBMs: k-steps Contrastive Divergence (CD-k). Ingredients: Markov chain using Gibbs sampling. Gradient approximation (gradient descent). (H.Larochelle.)

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 66/76

slide-76
SLIDE 76

Restricted Boltzmann Machines (RBMs)

Restricted Boltzmann Machines

Training of RBMs: k-steps Contrastive Divergence (CD-k). Ingredients: Markov chain using Gibbs sampling. Gradient approximation (gradient descent). (H.Larochelle.)

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 67/76

slide-77
SLIDE 77

Restricted Boltzmann Machines (RBMs)

Restricted Boltzmann Machines

Figure 7: Learned weights of the hidden units of RBM over Mnist.

(H.Larochelle. 2009)

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 68/76

slide-78
SLIDE 78

Restricted Boltzmann Machines (RBMs)

Example 1: Stacked RBMs

Dimensionality reduction (Hinton et al. 06).

Figure 8: Stacked RBMs

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 69/76

slide-79
SLIDE 79

Restricted Boltzmann Machines (RBMs)

Example 1: Stacked RBMs

Dimensionality reduction (Hinton et al. 06).

Figure 9: Top: origin. middle: reconstructed SRBMs. Bottom: reconstructed PCA.

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 70/76

slide-80
SLIDE 80

Restricted Boltzmann Machines (RBMs)

Example 1: Stacked RBMs

Dimensionality reduction (Hinton et al. 06).

Figure 10: A: original training images (1st two dimensions of PCA). B: two dimensional codes of 784- 1000-500-250-2.

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 71/76

slide-81
SLIDE 81

Restricted Boltzmann Machines (RBMs)

Example 2: Collaborative Filtering

Netflix recommendation (Salakhutdinov et al. 07). see CF (*.gif): https://en.wikipedia.org/wiki/File: Collaborative_filtering.gif

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 72/76

slide-82
SLIDE 82

Restricted Boltzmann Machines (RBMs)

Example 2: Collaborative Filtering

Netflix recommendation (Salakhutdinov et al. 07). see CF (*.gif): https://en.wikipedia.org/wiki/File: Collaborative_filtering.gif

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 72/76

slide-83
SLIDE 83

Restricted Boltzmann Machines (RBMs)

Example 2: Collaborative Filtering

Netflix recommendation (Salakhutdinov et al. 07). see CF (*.gif): https://en.wikipedia.org/wiki/File: Collaborative_filtering.gif

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 73/76

slide-84
SLIDE 84

Restricted Boltzmann Machines (RBMs)

Example 2: Collaborative Filtering

Netflix recommendation (Salakhutdinov et al. 07). see CF (*.gif): https://en.wikipedia.org/wiki/File: Collaborative_filtering.gif

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 74/76

slide-85
SLIDE 85

Conclusion

Conclusion

The way data is represented is important. Deep learning offers task-dependant representations. Exploit the large amount of unlabeled data: unsupervised learning:

Auto-encoders. Restricted Boltzmann Machines.

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 75/76

slide-86
SLIDE 86

Conclusion

Questions

Thank you for your attention, Questions? soufiane.belharbi@insa-rouen.fr https://sbelharbi.github.io

LITIS lab., Apprentissage team - INSA de Rouen, France Representation Learning 76/76