Deep-Learning: Unsupervised Generative models Deep Belief Networks - PDF document

Deep-Learning: Unsupervised Generative models Deep Belief Networks Deep Stacked AutoEncoders Generative Adversarial Networks Pr. Fabien MOUTARDE Center for Robotics MINES ParisTech PSL Université Paris Fabien.Moutarde@mines-paristech.fr http://people.mines-paristech.fr/fabien.moutarde Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 1 Acknowledgements During preparation of these slides, I got inspiration and borrowed some slide content from several sources, in particular: • Fei-Fei Li & J. Johnson & S. Yeung: course on Generative Models http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture13.pdf • I. Kokkinos: slides of a CentraleParis course on Deep Belief Networks http://cvn.ecp.fr/personnel/iasonas/course/DL5.pdf • I. Goodfellow : NIPS’2016 tutorial on Generative Adversarial Networks (GANs) https://media.nips.cc/Conferences/2016/Slides/6202-Slides.pdf • Binglin, Shashank & Bhargav: A short tutorial on Generative Adversarial Networks (GANs) http://slazebni.cs.illinois.edu/spring17/lec11_gan.pdf Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 2

Outline • Unsupervised Learning and Generative Models • Deep Belief Networks (DBN) and Deep Boltzman Machine (DBM) • Autoencoders • Generative Adversarial Networks (GAN) Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 3 Deep vs Shallow Learning techniques overview DEEP GAN SHALLOW Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 4

Supervised vs Unsupervised Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 5 Unsupervised Learning Examples: General framework: Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 6

Generative models Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 7 Why Generative? Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 8

Why generative? Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 9 Taxonomy of Generative Models Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 10

Outline • Unsupervised Learning and Generative Models • Deep Belief Networks (DBN) and Deep Boltzman Machine (DBM) • Autoencoders • Generative Adversarial Networks (GAN) Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 11 Deep Belief Networks (DBN) • One of first Deep-Learning models • Proposed by G. Hinton in 2006 • Generative probabilistic model (mostly UNSUPERVISED) For capturing high-order correlations of observed/visible data ( à pattern analysis, or synthesis); and/or characterizing joint statistical distributions of visible data Greedy successive UNSUPERVISED learning of layers of Restricted Boltzmann Machine (RBM) Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 12

Restricted Boltzmann Machine (RBM) h, hidden (~ latent variables) NB: connections are BI-DIRECTIONAL v, observed (with same weight) Modelling probability distribution as: with « Energy » E given by Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 13 Training RBM Finding q =(W,a,b) maximizing likelihood ! "#$ % & (v) of dataset S ó minimize NegLogLikelihood ' * +#, log % & (-) Independance within layers è % - 5) = A % - 3 5 and % 5 -) = A % 5 2 - 3 2 . / = argMin ' 0 0 log % & (- 2 ) So objective = find & +#, 1 % - 3 = 4 5) = 6 7 3 8 9 :;3 5 > ? In binary input case: with 6 < = > ? 8 4 % 5 2 = 4 -) = 6 @ 2 8 9 2;: - Algo: Contrastive Divergence » Gibbs sampling used inside a gradient descent procedure Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 14

Contrastive Divergence algo Repeat: 1. Take a training sample v , compute B C 1 = D +) = E F 1 8 G 1;: + and sample a vector h from this probability distribution 2. Compute positive gradient as outer product H I = +JC = +C K 3. From h , compute B +L N = D C) = E O N 8 G :;N C and sample reconstructed v' , then resample h' using B CL 1 = D +L) = E F 1 8 G 1;: +L [Gibbs sampling single step; should theoretically be repeated until convergence ] 4. Compute negative gradient as outer product H P = +LJCL = +LCL K 5. Update weight matrix by QG = R H I ' H P = R +C K ' + S CL K 6. Update biases a and b analogously: QO = R + ' +L and QF = R C ' CL Gibbs sampling Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 15 Use of trained RBM • Input data "completion" : set some v i then compute h, and generate compatible full samples • Generating representative samples • Classification if trained with inputs=data+label Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 16

Modeling of input data distribution from trained RBM Initial data is in blue, reconstructed in red (and green line connects each data point with reconstructed one). Learnt energy function: minima created where data points are Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 17 Interpretation of trained RBM hidden layer • Look at weights of hidden nodes à low-level features Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 18

Why go deeper with DBN ? DBN: upper layers à more « abstract » features Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 19 Learning of DBN Greedy learning of successive layers Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 20

Using low-dim final features for clustering Much better results than clustering in input space or using other dimension reduction (PCA, etc…) Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 21 Example application of DBN: Clustering of documents in database Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 22

Image Retrieval application example of DBN Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 23 DBN supervised tuning SUPERVISED UNSUPERVISED Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 24

Outline • Unsupervised Learning and Generative Models • Deep Belief Networks (DBN) and Deep Boltzman Machine (DBM) • Autoencoders • Generative Adversarial Networks (GAN) Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 25 Autoencoders Learn q q and p F in order to minimize reconstruction cost : X W U 'W U X = 0 T = 0 V B Y Z [ W U 'W U U U à unsupervised learning of latent variables, and of a generative model Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 26

Variants of autoencoders • Denoising autoencoders • Sparse autoencoders • Stochastic autoencoders • Contractive autoencoders • VARIATIONAL autoencoders • … Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 27 Deep Stacked Autoencoders Proposed by Yoshua Bengio in 2007 Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 28

Training of Stacked Autoencoers etc… Greedy layerwise training: for each layer k, use backpropagation to minimize || A k (h (k) )-h (k) || 2 (+ regularization cost l S ij |W ij | 2 ) possibly + additional term for "sparsity" Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 29 Variational AutoEncoders (VAE) KL = Kullback-Leibler divergence (a.k.a. ‘relative entropy’) KL(Q || P) measures how different are distributions Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 30