RESTRICTED BOLTZMANN MACHINES DANIEL KOHLSDORF LAST LECTURE: DEEP - PowerPoint PPT Presentation

RESTRICTED BOLTZMANN MACHINES DANIEL KOHLSDORF

LAST LECTURE: DEEP AUTO ENCODERS Directed Model Reconstructs the input Back propagation Today: Probabilistic Interpretation Undirected Model

DIRECTED VS UNDIRECTED MODELS VS

PROBABILISTIC UNDIRECTED MODELS

PRELIMINARIES: MARKOV RANDOM FIELD Probability Distribution Cliques

RESTRICTED BOLTZMANN MACHINE Hinton: A Practical Guide to Training Restricted Boltzmann Machines

GIBBS SAMPLING

GIBBS SAMPLING FOR RBM h 0 ~ p(h 0 | v 0, v 1, v 2, v 3, h 1, h 2 ) h 1 ~ p(h 1 | v 0, v 1, v 2, v 3, h 0, h 2 ) h 2 ~ p(h 2 | v 0, v 1, v 2, v 3, h 1, h 0 ) h 0, h 1, h 2 are independent

GIBBS SAMPLING FOR RBM h 0 ~ p(h 0 | v 0, v 1, v 2, v 3 ) h 1 ~ p(h 1 | v 0, v 1, v 2, v 3 ) h 2 ~ p(h 2 | v 0, v 1, v 2, v 3 )

RBM HIDDEN CONDITIONAL p(h 0 | v 0, v 1, v 2, v 3 )

RBM HIDDEN CONDITIONAL h 0 = p(h 0 = 1 | v 0, v 1, v 2, v 3 ) > Uniform(0, 1) h 1 = p(h 1 = 1 | v 0, v 1, v 2, v 3 ) > Uniform(0, 1) h 2 = p(h 2 = 1 | v 0, v 1, v 2, v 3 ) > Uniform(0, 1)

RBM VISIBLE CONDITIONAL v 0 = p(v 0 = 1 | h 0, h 1, h 2 ) > Uniform(0, 1) v 1 = p(v 1 = 1 | h 0, h 1, h 2 ) > Uniform(0, 1) v 2 = p(v 2 = 1 | h 0, h 1, h 2 ) > Uniform(0, 1) v 3 = p(v 3 = 1 | h 0, h 1, h 2 ) > Uniform(0, 1)

ALTERNATE GIBBS SAMPLING h 0 , h 1 , h 2 h 0 , h 1 , h 2 … v 0 , v 1 , v 2, v 3 v 0 , v 1 , v 2, v 3

LEARNING h 0 , h 1 , h 2 h 0 , h 1 , h 2 … v 0 , v 1 , v 2, v 3 v 0 , v 1 , v 2, v 3 Input Reconstruction Weight Update

LEARNING: CONTRASTIVE DIVERGENCE Just do it once! h 0 , h 1 , h 2 v 0 , v 1 , v 2, v 3 v 0 , v 1 , v 2, v 3 Reconstruction Input

Hinton, G. E. and Salakhutdinov, R. R. (2006) DEEP! Reducing the dimensionality of data with neural networks. Science

CONVOLUTIONAL RESTRICTED BOLTZMANN MACHINES Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y. Ng.

Edge Detector Gaussian From Aaron

From Aaron

Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y. Ng.

Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y. Ng.

Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y. Ng.

Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y. Ng.

Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y. Ng.