composing graphical models with neural networks for
play

Composing graphical models with neural networks for structured - PowerPoint PPT Presentation

Aug 21, 2022 •342 likes •1.02k views

Composing graphical models with neural networks for structured representations and fast inference Matt Johnson, David Duvenaud, Alex Wiltschko, Bob Datta, Ryan Adams 6 0 6 0 6 0 5 0 5 0 5 0 4 0 4 0 4 0 m m m m 3 0 m 3 0

  1. Composing graphical models with neural networks for structured representations and fast inference Matt Johnson, David Duvenaud, Alex Wiltschko, Bob Datta, Ryan Adams

  3. 6 0 6 0 6 0 5 0 5 0 5 0 4 0 4 0 4 0 m m m m 3 0 m 3 0 m 3 0 2 0 2 0 2 0 1 0 1 0 1 0 0 10 0 0 m m m 10 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 mm 10 m mm 2 0 10 20 30 40 50 60 70 20 10 20 30 40 50 60 70 20 30 3 0 3 0 40 40 40 m m m m 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 10 20 30 40 50 60 70 10 20 30 40 50 60 70 dart pause rear [1,2] /b/ /ax/ /n/ /ae/ /n/ /ax/ 60 60 50 50 40 40 mm mm 30 30 20 20 10 10 0 0 10 mm 10 mm 10 20 30 40 50 60 70 20 10 20 30 40 50 60 70 20 30 30 40 40 mm mm 10 20 30 40 50 60 70 10 20 30 40 50 60 70 60 50 40 mm 30 20 10 0 10 mm mm 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 20 30 40 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 [1] Lee and Glass. A Nonparametric Bayesian Approach to Acoustic Model Discovery. ACL 2012. [2] Lee. Discovering Linguistic Structures in Speech: Models and Applications. MIT Ph.D. Thesis 2014.

  5. Alexander Wiltschko, Matthew Johnson , et al., Neuron 2015.

  6. image manifold

  7. depth video image manifold

  8. depth video image manifold

  9. depth video image manifold

  10. depth video image manifold

  11. depth video image manifold

  12. depth video image manifold

  13. depth video image manifold manifold coordinates dart rear

  14. Recurrent neural networks? [1,2,3] v 3 ˆ v 2 ˆ v 1 ˆ Learned Representation W 1 W 1 copy W 2 W 2 v 1 v 2 v 3 v 3 v 3 v 2 v 2 Figure 2. LSTM Autoencoder Model Figure 1. LSTM unit Probabilistic graphical models? [4,5,6] [1] Srivastava, Mansimov, Salakhutdinov. Unsupervised learning of video representations using LSTMs. ICML 2015. [2] Ranzato, MarcAurelio, et al. Video (language) modeling: a baseline for generative models of natural videos. Preprint 2015. [3] Sutskever, Hinton, and Taylor. The Recurrent Temporal Restricted Boltzmann Machine. NIPS 2008. [4] Fox, Sudderth, Jordan, Willsky. Bayesian nonparametric inference of switching dynamic linear models. IEEE TSP 2011. [5] Johnson and Willsky. Bayesian nonparametric hidden semi-Markov models. JMLR 2013. [6] Murphy. Machine learning: a probabilistic perspective. MIT Press 2012.

  21. unsupervised learning supervised learning

  22. Probabilistic graphical models Deep learning + structured representations – neural net “goo” + priors and uncertainty – difficult parameterization + data and computational efficiency – can require lots of data – rigid assumptions may not fit + flexible – feature engineering + feature learning – top-down inference + recognition networks

  24. Modeling idea: graphical models on latent variables, neural network models for observations Inference: recognition networks output conjugate potentials, then apply fast graphical model inference Application: learn syllable representation of behavior from video 60 60 60 50 50 50 40 40 40 mm mm mm 30 30 30 20 20 20 10 10 10 0 0 10 0 mm mm 10 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 mm 10 2 mm 0 10 20 30 40 50 60 70 2 10 20 30 40 50 60 70 0 2 0 30 30 30 40 40 40 m m m m 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 10 20 30 40 50 60 70 10 20 30 40 50 60 70

  25. Modeling idea: graphical models on latent variables, neural network models for observations

  26. π (1)   z t +1 ∼ π ( z t ) π (2) π =   π (3) z 2 z 3 z 4 z 5 z 6 z 7 z 1 A (1) A (2) A (3) iid x t +1 = A ( z t ) x t + B ( z t ) u t ∼ N (0 , I ) u t B (1) B (2) B (3) 60 60 60 50 50 50 40 40 40 mm mm mm 30 30 30 20 20 20 10 10 10 0 10 0 0 mm mm 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 10 mm 10 mm 20 10 20 30 40 50 60 70 10 20 30 40 50 60 70 20 20 30 30 30 4 0 4 0 4 0 mm mm 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 10 20 30 40 50 60 70 10 20 30 40 50 60 70

  27. π (1)   π (2) π =   π (3) z 1 z 2 z 3 z 4 z 5 z 6 z 7 x 1 x 3 x 2 x 4 x 5 x 6 x 7 A (1) A (2) A (3) B (1) B (2) B (3) 60 60 60 50 50 50 40 40 40 mm mm mm 30 30 30 20 20 20 10 10 10 0 10 0 0 mm mm 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 10 mm 10 mm 20 10 20 30 40 50 60 70 10 20 30 40 50 60 70 20 20 30 30 30 4 0 4 0 4 0 mm mm 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 10 20 30 40 50 60 70 10 20 30 40 50 60 70

  28. θ z 1 z 2 z 3 z 4 z 5 z 6 z 7 x 1 x 3 x 2 x 4 x 5 x 6 x 7 60 60 60 50 50 50 40 40 40 mm mm mm 30 30 30 20 20 20 10 10 10 0 10 0 0 mm mm 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 10 mm 10 mm 20 10 20 30 40 50 60 70 10 20 30 40 50 60 70 20 20 30 30 30 4 0 4 0 4 0 mm mm 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 10 20 30 40 50 60 70 10 20 30 40 50 60 70

  29. θ z 1 z 2 z 3 z 4 z 5 z 6 z 7 x 1 x 3 x 2 x 4 x 5 x 6 x 7 y 1 y 2 y 3 y 4 y 5 y 6 y 7 60 60 60 50 50 50 40 40 40 mm mm mm 30 30 30 20 20 20 10 10 10 0 10 0 0 mm mm 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 10 mm 10 mm 20 10 20 30 40 50 60 70 10 20 30 40 50 60 70 20 20 30 30 30 4 0 4 0 4 0 mm mm 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 10 20 30 40 50 60 70 10 20 30 40 50 60 70

  30. θ z 1 z 2 z 3 z 4 z 5 z 6 z 7 x 1 x 3 x 2 x 4 x 5 x 6 x 7 γ y 1 y 2 y 3 y 4 y 5 y 6 y 7 x t y t | x t , γ ∼ N ( µ ( x t ; γ ) , Σ ( x t ; γ )) µ ( x t ; γ ) diag( Σ ( x t ; γ ))

  31. θ θ θ z n z 1 z 2 z 3 z 4 x n γ x n x 1 x 2 x 3 x 4 γ γ y n y n y 1 y 2 y 3 y 4 p ( θ ) conjugate prior on global variables θ p ( x | θ ) exponential family on local variables x n γ p ( γ ) any prior on observation parameters y n p ( y | x, γ ) neural network observation model

  32. [1] [2] [3] [4] Gaussian mixture model Linear dynamical system Hidden Markov model Switching LDS [5] [2] [6] [7] Mixture of Experts Driven LDS IO-HMM Factorial HMM [8,9] [10] Canonical correlations analysis admixture / LDA / NMF [1] Palmer, Wipf, Kreutz-Delgado, and Rao. Variational EM algorithms for non-Gaussian latent variable models. NIPS 2005. [2] Ghahramani and Beal. Propagation algorithms for variational Bayesian learning. NIPS 2001. [3] Beal. Variational algorithms for approximate Bayesian inference, Ch. 3. U of London Ph.D. Thesis 2003. [4] Ghahramani and Hinton. Variational learning for switching state-space models. Neural Computation 2000. [5] Jordan and Jacobs. Hierarchical Mixtures of Experts and the EM algorithm. Neural Computation 1994. [6] Bengio and Frasconi. An Input Output HMM Architecture. NIPS 1995. [7] Ghahramani and Jordan. Factorial Hidden Markov Models. Machine Learning 1997. [8] Bach and Jordan. A probabilistic interpretation of Canonical Correlation Analysis. Tech. Report 2005. [9] Archambeau and Bach. Sparse probabilistic projections. NIPS 2008. [10] Hoffman, Bach, Blei. Online learning for Latent Dirichlet Allocation. NIPS 2010.

  33. θ x n Inference? γ y n

  34. θ θ x 3 x 1 x 2 x 4 x 3 x 1 x 2 x 4 y 1 y 2 y 3 y 4 p ( x | θ ) is linear dynamical system q ( θ ) q ( x ) ≈ p ( θ , x | y ) p ( y | x, θ ) is linear-Gaussian p ( θ ) is conjugate prior h i log p ( θ ,x,y ) L [ q ( θ ) q ( x ) ] , E q ( θ ) q ( x ) q ( θ ) q ( x ) q ( θ ) ↔ η θ q ( x ) ↔ η x

  35. θ θ x 3 x 1 x 2 x 4 x 3 x 1 x 2 x 4 y 1 y 2 y 3 y 4 p ( x | θ ) is linear dynamical system q ( θ ) q ( x ) ≈ p ( θ , x | y ) p ( y | x, θ ) is linear-Gaussian p ( θ ) is conjugate prior h i log p ( θ ,x,y ) L ( η θ , η x ) , E q ( θ ) q ( x ) q ( θ ) q ( x ) x ( η θ ) , arg max L SVI ( η θ ) , L ( η θ , η ∗ L ( η θ , η x ) x ( η θ )) η ∗ η x Proposition (natural gradient SVI of Ho ff man et al. 2013) r L SVI ( η θ ) = η 0 e θ + E q ∗ ( x ) ( t xy ( x, y ) , 1) � η θ

  36. θ θ x 3 x 1 x 2 x 4 x 3 x 1 x 2 x 4 y 1 y 2 y 3 y 4 N N p ( x | θ ) is linear dynamical system q ( θ ) q ( x ) ≈ p ( θ , x | y ) p ( y | x, θ ) is linear-Gaussian p ( θ ) is conjugate prior h i log p ( θ ,x,y ) L ( η θ , η x ) , E q ( θ ) q ( x ) q ( θ ) q ( x ) x ( η θ ) , arg max L SVI ( η θ ) , L ( η θ , η ∗ L ( η θ , η x ) x ( η θ )) η ∗ η x Proposition (natural gradient SVI of Ho ff man et al. 2013) N X e r L SVI ( η θ ) = η 0 θ + E q ∗ ( x n ) ( t xy ( x n , y n ) , 1) � η θ n =1

  37. Step 1: compute evidence potentials [1] Johnson and Willsky. Stochastic variational inference for Bayesian time series models. ICML 2014. [2] Foti, Xu, Laird, and Fox. Stochastic variational inference for hidden Markov models. NIPS 2014.

  38. Step 1: compute evidence potentials [1] Johnson and Willsky. Stochastic variational inference for Bayesian time series models. ICML 2014. [2] Foti, Xu, Laird, and Fox. Stochastic variational inference for hidden Markov models. NIPS 2014.

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Lecture 18: Concluding Convolutional Neural Networks, Graphical Models as Foundation for Recurrent

Lecture 18: Concluding Convolutional Neural Networks, Graphical Models as Foundation for Recurrent

Lecture 18: Concluding Convolutional Neural Networks, Graphical Models as Foundation for Recurrent Neural Networks and Bayesian Networks Reference: We will be referring to sections etc of Deep Learning by Yoshua Bengio, Ian J. Goodfellow

974 views • 54 slides
Learning Discrete Graphical Models with Neural Networks Andrey Lokhov joint work with Abhijith

Learning Discrete Graphical Models with Neural Networks Andrey Lokhov joint work with Abhijith

Slide 1 Learning Discrete Graphical Models with Neural Networks Andrey Lokhov joint work with Abhijith Jayakumar, Sidhant Misra, Marc Vuffray UNCLASSIFIED Managed by Triad National Security, LLC for the U.S. Department of Energys NNSA

407 views • 15 slides
Directed Graphical Models: Bayesian Networks Probabilistic Graphical Models Sharif University of

Directed Graphical Models: Bayesian Networks Probabilistic Graphical Models Sharif University of

Directed Graphical Models: Bayesian Networks Probabilistic Graphical Models Sharif University of Technology Soleymani Spring 2018 Basics Multivariate distributions with large number of variables Independency assumptions are useful

503 views • 36 slides
Graphical Models Graphical Models Bayesian Networks Siamak Ravanbakhsh Fall 2019 Previously on

Graphical Models Graphical Models Bayesian Networks Siamak Ravanbakhsh Fall 2019 Previously on

Graphical Models Graphical Models Bayesian Networks Siamak Ravanbakhsh Fall 2019 Previously on Previously on Probabilistic Graphical Models Probabilistic Graphical Models Probability distribution and density functions Random variable

1.39k views • 104 slides
Graphical models for Neuroscience Part I Giuseppe Vinci Department of Statistics Rice

Graphical models for Neuroscience Part I Giuseppe Vinci Department of Statistics Rice

Graphical models for Neuroscience Part I Giuseppe Vinci Department of Statistics Rice University 1/1 Outline 1. Neural connectivity 2. Graphical models 3. Gaussian Graphical models 4. Functional connectivity in Macaque V4 2/1 Neural

454 views • 17 slides
Neural Networks Neural Networks can be : - Biological models MSE 2400 EaLiCaRA - Artificial

Neural Networks Neural Networks can be : - Biological models MSE 2400 EaLiCaRA - Artificial

Background Neural Networks Neural Networks can be : - Biological models MSE 2400 EaLiCaRA - Artificial models Dr. Tom Way Desire to produce artificial systems capable of sophisticated computations similar to the human brain. MSE 2400

215 views • 6 slides
Probabilistic Graphical Models Probabilistic Graphical Models Undirected Models Fall 2019

Probabilistic Graphical Models Probabilistic Graphical Models Undirected Models Fall 2019

Probabilistic Graphical Models Probabilistic Graphical Models Undirected Models Fall 2019 Siamak Ravanbakhsh Learning objectives Learning objectives Markov networks: how it represents a prob. dist. independence assumptions

941 views • 57 slides
Neural Networks Still seeking flexible, non-linear models for classfication and CS 335: Neural

Neural Networks Still seeking flexible, non-linear models for classfication and CS 335: Neural

Neural Networks Still seeking flexible, non-linear models for classfication and CS 335: Neural Networks regression Enter Neural Networks! Originally brain inspired Dan Sheldon Can (and will) avoid brain analogies: non-linear

298 views • 4 slides
Random Networks, Graphical Models and Exchangeability Alessandro Rinaldo Carnegie Mellon

Random Networks, Graphical Models and Exchangeability Alessandro Rinaldo Carnegie Mellon

Exchangeability A finite deFinetti theorem Bidirected graphical models Random Networks, Graphical Models and Exchangeability Alessandro Rinaldo Carnegie Mellon University joint work with Steffen Lauritzen and Kayvan Sadeghi October 4, 2015

647 views • 37 slides
Probabilistic Graphical Models Probabilistic Graphical Models Parameter learning in Bayesian

Probabilistic Graphical Models Probabilistic Graphical Models Parameter learning in Bayesian

Probabilistic Graphical Models Probabilistic Graphical Models Parameter learning in Bayesian networks Siamak Ravanbakhsh Fall 2019 Learning objectives Learning objectives likelihood function and MLE role of the sufficient statistics MLE for

644 views • 60 slides
Modular Neural Networks CPSC 533 Franco Lee Ian Ko Modular Neural Networks What is it ? Dif

Modular Neural Networks CPSC 533 Franco Lee Ian Ko Modular Neural Networks What is it ? Dif

Modular Neural Networks CPSC 533 Franco Lee Ian Ko Modular Neural Networks What is it ? Dif f erent models of neural net w orks combined int o a single syst em. Each single net w ork is made int o a module t hat can be f reely int

289 views • 25 slides
PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 8: GRAPHICAL MODELS Bayesian Networks Directed

PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 8: GRAPHICAL MODELS Bayesian Networks Directed

Some slides are removed by ATC (See PRML site for the original copy) PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 8: GRAPHICAL MODELS Bayesian Networks Directed Acyclic Graph (DAG) Bayesian Networks General Factorization Generative Models

525 views • 34 slides
graphical models Class 1 Rina Dechter Dechter-Morgan&claypool book (Dbook): Chapters 1-2

graphical models Class 1 Rina Dechter Dechter-Morgan&claypool book (Dbook): Chapters 1-2

Algorithms for reasoning with graphical models Class 1 Rina Dechter Dechter-Morgan&claypool book (Dbook): Chapters 1-2 class1 276-2018 Outline Graphical models: The constraint network, Probabilistic networks, cost networks and mixed

692 views • 44 slides
Probabilistic Graphical Models Lecture 3 Bayesian Networks Semantics CS/CNS/EE 155 Andreas

Probabilistic Graphical Models Lecture 3 Bayesian Networks Semantics CS/CNS/EE 155 Andreas

Probabilistic Graphical Models Lecture 3 Bayesian Networks Semantics CS/CNS/EE 155 Andreas Krause Bayesian networks Compact representation of distributions over large number of variables (Often) allows efficient exact inference

1.72k views • 35 slides
Probabilistic Graphical Models Lecture 2 Bayesian Networks Representation CS/CNS/EE 155

Probabilistic Graphical Models Lecture 2 Bayesian Networks Representation CS/CNS/EE 155

Probabilistic Graphical Models Lecture 2 Bayesian Networks Representation CS/CNS/EE 155 Andreas Krause Announcements Will meet in Steele 102 for now Still looking for another 1-2 TAs.. Homework 1 will be out soon. Start early!! 2

981 views • 32 slides
Transforming Graphical System Models to Graphical Attack Models ! Joint work with Marieta

Transforming Graphical System Models to Graphical Attack Models ! Joint work with Marieta

From Graphical System Models... ... to Graphical Attack Models ... ... to Risk Assessment Transforming Graphical System Models to Graphical Attack Models ! Joint work with Marieta Georgieva Ivanova, ! ! Ren e Rydhof Hansen, and Florian

650 views • 27 slides
What is Biological Diversity? Brook Milligan Department of Biology New Mexico State University

What is Biological Diversity? Brook Milligan Department of Biology New Mexico State University

What is Biological Diversity? Brook Milligan Department of Biology New Mexico State University Las Cruces, New Mexico 88003 brook@nmsu.edu Fall 2009 Brook Milligan What is Biological Diversity? Biological Diversity in General Part of the

145 views • 10 slides
PON1 polymorphisms and Wilms Tumor Gisele Moledo de Vasconcelos, PhD Gonalves BAA, Azevedo RM,

PON1 polymorphisms and Wilms Tumor Gisele Moledo de Vasconcelos, PhD Gonalves BAA, Azevedo RM,

PON1 polymorphisms and Wilms Tumor Gisele Moledo de Vasconcelos, PhD Gonalves BAA, Azevedo RM, Pombo de Oliveira MS, de Camargo B and the Brazilian Embryonic Tumor Group Pediatric Hematology Oncology Program Instituto Nacional de

623 views • 15 slides
On the Role of Auxiliary Assumptions in the Production of Evidence Corey Dethier University of

On the Role of Auxiliary Assumptions in the Production of Evidence Corey Dethier University of

Introduction Counterexamples Conflation Implications References On the Role of Auxiliary Assumptions in the Production of Evidence Corey Dethier University of Notre Dame Philosophy Department corey.dethier@gmail.com Nov. 3, 2018 PSA 2018

561 views • 26 slides
Theory of Natural Selection Definitions: Gene: Nucleotide sequence coding for, or regulating the

Theory of Natural Selection Definitions: Gene: Nucleotide sequence coding for, or regulating the

Theory of Natural Selection Definitions: Gene: Nucleotide sequence coding for, or regulating the expression of, a phenotypic trait Genotype: Set of alleles possessed by an individual at a locus (or several loci) Evolution: Change in allele

437 views • 19 slides
Types of Electric Vehicles Source: 2012 Chevrolet Volt plug-in hybrid, Photo by Mariordo, CC

Types of Electric Vehicles Source: 2012 Chevrolet Volt plug-in hybrid, Photo by Mariordo, CC

Prof. Dr. Pavol Bauer, Delft University of Technology Types of Electric Vehicles Source: 2012 Chevrolet Volt plug-in hybrid, Photo by Mariordo, CC BY-SA 3.0, Wikimedia.org Types of vehicles : Drive-train & Fuel Gasoline Conventional ICE

376 views • 18 slides
Mobility > Transportation The ability to move or be moved freely and easily. MOBIL ILIT ITY

Mobility > Transportation The ability to move or be moved freely and easily. MOBIL ILIT ITY

MOBIL ILIT ITY Mobility > Transportation The ability to move or be moved freely and easily. MOBIL ILIT ITY MOBIL ILIT ITY USA Today, 2/20/2017: Cheap gas and a surging economy are taxing the nations roads and contributing to

553 views • 9 slides
SENSATA FOURTH QUARTER AND FULL YEAR 2015 EARNINGS SUMMARY Forwardlooking Statements In

SENSATA FOURTH QUARTER AND FULL YEAR 2015 EARNINGS SUMMARY Forwardlooking Statements In

FEBRUARY 2016 SENSATA FOURTH QUARTER AND FULL YEAR 2015 EARNINGS SUMMARY Forwardlooking Statements In addition to historical facts, this earnings release, including any documents incorporated by reference herein, includes forward-looking

409 views • 20 slides
Electric Transportation Program Clean Energy Business and Market Development November 1, 2018

Electric Transportation Program Clean Energy Business and Market Development November 1, 2018

Electric Transportation Program Clean Energy Business and Market Development November 1, 2018 November 1, 2018 2 Overview Importance of Electric Transportation EV Terminology EV Trends NYS policies NYPA Programs

421 views • 18 slides

More recommend