doubly stochastic inference for deep gaussian processes
play

Doubly Stochastic Inference for Deep Gaussian Processes Hugh - PowerPoint PPT Presentation

Sep 05, 2022 •982 likes •1.53k views

Doubly Stochastic Inference for Deep Gaussian Processes Hugh Salimbeni Department of Computing Imperial College London 29/5/2017 Motivation DGPs promise much, but are difficult to train 2 Doubly Stochastic Inference for DGPs Hugh

  1. Doubly Stochastic Inference for Deep Gaussian Processes Hugh Salimbeni Department of Computing Imperial College London 29/5/2017

  2. Motivation § DGPs promise much, but are difficult to train 2 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  3. Motivation § DGPs promise much, but are difficult to train § Fully factorized VI doesn’t work well 2 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  4. Motivation § DGPs promise much, but are difficult to train § Fully factorized VI doesn’t work well § We seek a variational approach that works and scales 2 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  5. Motivation § DGPs promise much, but are difficult to train § Fully factorized VI doesn’t work well § We seek a variational approach that works and scales 2 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  6. Motivation § DGPs promise much, but are difficult to train § Fully factorized VI doesn’t work well § We seek a variational approach that works and scales Other recently proposed schemes [1, 2, 5] make additional approximations and require more machinery than VI 2 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  7. Talk outline 1. Summary : Model Inference Results 2. Details: Model Inference Results 3. Questions 3 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  8. Model We use the standard DGP model, with one addition: 4 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  9. Model We use the standard DGP model, with one addition: § We include a linear (identity) mean function for all the internal layers 4 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  10. Model We use the standard DGP model, with one addition: § We include a linear (identity) mean function for all the internal layers (1D example in [4]) 4 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  11. Inference § We use the model conditioned on the inducing points as a conditional variational posterior 5 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  12. Inference § We use the model conditioned on the inducing points as a conditional variational posterior § We impose Gaussians on the inducing points, (independent between layers but full rank within layers) 5 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  13. Inference § We use the model conditioned on the inducing points as a conditional variational posterior § We impose Gaussians on the inducing points, (independent between layers but full rank within layers) § We use sampling to deal with the intractable expectation 5 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  14. Inference § We use the model conditioned on the inducing points as a conditional variational posterior § We impose Gaussians on the inducing points, (independent between layers but full rank within layers) § We use sampling to deal with the intractable expectation 5 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  15. Inference § We use the model conditioned on the inducing points as a conditional variational posterior § We impose Gaussians on the inducing points, (independent between layers but full rank within layers) § We use sampling to deal with the intractable expectation We never compute N ˆ N matrices (we make no additional simplifications to variational posterior) 5 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  16. Results § We show significant improvement over single layer models on large ( „ 10 6 ) and massive ( „ 10 9 ) data 6 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  17. Results § We show significant improvement over single layer models on large ( „ 10 6 ) and massive ( „ 10 9 ) data § Big jump in improvement over single layer GP with 5 ˆ number of inducing points 6 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  18. Results § We show significant improvement over single layer models on large ( „ 10 6 ) and massive ( „ 10 9 ) data § Big jump in improvement over single layer GP with 5 ˆ number of inducing points § On small data we never do worse than the single layer model, and often better 6 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  19. Results § We show significant improvement over single layer models on large ( „ 10 6 ) and massive ( „ 10 9 ) data § Big jump in improvement over single layer GP with 5 ˆ number of inducing points § On small data we never do worse than the single layer model, and often better § We can get 98.1% on mnist with only 100 inducing points 6 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  20. Results § We show significant improvement over single layer models on large ( „ 10 6 ) and massive ( „ 10 9 ) data § Big jump in improvement over single layer GP with 5 ˆ number of inducing points § On small data we never do worse than the single layer model, and often better § We can get 98.1% on mnist with only 100 inducing points § We surpass all permutation invariant methods on rectangles-images (designed to test deep vs shallow architectures) 6 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  21. Results § We show significant improvement over single layer models on large ( „ 10 6 ) and massive ( „ 10 9 ) data § Big jump in improvement over single layer GP with 5 ˆ number of inducing points § On small data we never do worse than the single layer model, and often better § We can get 98.1% on mnist with only 100 inducing points § We surpass all permutation invariant methods on rectangles-images (designed to test deep vs shallow architectures) § Identical model/inference hyperparameters for all our models 6 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  22. Details: The Model We use the standard DGP model, with a linear mean function for all the internal layers: § If dimensions agree use the identity, otherwise PCA 7 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  23. Details: The Model We use the standard DGP model, with a linear mean function for all the internal layers: § If dimensions agree use the identity, otherwise PCA § Sensible alternative: initialize latents to identity (but linear mean function works better) 7 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  24. Details: The Model We use the standard DGP model, with a linear mean function for all the internal layers: § If dimensions agree use the identity, otherwise PCA § Sensible alternative: initialize latents to identity (but linear mean function works better) § Not so sensible alternative: random. Doesn’t work well (posterior is (very) multimodal) 7 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  25. The DGP: Graphical Model X Z 0 f 1 u 1 ✏ h 1 Z 1 f 2 u 2 ✏ h 2 Z 2 f 3 u 3 y 8 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  26. The DGP: Density likelihood hkkkkkikkkkkj N p p y , t h l , f l , u l u L π p p y i | f L l “ 1 q “ i q ˆ i “ 1 L π p p h l | f l q p p f l | u l ; h l ´ 1 , Z l ´ 1 q p p u l ; Z l ´ 1 q l “ 1 loooooooooooooooooooooooooomoooooooooooooooooooooooooon DGP prior 9 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  27. Factorised Variational Posterior X Z 0 X N ( u 1 | m 1 , S 1 ) f 1 u 1 f 1 u 1 ✏ 2 ) h 1 Z 1 i N ( h 1 i | µ 1 i , σ 1 h 1 Q i N ( u 2 | m 2 , S 2 ) f 2 u 2 f 2 u 2 ✏ 2 ) h 2 Z 2 Q i N ( h 2 i | µ 2 i , σ 2 h 2 i N ( u 3 | m 3 , S 3 ) f 3 u 3 f 3 u 3 y y 10 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  28. Our Variational Posterior X Z 0 X N ( u 1 | m 1 , S 1 ) f 1 u 1 f 1 u 1 ✏ ✏ h 1 Z 1 h 1 N ( u 2 | m 2 , S 2 ) f 2 u 2 f 2 u 2 ✏ ✏ h 2 Z 2 h 2 N ( u 3 | m 3 , S 3 ) f 3 u 3 f 3 u 3 y f 3 11 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  29. Recap: ‘GPs for Big Data’ [3] q p f , u q “ p p f | u ; X , Z q N p u | m , S q 12 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  30. Recap: ‘GPs for Big Data’ [3] q p f , u q “ p p f | u ; X , Z q N p u | m , S q Marginalise u from the variational posterior: ª p p f | u ; X , Z q N p u | m , S q d u “ N p f | µ , Σ q “ : q p f | m , S ; X , Z q (1) 12 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

  31. Recap: ‘GPs for Big Data’ [3] q p f , u q “ p p f | u ; X , Z q N p u | m , S q Marginalise u from the variational posterior: ª p p f | u ; X , Z q N p u | m , S q d u “ N p f | µ , Σ q “ : q p f | m , S ; X , Z q (1) Define the following mean and covariance functions: µ m , Z p x i q “ m p x i q ` α p x i q T p m ´ m p Z qq , Σ S , Z p x i , x j q “ k p x i , x j q ´ α p x i q T p k p Z , Z q ´ S q α p x j q . where α p x i q “ k p x i , Z q k p Z , Z q ´ 1 12 Doubly Stochastic Inference for DGPs Hugh Salimbeni @Amazon Berlin, 29/5/2017

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Doubly Stochastic Variational Inference for Neural Processes with Hierarchical Latent Variables

Doubly Stochastic Variational Inference for Neural Processes with Hierarchical Latent Variables

Doubly Stochastic Variational Inference for Neural Processes with Hierarchical Latent Variables Q. Wang & Herke van Hoof Amsterdam Machine Learning Lab ICML 2020 1 / 69 Highlights in this Work 2 / 69 Highlights in this Work A

965 views • 69 slides
Deep Gaussian Processes with Importance-Weighted Variational Inference Hugh Salimbeni Vincent

Deep Gaussian Processes with Importance-Weighted Variational Inference Hugh Salimbeni Vincent

Deep Gaussian Processes with Importance-Weighted Variational Inference Hugh Salimbeni Vincent Dutordoir, James Hensman, Marc P Deisenroth Problem setting Problem setting Bimodal density Problem setting Changes with input Problem setting

1.16k views • 43 slides
Gaussian Processes for Big Data James Hensman joint work with Nicol o Fusi, Neil D. Lawrence

Gaussian Processes for Big Data James Hensman joint work with Nicol o Fusi, Neil D. Lawrence

Gaussian Processes for Big Data James Hensman joint work with Nicol o Fusi, Neil D. Lawrence Overview Motivation Sparse Gaussian Processes Stochastic Variational Inference Examples Overview Motivation Sparse Gaussian Processes

533 views • 36 slides
Modern Gaussian Processes: Scalable Inference and Novel Applications (Part II-b) Approximate

Modern Gaussian Processes: Scalable Inference and Novel Applications (Part II-b) Approximate

Modern Gaussian Processes: Scalable Inference and Novel Applications (Part II-b) Approximate Inference Edwin V. Bonilla and Maurizio Filippone CSIROs Data61, Sydney, Australia and EURECOM, Sophia Antipolis, France July 14 th , 2019 1

1.14k views • 44 slides
Stochastic (partial) differential equations and Gaussian processes Simo Srkk Aalto

Stochastic (partial) differential equations and Gaussian processes Simo Srkk Aalto

Stochastic (partial) differential equations and Gaussian processes Simo Srkk Aalto University, Finland Contents Basic ideas 1 Stochastic differential equations and Gaussian processes 2 Stochastic partial differential equations and

2.22k views • 79 slides
Deep Gaussian Processes (IPVI DGP) Haibin Yu*, Yizhou Chen* Zhongxiang Dai Bryan Kian Hsiang Low

Deep Gaussian Processes (IPVI DGP) Haibin Yu*, Yizhou Chen* Zhongxiang Dai Bryan Kian Hsiang Low

Implicit Posterior Variational Inference for Deep Gaussian Processes (IPVI DGP) Haibin Yu*, Yizhou Chen* Zhongxiang Dai Bryan Kian Hsiang Low and Patrick Jaillet Department of Computer Science National University of Singapore Department of

511 views • 25 slides
Another introduction to Gaussian Processes Richard Wilkinson School of Maths and Statistics

Another introduction to Gaussian Processes Richard Wilkinson School of Maths and Statistics

Another introduction to Gaussian Processes Richard Wilkinson School of Maths and Statistics University of Sheffield GP summer school September 2017 Why use Gaussian processes? Why use Gaussian processes? A stochastic process is a collection

1.69k views • 152 slides
Gaussian Processes Dan Cervone NYU CDS November 10, 2015 Dan Cervone (NYU CDS) Gaussian

Gaussian Processes Dan Cervone NYU CDS November 10, 2015 Dan Cervone (NYU CDS) Gaussian

Gaussian Processes Dan Cervone NYU CDS November 10, 2015 Dan Cervone (NYU CDS) Gaussian Processes November 10, 2015 1 / 22 What are Gaussian processes? GPs let us do Bayesian inference on functions . Using GPs we can: Interpolate spatial

728 views • 54 slides
Modern Gaussian Processes: Scalable Inference and Novel Applications (Part III) Applications,

Modern Gaussian Processes: Scalable Inference and Novel Applications (Part III) Applications,

Modern Gaussian Processes: Scalable Inference and Novel Applications (Part III) Applications, Challenges & Opportunities Edwin V. Bonilla and Maurizio Filippone CSIROs Data61, Sydney, Australia and EURECOM, Sophia Antipolis, France July 14

551 views • 38 slides
Stochastic (Partial) Differential Equations and Gaussian Processes Simo Srkk Aalto

Stochastic (Partial) Differential Equations and Gaussian Processes Simo Srkk Aalto

Stochastic (Partial) Differential Equations and Gaussian Processes Simo Srkk Aalto University, Finland Why use S(P)DE solvers for GPs? The O ( n 3 ) computational complexity is always a challenge. Latent force models combine PDE/ODEs with

482 views • 12 slides
Modern Gaussian Processes: Scalable Inference and Novel Applications (Part IV) Theory & Code

Modern Gaussian Processes: Scalable Inference and Novel Applications (Part IV) Theory & Code

Modern Gaussian Processes: Scalable Inference and Novel Applications (Part IV) Theory & Code Edwin V. Bonilla and Maurizio Filippone CSIROs Data61, Sydney, Australia and EURECOM, Sophia Antipolis, France July 14 th , 2019 1 Outline 1

529 views • 11 slides
Stochastic Processes Will Perkins March 7, 2013 Stochastic Processes Q: What is a Stochastic

Stochastic Processes Will Perkins March 7, 2013 Stochastic Processes Q: What is a Stochastic

Stochastic Processes Will Perkins March 7, 2013 Stochastic Processes Q: What is a Stochastic Process? A: A collection of random variables defined on the same probability space and indexed by a time parameter. { Z t } t T where each Z

1.52k views • 17 slides
Assembling stochastic quasi-Newton algorithms using Gaussian processes Thomas Sch on, Uppsala

Assembling stochastic quasi-Newton algorithms using Gaussian processes Thomas Sch on, Uppsala

Assembling stochastic quasi-Newton algorithms using Gaussian processes Thomas Sch on, Uppsala University, Sweden. Joint work with Adrian Wills , University of Newcastle, Australia. LCCC Workshop on Learning and Adaptation for Sensorimotor

434 views • 39 slides
Scalable Exact Inference in Multi-Output Gaussian Processes Wessel P. Bruinsma 1 , 2 , Eric Perim

Scalable Exact Inference in Multi-Output Gaussian Processes Wessel P. Bruinsma 1 , 2 , Eric Perim

Scalable Exact Inference in Multi-Output Gaussian Processes Wessel P. Bruinsma 1 , 2 , Eric Perim 2 , Will Tebbutt 1 , J. Scott Hosking 3 , 4 , Arno Solin 5 , Richard E. Turner 1 , 6 1 University of Cambridge, 2 Invenia Labs, 3 British Antarctic

340 views • 23 slides
Power Expectation Propagation for Deep Gaussian Processes Dr. Richard E. Turner ( ret26@cam.ac.uk

Power Expectation Propagation for Deep Gaussian Processes Dr. Richard E. Turner ( ret26@cam.ac.uk

Power Expectation Propagation for Deep Gaussian Processes Dr. Richard E. Turner ( ret26@cam.ac.uk ) Computational and Biological Learning Lab, Department of Engineering, University of Cambridge with Thang Bui, Yingzhen Li, Jos e Miguel Hern

957 views • 69 slides
Deep Neural Networks as Gaussian Processes Jaehoon Lee Google Brain Workshop on Accelerating the

Deep Neural Networks as Gaussian Processes Jaehoon Lee Google Brain Workshop on Accelerating the

Deep Neural Networks as Gaussian Processes Jaehoon Lee Google Brain Workshop on Accelerating the Search for Dark Matter with Machine Learning April 10, 2019 Based on Published in ICLR 2018, htups://arxiv.org/abs/1711.00165 Open source

501 views • 33 slides
A quantum algorithm for model independent searches for new physics Prasanth Shyamsundar

A quantum algorithm for model independent searches for new physics Prasanth Shyamsundar

A quantum algorithm for model independent searches for new physics Prasanth Shyamsundar University of Florida based on [arXiv:2003.02181] Prof. Konstantin T. Matchev Prasanth Shyamsundar Dr. Jordan Smolinsky Pheno 2020 May 4-6, 2020

173 views • 13 slides
Investor Presentation November 2019 Safe Harbor Statement The offering to which this

Investor Presentation November 2019 Safe Harbor Statement The offering to which this

Investor Presentation November 2019 Safe Harbor Statement The offering to which this presentation (the "Presentation") relates will be made pursuant to an offering Forward-looking information contained in this Presentation is based on

423 views • 20 slides
Supervised Topic Models Atallah Hezbor and Anant Kharkar Outline (optional. Mostly for our

Supervised Topic Models Atallah Hezbor and Anant Kharkar Outline (optional. Mostly for our

Supervised Topic Models Atallah Hezbor and Anant Kharkar Outline (optional. Mostly for our reference) Intro [AK] LDA [AK] Objective Diagram Motivation for sLDA sLDA Expectation Maximization [AH]

328 views • 13 slides
ADVANCEMENT Review of RFP for Next W.M. Keck Foundation Research Program Funding Cycle June 9,

ADVANCEMENT Review of RFP for Next W.M. Keck Foundation Research Program Funding Cycle June 9,

ADVANCEMENT Review of RFP for Next W.M. Keck Foundation Research Program Funding Cycle June 9, 2020 Marco Monoc, Ph.D. Liz Smith Executive Director of Foundation Relations Director of Foundation Relations Why Pursue Funding from Keck?

277 views • 15 slides
Dive Deeper in Finance GTC 2017 San Jos California Daniel Egloff Dr. sc. math.

Dive Deeper in Finance GTC 2017 San Jos California Daniel Egloff Dr. sc. math.

Dive Deeper in Finance GTC 2017 San Jos California Daniel Egloff Dr. sc. math. Managing Director QuantAlea May 7, 2017 Today Generative models for financial time series Sequential latent Gaussian Variational Autoencoder

856 views • 43 slides
Accelerating Best Response Calculation in Large Extensive Games Q J # $ K 10 P " C

Accelerating Best Response Calculation in Large Extensive Games Q J # $ K 10 P " C

Accelerating Best Response Calculation in Large Extensive Games Q J # $ K 10 P " C R ! July 21, 2011 U G A V ! " # Michael Johanson, Kevin Waugh, ! K Q $ A J ! 10 University of Alberta Michael Bowling, Martin

1.52k views • 120 slides
Multi-phase Particles Morphology Formation: Model & Methods Simone Rusconi BCAM Basque

Multi-phase Particles Morphology Formation: Model & Methods Simone Rusconi BCAM Basque

Multi-phase Particles Morphology Formation: Model & Methods Simone Rusconi BCAM Basque Center for Applied Mathematics December 21, 2018 Multi-phase Particles Morphology (MPM) Multi-phase Particles : comprise phase-separated polymers

585 views • 30 slides
Fresh Start to End Family Homelessness Department of Human Services Guiding Principles

Fresh Start to End Family Homelessness Department of Human Services Guiding Principles

Fresh Start to End Family Homelessness Department of Human Services Guiding Principles Homelessness in DC is not an intractable problem it can be solved. When members of our community experience homelessness, we all suffer. Our entire

256 views • 25 slides

More recommend