gaussian process
play

Gaussian Process Lei Tang Arizona State University Jul. 31th, 2007 - PowerPoint PPT Presentation

Oct 14, 2022 •30 likes •340 views

Gaussian Process Lei Tang Arizona State University Jul. 31th, 2007 Lei Tang (ASU) Gaussian Process Jul. 31th, 2007 1 / 22 Gaussian Process, (kriging in geostatistics) Autoregressive moving average model, Kalman filters, and radial basis

  1. Gaussian Process Lei Tang Arizona State University Jul. 31th, 2007 Lei Tang (ASU) Gaussian Process Jul. 31th, 2007 1 / 22

  2. Gaussian Process, (kriging in geostatistics) Autoregressive moving average model, Kalman filters, and radial basis function networks can be viewed as forms of Gaussian process models. Lei Tang (ASU) Gaussian Process Jul. 31th, 2007 2 / 22

  3. Linear regression revisited w T φ ( x ) y ( x ) = N ( w | 0 , α − 1 I ) p ( w ) = y = Φ w E [ y ] = Φ E [ w ] E [ yy T ] = Φ E [ ww T ]Φ T = 1 α ΦΦ T = K cov [ y ] = where K is Gram matrix with elements K nm = k ( x n , x m ) = 1 αφ ( x n ) T φ ( x m ) Lei Tang (ASU) Gaussian Process Jul. 31th, 2007 3 / 22

  4. Gaussian Process A Gaussian process is defined as a probability distribution over functions y ( x ) suth that the set of values of y ( x ) evaluated at an arbitrary set of points x 1 , · · · , x N jointly have a Gaussian distribution. Gaussian random field : when the input vector x is two-dimentional. Stochastic process : y ( x ) is specified by giving the joint probability distirubtion for any finite set of values y ( x 1 ) , · · · , y ( x N ) in a consistent manner. Lei Tang (ASU) Gaussian Process Jul. 31th, 2007 4 / 22

  5. GP Connection to Kernel For Gaussian stochstic process, the joint distribution over N variables y 1 , · · · , y N is specified completely by the second-order statistics. For most applications, we have no prior knowledge, so by symmetry(also for sparsity) we take the mean of y ( x ) to be zero. Then the Gaussian process is deteremined by the covariance of y ( x ) which is specified by the kernel function: E [ y ( x n ) , y ( x m )] = k ( x n , x m ) Lei Tang (ASU) Gaussian Process Jul. 31th, 2007 5 / 22

  6. Two Examples of GP Specificy the covaraince (kernel) directly. 1 Gaussian Kernel: k ( x , x ′ ) = exp ( −|| x − x ′ || 2 / 2 σ 2 ) 2 Exponential Kernel: k ( x , x ′ ) = exp ( − θ | x − x ′ | ) (correpsonds to the Ornstein-Uhlenbeck process original introduced for Brownian motion) 3 3 1.5 1.5 0 0 −1.5 −1.5 −3 −3 −1 −0.5 0 0.5 1 −1 −0.5 0 0.5 1 Lei Tang (ASU) Gaussian Process Jul. 31th, 2007 6 / 22

  7. GP for Regression with Random Noise If the noise on the observed target values are considered: N ( t n | y n , β − 1 ) p ( t n | y n ) = N ( t | y , β − 1 I N ) p ( t | y ) = N ( y | 0 , K ) p ( y ) = � p ( t ) = p ( t | y ) p ( y ) d y = N ( t | 0 , C ) where C ( x n , x m ) = k ( x n , x m ) + β − 1 δ nm . Covraince simply add. Hint: matrix inverse lemma [ B − 1 + CD − 1 C T ] − 1 = B − BC ( D + C T BC ) − 1 C T B Lei Tang (ASU) Gaussian Process Jul. 31th, 2007 7 / 22

  8. GP for Regression with Random Noise If the noise on the observed target values are considered: N ( t n | y n , β − 1 ) p ( t n | y n ) = N ( t | y , β − 1 I N ) p ( t | y ) = N ( y | 0 , K ) p ( y ) = � p ( t ) = p ( t | y ) p ( y ) d y = N ( t | 0 , C ) where C ( x n , x m ) = k ( x n , x m ) + β − 1 δ nm . Covraince simply add. Hint: matrix inverse lemma [ B − 1 + CD − 1 C T ] − 1 = B − BC ( D + C T BC ) − 1 C T B Lei Tang (ASU) Gaussian Process Jul. 31th, 2007 7 / 22

  9. GP for Regression with Random Noise If the noise on the observed target values are considered: N ( t n | y n , β − 1 ) p ( t n | y n ) = N ( t | y , β − 1 I N ) p ( t | y ) = N ( y | 0 , K ) p ( y ) = � p ( t ) = p ( t | y ) p ( y ) d y = N ( t | 0 , C ) where C ( x n , x m ) = k ( x n , x m ) + β − 1 δ nm . Covraince simply add. Hint: matrix inverse lemma [ B − 1 + CD − 1 C T ] − 1 = B − BC ( D + C T BC ) − 1 C T B Lei Tang (ASU) Gaussian Process Jul. 31th, 2007 7 / 22

  10. � ✁ 3 0 −3 −1 0 1 Lei Tang (ASU) Gaussian Process Jul. 31th, 2007 8 / 22

  11. Commonly Used Kernel for Regression � � − θ 1 2 || x n − x m || 2 + θ 2 + θ 3 x T k ( x n , x m ) = θ 0 exp n x m Lei Tang (ASU) Gaussian Process Jul. 31th, 2007 9 / 22

  12. GP for Prediction p ( t N +1 ) = N ( t N +1 | 0 , C N +1 ) � C N � k C N +1 = k T c k T C − 1 m ( x N +1 ) = N t σ 2 ( x N +1 ) c − k T C − 1 = N k If we rewrite m ( x N +1 ) = � N n =1 a n k ( x n , x N +1 ), and define a kernel function depending only on the distance || x n − x m || , we obtain an expansion in radial basis function. Lei Tang (ASU) Gaussian Process Jul. 31th, 2007 10 / 22

  13. GP for Prediction p ( t N +1 ) = N ( t N +1 | 0 , C N +1 ) � C N � k C N +1 = k T c k T C − 1 m ( x N +1 ) = N t σ 2 ( x N +1 ) c − k T C − 1 = N k If we rewrite m ( x N +1 ) = � N n =1 a n k ( x n , x N +1 ), and define a kernel function depending only on the distance || x n − x m || , we obtain an expansion in radial basis function. Lei Tang (ASU) Gaussian Process Jul. 31th, 2007 10 / 22

  14. 1 0.5 0 −0.5 −1 0 0.2 0.4 0.6 0.8 1 Lei Tang (ASU) Gaussian Process Jul. 31th, 2007 11 / 22

  15. Computation time for GP regression 1 Training: GP: inversion of a N × N matrix O ( N 3 ) + O ( N 2 ). Linear basis function model: inversion of a M × M matrix O ( M 3 ) + O ( M 2 ). 2 Prediction: GP: O ( N ). Linear basis function: O ( M ). Advantages of GP If the number of basis functions is larger than the number of data points, GP is computionally more efficient. Donot need to construct the basis function. Can learn the hyperparameters (maximum likelihood estimation) Lei Tang (ASU) Gaussian Process Jul. 31th, 2007 12 / 22

  16. Computation time for GP regression 1 Training: GP: inversion of a N × N matrix O ( N 3 ) + O ( N 2 ). Linear basis function model: inversion of a M × M matrix O ( M 3 ) + O ( M 2 ). 2 Prediction: GP: O ( N ). Linear basis function: O ( M ). Advantages of GP If the number of basis functions is larger than the number of data points, GP is computionally more efficient. Donot need to construct the basis function. Can learn the hyperparameters (maximum likelihood estimation) Lei Tang (ASU) Gaussian Process Jul. 31th, 2007 12 / 22

  17. Automatic relevance determination Previous example doesn’t consider the relevave importance of each dimension. Define a kernel as � 2 � − 1 � γ i ( x i − x ′ i ) 2 k ( x , x ′ ) = θ 0 exp 2 i =1 Atuomatically learn the hyperparameters resulting ARD which automatically determine the relative importance of each basis. Lei Tang (ASU) Gaussian Process Jul. 31th, 2007 13 / 22

  18. GP for Classification Similar to logistic/probit regression, using a nonlinear activation function to transform ( −∞ , + ∞ ) into probability interval (0 , 1). Assume latent variable a and the target output given latent variables are determined: σ ( a ) t (1 − σ ( a )) 1 − t p ( t | a ) = Latent variables a follows the Gaussian Process For prediction, � p ( t N +1 = 1 | t N ) = p ( t N +1 = 1 | a N +1 ) p ( a N +1 | t N ) da N +1 Unfortunately, this is analytically intractable and may be approximated using sampling methods or analytical approximation. Lei Tang (ASU) Gaussian Process Jul. 31th, 2007 14 / 22

  19. GP for Classification Similar to logistic/probit regression, using a nonlinear activation function to transform ( −∞ , + ∞ ) into probability interval (0 , 1). Assume latent variable a and the target output given latent variables are determined: σ ( a ) t (1 − σ ( a )) 1 − t p ( t | a ) = Latent variables a follows the Gaussian Process For prediction, � p ( t N +1 = 1 | t N ) = p ( t N +1 = 1 | a N +1 ) p ( a N +1 | t N ) da N +1 Unfortunately, this is analytically intractable and may be approximated using sampling methods or analytical approximation. Lei Tang (ASU) Gaussian Process Jul. 31th, 2007 14 / 22

  20. GP for Classification Similar to logistic/probit regression, using a nonlinear activation function to transform ( −∞ , + ∞ ) into probability interval (0 , 1). Assume latent variable a and the target output given latent variables are determined: σ ( a ) t (1 − σ ( a )) 1 − t p ( t | a ) = Latent variables a follows the Gaussian Process For prediction, � p ( t N +1 = 1 | t N ) = p ( t N +1 = 1 | a N +1 ) p ( a N +1 | t N ) da N +1 Unfortunately, this is analytically intractable and may be approximated using sampling methods or analytical approximation. Lei Tang (ASU) Gaussian Process Jul. 31th, 2007 14 / 22

  21. GP for classification prediction Gaussin approximation to the posterior distribution over a N +1 . � p ( a N +1 | t N ) = p ( a N +1 | a N ) p ( a N | t N ) d a N N ( a N +1 | k T C − 1 N a N , c − k T C − 1 p ( a N +1 | a N ) = N k ) Need to estimate p ( a N | t N ): use Gaussian Approximation The shape of single-mode distribution is close to Gaussian distribution. Increasing the number of data points falling in a fixed region of x space, then the corresponding uncertainty in the function a ( x ) will decrease, asymptotically leading to a Gaussian. Lei Tang (ASU) Gaussian Process Jul. 31th, 2007 15 / 22

  22. GP for classification prediction Gaussin approximation to the posterior distribution over a N +1 . � p ( a N +1 | t N ) = p ( a N +1 | a N ) p ( a N | t N ) d a N N ( a N +1 | k T C − 1 N a N , c − k T C − 1 p ( a N +1 | a N ) = N k ) Need to estimate p ( a N | t N ): use Gaussian Approximation The shape of single-mode distribution is close to Gaussian distribution. Increasing the number of data points falling in a fixed region of x space, then the corresponding uncertainty in the function a ( x ) will decrease, asymptotically leading to a Gaussian. Lei Tang (ASU) Gaussian Process Jul. 31th, 2007 15 / 22

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Gaussian Processes Seung-Hoon Na Chonbuk National University Gaussian Process Regression

Gaussian Processes Seung-Hoon Na Chonbuk National University Gaussian Process Regression

Gaussian Processes Seung-Hoon Na Chonbuk National University Gaussian Process Regression Predictions using noisy observations The case of a single test input: Gaussian Process Regression Computational and numerical issues It is

444 views • 29 slides
Lecture 13 Gaussian Process Models - Part 2 3/06/2018 1 EDA and GPs 2 Variogram When fitting

Lecture 13 Gaussian Process Models - Part 2 3/06/2018 1 EDA and GPs 2 Variogram When fitting

Lecture 13 Gaussian Process Models - Part 2 3/06/2018 1 EDA and GPs 2 Variogram When fitting a Gaussian process model, it is often difficult to fit the covariance parameters (hard to identify). Today we will discuss some EDA approaches for

1.15k views • 49 slides
Bayesian Neural Networks from a Gaussian Process Perspective Andrew Gordon Wilson

Bayesian Neural Networks from a Gaussian Process Perspective Andrew Gordon Wilson

Bayesian Neural Networks from a Gaussian Process Perspective Andrew Gordon Wilson https://cims.nyu.edu/~andrewgw Courant Institute of Mathematical Sciences Center for Data Science New York University Gaussian Process Summer School September

508 views • 47 slides
Outline 1. Motivation 2. Gaussian process introduction 3. Change surface model 4. Analysis of

Outline 1. Motivation 2. Gaussian process introduction 3. Change surface model 4. Analysis of

5/22/2016 Scalable Gaussian Processes for Characterizing Multidimensional Change Surfaces April 18, 2016 William Herlands Committee: Daniel Neill, Alex Smola, Wilbert Van Panhuis Chair: Dave Choi Outline 1. Motivation 2. Gaussian process

1k views • 21 slides
Ruin problem for integrated stationary Gaussian process Kobelkov S. G. 1 Consider a random process

Ruin problem for integrated stationary Gaussian process Kobelkov S. G. 1 Consider a random process

Ruin problem for integrated stationary Gaussian process Kobelkov S. G. 1 Consider a random process t 0 X s ds ct , Y t = where X t , t 0 is a stationary real-valued zero-mean Gaussian process with continuous trajectories and twice

343 views • 5 slides
Gaussian Process Implicit Surfaces Oliver Williams 1 Microsoft Research and Trinity Hall

Gaussian Process Implicit Surfaces Oliver Williams 1 Microsoft Research and Trinity Hall

Implicit surface modelling Spline regularization as a Gaussian process GPIS for 2D curves GPIS for 3D surfaces Summary Gaussian Process Implicit Surfaces Oliver Williams 1 Microsoft Research and Trinity Hall Cambridge, UK Gaussian Processes

647 views • 40 slides
Scalable Training of Inference Networks for Gaussian-Process Models Jiaxin Shi Tsinghua

Scalable Training of Inference Networks for Gaussian-Process Models Jiaxin Shi Tsinghua

Scalable Training of Inference Networks for Gaussian-Process Models Jiaxin Shi Tsinghua University Joint work with Mohammad Emtiyaz Khan Jun Zhu Gaussian Process mean function covariance function / kernel inducing points Posterior

483 views • 13 slides
A prior near-ignorance Gaussian Process model for nonparametric regression Francesca Mangili

A prior near-ignorance Gaussian Process model for nonparametric regression Francesca Mangili

Introduction Gaussian Process (GP) Imprecise GP (IGP) Constant mean IGP Application Conclusions A prior near-ignorance Gaussian Process model for nonparametric regression Francesca Mangili francesca@idsia.ch Istituto Dalle Molle di

309 views • 15 slides
Probabilistic Graphical Models Lecture 21: Advanced Gaussian Processes Andrew Gordon Wilson

Probabilistic Graphical Models Lecture 21: Advanced Gaussian Processes Andrew Gordon Wilson

Probabilistic Graphical Models Lecture 21: Advanced Gaussian Processes Andrew Gordon Wilson www.cs.cmu.edu/~andrewgw Carnegie Mellon University April 1, 2015 1 / 100 Gaussian process review Definition A Gaussian process (GP) is a collection

1.36k views • 100 slides
How to choose the covariance for Gaussian process regression independently of the basis Workshop

How to choose the covariance for Gaussian process regression independently of the basis Workshop

How to choose the covariance for Gaussian process regression independently of the basis Workshop Gaussian Processes in Practice Matthias O. Franz and Peter V. Gehler June 12th, 2006 Motivation: Nonlinear system identification using Volterra

541 views • 4 slides
Determining the PSF over the Full FoV of LSST using Anisotropic Gaussian Processes

Determining the PSF over the Full FoV of LSST using Anisotropic Gaussian Processes

Determining the PSF over the Full FoV of LSST using Anisotropic Gaussian Processes Pierre-Franois Lget, Stanford University - KIPAC Gaussian process (GP) interpolation: Gaussian processes are a non-parametric way to interpolate data

281 views • 7 slides
Learning to Control an Octopus Arm with Gaussian Process Temporal Difference Methods Yaakov

Learning to Control an Octopus Arm with Gaussian Process Temporal Difference Methods Yaakov

Learning to Control an Octopus Arm with Gaussian Process Temporal Difference Methods Yaakov Engel Joint work with Peter Szabo and Dmitry Volkinshtein (ex. Technion) Gaussian Processes in Practice Workshop Why use GPs in RL? A Bayesian

271 views • 23 slides
Gaussian Filter The Gaussian filter 1 2 1 A Gaussian kernel gives less 1 2 4 2 weight to

Gaussian Filter The Gaussian filter 1 2 1 A Gaussian kernel gives less 1 2 4 2 weight to

Gaussian Filter The Gaussian filter 1 2 1 A Gaussian kernel gives less 1 2 4 2 weight to pixels further from 16 the center of the window 1 2 1 This kernel is an approximation of a Gaussian function Gaussian filtering versus mean

405 views • 11 slides
Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models

Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models

Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models Yarin Gal Mark van der Wilk Carl E. Rasmussen yg279@cam.ac.uk June 25th, 2014 Outline Gaussian process regression and latent variable

1.18k views • 54 slides
Hands on Demos for Gaussian Process using R Software Tak (Hyungsuk) Tak & David Jones SAMSI

Hands on Demos for Gaussian Process using R Software Tak (Hyungsuk) Tak & David Jones SAMSI

Hands on Demos for Gaussian Process using R Software Tak (Hyungsuk) Tak & David Jones SAMSI Undergraduate Workshop 24 Oct 2016 1 / 1 Gaussian Processes X Y (= f ( X )) Science: What is f ( )? What is f ( X ) if I have X ?

337 views • 11 slides
Non-Gaussian likelihoods for Gaussian Processes Alan Saul Outline Motivation Non-Gaussian

Non-Gaussian likelihoods for Gaussian Processes Alan Saul Outline Motivation Non-Gaussian

Non-Gaussian likelihoods for Gaussian Processes Alan Saul Outline Motivation Non-Gaussian posteriors Approximate methods Laplace approximation Variational bayes Expectation propagation Comparisons GP regression - recap so far Model the

1.4k views • 113 slides
Bayesian Optimization of Gaussian Processes applied to Performance Tuning Ramki Ramakrishna

Bayesian Optimization of Gaussian Processes applied to Performance Tuning Ramki Ramakrishna

Bayesian Optimization of Gaussian Processes applied to Performance Tuning Ramki Ramakrishna @ysr1729 #TwitterVMTeam QCon Sao Paulo, 2019 A JVM Engineer talks to a Data Scientist 2 Many Hundreds of Services Several Tens of Thousands

1.97k views • 60 slides
Fast Laplace Approximation for Gaussian Ketter Processes with a Tensor Product Kernel

Fast Laplace Approximation for Gaussian Ketter Processes with a Tensor Product Kernel

Perry Groot , Markus Peters Tom Heskes, Wolgang Fast Laplace Approximation for Gaussian Ketter Processes with a Tensor Product Kernel Introduction Background GPs Laplace Approximation Perry Groot a Markus Peters b Kronecker product Tom

375 views • 18 slides
Kernel Methods Lei Tang Arizona State University Jul. 26th, 2007 Lei Tang Kernel Methods

Kernel Methods Lei Tang Arizona State University Jul. 26th, 2007 Lei Tang Kernel Methods

Kernel Methods Lei Tang Arizona State University Jul. 26th, 2007 Lei Tang Kernel Methods Introduction Linear parametric models for regression and classification. Memory-based methods: Parzen probability density estimation, k-nearest

1.01k views • 63 slides
Convex Analysis in Stochastic Teams and Asymptotic Optimality of Finite Model Representations and

Convex Analysis in Stochastic Teams and Asymptotic Optimality of Finite Model Representations and

Convex Analysis in Stochastic Teams and Asymptotic Optimality of Finite Model Representations and Quantized Policies Serdar Yksel Part of this is joint work with Naci Saldi and Tams Linder Queens University, Canada Department of

1.17k views • 81 slides
TV Ads Attribution and Gaussian Processes Adrin Jalali November 16, 2016 1 / 27 Problem

TV Ads Attribution and Gaussian Processes Adrin Jalali November 16, 2016 1 / 27 Problem

TV Ads Attribution and Gaussian Processes Adrin Jalali November 16, 2016 1 / 27 Problem Definition Website 2 / 27 Problem Definition Website Sources of traffic TV Ads Google Ads . . . 2 / 27 Problem Definition

892 views • 29 slides
An Extension of the Divergence Operator for Gaussian Processes Jorge A. Len Departamento de

An Extension of the Divergence Operator for Gaussian Processes Jorge A. Len Departamento de

An Extension of the Divergence Operator for Gaussian Processes Jorge A. Len Departamento de Control Automtico Cinvestav del IPN Spring School "Stochastic Control in Finance", Roscoff 2010 Jointly with David Nualart and Jaime San

816 views • 80 slides
Bayesian Optimization in Reduced Eigenbases David Gaudrie 1 , Rodolphe Le Riche 2 , Victor Picheny

Bayesian Optimization in Reduced Eigenbases David Gaudrie 1 , Rodolphe Le Riche 2 , Victor Picheny

Parametric shape optimization Intrinsic dimensionality Meta-modeling in eigenbasis Optimization in eigenbasis Conclusions Bayesian Optimization in Reduced Eigenbases David Gaudrie 1 , Rodolphe Le Riche 2 , Victor Picheny 3 , t Enaux 1 ,

581 views • 55 slides
Reduced-Set Models for Improving the Training and Execution Speed of Kernel Methods Hassan A.

Reduced-Set Models for Improving the Training and Execution Speed of Kernel Methods Hassan A.

Introduction RSKPCA: Basics RSKPCA: Applications GP-MRAC Conclusion Questions Reduced-Set Models for Improving the Training and Execution Speed of Kernel Methods Hassan A. Kingravi IVALab 1 Introduction RSKPCA: Basics RSKPCA:

1.19k views • 76 slides

More recommend