A Unifying Framework for Sparse Gaussian Process Approximation using - - PowerPoint PPT Presentation

A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation

• Dr. Richard E. Turner (ret26@cam.ac.uk)

Computational and Biological Learning Lab, Department of Engineering, University of Cambridge

...joint work with Thang Bui, Cuong Nguyen and Josiah Yan

1 / 22

Motivation: Gaussian Process Regression inputs

• utputs

2 / 22

Motivation: Gaussian Process Regression inputs

• utputs

?

2 / 22

Motivation: Gaussian Process Regression inputs

• utputs

?

2 / 22

Motivation: Gaussian Process Regression inputs

• utputs

?

inference & learning

2 / 22

Motivation: Gaussian Process Regression inputs

• utputs

?

inference & learning intractabilities computational analytic

2 / 22

A Brief History of Gaussian Process Approximations

FITC: Snelson et al. “Sparse Gaussian Processes using Pseudo-inputs” PITC: Snelson et al. “Local and global sparse Gaussian process approximations” EP: Csato and Opper 2002 / Qi et al. "Sparse-posterior Gaussian Processes for general likelihoods.” VFE: Titsias “Variational Learning of Inducing Variables in Sparse Gaussian Processes” DTC / PP: Seeger et al. “Fast Forward Selection to Speed Up Sparse Gaussian Process Regression”

3 / 22

A Brief History of Gaussian Process Approximations

approximate generative model exact inference

FITC: Snelson et al. “Sparse Gaussian Processes using Pseudo-inputs” PITC: Snelson et al. “Local and global sparse Gaussian process approximations” EP: Csato and Opper 2002 / Qi et al. "Sparse-posterior Gaussian Processes for general likelihoods.” VFE: Titsias “Variational Learning of Inducing Variables in Sparse Gaussian Processes” DTC / PP: Seeger et al. “Fast Forward Selection to Speed Up Sparse Gaussian Process Regression”

3 / 22

A Brief History of Gaussian Process Approximations

approximate generative model exact inference methods employing pseudo-data

FITC: Snelson et al. “Sparse Gaussian Processes using Pseudo-inputs” PITC: Snelson et al. “Local and global sparse Gaussian process approximations” EP: Csato and Opper 2002 / Qi et al. "Sparse-posterior Gaussian Processes for general likelihoods.” VFE: Titsias “Variational Learning of Inducing Variables in Sparse Gaussian Processes” DTC / PP: Seeger et al. “Fast Forward Selection to Speed Up Sparse Gaussian Process Regression”

3 / 22

A Brief History of Gaussian Process Approximations

approximate generative model exact inference methods employing pseudo-data

FITC: Snelson et al. “Sparse Gaussian Processes using Pseudo-inputs” PITC: Snelson et al. “Local and global sparse Gaussian process approximations” EP: Csato and Opper 2002 / Qi et al. "Sparse-posterior Gaussian Processes for general likelihoods.” VFE: Titsias “Variational Learning of Inducing Variables in Sparse Gaussian Processes” DTC / PP: Seeger et al. “Fast Forward Selection to Speed Up Sparse Gaussian Process Regression”

FITC PITC DTC

3 / 22

A Brief History of Gaussian Process Approximations

approximate generative model exact inference methods employing pseudo-data

FITC: Snelson et al. “Sparse Gaussian Processes using Pseudo-inputs” PITC: Snelson et al. “Local and global sparse Gaussian process approximations” EP: Csato and Opper 2002 / Qi et al. "Sparse-posterior Gaussian Processes for general likelihoods.” VFE: Titsias “Variational Learning of Inducing Variables in Sparse Gaussian Processes” DTC / PP: Seeger et al. “Fast Forward Selection to Speed Up Sparse Gaussian Process Regression”

FITC PITC DTC

A Unifying View of Sparse Approximate Gaussian Process Regression Quinonero-Candela & Rasmussen, 2005 (FITC, PITC, DTC)

3 / 22

A Brief History of Gaussian Process Approximations

approximate generative model exact inference exact generative model approximate inference methods employing pseudo-data

FITC: Snelson et al. “Sparse Gaussian Processes using Pseudo-inputs” PITC: Snelson et al. “Local and global sparse Gaussian process approximations” EP: Csato and Opper 2002 / Qi et al. "Sparse-posterior Gaussian Processes for general likelihoods.” VFE: Titsias “Variational Learning of Inducing Variables in Sparse Gaussian Processes” DTC / PP: Seeger et al. “Fast Forward Selection to Speed Up Sparse Gaussian Process Regression”

FITC PITC DTC

A Unifying View of Sparse Approximate Gaussian Process Regression Quinonero-Candela & Rasmussen, 2005 (FITC, PITC, DTC)

3 / 22

A Brief History of Gaussian Process Approximations

approximate generative model exact inference exact generative model approximate inference methods employing pseudo-data

FITC: Snelson et al. “Sparse Gaussian Processes using Pseudo-inputs” PITC: Snelson et al. “Local and global sparse Gaussian process approximations” EP: Csato and Opper 2002 / Qi et al. "Sparse-posterior Gaussian Processes for general likelihoods.” VFE: Titsias “Variational Learning of Inducing Variables in Sparse Gaussian Processes” DTC / PP: Seeger et al. “Fast Forward Selection to Speed Up Sparse Gaussian Process Regression”

VFE EP PP FITC PITC DTC

A Unifying View of Sparse Approximate Gaussian Process Regression Quinonero-Candela & Rasmussen, 2005 (FITC, PITC, DTC)

3 / 22

A Brief History of Gaussian Process Approximations

approximate generative model exact inference exact generative model approximate inference methods employing pseudo-data

FITC: Snelson et al. “Sparse Gaussian Processes using Pseudo-inputs” PITC: Snelson et al. “Local and global sparse Gaussian process approximations” EP: Csato and Opper 2002 / Qi et al. "Sparse-posterior Gaussian Processes for general likelihoods.” VFE: Titsias “Variational Learning of Inducing Variables in Sparse Gaussian Processes” DTC / PP: Seeger et al. “Fast Forward Selection to Speed Up Sparse Gaussian Process Regression”

VFE EP PP FITC PITC DTC

A Unifying View of Sparse Approximate Gaussian Process Regression Quinonero-Candela & Rasmussen, 2005 (FITC, PITC, DTC)

3 / 22

A Brief History of Gaussian Process Approximations

approximate generative model exact inference exact generative model approximate inference methods employing pseudo-data

FITC: Snelson et al. “Sparse Gaussian Processes using Pseudo-inputs” PITC: Snelson et al. “Local and global sparse Gaussian process approximations” EP: Csato and Opper 2002 / Qi et al. "Sparse-posterior Gaussian Processes for general likelihoods.” VFE: Titsias “Variational Learning of Inducing Variables in Sparse Gaussian Processes” DTC / PP: Seeger et al. “Fast Forward Selection to Speed Up Sparse Gaussian Process Regression”

VFE EP PP FITC PITC DTC

A Unifying View of Sparse Approximate Gaussian Process Regression Quinonero-Candela & Rasmussen, 2005 (FITC, PITC, DTC) A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation Bui, Yan and Turner, 2016 (VFE, EP, FITC, PITC ...)

3 / 22

EP pseudo-point approximation

true posterior

4 / 22

EP pseudo-point approximation

true posterior

4 / 22

EP pseudo-point approximation

true posterior

marginal likelihood posterior

4 / 22

EP pseudo-point approximation

true posterior approximate posterior

marginal likelihood posterior

4 / 22

EP pseudo-point approximation

true posterior approximate posterior

marginal likelihood posterior

4 / 22

EP pseudo-point approximation

true posterior approximate posterior

marginal likelihood posterior

4 / 22

EP pseudo-point approximation

true posterior approximate posterior

marginal likelihood posterior

4 / 22

EP pseudo-point approximation

input locations of 'pseudo' data

• utputs and covariance

'pseudo' data

true posterior approximate posterior

marginal likelihood posterior exact joint

• f new GP

regression model

4 / 22

EP algorithm

5 / 22

EP algorithm

• 1. remove

take out one pseudo-observation likelihood

cavity

5 / 22

EP algorithm

• 1. remove
• 2. include

take out one pseudo-observation likelihood add in one true observation likelihood

cavity tilted

5 / 22

EP algorithm

• 1. remove
• 2. include
• 3. project

take out one pseudo-observation likelihood add in one true observation likelihood project onto approximating family

cavity tilted KL between unnormalised stochastic processes

5 / 22

EP algorithm

• 1. remove
• 2. include
• 3. project
• 4. update

take out one pseudo-observation likelihood add in one true observation likelihood project onto approximating family update pseudo-observation likelihood

cavity tilted KL between unnormalised stochastic processes

5 / 22

EP algorithm

• 1. remove
• 2. include
• 3. project
• 4. update

take out one pseudo-observation likelihood add in one true observation likelihood project onto approximating family update pseudo-observation likelihood

cavity tilted

• 1. minimum: moments matched at pseudo-inputs
• 2. Gaussian regression: matches moments everywhere

KL between unnormalised stochastic processes

5 / 22

EP algorithm

• 1. remove
• 2. include
• 3. project
• 4. update

take out one pseudo-observation likelihood add in one true observation likelihood project onto approximating family update pseudo-observation likelihood

cavity tilted

• 1. minimum: moments matched at pseudo-inputs
• 2. Gaussian regression: matches moments everywhere

KL between unnormalised stochastic processes rank 1

5 / 22

Fixed points of EP = FITC approximation

6 / 22

Fixed points of EP = FITC approximation

suppressed &

6 / 22

Fixed points of EP = FITC approximation

suppressed &

6 / 22

Fixed points of EP = FITC approximation

suppressed &

6 / 22

Fixed points of EP = FITC approximation

suppressed &

7 / 22

Fixed points of EP = FITC approximation

suppressed &

7 / 22

Fixed points of EP = FITC approximation

suppressed &

7 / 22

Fixed points of EP = FITC approximation

suppressed &

8 / 22

Fixed points of EP = FITC approximation

suppressed &

8 / 22

Fixed points of EP = FITC approximation

suppressed &

8 / 22

Fixed points of EP = FITC approximation

suppressed &

8 / 22

Fixed points of EP = FITC approximation

suppressed &

9 / 22

Fixed points of EP = FITC approximation

suppressed &

=

equivalent

9 / 22

Fixed points of EP = FITC approximation

Csato & Opper (2002) Qi, Abdel-Gawad & Minka (2010) suppressed &

=

equivalent

9 / 22

Fixed points of EP = FITC approximation

Csato & Opper (2002) Qi, Abdel-Gawad & Minka (2010)

Interpretation resolves philosophical issues with FITC (increase M with N) FITC likelihood > GP likelihood => EP over-estimates (marginal) likelihood

suppressed &

=

equivalent

9 / 22

EP algorithm

• 1. remove
• 2. include
• 3. project
• 4. update

take out one pseudo-observation likelihood add in one true observation likelihood project onto approximating family update pseudo-observation likelihood

cavity tilted

• 1. minimum: moments matched at pseudo-inputs
• 2. Gaussian regression: matches moments everywhere

KL between unnormalised stochastic processes rank 1

10 / 22

Power EP algorithm (as tractable as EP)

• 1. remove
• 2. include
• 3. project
• 4. update

take out fraction of pseudo-observation likelihood add in fraction of true observation likelihood project onto approximating family update pseudo-observation likelihood

cavity tilted

• 1. minimum: moments matched at pseudo-inputs
• 2. Gaussian regression: matches moments everywhere

KL between unnormalised stochastic processes rank 1

11 / 22

Power EP: a unifying framework

FITC Csato and Opper, 2002 Snelson and Ghahramani, 2005 VFE Titsias, 2009

12 / 22

Power EP: a unifying framework

Approximate blocks of data: structured approximations

13 / 22

Power EP: a unifying framework

Approximate blocks of data: structured approximations

13 / 22

Power EP: a unifying framework

PITC / BCM Schwaighofer & Tresp, 2002, Snelson 2006, VFE Titsias, 2009

Approximate blocks of data: structured approximations

13 / 22

Power EP: a unifying framework

PITC / BCM Schwaighofer & Tresp, 2002, Snelson 2006, VFE Titsias, 2009

Approximate blocks of data: structured approximations Place pseudo-data in different space: interdomain transformations (linear transform)

13 / 22

Power EP: a unifying framework

PITC / BCM Schwaighofer & Tresp, 2002, Snelson 2006, VFE Titsias, 2009

Approximate blocks of data: structured approximations Place pseudo-data in different space: interdomain transformations (linear transform)

13 / 22

Power EP: a unifying framework

PITC / BCM Schwaighofer & Tresp, 2002, Snelson 2006, VFE Titsias, 2009

Approximate blocks of data: structured approximations Place pseudo-data in different space: interdomain transformations (linear transform) pseudo-data in new space

13 / 22

Power EP: a unifying framework

PITC / BCM Schwaighofer & Tresp, 2002, Snelson 2006, VFE Titsias, 2009 Figueiras-Vidal & Lázaro-Gredilla 2009 T

• bar et al. 2015

Matthews et al, 2016

Approximate blocks of data: structured approximations Place pseudo-data in different space: interdomain transformations (linear transform) pseudo-data in new space

13 / 22

Power EP: a unifying framework

GP Regression GP Classification

[1] Quiñonero-Candela et al. 2005 [2] Snelson et al., 2005 [3] Snelson, 2006 [4] Schwaighofer, 2002 [13] Matthews et al., 2016 [14] Figueiras-Vidal et al., 2009 [15] Alverez et al. 2010 [5] Titsias, 2009 [6] Csató, 2002 [7] Csató et al., 2002 [8] Seeger et al., 2003 [9] Naish-Guzman et al, 2007 [10] Qi et al., 2010 [11] Hensman et al., 2015 [12] Hernández-Lobato et al., 2016 * = optimised pseudo-inputs ** = structured versions of VFE recover VFE (Remark 5)

PEP VFE EP inter-domain

[7,1,2*] [14,10] [6,8,5*] [15*,13*]

s t r u c t u r e d a p p r

• x

.

(FITC) [4,1*,3*] (PITC) ** [11*] [9*,12*] [10]

inter-domain s t r u c t u r e d a p p r

• x

. PEP VFE EP

** 14 / 22

How should I set the power parameter α?

6 UCI classification datasets 20 random splits M = 10, 50, 100 hypers and inducing inputs optimised 8 UCI regression datasets 20 random splits M = 0 - 200 hypers and inducing inputs optimised

0.0 0.2 0.4 0.6 0.8 1.0 2 3 4 5 6 7 8 0.0 0.2 0.4 0.6 0.8 1.0 2 3 4 5 6 7 8

MSE rank error rank log-loss rank log-loss rank = 0.5 does well on average

0.0 0.2 0.4 0.6 0.8 1.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 0.0 0.2 0.4 0.6 0.8 1.0 1 2 3 4 5 6 7

15 / 22

How should I set the power parameter α?

6 UCI classification datasets 8 UCI regression datasets MSE error rank log-loss rank log-loss rank = 0.5 does well on average

V FE V FE 0.05 0.1 0.2 0.4 0.5 0.6 0.8 EP 0.05 0.1 0.2 0.4 0.5 0.6 0.8 EP V FE V FE 0.05 0.1 0.2 0.4 0.5 0.6 0.8 EP 0.05 0.1 0.2 0.4 0.5 0.6 0.8 EP V FE V FE 0.01 0.1 0.2 0.4 0.5 0.6 0.8 EP 0.01 0.1 0.2 0.4 0.5 0.6 0.8 EP V FE V FE 0.01 0.1 0.2 0.4 0.5 0.6 0.8 EP 0.01 0.1 0.2 0.4 0.5 0.6 0.8 EP

1 EP beats VFE in 40% of tests 0.4

16 / 22

Streaming / Online Sparse Approximations Goal: Online posterior update (using old posterior and new data batch). Two new innovations for online learning and inducing input

• ptimisation
• 1. na¨

ıve approach: use previous approximate posterior as prior

new posterior

• q(new)(f ) ≈

new likelihood

• p(y(new)|f )
• ld posterior
• q(old)(f )
• ld likelihoods
• q(old)(f )

p(f |θ(old))

17 / 22

Streaming / Online Sparse Approximations Goal: Online posterior update (using old posterior and new data batch). Two new innovations for online learning and inducing input

• ptimisation
• 1. better approach: only take likelihood terms from old posterior

new posterior

• q(new)(f ) ≈

new likelihood

• p(y(new)|f )
• ld likelihoods
• q(old)(f )

p(f |θ(old))

• riginal prior
• p(f |θ(new))

17 / 22

Streaming / Online Sparse Approximations Goal: Online posterior update (using old posterior and new data batch). Two new innovations for online learning and inducing input

• ptimisation
• 1. better approach: only take likelihood terms from old posterior

new posterior

• q(new)(f ) ≈

new likelihood

• p(y(new)|f )
• ld likelihoods
• q(old)(f )

p(f |θ(old))

• riginal prior
• p(f |θ(new))
• 2. na¨

ıve approach: use same pseudo-points throughout q(old)(f ) = p(f=u|u, θ(old))q(u) q(new)(f ) = p(f=u|u, θ(new))q(u)

17 / 22

Streaming / Online Sparse Approximations Goal: Online posterior update (using old posterior and new data batch). Two new innovations for online learning and inducing input

• ptimisation
• 1. better approach: only take likelihood terms from old posterior

new posterior

• q(new)(f ) ≈

new likelihood

• p(y(new)|f )
• ld likelihoods
• q(old)(f )

p(f |θ(old))

• riginal prior
• p(f |θ(new))
• 2. better approach: decouple sets of pseudo-points

q(old)(f ) = p(f=u(old)|u(old), θ(old))q(u(old)) q(new)(f ) = p(f=u(new)|u(new), θ(new))q(u(new))

17 / 22

Streaming / Online Sparse Approximations Goal: Online posterior update (using old posterior and new data batch). Two new innovations for online learning and inducing input

• ptimisation
• 1. better approach: only take likelihood terms from old posterior

new posterior

• q(new)(f ) ≈

new likelihood

• p(y(new)|f )
• ld likelihoods
• q(old)(f )

p(f |θ(old))

• riginal prior
• p(f |θ(new))
• 2. better approach: decouple sets of pseudo-points

q(old)(f ) = p(f=u(old)|u(old), θ(old))q(u(old)) q(new)(f ) = p(f=u(new)|u(new), θ(new))q(u(new)) VFE is now the best Power EP method (inducing point clumping)

17 / 22

Online Sparse GP Approximations: Regression

• 2.0
• 1.0

0.0 1.0 2.0

y

−2 2 4 6 8 10 12

x

18 / 22

Online Sparse GP Approximations: Regression

• 2.0
• 1.0

0.0 1.0 2.0

y

• 2.0
• 1.0

0.0 1.0 2.0

y

−2 2 4 6 8 10 12

x

18 / 22

Online Sparse GP Approximations: Regression

• 2.0
• 1.0

0.0 1.0 2.0

y

• 2.0
• 1.0

0.0 1.0 2.0

y

• 2.0
• 1.0

0.0 1.0 2.0

y

−2 2 4 6 8 10 12

x

18 / 22

Online Sparse GP Approximations: Regression

• 2.0
• 1.0

0.0 1.0 2.0

y

• 2.0
• 1.0

0.0 1.0 2.0

y

• 2.0
• 1.0

0.0 1.0 2.0

y

−2 2 4 6 8 10 12

x

• 2.0
• 1.0

0.0 1.0 2.0

y

18 / 22

Online Sparse GP Approximations: Classification

−2 2

x1

−2 2

x2

❡rr♦r❂✵✳✷✽

−2 2

x1

❡rr♦r❂✵✳✶✺

−2 2

x1

❡rr♦r❂✵✳✶✵

−2 2

x1

❡rr♦r❂✵✳✶✵

19 / 22

Streaming / Online Sparse Approximations: Time-series Regression

mean held out log-likelihood

1 10 100 1000

• 8
• 6
• 4
• 2

2 4

• nline variational

exact batch VFE minibatch VFE

accumulated running time /s

20 / 22

Summary Provided a unifying framework for Gaussian Process Approximation methods using pseudo-points via PEP FITC and PITC are EP in disguise and they use the same approximating distribution as VFE Intermediate powers in PEP perform best on average in batch setting (more theory and empirical work needed) VFE methods perform best in the online setting Core material: A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation, arXiv preprint 2016 Streaming Sparse Gaussian Process Approximations, arXiv preprint 2017

21 / 22

VFE is best for online inference and learning

5 10 15 20 25

❜❛t❝❤ ✐♥❞❡①

1.0 1.5 2.0 2.5

♠❡❛♥ ❧♦❣✲❧✐❦❡❧✐❤♦♦❞

✵✳✵✶ ✵✳✷✵ ✵✳✺✵ ✵✳✽✵ ✶✳✵✵

22 / 22