Gaussian processes - Refresher and some more in insig ights - - PowerPoint PPT Presentation

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Gaussian processes

• Refresher and some more in

insig ights

Marcel Lüthi

Graphics and Vision Research Group Department of Mathematics and Computer Science University of Basel

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Outline

• Gaussian process – refresher
• Vector-valued and scalar valued Gaussian processes
• The space of samples
• Gaussian process regression

2

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Gaussian process: Formal definition

A Gaussian process 𝑞 𝑣 = 𝐻𝑄 𝜈, 𝑙 is a probability distribution over functions 𝑣 ∶ 𝒴 → ℝ𝑒 such that every finite restriction to function values 𝑣𝑌 = (𝑣 𝑦1 , … , 𝑣 𝑦𝑜 ) is a multivariate normal distribution 𝑞(𝑣𝑌) = 𝑂 𝜈𝑌, 𝑙𝑌𝑌 .

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Gaussian process: Illustration

Restriction to values at points 𝑌 = {𝑦} 𝑣 𝑦 = 𝑣1 𝑦 𝑣2 𝑦 ∼ 𝑂 𝜈𝑌, 𝑙𝑌𝑌 = 𝑂 𝜈1(𝑦) 𝜈2(𝑦) , 𝑙11(𝑦, 𝑦) 𝑙12(𝑦, 𝑦) 𝑙21(𝑦, 𝑦) 𝑙22(𝑦, 𝑦)

𝜈(𝑦) 𝑦

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Restriction to values at points 𝑌 = {𝑦, 𝑦′} 𝑣(𝑦) 𝑣(𝑦′) = 𝑣1 𝑦 𝑣2(𝑦) 𝑣1 𝑦′ 𝑣2(𝑦′) ∼ 𝑂 𝜈𝑌, 𝑙𝑌𝑌 = 𝑂 𝜈1(𝑦) 𝜈2(𝑦) 𝜈1(𝑦′) 𝜈2(𝑦′) , k11(𝑦, 𝑦) k12(𝑦, 𝑦) k21(𝑦, 𝑦) k22(𝑦, 𝑦) k11(𝑦, 𝑦′) k12(𝑦, 𝑦′) k21(𝑦, 𝑦′) k22(𝑦, 𝑦′) k11(𝑦′, 𝑦) k12(𝑦′, 𝑦) k21(𝑦′, 𝑦) k22(𝑦′, 𝑦) k11(𝑦′, 𝑦′) k12(𝑦′, 𝑦′) k21(𝑦′, 𝑦′) k22(𝑦′, 𝑦′)

𝑣(𝑦′) 𝑦′ 𝑣(𝑦) 𝑦

Gaussian process: Illustration

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

ex

Defining a Gaussian process

A Gaussian process 𝐻𝑄 𝜈, 𝑙 is completely specified by a mean function 𝜈 and covariance function (or kernel) 𝑙.

• 𝜈: 𝒴 → ℝ𝑒 defines how the average deformation looks like
• 𝑙: 𝒴 × 𝒴 → ℝ𝑒×𝑒 defines how it can deviate from the mean
• Must be positive semi-definite

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Marginalization property

Let 𝑌 = 𝑦1, … , 𝑦𝑜 and 𝑍 = 𝑧1, … , 𝑧𝑛 p 𝑌, 𝑍 = 𝑂 𝜈𝑌 𝜈𝑍 , Σ𝑌𝑌 Σ𝑌𝑍 Σ𝑍𝑌 Σ𝑍𝑍 The marginal distribution 𝑞 𝑌 = ∫ 𝑞 𝑌, 𝑍 𝑒𝑍 is given by 𝑞 𝑌 = 𝑂 𝜈𝑌, Σ𝑌𝑌 .

• Evaluating the Gaussian process 𝐻𝑄 𝜈, 𝑙 defined on domain 𝒴 at the points 𝑌 =

(𝑦1, … , 𝑦𝑜) is marginalizing out (ignoring) all random variables 𝒴 \ 𝑌

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Conceptual formulation: Continuous: 𝐻𝑄(𝜈, 𝑙) Practical implementation: Discrete: 𝑂(𝜈, 𝐿)

From continuous to discrete

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

We can write u ∼ 𝐻𝑄 𝜈, 𝑙 as 𝑣 ∼ 𝜈 + σ𝑗=1

∞ 𝛽𝑗

𝜇𝑗 𝜚𝑗, 𝛽𝑗 ∼ 𝑂(0, 1)

• 𝜚𝑗 is the eigenfunction with associated eigenvalue 𝜇𝑗 of the

linear operator [𝑈𝑙𝑣](𝑦) = ∫ 𝑙 𝑦, 𝑡 𝑣 𝑡 𝑒𝑡

The Karhunen-Loève expansion

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

• The total variance of the process

𝑣 ∼ 𝐻𝑄 𝜈, 𝑙 is given by σ𝑗=1

∞ 𝜇𝑗 .

• Observatio

ion: Most variance is explained by the first eigenfunctions

• Eigenvalue 𝜇𝑗
• Interpretation: Variance of 𝛽𝑗

𝜇𝑗𝜚𝑗

Eigenvalues and variance

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Main idea: Represent process using only the first 𝑠 components

• We have a finite, parametric representation of the process.
• Any deformation 𝑣 is determined by the coefficients

𝛽 = 𝛽1, … , 𝛽𝑠 𝑞 𝑣 = 𝑞 𝛽 = ෑ

𝑗=1 𝑠

1 2𝜌 exp(−𝛽𝑗

2/2)

𝑣 = 𝜈 + ෍

𝑗=1 𝑠

𝛽𝑗 𝜇𝑗 𝜚𝑗, 𝛽𝑗 ∼ 𝑂(0, 1)

Low-rank approximation

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Vector-valued and single valued Gaussian processes

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Scalar-valued Gaussian processes

Vector-valu lued (th (this is cou

• urse)
• Samples u are deformation fields:

𝑣: ℝ𝑜 → ℝ𝑒 Sc Scalar-valu lued (m (more common)

• Samples f are real-valued functions

𝑔 ∶ ℝ𝑜 → ℝ

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Scalar-valued Gaussian processes

Vector-valu lued (th (this is cou

• urse)

𝑣 ∼ 𝐻𝑄 Ԧ 𝜈, 𝒍 Ԧ 𝜈: 𝒴 → ℝ𝑒 𝒍: 𝒴 × 𝒴 → ℝ𝑒×𝑒 Sc Scalar-valu lued (m (more common) 𝑔 ∼ 𝐻𝑄 𝜈, 𝑙 𝜈: 𝒴 → ℝ 𝑙: 𝒴 × 𝒴 → ℝ

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

A connection

Matrix-valued kernels can be reinterpreted as scalar-valued kernels: Matrix valued kernel: 𝒍: 𝒴 × 𝒴 → ℝ𝒆×𝒆 Scalar valued kernel: 𝑙: 𝒴 × 1. . 𝑒 × 𝒴 × 1. . 𝑒 → ℝ Bijection: : Define 𝑙( 𝑦, 𝑗 , 𝑦′, 𝑘 = 𝒍 𝑦′, 𝑦′ 𝑗,𝑘

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

GP Regression – Vector-valued case

21

𝑳 = 𝑙11 𝑦1, 𝑦1 𝑙12 𝑦1, 𝑦1 𝑙21 𝑦1, 𝑦1 𝑙22 𝑦1, 𝑦1 … 𝑙11 𝑦1, 𝑦𝑜 𝑙12 𝑦1, 𝑦𝑜 𝑙21 𝑦1, 𝑦𝑜 𝑙22 𝑦1, 𝑦𝑜 ⋮ ⋮ 𝑙11 𝑦𝑜, 𝑦1 𝑙12 𝑦𝑜, 𝑦1 𝑙21 𝑦𝑜, 𝑦1 𝑙22 𝑦𝑜, 𝑦1 … 𝑙11 𝑦𝑜, 𝑦𝑜 𝑙12 𝑦𝑜, 𝑦𝑜 𝑙21 𝑦𝑜, 𝑦𝑜 𝑙22 𝑦𝑜, 𝑦𝑜

𝐿 = 𝑙 (𝑦1, 1), (𝑦1, 1) 𝑙 (𝑦1, 1), (𝑦1, 2) 𝑙 𝑦1, 2 , (𝑦1, 1) 𝑙 𝑦1, 2 , (𝑦1, 2) … 𝑙 (𝑦1, 1), (𝑦𝑜, 1) 𝑙 (𝑦1, 1), (𝑦𝑜, 2) 𝑙 𝑦1, 2 , (𝑦𝑜, 1) 𝑙 𝑦1, 2 , (𝑦𝑜, 2) ⋮ ⋮ 𝑙 (𝑦𝑜, 1), (𝑦1, 1) 𝑙 (𝑦𝑜, 1), (𝑦1, 2) 𝑙 𝑦𝑜, 2 , (𝑦1, 1) 𝑙 𝑦𝑜, 2 , (𝑦1, 2) … 𝑙 (𝑦𝑜, 1), (𝑦𝑜, 1) 𝑙 (𝑦𝑜, 1), (𝑦𝑜, 2) 𝑙 𝑦𝑜, 2 , (𝑦𝑜, 1) 𝑙 𝑦𝑜, 2 , (𝑦𝑜, 2)

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

A connection

Matrix-valued kernels can be reinterpreted as scalar-valued kernels: Matrix valued kernel: 𝒍: 𝒴 × 𝒴 → ℝ𝒆×𝒆 Scalar valued kernel: 𝑙: 𝒴 × 1. . 𝑒 × 𝒴 × 1. . 𝑒 → ℝ Bijection: : Define 𝑙( 𝑦, 𝑗 , 𝑦′, 𝑘 = 𝒍 𝑦′, 𝑦′ 𝑗,𝑘 All the theory developed for the scalar-valued GPs holds also for vector-valued GPs!

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Sampling revisited

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Finite views on infinite objects

24

Gaussian process Infinite dimensional Finite dimensional Continuous domain Finite domain (Marginalization) Finite rank (KL- Expansion)

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

The space of samples

Sampling from 𝐻𝑄 𝜈, 𝑙 is done using the corresponding normal distribution 𝑂( Ԧ 𝜈, K) Algorithm for sampling (slightly inefficient) 1. Do an SVD: K = 𝑉𝐸2𝑉𝑈

• 2. Draw a normal vector 𝛽 ∼ 𝑂 0, 𝐽𝑜×𝑜
• 3. Compute Ԧ

𝜈 + 𝑉𝐸𝛽

25

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

The space of samples

• From K = 𝑉𝐸2𝑉𝑈(using that 𝑉𝑈𝑉 = 𝐽) we have that

K𝑉𝐸−1 = 𝑉𝐸

• Any sample

𝑡 = Ԧ 𝜈 + 𝑉𝐸𝛽 = Ԧ 𝜈 + K𝑉𝐸−1𝛽 = 𝜈 + K𝛾 is a linear combinations of the columns of K. Two ways to represent sample:

• 1. KL-Expansion: 𝑡 = Ԧ

𝜈 + σ𝑗 𝑒𝑗𝛽𝑗𝑣𝑗

• 2. Linear combination of kernels: 𝑡 = Ԧ

𝜈 + σ𝑘 𝛾𝑙𝑘

26

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Four examples covariance functions

27

𝑙 𝑦, 𝑦′ = ෍

𝑗=1 3

𝑔

𝑗 𝑦 𝑔 𝑗(𝑦′)

𝑔

1 𝑦 = sin 𝑦 , 𝑔 2 𝑦 = 𝑦, 𝑔 3 𝑦 = cos(𝑦 2)

𝑙 𝑦, 𝑦′ = 𝑔 𝑦 𝑔 𝑦′ f x = (1 − 𝑡 𝑦 )2𝑦2 + 𝑡 𝑦 sin 𝑦2

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Four examples covariance functions

28

𝑙 𝑦, 𝑦′ = 𝜀(𝑦, 𝑦′)

𝑙 𝑦, 𝑦′ = exp − 𝑦 − 𝑦′ 2 9

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Example 1

29

𝑙 𝑦, 𝑦′ = 𝑔 𝑦 𝑔(𝑦′)

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Example 1

30

𝑙 𝑦, 𝑦′ = 𝑔 𝑦 𝑔(𝑦′)

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Example 1

31

𝑙 𝑦, 𝑦′ = 𝑔 𝑦 𝑔(𝑦′)

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Example 2

32

𝑙 𝑦, 𝑦′ = ෍

𝑗=1 3

𝑔

𝑗 𝑦 𝑔 𝑗(𝑦′)

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Example 2

33 𝑙 𝑦, 𝑦′ = ෍

𝑗=1 3

𝑔

𝑗 𝑦 𝑔 𝑗(𝑦′)

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Example 2

34

𝑙 𝑦, 𝑦′ = ෍

𝑗=1 3

𝑔

𝑗 𝑦 𝑔 𝑗(𝑦′)

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Example 3

35

𝑙 𝑦, 𝑦′ = exp − 𝑦 − 𝑦′ 2 9

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Example 3

36

𝑙 𝑦, 𝑦′ = exp − 𝑦 − 𝑦′ 2 9

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Example 3

37

𝑙 𝑦, 𝑦′ = exp − 𝑦 − 𝑦′ 2 9

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Example 4

38

𝑙 𝑦, 𝑦′ = 𝜀(𝑦, 𝑦′)

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Example 4

39

𝑙 𝑦, 𝑦′ = 𝜀(𝑦, 𝑦′)

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Example 4

40

𝑙 𝑦, 𝑦′ = 𝜀(𝑦, 𝑦′)

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Gaussian process regression revisited

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Gaussian process regression

• Given: observations {(𝑦1, 𝑧1), … , 𝑦𝑜, 𝑧𝑜 }
• Model: 𝑧𝑗 = 𝑔 𝑦𝑗 + 𝜗, 𝑔 ∼ 𝐻𝑄(𝜈, 𝑙)
• Goal: compute p(𝑧∗|𝑦∗, 𝑦1, … , 𝑦𝑜, 𝑧1, … , 𝑧𝑜)

53

𝑦1 𝑦2 𝑦𝑜 𝑦∗ 𝑧∗

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Gaussian process regression

• Solution given by posterior process 𝐻𝑄 𝜈𝑞, 𝑙𝑞 with

𝜈𝑞(𝑦∗) = 𝐿 𝑦∗, 𝑌 𝐿 𝑌, 𝑌 + 𝜏2𝐽 −1𝑧 𝑙𝑞 𝑦∗, 𝑦∗′ = 𝑙 𝑦∗, 𝑦∗′ − 𝐿 𝑦∗, 𝑌 𝐿 𝑌, 𝑌 + 𝜏2𝐽 −1𝐿 𝑌, 𝑦∗

′

• The covariance is independent of the value at the training points
• Structure of posterior GP determined solely by kernel.

54

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Examples

60

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Example: Gaussian kernel

61

σ = 1

𝑙 𝑦, 𝑦′ = exp − 𝑦 − 𝑦′ 2 𝜏2

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Examples

62

• Gaussian kernel (𝜏 = 1)

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Example: Gaussian kernel

63

𝑙 𝑦, 𝑦′ = exp − 𝑦 − 𝑦′ 2 𝜏2

σ = 3

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Examples

64

• Gaussian kernel (𝜏 = 5)

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Periodic kernels

• Define 𝑣 𝑦 =

cos 𝑦 sin(𝑦)

• 𝑙 𝑦, 𝑦′ = exp(−‖(𝑣 𝑦 − 𝑣 𝑦′ ‖2= exp(−4 sin2

‖𝑦 −𝑦′‖ 𝜏2

)

65

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Examples

66

• Periodic kernel

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Changepoint kernels

• 𝑙 𝑦, 𝑦′ = 𝑡 𝑦 𝑙1 𝑦, 𝑦′ 𝑡 𝑦′ + (1 − 𝑡 𝑦 )𝑙2(𝑦, 𝑦′)(1 − 𝑡 𝑦′ )
• s 𝑦 =

1 1+exp( −𝑦)

67

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Examples

68

• Changepoint kernel

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Symmetric kernels

• Enforce that f(x) = f(-x)
• 𝑙 𝑦, 𝑦′ = 𝑙 −𝑦, 𝑦′ + 𝑙(𝑦, 𝑦′)

69

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Examples

70

• Symmetric kernel

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL

Summary

• Gaussian processes are an extremely rich toolbox for modelling functions / deformation

fields

• Possible functions are linear combinations of the kernels 𝑙(⋅, 𝑦), fixed at one point 𝑦
• Kernels 𝑙(⋅, 𝑦) form the basis of the space of possible functions
• Regularity/smoothness of kernels is transferred to samples
• In inference tasks, the structure of the kernel determines the prediction
• => Extremely important to model it well