,Y\ 00 @ hi 0 1 . 5h45 , Y } QOO y yz , *=E[ be - - PowerPoint PPT Presentation

SLIDE 1 Posterior Predictive Distribution . 1 , y

,Y\

yz Y } 5h45 , Small be ya9uy5

@

00

1

QOO

' , 7=5 y* SZ ' l

• hi

co i , y # / Large A

• o0

µ

y # y* P(y*ly , :n ) =

|df

pcytlfspcfly , :µ )

µ*=E[

4*14 , :b ) Gaussian Processes Kernel Ridge Regression

SLIDE 2 Gaussian Processes

g

Infinite / Unbounded Degrees

• f

freedom Formal View : Nonparametric Distribution

• n

Functions f ~ p ( f 1 Y , :w ) ° µ( × ) Mean function

• *

)

• .

...

h( × ,× ' ) Coviarana function fcx) .

• .

' . F ~ GPC µ , h ) Prior an functions

YuIF=f~

Norm (

flIn

) ,6 ) Likelihood ×

• Likelihood

prior Posterior : p( fly , :µ ) = p( y , in , 1 f) pcf ) pcyi :n ) Evidence / Marginal

SLIDE 3 Gaussian Processes Practical View : Generalization

• f

Multivariate Normal plfcx ) 1 blini ) µn

i=

µ (In ) £m÷ k( In ,Im ) fk ' E~ Nonmlpi , 2)

TIFIF

• Norm

If ,o2I ) x * t.IE I

g*=KII

't split

" Predictive :

Pc f

*

15

)

=/

piylfspc # f) DI

t Pc 5)

SLIDE 4 Joint Distributions an Function Values

nleext

• Nonmllyu

"¥ , ] ,KuYx¥xthuYx¥IH

)

:

uk ) =(µlIn , ... ,pkI , ) ) M # taxes , ... ,µ#n na ,×*,= h

?

"&%" she , .nl HKI, .si?l.....hlxInFn ) Y = E + E E

• µom(µx

) , hlx ,Xl ) 4- ~ Norm ( Mk) , kk ,X)tEI ) E ~ Norm (

• ,

621 ) ( ¥* )

• Nonm( I

,yYI!,/

,

µ×#

to 's hkx 't . HKIH ' hcx:o))

SLIDE 5 Properties

• f

Multivariate Normans Tf ) : Probability density for jointly Gaussian variables µ M µ N N×N NXM piers =

Nonmflfililabt

, lets ) ) M M M×µ MXM Marginal Conditional N NXMMXMM pH ) = Norm # i. A) pc

ftp.t.N/a;a+cB'lpi.bl.pCp)=Norm(p;5iB

) A . CB "Ct ) 14×14 NXM MXMMXN

SLIDE 6

Predictive

Distributions

• n

Function Values ys ' =5

• µK )
• l¥*]~N0nmlkµY¥

, ]

.MY#xxtt*huYxYIM

)

Can define 51=5

• Mk

) Predictive an new function values I* µi(×)=o

pct*lF=j)=

Normlt

;FK*hkKK*

) ) /

Fla

= µcx*s + hk ,*xl ( kk,×l+iIYh

.µ¢x

)) M m Mxkl NXN X '

hiya

= hk ,# + hcxxixllhlx ,x+MIhH,x* ) MXM MXM MXN µxN µ×M \

SLIDE 7 kernel Ridge Regression f * = li ( k + Its ' ' g 5=14 ' , ' ' ' , Yn ) Knm = be ( In ,Im ) ten = lr ( I* , In )

SLIDE 8 Gaussian Processes vs Kennel Ridge Regression Reformulation

• f

Kennel Ridge Regression it ~ Norm ( E. S . ) fix

's

= &?wa¢ak 's e- ~ Norm ( 8,621 ) 5 = f (E) + E lek ; ,I ;) = 4tEilTS.4kj7-d@e0aKilSo.de ctekj )

• Ft [ 4*15=5 ]

= h(x*,x ) ( hk ,x ) +6215 ' 5 =file ) ( µKl=o ) Kernel Ridge Regression c→ Mean

• f

GP Regression

SLIDE 9 Regularization in Kennel

• based

Regression . l y

,Y\

yz 4 } 5h55 , Small

00

be ya9uy5

@

8

@@8

1

HE

y* SZ ' 1 D= B

• oo#

15.12 y # / Large #

• ooµ

y # y* . absorbed into Cov , fwnc ! Choice

• f

kernel h( Ei ,I ;) =

{

¢d( Ii ) So.de/0elEj ) function implies e choice

• f

9

SLIDE 10 Choosing Kernel Hyperparameters Source : Carl Rasmussen Squared

• Exponential

: h( x. x ' ) = exp (-1×2*3) Large l means stronger regularization

SLIDE 11 Matern kernels Idea : prior

• ver

h times differentiable functions then

#inga

, Tom Rainforth , PhD Thesis k=w

• D

Gamma

• leu (

E ,I ' ) = 6 , }l÷g KfM2q HE

• I 'll )

0=312,512 , " .

SLIDE 12 Basis Kernels Source : David Duvenaud , PhD Thesis

SLIDE 13 Combinations

• f

kernels Source : David Duvenaud , PhD Thesis Sumi . h( I ,E ' ) = be ,( I. In thzk ,I ' ) Product : hl I ,k ' ) = be ,( I. In . hz( I ,I ' )

SLIDE 14 Inner Products Definition : Given It vector space

• ver

R a function ( ; a) µ : HxH→R is an inner product when it is 1 . Linear ( w , f , + wzfz , g) µ = W , ( f , ,g >

twz

< fz , g) 2 . Symmetric ( f. g 7 µ = ( g. f7µ 3 . Positive definite 2 f. f7µ > ,

• ( f.

f) =o iff , f=o

SLIDE 15 Hilbert Spaces & Kennels Definition : A vector space H equipped With an inner product (containing Cauchy limit sequences ) Is a Hilbert space y need not be vector space ! Definition : A function h :4xX→R Videos document is a kernel , if there is a function numbers Feature ¢ :X → H such That f×,× , € z teeters mapµEF ,IF . .= Lax ) .dk ' ' )H

SLIDE 16 Infinite Sequences Definition ; lz ( square summate sequences ) comprises all W ÷ ( Wa ) dz , Such that as HWHE , = I War < as D= , Given a sequence ( loidk ) )d , , in lz where ¢d :X → R is the d- th coordinate be ( × ,x ' ) := §|¢d(×)¢d( × ' ) = ( 441,4k 'Dµ

SLIDE 17 Positive Definiteness Theorem : If H is a Hibbert space ¢ :X → H is a feature map ( and X is a non . empty set ) then be ( × ,x' 1 ÷ { 4kt , ¢ 1y ) 7 µ is positive definite Theorem . ( Moore . Anonsujn ) : For any positive definite h( x. x ' )

there

is a unique Reproducing kernel Hibert Space ( RKHS )

SLIDE 18 Reproducing Kernels Definition : Suppose that H

is

a Hilbert space

• ver

functions f :f→R then 71 . is a reproducing kernel Hilbert space ( RKHS ) When

• 1. k(

. , x ) EH txeX 2. ( fl . ) , hl . , × ) ? µ = fix ) ( Kernel trick ) ( Reproducing Property) Implication h( x. × ' ) = ( 441,4 K ' )7µ=LkC . ,x ) ,kl;× 'D

SLIDE 19 Function Space Equivalence Classes Reproducing Positive Definite Kernels Functions Hilbert function spaces with bounded point evaluation

SLIDE 20 Classic Example : XOR , 1 , , , ' "

Xzo

• ,

, ,

• .

. I I I × , Features dk , ,xz)=

[ ¥z×z|

Kennel hk.pt/#zaMy?l g.)

SLIDE 21 Classic Example : XOR Features

• tlxl

= [ ¥g×z| Kennel

hkijs

• . III.MY

;D = ( ¢ki ,

41517,3

Functions : fix ) = W , × , twzxz + W

}×i×z

= (8,4K¥ , } Rpnodnny

flit

= { in ,¢KDp } = 2ft ) ,¢KDµ Property ¢ # , = kl . ,I )

SLIDE 22 Implications fan Gaussian Processes F ~ GP(

• ,

k ) . h( x. x ' I is positive definite → f lies in RKHS Posterior : F 15=5 ~ GPC

F.

k )

picas

=

kC¥×l(

kcx ,x)+iI5' 5 =

{mhlxsixnllhkixltttttnmym

pic . )= { kl . ,En) wn