Latent Force Models and Multiple Output Gaussian Processes Neil D. - PowerPoint PPT Presentation

Latent Force Models and Multiple Output Gaussian Processes Neil D. Lawrence work with Magnus Rattray, Mauricio Alvarez, Pei Gao, Antti Honkela, David Luengo, Guido Sanguinetti, Michalis Titsias, Jennifer Withers SLIM Meeting 23rd July 2009 Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 1 / 36

Outline Introduction 1 Latent Force Covariance Functions 2 Cascaded Differential Equations 3 Discussion and Future Work 4 Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 2 / 36

Outline Introduction 1 Latent Force Covariance Functions 2 Cascaded Differential Equations 3 Discussion and Future Work 4 Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 3 / 36

Dimensionality Reduction I Linear relationship between the data, X ∈ ℜ N × d , and a reduced dimensional representation, F ∈ ℜ N × q , where q ≪ d . X = FW + ǫ , ǫ ∼ N ( 0 , Σ ) Integrate out F , optimize with respect to W . For temporal data and a particular Gaussian prior in the latent space: Kalman filter/smoother. More generally consider a Gaussian process (GP) prior, q � � � p ( F | t ) = N f : , i | 0 , K f : , i , f : , i . i =1 Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 4 / 36

Dimensionality Reduction II Given the covariance functions for { f i ( t ) } the implied covariance functions for { x i ( t ) } — semi-parametric latent factor model (Teh et al., 2005). Linear Models of Coregionalization. Kalman filter/smoother approach has been preferred ◮ linear computational complexity in N . ◮ Advances in sparse approximations have made the general GP framework practical. (Snelson and Ghahramani, 2006; Qui˜ nonero Candela and Rasmussen, 2005; Titsias, 2009) . Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 5 / 36

Mechanical Analogy These models rely on the latent variables to provide the dynamic information. We now introduce a further dynamical system with a mechanistic inspiration. Physical Interpretation: ◮ the latent functions, f i ( t ) are q forces. ◮ We observe the displacement of d springs to the forces., ◮ Interpret system as the force balance equation, XD = FS + ǫ . ◮ Forces act, e.g. through levers — a matrix of sensitivities, S ∈ ℜ q × d . ◮ Diagonal matrix of spring constants, D ∈ ℜ d × d . ◮ Original System: W = SD − 1 . Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 6 / 36

Extend Model Add a damper and give the system mass. FS = ¨ XM + ˙ XC + XD + ǫ . Now have a second order mechanical system. It will exhibit inertia and resonance. There are many systems that can also be represented by differential equations. ◮ When being forced by latent function(s), { f i ( t ) } q i =1 , we call this a latent force model . Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 7 / 36

Gaussian Process priors and Latent Force Models Driven Harmonic Oscillator For Gaussian process we can compute the covariance matrices for the output displacements. For one displacement the model is M � m k ¨ x k ( t ) + c k ˙ x k ( t ) + d k x k ( t ) = b k + s ik f i ( t ) , (1) i =0 where, m k is the k th diagonal element from M and similarly for c k and d k . s ik is the i , k th element of S . Model the latent forces as q independent, GPs with RBF covariances − ( t − t ′ ) 2 � � k f i f l ( t , t ′ ) = exp δ il . σ 2 i Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 8 / 36

Covariance for ODE Model RBF Kernel function for f ( t ) � t q 1 � x j ( t ) = S ji exp( − α j t ) f i ( u ) exp( α j u ) sin( ω j ( t − u )) d u m j ω j 0 i =1 Joint distribution f(t) 0.8 for x 1 ( t ), x 2 ( t ), 0.6 y 1 (t) x 3 ( t ) and f ( t ). 0.4 Damping ratios: 0.2 y 2 (t) 0 ζ 1 ζ 2 ζ 3 −0.2 0.125 2 1 y 3 (t) −0.4 f(t) y 1 (t) y 2 (t) y 3 (t) Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 9 / 36

Joint Sampling of x ( t ) and f ( t ) demLfmSample 2 1.5 1 0.5 0 −0.5 −1 −1.5 −2 −2.5 0 5 10 15 20 Figure: Joint samples from the ODE covariance, cyan : f ( t ), red : x 1 ( t )(underdamped) and green : x 2 ( t ) (overdamped) and blue : x 3 ( t ) (critically damped). Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 10 / 36

Joint Sampling of x ( t ) and f ( t ) demLfmSample 2 1.5 1 0.5 0 −0.5 −1 −1.5 −2 −2.5 0 5 10 15 20 Figure: Joint samples from the ODE covariance, cyan : f ( t ), red : x 1 ( t )(underdamped) and green : x 2 ( t ) (overdamped) and blue : x 3 ( t ) (critically damped). Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 10 / 36

Joint Sampling of x ( t ) and f ( t ) demLfmSample 2 1.5 1 0.5 0 −0.5 −1 0 5 10 15 20 Figure: Joint samples from the ODE covariance, cyan : f ( t ), red : x 1 ( t )(underdamped) and green : x 2 ( t ) (overdamped) and blue : x 3 ( t ) (critically damped). Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 10 / 36

Joint Sampling of x ( t ) and f ( t ) demLfmSample 2 1.5 1 0.5 0 −0.5 −1 −1.5 −2 −2.5 0 5 10 15 20 Figure: Joint samples from the ODE covariance, cyan : f ( t ), red : x 1 ( t )(underdamped) and green : x 2 ( t ) (overdamped) and blue : x 3 ( t ) (critically damped). Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 10 / 36

Covariance for ODE RBF Kernel function for f ( t ) � t q 1 � x j ( t ) = S ji exp( − α j t ) f i ( u ) exp( α j u ) sin( ω j ( t − u )) d u m j ω j 0 i =1 Joint distribution f(t) 0.8 for x 1 ( t ), x 2 ( t ), 0.6 y 1 (t) x 3 ( t ) and f ( t ). 0.4 0.2 Damping ratios: y 2 (t) 0 ζ 1 ζ 2 ζ 3 −0.2 y 3 (t) 0.125 2 1 −0.4 f(t) y 1 (t) y 2 (t) y 3 (t) Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 11 / 36

Example: Motion Capture Mauricio Alvarez and David Luengo (´ Alvarez et al., 2009) Motion capture data: used for animating human motion. Multivariate time series of angles representing joint positions. Objective: generalize from training data to realistic motions. Use 2nd Order Latent Force Model with mass/spring/damper (resistor inductor capacitor) at each joint. demAistats Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 12 / 36

Example: Motion Capture Mauricio Alvarez and David Luengo (´ Alvarez et al., 2009) Motion capture data: used for animating human motion. Multivariate time series of angles representing joint positions. Objective: generalize from training data to realistic motions. Use 2nd Order Latent Force Model with mass/spring/damper (resistor inductor capacitor) at each joint. demAistats Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 12 / 36

Example: Motion Capture Mauricio Alvarez and David Luengo (´ Alvarez et al., 2009) Motion capture data: used for animating human motion. Multivariate time series of angles representing joint positions. Objective: generalize from training data to realistic motions. Use 2nd Order Latent Force Model with mass/spring/damper (resistor inductor capacitor) at each joint. demAistats Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 12 / 36

Example: Motion Capture Mauricio Alvarez and David Luengo (´ Alvarez et al., 2009) Motion capture data: used for animating human motion. Multivariate time series of angles representing joint positions. Objective: generalize from training data to realistic motions. Use 2nd Order Latent Force Model with mass/spring/damper (resistor inductor capacitor) at each joint. demAistats Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 12 / 36

Example: Transcriptional Regulation r First Order Differential Equation d x j ( t ) = B j + S j f ( t ) − D j x j ( t ) d t Can be used as a model of gene transcription: Barenco et al., 2006; Gao et al., 2008. x j ( t ) – concentration of gene j ’s mRNA f ( t ) – concentration of active transcription factor Model parameters: baseline B j , sensitivity S j and decay D j Application: identifying co-regulated genes (targets) Problem: how do we fit the model when f ( t ) is not observed? Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 13 / 36

Example: Transcriptional Regulation r First Order Differential Equation d x j ( t ) = B j + S j f ( t ) − D j x j ( t ) d t Can be used as a model of gene transcription: Barenco et al., 2006; Gao et al., 2008. x j ( t ) – concentration of gene j ’s mRNA f ( t ) – concentration of active transcription factor Model parameters: baseline B j , sensitivity S j and decay D j Application: identifying co-regulated genes (targets) Problem: how do we fit the model when f ( t ) is not observed? Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 13 / 36

Example: Transcriptional Regulation r First Order Differential Equation d x j ( t ) = B j + S j f ( t ) − D j x j ( t ) d t Can be used as a model of gene transcription: Barenco et al., 2006; Gao et al., 2008. x j ( t ) – concentration of gene j ’s mRNA f ( t ) – concentration of active transcription factor Model parameters: baseline B j , sensitivity S j and decay D j Application: identifying co-regulated genes (targets) Problem: how do we fit the model when f ( t ) is not observed? Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 13 / 36