Latent Force Models and Multiple Output Gaussian Processes Neil D. - - PowerPoint PPT Presentation

▶
latent force models and multiple output gaussian processes
SMART_READER_LITE
LIVE PREVIEW

Latent Force Models and Multiple Output Gaussian Processes Neil D. - - PowerPoint PPT Presentation

Dec 27, 2023 153 likes •1.08k views

Latent Force Models and Multiple Output Gaussian Processes Neil D. Lawrence work with Magnus Rattray, Mauricio Alvarez, Pei Gao, Antti Honkela, David Luengo, Guido Sanguinetti, Michalis Titsias, Jennifer Withers SLIM Meeting 23rd July 2009

slide-1
SLIDE 1

Latent Force Models and Multiple Output Gaussian Processes

Neil D. Lawrence work with Magnus Rattray, Mauricio Alvarez, Pei Gao, Antti Honkela, David Luengo, Guido Sanguinetti, Michalis Titsias, Jennifer Withers

SLIM Meeting

23rd July 2009

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 1 / 36

slide-2
SLIDE 2

Outline

1

Introduction

2

Latent Force Covariance Functions

3

Cascaded Differential Equations

4

Discussion and Future Work

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 2 / 36

slide-3
SLIDE 3

Outline

1

Introduction

2

Latent Force Covariance Functions

3

Cascaded Differential Equations

4

Discussion and Future Work

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 3 / 36

slide-4
SLIDE 4

Dimensionality Reduction I

Linear relationship between the data, X ∈ ℜN×d, and a reduced dimensional representation, F ∈ ℜN×q, where q ≪ d. X = FW + ǫ, ǫ ∼ N (0, Σ) Integrate out F, optimize with respect to W. For temporal data and a particular Gaussian prior in the latent space: Kalman filter/smoother. More generally consider a Gaussian process (GP) prior, p (F|t) =

q

  • i=1

N

  • f:,i|0, Kf:,i,f:,i
  • .

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 4 / 36

slide-5
SLIDE 5

Dimensionality Reduction II

Given the covariance functions for {fi(t)} the implied covariance functions for {xi(t)} — semi-parametric latent factor model (Teh et al., 2005). Linear Models of Coregionalization. Kalman filter/smoother approach has been preferred

◮ linear computational complexity in N. ◮ Advances in sparse approximations have made the general GP

framework practical. (Snelson and Ghahramani, 2006; Qui˜

nonero Candela and Rasmussen, 2005; Titsias, 2009).

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 5 / 36

slide-6
SLIDE 6

Mechanical Analogy

These models rely on the latent variables to provide the dynamic information. We now introduce a further dynamical system with a mechanistic inspiration. Physical Interpretation:

◮ the latent functions, fi(t) are q forces. ◮ We observe the displacement of d springs to the forces., ◮ Interpret system as the force balance equation, XD = FS + ǫ. ◮ Forces act, e.g. through levers — a matrix of sensitivities, S ∈ ℜq×d. ◮ Diagonal matrix of spring constants, D ∈ ℜd×d. ◮ Original System: W = SD−1. Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 6 / 36

slide-7
SLIDE 7

Extend Model

Add a damper and give the system mass. FS = ¨ XM + ˙ XC + XD + ǫ. Now have a second order mechanical system. It will exhibit inertia and resonance. There are many systems that can also be represented by differential equations.

◮ When being forced by latent function(s), {fi(t)}q

i=1, we call this a

latent force model.

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 7 / 36

slide-8
SLIDE 8

Gaussian Process priors and Latent Force Models

Driven Harmonic Oscillator

For Gaussian process we can compute the covariance matrices for the

  • utput displacements.

For one displacement the model is mk¨ xk(t) + ck ˙ xk(t) + dkxk(t) = bk +

M

  • i=0

sikfi(t), (1) where, mk is the kth diagonal element from M and similarly for ck and dk. sik is the i, kth element of S. Model the latent forces as q independent, GPs with RBF covariances kfifl(t, t′) = exp

  • −(t − t′)2

σ2

i

  • δil.

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 8 / 36

slide-9
SLIDE 9

Covariance for ODE Model

RBF Kernel function for f (t) xj(t) = 1 mjωj

q

  • i=1

Sji exp(−αjt) t fi(u) exp(αju) sin(ωj(t − u))du Joint distribution for x1 (t), x2 (t), x3 (t) and f (t). Damping ratios:

ζ1 ζ2 ζ3

0.125 2 1

f(t) y1(t) y2(t) y3(t) f(t) y1(t) y2(t) y3(t)

−0.4 −0.2 0.2 0.4 0.6 0.8

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 9 / 36

slide-10
SLIDE 10

Joint Sampling of x (t) and f (t)

demLfmSample

5 10 15 20 −2.5 −2 −1.5 −1 −0.5 0.5 1 1.5 2

Figure: Joint samples from the ODE covariance, cyan: f (t), red: x1 (t)(underdamped) and green: x2 (t) (overdamped) and blue: x3 (t) (critically damped).

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 10 / 36

slide-11
SLIDE 11

Joint Sampling of x (t) and f (t)

demLfmSample

5 10 15 20 −2.5 −2 −1.5 −1 −0.5 0.5 1 1.5 2

Figure: Joint samples from the ODE covariance, cyan: f (t), red: x1 (t)(underdamped) and green: x2 (t) (overdamped) and blue: x3 (t) (critically damped).

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 10 / 36

slide-12
SLIDE 12

Joint Sampling of x (t) and f (t)

demLfmSample

5 10 15 20 −1 −0.5 0.5 1 1.5 2

Figure: Joint samples from the ODE covariance, cyan: f (t), red: x1 (t)(underdamped) and green: x2 (t) (overdamped) and blue: x3 (t) (critically damped).

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 10 / 36

slide-13
SLIDE 13

Joint Sampling of x (t) and f (t)

demLfmSample

5 10 15 20 −2.5 −2 −1.5 −1 −0.5 0.5 1 1.5 2

Figure: Joint samples from the ODE covariance, cyan: f (t), red: x1 (t)(underdamped) and green: x2 (t) (overdamped) and blue: x3 (t) (critically damped).

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 10 / 36

slide-14
SLIDE 14

Covariance for ODE

RBF Kernel function for f (t) xj(t) = 1 mjωj

q

  • i=1

Sji exp(−αjt) t fi(u) exp(αju) sin(ωj(t − u))du Joint distribution for x1 (t), x2 (t), x3 (t) and f (t). Damping ratios:

ζ1 ζ2 ζ3

0.125 2 1

f(t) y1(t) y2(t) y3(t) f(t) y1(t) y2(t) y3(t)

−0.4 −0.2 0.2 0.4 0.6 0.8

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 11 / 36

slide-15
SLIDE 15

Example: Motion Capture

Mauricio Alvarez and David Luengo (´ Alvarez et al., 2009) Motion capture data: used for animating human motion. Multivariate time series of angles representing joint positions. Objective: generalize from training data to realistic motions. Use 2nd Order Latent Force Model with mass/spring/damper (resistor inductor capacitor) at each joint. demAistats

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 12 / 36

slide-16
SLIDE 16

Example: Motion Capture

Mauricio Alvarez and David Luengo (´ Alvarez et al., 2009) Motion capture data: used for animating human motion. Multivariate time series of angles representing joint positions. Objective: generalize from training data to realistic motions. Use 2nd Order Latent Force Model with mass/spring/damper (resistor inductor capacitor) at each joint. demAistats

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 12 / 36

slide-17
SLIDE 17

Example: Motion Capture

Mauricio Alvarez and David Luengo (´ Alvarez et al., 2009) Motion capture data: used for animating human motion. Multivariate time series of angles representing joint positions. Objective: generalize from training data to realistic motions. Use 2nd Order Latent Force Model with mass/spring/damper (resistor inductor capacitor) at each joint. demAistats

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 12 / 36

slide-18
SLIDE 18

Example: Motion Capture

Mauricio Alvarez and David Luengo (´ Alvarez et al., 2009) Motion capture data: used for animating human motion. Multivariate time series of angles representing joint positions. Objective: generalize from training data to realistic motions. Use 2nd Order Latent Force Model with mass/spring/damper (resistor inductor capacitor) at each joint. demAistats

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 12 / 36

slide-19
SLIDE 19

Example: Transcriptional Regulation

r First Order Differential Equation dxj (t) dt = Bj + Sjf (t) − Djxj (t) Can be used as a model of gene transcription: Barenco et al., 2006; Gao et al., 2008. xj(t) – concentration of gene j’s mRNA f (t) – concentration of active transcription factor Model parameters: baseline Bj, sensitivity Sj and decay Dj Application: identifying co-regulated genes (targets) Problem: how do we fit the model when f (t) is not observed?

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 13 / 36

slide-20
SLIDE 20

Example: Transcriptional Regulation

r First Order Differential Equation dxj (t) dt = Bj + Sjf (t) − Djxj (t) Can be used as a model of gene transcription: Barenco et al., 2006; Gao et al., 2008. xj(t) – concentration of gene j’s mRNA f (t) – concentration of active transcription factor Model parameters: baseline Bj, sensitivity Sj and decay Dj Application: identifying co-regulated genes (targets) Problem: how do we fit the model when f (t) is not observed?

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 13 / 36

slide-21
SLIDE 21

Example: Transcriptional Regulation

r First Order Differential Equation dxj (t) dt = Bj + Sjf (t) − Djxj (t) Can be used as a model of gene transcription: Barenco et al., 2006; Gao et al., 2008. xj(t) – concentration of gene j’s mRNA f (t) – concentration of active transcription factor Model parameters: baseline Bj, sensitivity Sj and decay Dj Application: identifying co-regulated genes (targets) Problem: how do we fit the model when f (t) is not observed?

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 13 / 36

slide-22
SLIDE 22

Example: Transcriptional Regulation

r First Order Differential Equation dxj (t) dt = Bj + Sjf (t) − Djxj (t) Can be used as a model of gene transcription: Barenco et al., 2006; Gao et al., 2008. xj(t) – concentration of gene j’s mRNA f (t) – concentration of active transcription factor Model parameters: baseline Bj, sensitivity Sj and decay Dj Application: identifying co-regulated genes (targets) Problem: how do we fit the model when f (t) is not observed?

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 13 / 36

slide-23
SLIDE 23

Example: Transcriptional Regulation

r First Order Differential Equation dxj (t) dt = Bj + Sjf (t) − Djxj (t) Can be used as a model of gene transcription: Barenco et al., 2006; Gao et al., 2008. xj(t) – concentration of gene j’s mRNA f (t) – concentration of active transcription factor Model parameters: baseline Bj, sensitivity Sj and decay Dj Application: identifying co-regulated genes (targets) Problem: how do we fit the model when f (t) is not observed?

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 13 / 36

slide-24
SLIDE 24

Example: Transcriptional Regulation

r First Order Differential Equation dxj (t) dt = Bj + Sjf (t) − Djxj (t) Can be used as a model of gene transcription: Barenco et al., 2006; Gao et al., 2008. xj(t) – concentration of gene j’s mRNA f (t) – concentration of active transcription factor Model parameters: baseline Bj, sensitivity Sj and decay Dj Application: identifying co-regulated genes (targets) Problem: how do we fit the model when f (t) is not observed?

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 13 / 36

slide-25
SLIDE 25

Example: Transcriptional Regulation

r First Order Differential Equation dxj (t) dt = Bj + Sjf (t) − Djxj (t) Can be used as a model of gene transcription: Barenco et al., 2006; Gao et al., 2008. xj(t) – concentration of gene j’s mRNA f (t) – concentration of active transcription factor Model parameters: baseline Bj, sensitivity Sj and decay Dj Application: identifying co-regulated genes (targets) Problem: how do we fit the model when f (t) is not observed?

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 13 / 36

slide-26
SLIDE 26

[labels=skipGPProperties]Covariance for Transcription Model

RBF covariance function for f (t) xi (t) = Bi Di + Si exp (−Dit) t f (u) exp (Diu) du. Joint distribution for x1 (t), x2 (t) and f (t).

◮ Here:

D1 S1 D2 S2 5 5 0.5 0.5

f(t) x1(t) x2(t) f(t) x1(t) x2(t)

−1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 14 / 36

slide-27
SLIDE 27

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-28
SLIDE 28

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-29
SLIDE 29

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-30
SLIDE 30

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-31
SLIDE 31

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-32
SLIDE 32

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-33
SLIDE 33

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-34
SLIDE 34

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-35
SLIDE 35

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-36
SLIDE 36

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-37
SLIDE 37

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-38
SLIDE 38

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-39
SLIDE 39

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-40
SLIDE 40

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-41
SLIDE 41

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-42
SLIDE 42

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-43
SLIDE 43

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-44
SLIDE 44

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-45
SLIDE 45

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-46
SLIDE 46

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-47
SLIDE 47

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-48
SLIDE 48

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-49
SLIDE 49

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-50
SLIDE 50

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-51
SLIDE 51

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-52
SLIDE 52

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-53
SLIDE 53

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-54
SLIDE 54

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-55
SLIDE 55

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-56
SLIDE 56

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-57
SLIDE 57

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-58
SLIDE 58

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-59
SLIDE 59

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-60
SLIDE 60

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-61
SLIDE 61

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-62
SLIDE 62

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-63
SLIDE 63

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 15 / 36

slide-64
SLIDE 64

p53“Guardian of the Cell”

Responsible for Repairing DNA damage Activates DNA Repair proteins Pauses the Cell Cycle (prevents replication of damage DNA) Initiates apoptosis (cell death) in the case where damage can’t be repaired. Large scale feeback loop with NF-κB.

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 16 / 36

slide-65
SLIDE 65

p53 DNA Damage Repair

Figure: p53. Left unbound, Right bound to DNA. Images by David S. Goodsell from http://www.rcsb.org/ (see the“Molecule of the Month”feature).

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 17 / 36

slide-66
SLIDE 66

p53

Figure: Repair of DNA damage by p53. Image fromGoodsell (1999).

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 18 / 36

slide-67
SLIDE 67

Modelling Assumption

Assume p53 affects targets as a single input module network motif (SIM).

p53 p21 DDB2 PA26 BIK

TNFRSF10b

Figure: p53 SIM network motif as modelled by Barenco et al. 2006.

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 19 / 36

slide-68
SLIDE 68

p53 (RBF covariance)

Pei Gao

2 4 6 8 10 12 −0.5 0.5 1 1.5 2 2.5 3

Inferred p53 protein

2 4 6 8 10 12 1 2 3 4

gene TNFRSF20b mRNA

B = 0.4489 D = 0.4487 S = 0.40601

2 4 6 8 10 12 1 2 3 4

gene DDB2 mRNA

B = 2.0719 D = 0.31956 S = 1.7843

2 4 6 8 10 12 −1 1 2 3 4

gene p21 mRNA

B = 0.22518 D = 0.8 S = 1

2 4 6 8 10 12 1 2 3 4

gene BIK mRNA

B = 1.0637 D = 0.61474 S = 0.71201

2 4 6 8 10 12 1 2 3 4 5

gene hPA26 mRNA

B = 1.1904 D = 0.42333 S = 0.4787

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 20 / 36

slide-69
SLIDE 69

Ranking with ERK Signalling

Target Ranking for Elk-1. Elk-1 is phosphorylated by ERK from the EGF signalling pathway. Predict concentration of Elk-1 from known targets. Rank other targets of Elk-1.

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 21 / 36

slide-70
SLIDE 70

Elk-1 (MLP covariance)

Jennifer Withers

2 4 6 8 −2 2 4 6 8

time (h) TF concentration Transcription factor concentration over time 1 2 3 4 5 6 7 8 0.5 1 1.5 2 2.5 3 Training Gene 1 time (h) 1 2 3 4 5 6 7 8 0.5 1 1.5 2 2.5 3 Training Gene 2 time(h) 1 2 3 4 5 6 7 8 0.5 1 1.5 2 2.5 3 3.5 Training Gene 3 time (h) 1 2 3 4 5 6 7 8 0.5 1 1.5 2 2.5 3 3.5 Training Gene 4 time (h) 1 2 3 4 5 6 7 8 −0.5 0.5 1 1.5 2 2.5 Training Gene 5 time (h)

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 22 / 36

slide-71
SLIDE 71

Elk-1 target selection

Fitted model used to rank potential targets of Elk-1

1 2 3 4 5 6 7 8 0.5 1 1.5 2 2.5 3

Predicted target gene time (h)

1 2 3 4 5 6 7 8 −0.2 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8

Predicted non−target gene time (h)

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 23 / 36

slide-72
SLIDE 72

Roadmap

1

Introduction

2

Latent Force Covariance Functions

3

Cascaded Differential Equations

4

Discussion and Future Work

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 24 / 36

slide-73
SLIDE 73

Cascaded Differential Equations

Antti Honkela Transcription factor protein also has governing mRNA. This mRNA can be measured. In signalling systems this measurement can be misleading because it is activated (phosphorylated) transcription factor that counts. In development phosphorylation plays less of a role.

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 25 / 36

slide-74
SLIDE 74

Drosophila Mesoderm Development

Collaboration with Furlong Lab in EMBL Heidelberg. Mesoderm development in Drosophila melanogaster (fruit fly). Mesoderm forms in triplobastic animals (along with ectoderm and endoderm). Mesoderm develops into muscles, and circulatory system. The transcription factor Twist initiates Drosophila mesoderm development, resulting in the formation of heart, somatic muscle, and

  • ther cell types.

Wildtype microarray experiments publicly available. Can we use the cascade model to predict viable targets of Twist?

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 26 / 36

slide-75
SLIDE 75

Cascaded Differential Equations

Antti Honkela We take the production rate of active transcription factor to be given by df (t) dt = σy (t) − δf (t) dxj (t) dt = Bj + Sjf (t) − Djxj (t) The solution for f (t), setting transient terms to zero, is f (t) = σ exp (−δt) t y(u) exp (δu) du .

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 27 / 36

slide-76
SLIDE 76

Covariance for Translation/Transcription Model

RBF covariance function for y (t)

f (t) = σ exp (−δt) Z t y(u) exp (δu) du xi (t) = Bi Di + Si exp (−Dit) Z t f (u) exp (Diu) du.

Joint distribution for x1 (t), x2 (t), f (t) and y (t). Here: δ

D1 S1 D2 S2

0.1 5

5 0.5 0.5

y(t) f(t) x1(t) x2(t) y(t) f(t) x1(t) x2(t)

−1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 28 / 36

slide-77
SLIDE 77

Twist Results

Use mRNA of Twist as driving input. For each gene build a cascade model that forces Twist to be the only TF. Compare fit of this model to a baseline (e.g. similar model but sensitivity zero). Rank according to the likelihood above the baseline.

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 29 / 36

slide-78
SLIDE 78

Results for Twi using the Cascade model

2 4 6 8 10 12 −1 1 2 3 4 5 6 Inferred twi protein 2 4 6 8 10 12 −1 1 2 3 4 Driving Input 2 4 6 8 10 12 −1 1 2 3 4 FBgn0002526 0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1 delta 0.0768465 sigma 1 D 0.0760771 S 0.0956793 B 0.000847107

Figure: Model for flybase gene identity FBgn0002526.

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 30 / 36

slide-79
SLIDE 79

Results for Twi using the Cascade model

2 4 6 8 10 12 −1 1 2 3 4 5 6 x 10

−3

Inferred twi protein 2 4 6 8 10 12 −1 1 2 3 4 Driving Input 2 4 6 8 10 12 −1 1 2 3 4 FBgn0003486 0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1 delta 517.034 sigma 1 D 542.062 S 266101 B 3.81368e−06

Figure: Model for flybase gene identity FBgn0003486.

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 30 / 36

slide-80
SLIDE 80

Results for Twi using the Cascade model

2 4 6 8 10 12 −1 1 2 3 4 5 6 Inferred twi protein 2 4 6 8 10 12 −1 1 2 3 4 Driving Input 2 4 6 8 10 12 −1 1 2 3 4 FBgn0011206 0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1 delta 0.0543985 sigma 1 D 0.0502381 S 0.0823117 B 0.000447727

Figure: Model for flybase gene identity FBgn0011206.

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 30 / 36

slide-81
SLIDE 81

Results for Twi using the Cascade model

2 4 6 8 10 12 −2 2 4 6 8 Inferred twi protein 2 4 6 8 10 12 −1 1 2 3 4 Driving Input 2 4 6 8 10 12 −1 1 2 3 4 FBgn0030955 0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1 delta 3.17042e−05 sigma 1 D 0.000118374 S 0.0531884 B 7.20183e−08

Figure: Model for flybase gene identity FBgn00309055.

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 30 / 36

slide-82
SLIDE 82

Results for Twi using the Cascade model

2 4 6 8 10 12 −2 2 4 6 8 Inferred twi protein 2 4 6 8 10 12 −1 1 2 3 4 Driving Input 2 4 6 8 10 12 −1 1 2 3 4 FBgn0031907 0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1 delta 0.000381468 sigma 1 D 0.000540422 S 0.0520367 B 3.83826e−06

Figure: Model for flybase gene identity FBgn0031907.

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 30 / 36

slide-83
SLIDE 83

Results for Twi using the Cascade model

2 4 6 8 10 12 −1 1 2 3 4 5 6 Inferred twi protein 2 4 6 8 10 12 −1 1 2 3 4 Driving Input 2 4 6 8 10 12 −1 1 2 3 4 FBgn0035257 0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1 delta 0.0200954 sigma 1 D 0.0211176 S 0.0661116 B 0.000204487

Figure: Model for flybase gene identity FBgn0035257.

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 30 / 36

slide-84
SLIDE 84

Results for Twi using the Cascade model

2 4 6 8 10 12 −0.05 0.05 0.1 0.15 0.2 0.25 0.3 Inferred twi protein 2 4 6 8 10 12 −1 1 2 3 4 Driving Input 2 4 6 8 10 12 −1 1 2 3 4 FBgn0039286 0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1 delta 11.5089 sigma 1 D 119.017 S 1380.22 B 0.00532375

Figure: Model for flybase gene identity FBgn0039286.

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 30 / 36

slide-85
SLIDE 85

Results of Ranking

10

1

10

2

10

3

0.1 0.2 0.3 0.4 0.5 0.6 0.7 In−situ validation: twi Top N to consider Percentage enrichment Filtered GPDISIM Normalised GPDISIM Normalised GPSIM TSNI (Della Gatta et al.) Knock−outs Correlation Random

Figure: Percentage enrichment for top N targets for relevant terms in Drosophila in situs.

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 31 / 36

slide-86
SLIDE 86

Results of Ranking

10

1

10

2

10

3

0.1 0.2 0.3 0.4 0.5 0.6 0.7 ChIP validation: twi (10 kb) Top N to consider Percentage enrichment Filtered GPDISIM Normalised GPDISIM Normalised GPSIM TSNI (Della Gatta et al.) Knock−outs Correlation Random

Figure: Percentage enrichment for top N targets for ChIP-chip confirmed targets.

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 31 / 36

slide-87
SLIDE 87

Summary

Cascade models allow genomewide analysis of potential targets given

  • nly expression data.

Once a set of potential candidate targets have been identified, they can be modelled in a more complex manner. We don’t have ground truth, but evidence indicates that the approach can perform as well as knockouts.

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 32 / 36

slide-88
SLIDE 88

Outline

1

Introduction

2

Latent Force Covariance Functions

3

Cascaded Differential Equations

4

Discussion and Future Work

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 33 / 36

slide-89
SLIDE 89

Discussion and Future Work

Integration of probabilistic inference with mechanistic models. These results are small simple systems. Other aspects:

◮ Non-linear responses in differential equations (Michalis Titsias’s work

— turn to sampling, Pei Gao — use Laplace approximation).

◮ Scaling up to larger systems (Mauricio’s Talk). ◮ Applications to other types of system, e.g. spatial systems etc. (using

PDEs (´ Alvarez et al., 2009))

◮ Stochastic differential equations (financial time series example). Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 34 / 36

slide-90
SLIDE 90

Acknowledgements

Investigators: Neil Lawrence and Magnus Rattray Researchers: Peo Gao, Antti Honkela, Michalis Titsias, Mauricio Alvarez, David Luengo and Jennifer Withers Charles Girardot and Eileen Furlong of EMBL in Heidelberg (mesoderm development in D. Melanogaster). Martino Barenco and Mike Hubank at the Institute of Child Health in UCL (p53 pathway).

Funded by the BBSRC award“Improved Processing of microarray data using probabilistic models”and EPSRC award“Gaussian Processes for Systems Identification with applications in Systems Biology”

Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 35 / 36

slide-91
SLIDE 91

References I

  • M. ´

Alvarez, D. Luengo, and N. D. Lawrence. Latent force models. In van Dyk and Welling (2009), pages 9–16. [PDF].

  • M. Barenco, D. Tomescu, D. Brewer, R. Callard, J. Stark, and M. Hubank. Ranked prediction of p53 targets using hidden

variable dynamic modeling. Genome Biology, 7(3):R25, 2006. [PDF].

  • P. Gao, A. Honkela, M. Rattray, and N. D. Lawrence. Gaussian process modelling of latent chemical species: Applications to

inferring transcription factor activities. Bioinformatics, 24:i70–i75, 2008. [PDF]. [DOI].

  • D. S. Goodsell. The molecular perspective: p53 tumor suppressor. The Oncologist, Vol. 4, No. 2, 138-139, April 1999, 4(2):

138–139, 1999.

  • J. Qui˜

nonero Candela and C. E. Rasmussen. A unifying view of sparse approximate Gaussian process regression. Journal of Machine Learning Research, 6:1939–1959, 2005. [PDF].

  • E. Snelson and Z. Ghahramani. Sparse Gaussian processes using pseudo-inputs. In Y. Weiss, B. Sch¨
  • lkopf, and J. C. Platt,

editors, Advances in Neural Information Processing Systems, volume 18, Cambridge, MA, 2006. MIT Press. [PDF].

  • Y. W. Teh, M. Seeger, and M. I. Jordan. Semiparametric latent factor models. In R. G. Cowell and Z. Ghahramani, editors,

Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics, pages 333–340, Barbados, 6-8 January 2005. Society for Artificial Intelligence and Statistics. [PDF].

  • M. K. Titsias. Variational learning of inducing variables in sparse Gaussian processes. In van Dyk and Welling (2009), pages

567–574.

  • D. van Dyk and M. Welling, editors. Artificial Intelligence and Statistics, volume 5, Clearwater Beach, FL, 16-18 April 2009.

JMLR W&CP 5. Neil D. Lawrence (Manchester) Latent Force Models 23rd July 2009 36 / 36