Latent Force Models with Gaussian Processes Neil D. Lawrence - - PowerPoint PPT Presentation

▶
latent force models with gaussian processes
SMART_READER_LITE
LIVE PREVIEW

Latent Force Models with Gaussian Processes Neil D. Lawrence - - PowerPoint PPT Presentation

Oct 26, 2022 535 likes •1.54k views

Latent Force Models with Gaussian Processes Neil D. Lawrence Bayesian Research Kitchen, Wordsworth Hotel, Grasmere 6th September 2008 Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 1 / 51 Outline Introduction 1

slide-1
SLIDE 1

Latent Force Models with Gaussian Processes

Neil D. Lawrence

Bayesian Research Kitchen, Wordsworth Hotel, Grasmere

6th September 2008

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 1 / 51

slide-2
SLIDE 2

Outline

1

Introduction

2

Covariance Functions

3

Convolutions and Computational Complexity

4

Non-linear Response Models

5

Cascaded Differential Equations

6

Discussion and Future Work

7

Acknowledgements

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 2 / 51

slide-3
SLIDE 3

Outline

1

Introduction

2

Covariance Functions

3

Convolutions and Computational Complexity

4

Non-linear Response Models

5

Cascaded Differential Equations

6

Discussion and Future Work

7

Acknowledgements

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 3 / 51

slide-4
SLIDE 4

Dimensionality Reduction I

Linear relationship between the data, X ∈ ℜN×d, and a reduced dimensional representation, F ∈ ℜN×q, where q ≪ d. X = FW + ǫ, ǫ ∼ N (0, Σ) Integrate out X, optimize with respect to W. For temporal data and a particular Gaussian prior in the latent space: Kalman filter/smoother More generally consider a Gaussian process (GP) prior, p (F|t) =

q

  • i=1

N

  • f:,i|0, Kf:,i,f:,i
  • .

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 4 / 51

slide-5
SLIDE 5

Dimensionality Reduction II

Given the covariance functions for {fi(t)} the implied covariance functions for {xi(t)} — semi-parametric latent factor model (Teh et al., 2005). Kalman filter/smoother approach has been preferred

◮ linear computational complexity in N. ◮ Advances in sparse approximations have made the general GP

framework practical. (Snelson and Ghahramani, 2006; Qui˜

nonero Candela and Rasmussen, 2005, also Titsias tomorrow).

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 5 / 51

slide-6
SLIDE 6

Mechanical Analogy

These models rely on the latent variables to provide the dynamic information. We now introduce a further dynamical system with a mechanistic inspiration. Physical Interpretation:

◮ the latent functions, fi(t) are q forces. ◮ We observe the displacement of d springs to the forces., ◮ Interpret system as the force balance equation, XD = FSǫ. ◮ Forces act, e.g. through levers — a matrix of sensitivities, S ∈ ℜq×d. ◮ Diagonal matrix of spring constants, D ∈ ℜd×d. ◮ Original System: W = SD−1. Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 6 / 51

slide-7
SLIDE 7

Extend Model

Add a damper and give the system mass. FS = ¨ XM + ˙ XC + XD + ǫ. Now have a second order mechanical system. It will exhibit inertia and resonance. There are many systems that can also be represented by differential equations.

◮ When being forced by latent function(s), {fi(t)}q

i=1, we call this a

latent force model.

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 7 / 51

slide-8
SLIDE 8

Gaussian Process priors and Latent Force Models

For Gaussian process we can compute the covariance matrices for the

  • utput displacements.

For one displace the model is mk¨ xk(t) + ck ˙ xk(t) + dkxk(t) = bk +

M

  • i=0

sikfi(t), (1) where, mk is the kth diagonal element from M and similarly for ck and dk. sik is the i, kth element of S. Model the latent forces as q independent, GPs with RBF covariances kfifl(t, t′) = exp

  • −(t − t′)2

σ2

i

  • δil.

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 8 / 51

slide-9
SLIDE 9

Covariance for ODE Model

RBF Kernel function for f (t) xj(t) = 1 mjωj

q

  • i=1

Sji exp(−αjt) t fi(u) exp(αju) sin(ωj(t − u))du Joint distribution for x1 (t), x2 (t), x3 (t) and f (t). Damping ratios:

ζ1 ζ2 ζ3

0.125 2 1

f(t) y1(t) y2(t) y3(t) f(t) y1(t) y2(t) y3(t)

−0.4 −0.2 0.2 0.4 0.6 0.8

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 9 / 51

slide-10
SLIDE 10

Joint Sampling of x (t) and f (t)

demLfmSample

5 10 15 20 −2.5 −2 −1.5 −1 −0.5 0.5 1 1.5 2

Figure: Joint samples from the ODE covariance, cyan: f (t), red: x1 (t)(underdamped) and green: x2 (t) (overdamped) and blue: x3 (t) (critically damped).

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 10 / 51

slide-11
SLIDE 11

Joint Sampling of x (t) and f (t)

demLfmSample

5 10 15 20 −2.5 −2 −1.5 −1 −0.5 0.5 1 1.5 2

Figure: Joint samples from the ODE covariance, cyan: f (t), red: x1 (t)(underdamped) and green: x2 (t) (overdamped) and blue: x3 (t) (critically damped).

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 10 / 51

slide-12
SLIDE 12

Joint Sampling of x (t) and f (t)

demLfmSample

5 10 15 20 −1 −0.5 0.5 1 1.5 2

Figure: Joint samples from the ODE covariance, cyan: f (t), red: x1 (t)(underdamped) and green: x2 (t) (overdamped) and blue: x3 (t) (critically damped).

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 10 / 51

slide-13
SLIDE 13

Joint Sampling of x (t) and f (t)

demLfmSample

5 10 15 20 −2.5 −2 −1.5 −1 −0.5 0.5 1 1.5 2

Figure: Joint samples from the ODE covariance, cyan: f (t), red: x1 (t)(underdamped) and green: x2 (t) (overdamped) and blue: x3 (t) (critically damped).

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 10 / 51

slide-14
SLIDE 14

Covariance for ODE

RBF Kernel function for f (t) xj(t) = 1 mjωj

q

  • i=1

Sji exp(−αjt) t fi(u) exp(αju) sin(ωj(t − u))du Joint distribution for x1 (t), x2 (t), x3 (t) and f (t). Damping ratios:

ζ1 ζ2 ζ3

0.125 2 1

f(t) y1(t) y2(t) y3(t) f(t) y1(t) y2(t) y3(t)

−0.4 −0.2 0.2 0.4 0.6 0.8

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 11 / 51

slide-15
SLIDE 15

Example: Transcriptional Regulation

First Order Differential Equation dxj (t) dt = Bj + Sjf (t) − Djxj (t) Can be used as a model of gene transcription: Barenco et al., 2006. xj(t) – concentration of gene j’s mRNA f (t) – concentration of active transcription factor Model parameters: baseline Bj, sensitivity Sj and decay Dj Application: identifying co-regulated genes (targets) Problem: how do we fit the model when f (t) is not observed?

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 12 / 51

slide-16
SLIDE 16

Example: Transcriptional Regulation

First Order Differential Equation dxj (t) dt = Bj + Sjf (t) − Djxj (t) Can be used as a model of gene transcription: Barenco et al., 2006. xj(t) – concentration of gene j’s mRNA f (t) – concentration of active transcription factor Model parameters: baseline Bj, sensitivity Sj and decay Dj Application: identifying co-regulated genes (targets) Problem: how do we fit the model when f (t) is not observed?

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 12 / 51

slide-17
SLIDE 17

Example: Transcriptional Regulation

First Order Differential Equation dxj (t) dt = Bj + Sjf (t) − Djxj (t) Can be used as a model of gene transcription: Barenco et al., 2006. xj(t) – concentration of gene j’s mRNA f (t) – concentration of active transcription factor Model parameters: baseline Bj, sensitivity Sj and decay Dj Application: identifying co-regulated genes (targets) Problem: how do we fit the model when f (t) is not observed?

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 12 / 51

slide-18
SLIDE 18

Example: Transcriptional Regulation

First Order Differential Equation dxj (t) dt = Bj + Sjf (t) − Djxj (t) Can be used as a model of gene transcription: Barenco et al., 2006. xj(t) – concentration of gene j’s mRNA f (t) – concentration of active transcription factor Model parameters: baseline Bj, sensitivity Sj and decay Dj Application: identifying co-regulated genes (targets) Problem: how do we fit the model when f (t) is not observed?

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 12 / 51

slide-19
SLIDE 19

Example: Transcriptional Regulation

First Order Differential Equation dxj (t) dt = Bj + Sjf (t) − Djxj (t) Can be used as a model of gene transcription: Barenco et al., 2006. xj(t) – concentration of gene j’s mRNA f (t) – concentration of active transcription factor Model parameters: baseline Bj, sensitivity Sj and decay Dj Application: identifying co-regulated genes (targets) Problem: how do we fit the model when f (t) is not observed?

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 12 / 51

slide-20
SLIDE 20

Example: Transcriptional Regulation

First Order Differential Equation dxj (t) dt = Bj + Sjf (t) − Djxj (t) Can be used as a model of gene transcription: Barenco et al., 2006. xj(t) – concentration of gene j’s mRNA f (t) – concentration of active transcription factor Model parameters: baseline Bj, sensitivity Sj and decay Dj Application: identifying co-regulated genes (targets) Problem: how do we fit the model when f (t) is not observed?

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 12 / 51

slide-21
SLIDE 21

Example: Transcriptional Regulation

First Order Differential Equation dxj (t) dt = Bj + Sjf (t) − Djxj (t) Can be used as a model of gene transcription: Barenco et al., 2006. xj(t) – concentration of gene j’s mRNA f (t) – concentration of active transcription factor Model parameters: baseline Bj, sensitivity Sj and decay Dj Application: identifying co-regulated genes (targets) Problem: how do we fit the model when f (t) is not observed?

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 12 / 51

slide-22
SLIDE 22

Covariance for Transcription Model

RBF covariance function for f (t) xi (t) = Bi Di + Si exp (−Dit) t f (u) exp (Diu) du. Joint distribution for x1 (t), x2 (t) and f (t).

◮ Here:

D1 S1 D2 S2 5 5 0.5 0.5

f(t) x1(t) x2(t) f(t) x1(t) x2(t)

−1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 13 / 51

slide-23
SLIDE 23

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-24
SLIDE 24

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-25
SLIDE 25

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-26
SLIDE 26

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-27
SLIDE 27

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-28
SLIDE 28

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-29
SLIDE 29

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-30
SLIDE 30

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-31
SLIDE 31

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-32
SLIDE 32

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-33
SLIDE 33

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-34
SLIDE 34

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-35
SLIDE 35

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-36
SLIDE 36

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-37
SLIDE 37

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-38
SLIDE 38

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-39
SLIDE 39

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-40
SLIDE 40

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-41
SLIDE 41

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-42
SLIDE 42

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-43
SLIDE 43

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-44
SLIDE 44

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-45
SLIDE 45

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-46
SLIDE 46

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-47
SLIDE 47

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-48
SLIDE 48

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-49
SLIDE 49

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-50
SLIDE 50

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-51
SLIDE 51

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-52
SLIDE 52

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-53
SLIDE 53

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-54
SLIDE 54

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-55
SLIDE 55

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-56
SLIDE 56

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-57
SLIDE 57

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-58
SLIDE 58

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-59
SLIDE 59

Artificial Example: Inferring f (t)

5 10 15 5 10 15 20 5 10 15 −2 −1 1 2 3 4

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 14 / 51

slide-60
SLIDE 60

p53“Guardian of the Cell”

Responsible for Repairing DNA damage Activates DNA Repair proteins Pauses the Cell Cycle (prevents replication of damage DNA) Initiates apoptosis (cell death) in the case where damage can’t be repaired. Large scale feeback loop with NF-κB.

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 15 / 51

slide-61
SLIDE 61

p53 DNA Damage Repair

Figure: p53. Left unbound, Right bound to DNA. Images by David S. Goodsell from http://www.rcsb.org/ (see the“Molecule of the Month”feature).

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 16 / 51

slide-62
SLIDE 62

p53

Figure: Repair of DNA damage by p53. Image fromGoodsell (1999).

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 17 / 51

slide-63
SLIDE 63

Modelling Assumption

Assume p53 affects targets as a single input module network motif (SIM).

p53 p21 DDB2 PA26 BIK

TNFRSF10b

Figure: p53 SIM network motif as modelled by Barenco et al. 2006.

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 18 / 51

slide-64
SLIDE 64

p53 (RBF covariance)

Pei Gao

2 4 6 8 10 12 −0.5 0.5 1 1.5 2 2.5 3

Inferred p53 protein

2 4 6 8 10 12 1 2 3 4

gene TNFRSF20b mRNA

B = 0.4489 D = 0.4487 S = 0.40601

2 4 6 8 10 12 1 2 3 4

gene DDB2 mRNA

B = 2.0719 D = 0.31956 S = 1.7843

2 4 6 8 10 12 −1 1 2 3 4

gene p21 mRNA

B = 0.22518 D = 0.8 S = 1

2 4 6 8 10 12 1 2 3 4

gene BIK mRNA

B = 1.0637 D = 0.61474 S = 0.71201

2 4 6 8 10 12 1 2 3 4 5

gene hPA26 mRNA

B = 1.1904 D = 0.42333 S = 0.4787

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 19 / 51

slide-65
SLIDE 65

Ranking with ERK Signalling

Target Ranking for Elk-1. Elk-1 is phosphorylated by ERK from the EGF signalling pathway. Predict concentration of Elk-1 from known targets. Rank other targets of Elk-1.

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 20 / 51

slide-66
SLIDE 66

Elk-1 (MLP covariance)

Jennifer Withers

2 4 6 8 −2 2 4 6 8

time (h) TF concentration Transcription factor concentration over time 1 2 3 4 5 6 7 8 0.5 1 1.5 2 2.5 3 Training Gene 1 time (h) 1 2 3 4 5 6 7 8 0.5 1 1.5 2 2.5 3 Training Gene 2 time(h) 1 2 3 4 5 6 7 8 0.5 1 1.5 2 2.5 3 3.5 Training Gene 3 time (h) 1 2 3 4 5 6 7 8 0.5 1 1.5 2 2.5 3 3.5 Training Gene 4 time (h) 1 2 3 4 5 6 7 8 −0.5 0.5 1 1.5 2 2.5 Training Gene 5 time (h)

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 21 / 51

slide-67
SLIDE 67

Elk-1 target selection

Fitted model used to rank potential targets of Elk-1

1 2 3 4 5 6 7 8 0.5 1 1.5 2 2.5 3

Predicted target gene time (h)

1 2 3 4 5 6 7 8 −0.2 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8

Predicted non−target gene time (h)

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 22 / 51

slide-68
SLIDE 68

Outline

1

Introduction

2

Covariance Functions

3

Convolutions and Computational Complexity

4

Non-linear Response Models

5

Cascaded Differential Equations

6

Discussion and Future Work

7

Acknowledgements

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 23 / 51

slide-69
SLIDE 69

Convolutions and Computational Complexity

Mauricio Alvarez Solutions to these differential equations is normally as a convolution. xi (t) =

  • f (u) ki (u − t) du + hi (t)

xi (t) = t f (u) gi (u) du + hi (t) Convolution Processes (Higdon, 2002; Boyle and Frean, 2005). Convolutions lead to N × d size covariance matrices O

  • N3d3

complexity, O

  • N2d2

storage. Model is conditionally independent over {xi (t)}d

i=1 given f (t).

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 24 / 51

slide-70
SLIDE 70

Independence Assumption

Mauricio Alvarez Can assume conditional independence given given {f (ti)}k

i=1.

◮ Result is very similar to PITC approximation (Qui˜

nonero Candela and Rasmussen, 2005).

◮ Reduces to O

  • N3dk2

complexity, O

  • N2dk
  • storage.

◮ Can also do a FITC style approximation (Snelson and Ghahramani, 2006). ◮ Reduces to O

  • Ndk2

complexity, O (Ndk) storage.

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 25 / 51

slide-71
SLIDE 71

Tide Sensor Network

Mauricio Alvarez Network of tide height sensors in the solent — tide heights are correlated. Data kindly provided by Alex Rogers (see Rogers et al., 2008). d = 3 and N = 1000 of the 4320 for the training set. Simulate sensor failure by knocking out onse sensor for a given time. For the other two sensors we used all 1000 training observations. Take k = 100.

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 26 / 51

slide-72
SLIDE 72

Tide Height Results

Mauricio Alvarez

0.5 1 1.5 2 2.5 3 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 Tide Height (m) Time (days)

(a) Bramblemet Indepen- dent

0.5 1 1.5 2 2.5 3 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 Tide Height (m) Time (days)

(b) Bramblemet PITC

0.5 1 1.5 2 2.5 3 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 Tide Height (m) Time (days) 0.5 1 1.5 2 2.5 3 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 Tide Height (m) Time (days)

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 27 / 51

slide-73
SLIDE 73

Cokriging Jura

Mauricio Alvarez Jura dataset — concentrations of several heavy metals. Prediction 259 data, validation 100 data points. Predict primary variables (cadmium and copper) at prediction locations in conjunction with some secondary variables (nickel and zinc for cadmium; lead, nickel and zinc for copper) (Goovaerts, 1997,

  • p. 248,249).

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 28 / 51

slide-74
SLIDE 74

Swiss Jura Results

Mauricio Alvarez

IGP P(50) P(100) P(200) P(500) FGP CK 0.42 0.44 0.46 0.48 0.5 0.52 0.54 0.56 0.58

MEAN ABSOLUTE ERROR Cd

(a) Cadmium

IGP P(50) P(100) P(200) P(500) FGP CK 7 8 9 10 11 12 13 14 15 16

MEAN ABSOLUTE ERROR Cu

(b) Copper

Figure: Mean absolute error. IGP stands for independent GP, P(M) stands for PITC with M inducing values, FGP stands for full GP and CK stands for ordinary co-kriging.

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 29 / 51

slide-75
SLIDE 75

Outline

1

Introduction

2

Covariance Functions

3

Convolutions and Computational Complexity

4

Non-linear Response Models

5

Cascaded Differential Equations

6

Discussion and Future Work

7

Acknowledgements

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 30 / 51

slide-76
SLIDE 76

Models of non-linear regulation

Non-linear Activation: Michaelis-Menten Kinetics dxi (t) dt = Bi + Sif (t) γi + f (t) − Dixi (t) used by Rogers and Girolami (2006) Non-linear Repression dxi (t) dt = Bi + Si γi + f (t) − Dixi (t) used by Khanin et al., 2006, PNAS 103

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 31 / 51

slide-77
SLIDE 77

Models of non-linear regulation

Non-linear Activation: Michaelis-Menten Kinetics dxi (t) dt = Bi + Sif (t) γi + f (t) − Dixi (t) used by Rogers and Girolami (2006) Non-linear Repression dxi (t) dt = Bi + Si γi + f (t) − Dixi (t) used by Khanin et al., 2006, PNAS 103

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 31 / 51

slide-78
SLIDE 78

Models of non-linear regulation

Non-linear Activation: Michaelis-Menten Kinetics dxi (t) dt = Bi + Sif (t) γi + f (t) − Dixi (t) used by Rogers and Girolami (2006) Non-linear Repression dxi (t) dt = Bi + Si γi + f (t) − Dixi (t) used by Khanin et al., 2006, PNAS 103

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 31 / 51

slide-79
SLIDE 79

MAP Laplace Approximation

Consider the following modification to the model, dxj (t) dt = Bj + Sjg (f (t)) − Djxj (t) , where g (·) is a non-linear function. The differential equation can still be solved, xj (t) = Bj Dj + Sj t e−Dj(t−u)gj (f (u)) du Use Laplace’s method (Laplace, 1774), p (f | x) = N

  • ˆ

f, A−1 ∝ exp

  • −1

2

  • f − ˆ

f T A

  • f − ˆ

f

  • where ˆ

f = argmaxp(f | x) and A = −∇∇ log p (f | y) |f=ˆ

f is the Hessian

  • f the negative posterior at that point.

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 32 / 51

slide-80
SLIDE 80

p53 and Michaelis-Menten Kinetics

Pei Gao The Michaelis-Menten activation model uses the following non-linearity gj (f (t)) = ef (t) γj + ef (t) , where we are using a GP f (t) to model the log of the TF activity.

2 4 6 8 10 12 −1 1 2 3 4

Inferred p53 protein

2 4 6 8 10 12 0.5 1 1.5 2

Inferred p53 protein

(a)

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 33 / 51

slide-81
SLIDE 81

Valdiation of Laplace Approximation

Michalis Titsias

2 4 6 8 10 12 1 2 3 4 Figure: Laplace approximation error bars along with samples from the true posterior distribution.

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 34 / 51

slide-82
SLIDE 82

Use Samples to Represent Posterior

Michalis Titsias Sample in Gaussian processes p (f|x) ∝ p (x|f) p (f) Likelihood relates GP to data through xj (t) = αje−Djt + Bj Dj + Sj t e−Dj(t−u)gj(f (u))du We use control points for fast sampling.

◮ Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 35 / 51

slide-83
SLIDE 83

Sampling using control points

Separate the points in f into two groups:

◮ few control points fc ◮ and the large majority of the remaining points fρ = f \ fc

Sample the control points fc using a proposal q

  • f(t+1)

c

|f(t)

c

  • Sample the remaining points fρ using the conditional GP prior

p

  • f(t+1)

ρ

|f(t+1)

c

  • The whole proposal is

Q

  • f(t+1)|f(t)

= p

  • f(t+1)

ρ

|f(t+1)

c

  • q
  • f(t+1)

c

|f(t)

c

  • Its like sampling from the prior p(f) but imposing random walk

behaviour through the control points.

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 36 / 51

slide-84
SLIDE 84

p53 System Again

One transcription factor (p53) that acts as an activator. We consider the Michaelis-Menten kinetic equation dxj(t) dt = Bj + Sj exp(f (t)) exp(f (t)) + γj − Djxj(t) MCMC details:

◮ 7 control points are used (placed in a equally spaced grid) ◮ Running time 4/5 hours for 2 million sampling iterations plus burn in ◮ Acceptance rate for f after burn in was between 15% − 25% Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 37 / 51

slide-85
SLIDE 85

Data used by Barenco et al. (2006): Predicted gene expressions for the 1st replica

2 4 6 8 10 12 0.5 1 1.5 2 2.5 3 3.5 DDB2 Gene − first Replica 2 4 6 8 10 12 −1 1 2 3 4 5 BIK Gene − first Replica 2 4 6 8 10 12 −0.5 0.5 1 1.5 2 2.5 3 3.5 TNFRSF10b Gene − first Replica 2 4 6 8 10 12 −1 1 2 3 4 5 CIp1/p21 Gene − first Replica 2 4 6 8 10 12 0.5 1 1.5 2 2.5 3 3.5 p26 sesn1 Gene − first Replica

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 38 / 51

slide-86
SLIDE 86

Data used by Barenco et al. (2006): Protein concentrations

2 4 6 8 10 12 0.5 1 1.5 2

Inferred p53 protein

2 4 6 8 10 12 −0.5 0.5 1 1.5 2

Inferred p53 protein

2 4 6 8 10 12 −0.5 0.5 1 1.5 2 2.5 3

Inferred p53 protein

Linear model (Barenco et al. predictions are shown as crosses)

2 4 6 8 10 12 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Inferred protein 2 4 6 8 10 12 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Inferred protein 2 4 6 8 10 12 0.5 1 1.5 2 2.5 3 Inferred protein

Nonlinear (Michaelis-Menten kinetic equation)

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 39 / 51

slide-87
SLIDE 87

p53 Data Kinetic parameters

DDB2 p26 sesn1 TNFRSF10b CIp1/p21 BIK 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6

Basal rates

DDB2 p26 sesn1 TNFRSF10b CIp1/p21 BIK 1 2 3 4 5 6 7 8 9 10

Decay rates

DDB2 p26 sesn1 TNFRSF10b CIp1/p21 BIK 5 10 15 20 25 30

Sensitivities

DDB2 p26 sesn1 TNFRSF10b CIp1/p21 BIK 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Gamma parameters

Our results (grey) compared with Barenco et al. (2006) (black). Note that Barenco et al. use a linear model

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 40 / 51

slide-88
SLIDE 88

Outline

1

Introduction

2

Covariance Functions

3

Convolutions and Computational Complexity

4

Non-linear Response Models

5

Cascaded Differential Equations

6

Discussion and Future Work

7

Acknowledgements

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 41 / 51

slide-89
SLIDE 89

Cascaded Differential Equations

Antti Honkela Transcription factor protein also has governing mRNA. This mRNA can be measured. In signalling systems this measurement can be misleading because it is activated (phosphorylated) transcription factor that counts. In development phosphorylation plays less of a role.

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 42 / 51

slide-90
SLIDE 90

Drosophila Mesoderm Development

Data from Furlong Lab in EMBL Heidelberg. Describe mesoderm development.

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 43 / 51

slide-91
SLIDE 91

Cascaded Differential Equations

Antti Honkela We take the production rate of active transcription factor to be given by df (t) dt = σy (t) − δf (t) dxj (t) dt = Bj + Sjf (t) − Djxj (t) The solution for f (t), setting transient terms to zero, is f (t) = σ exp (−δt) t y(u) exp (δu) du .

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 44 / 51

slide-92
SLIDE 92

Covariance for Translation/Transcription Model

RBF covariance function for y (t)

f (t) = σ exp (−δt) Z t y(u) exp (δu) du xi (t) = Bi Di + Si exp (−Dit) Z t f (u) exp (Diu) du.

Joint distribution for x1 (t), x2 (t), f (t) and y (t). Here: δ

D1 S1 D2 S2

0.1 5

5 0.5 0.5

y(t) f(t) x1(t) x2(t) y(t) f(t) x1(t) x2(t)

−1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 45 / 51

slide-93
SLIDE 93

Results for Mef2 using the Cascade model

2 4 6 8 10 12 −1 1 2 3 4 5

Driving Input mRNA

2 4 6 8 10 12 −0.1 0.1 0.2 0.3 0.4

Inferred Mef2 Protein

2 4 6 8 10 12 −1 1 2 3 4

Gene Rya−r44F mRNA

2 4 6 8 10 12 −1 1 2 3 4

Gene ttk mRNA

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 46 / 51

slide-94
SLIDE 94

Outline

1

Introduction

2

Covariance Functions

3

Convolutions and Computational Complexity

4

Non-linear Response Models

5

Cascaded Differential Equations

6

Discussion and Future Work

7

Acknowledgements

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 47 / 51

slide-95
SLIDE 95

Discussion and Future Work

Integration of probabilistic inference with mechanistic models. These results are small simple systems. Ongoing work:

◮ Scaling up to larger systems ◮ Applications to other types of system, e.g. non-steady-state

metabolomics, spatial systems etc.

◮ Improved approximations. ◮ Stochastic differential equations Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 48 / 51

slide-96
SLIDE 96

Outline

1

Introduction

2

Covariance Functions

3

Convolutions and Computational Complexity

4

Non-linear Response Models

5

Cascaded Differential Equations

6

Discussion and Future Work

7

Acknowledgements

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 49 / 51

slide-97
SLIDE 97

Acknowledgements

Investigators: Neil Lawrence and Magnus Rattray Researchers: Peo Gao, Antti Honkela, Michalis Titsias, Mauricio Alvarez and Jennifer Withers Charles Girardot and Eileen Furlong of EMBL in Heidelberg (mesoderm development in D. Melanogaster). Martino Barenco and Mike Hubank at the Institute of Child Health in UCL (p53 pathway).

Funded by the BBSRC award“Improved Processing of microarray data using probabilistic models”and EPSRC award“Gaussian Processes for Systems Identification with applications in Systems Biology”

Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 50 / 51

slide-98
SLIDE 98

References I

  • M. Barenco, D. Tomescu, D. Brewer, R. Callard, J. Stark, and M. Hubank. Ranked prediction of p53 targets using hidden

variable dynamic modeling. Genome Biology, 7(3):R25, 2006. [PDF].

  • P. Boyle and M. Frean. Dependent Gaussian processes. In L. Saul, Y. Weiss, and L. Bouttou, editors, Advances in Neural

Information Processing Systems, volume 17, pages 217–224, Cambridge, MA, 2005. MIT Press. [PDF].

  • D. S. Goodsell. The molecular perspective: p53 tumor suppressor. The Oncologist, Vol. 4, No. 2, 138-139, April 1999, 4(2):

138–139, 1999.

  • P. Goovaerts. Geostatistics For Natural Resources Evaluation. Oxford University Press, 1997. [Google Books] .
  • D. M. Higdon. Space and space-time modelling using process convolutions. In C. Anderson, V. Barnett, P. Chatwin, and
  • A. El-Shaarawi, editors, Quantitative methods for current environmental issues, pages 37–56. Springer-Verlag, 2002.
  • R. Khanin, V. Viciotti, and E. Wit. Reconstructing repressor protein levels from expression of gene targets in E. Coli. Proc. Natl.
  • Acad. Sci. USA, 103(49):18592–18596, 2006. [PDF]. [DOI].
  • P. S. Laplace. M´

emoire sur la probabilit´ e des causes par les ´ ev`

  • enemens. In M´

emoires de math` ematique et de physique, present´ es ` a lAcad´ emie Royale des Sciences, par divers savans, & l` u dans ses assembl´ ees 6, pages 621–656, 1774. Translated in Stigler (1986).

  • J. Qui˜

nonero Candela and C. E. Rasmussen. A unifying view of sparse approximate Gaussian process regression. Journal of Machine Learning Research, 6:1939–1959, 2005. [PDF].

  • A. Rogers, M. A. Osborne, S. D. Ramchurn, S. J. Roberts, and N. R. Jennings. Towards real-time information processing of

sensor network data using computationally efficient multi-output Gaussian processes. In Proceedings of the International Conference on Information Processing in Sensor Networks (IPSN 2008), 2008. In press.

  • S. Rogers and M. Girolami. Model based identification of transcription factor regulatory activity via Markov chain Monte Carlo.

Presentation at MASAMB ’06, 2006.

  • E. Snelson and Z. Ghahramani. Sparse Gaussian processes using pseudo-inputs. In Y. Weiss, B. Sch¨
  • lkopf, and J. C. Platt,

editors, Advances in Neural Information Processing Systems, volume 18, Cambridge, MA, 2006. MIT Press. [PDF].

  • S. M. Stigler. Laplace’s 1774 memoir on inverse probability. Statistical Science, 1:359–378, 1986.
  • Y. W. Teh, M. Seeger, and M. I. Jordan. Semiparametric latent factor models. In R. G. Cowell and Z. Ghahramani, editors,

Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics, pages 333–340, Barbados, 6-8 January 2005. Society for Artificial Intelligence and Statistics. [PDF]. Neil D. Lawrence (BARK 08) Latent Force Models 6th September 2008 51 / 51