Polynomial Chaos Acceleration for the Bayesian Inference of Random - - PowerPoint PPT Presentation

Bayesian Inference Coordinate transformation for Uncertain Correlation Function PC surrogate model Inference Example

Polynomial Chaos Acceleration for the Bayesian Inference of Random Fields with Gaussian Priors and Uncertain Covariance Hyper-Parameters

Ihab Sraj1, Olivier Le Maître2,1,3, Omar Knio1,3 and Ibrahim Hoteit1

1KAUST, CEMSE and PSE SRI-UQ Center Ihab.Sraj@kaust.edu.sa Ibrahim.Hoteit@kaust.edu.sa Omar.Knio@kaust.edu.sa Olivier.LeMaitre@kaust.edu.sa 2LIMSI-CNRS UPR-3251, Orsay, France

• lm@limsi.fr

www.limsi.fr/Individu/olm 3Duke University Mechanical Engineering and Materials Science

• mar.knio@duke.edu
• livier.le.maitre@duke.edu

KAUST - MCS 350 - UQ course

O.P . Le Maître et.al Change of Coordinates & PC Acceleration for Bayesian Inference

Bayesian Inference Coordinate transformation for Uncertain Correlation Function PC surrogate model Inference Example 1

Bayesian Inference

2

Coordinate transformation for Uncertain Correlation Function

3

PC surrogate model

4

Inference Example

O.P . Le Maître et.al Change of Coordinates & PC Acceleration for Bayesian Inference

Bayesian Inference Coordinate transformation for Uncertain Correlation Function PC surrogate model Inference Example

Inference of parameter field We want to infer a parameter field M ∈ L2(Ω), from a set of observations d ∈ Rm of a given process, a model u(M) ∈ Rm that predicts the observation, the Bayesian rule to update our knowledge of M. p(M, σ2

• |d) ∝ p(d|M, σ2
• )pM(M)po(σ2
• )

p(d|M, σ2

• ) is the likelihood of the observations,

pM(M) is the Gaussian field’s prior, σ2

• is an error model hyper-parameter with prior of po(σ2
• ).

Classical choices are i.i.d. model errors with Gaussian distribution N(0, σ2

• ) leading to

p(d|M, σ2

• ) =

m

• i=1

pǫ(di − ui(M), σ2

• ),

pǫ(x, σ2

• ) .

= 1

• 2πσ2
• exp
• − x2

2σ2

• with uninformative Jeffrey’s prior for σo.

O.P . Le Maître et.al Change of Coordinates & PC Acceleration for Bayesian Inference

Bayesian Inference Coordinate transformation for Uncertain Correlation Function PC surrogate model Inference Example

Gaussian field’s prior We shall consider prior M that are centered Gaussian processes with covariance function C(x, x′). The prior M(x) can then be decomposed in Principal Orthogonal Components (KL decomposition), C(x, x′) =

∞

• k=1

λkφk(x)φk(x′), M(x) =

∞

• k=1
• λkΦk(x)ηk,

where the ηk’s are iid standard Gaussian random variables. Upon truncation of the expansion of M to its K dominant terms, M(x) ≈ MK (x) =

K

• k=1
• λkΦk(x)ηk,

Inference problem for the stochastic coordinates ηk’s : p(η, σ2

• |d) ∝ p(d|η, σ2
• )pη(η)po(σ2
• ),

with pη(η) = 1 (2π)K/2 exp

• −η2/2
• ,

p(d|η, σ2

• ) =

m

• i=1

pǫ(di − ui(η), σ2

• ).

O.P . Le Maître et.al Change of Coordinates & PC Acceleration for Bayesian Inference

Bayesian Inference Coordinate transformation for Uncertain Correlation Function PC surrogate model Inference Example

Uncertainty in the covariance function The selection of the covariance function affects the inference procedure and C is in general uncertain. ⇒ family of covariance functions C(q) with hyper-parameters q having prior pq(q) (also inferred). Following this approach, we write M(x, q) ≈ MK (x, q) =

K

• k=1
• λk(q)Φk(x, q)ηk,

where the ηk’s are still i.i.d. standard Gaussian random variables and (λk(q), Φk(q)) are the dominant proper elements of C(x, x′, q). p(η, q, σ2

• |d) ∝ p(d|η, q, σ2
• )pη(η)pq(q)po(σ2
• ).

KL decomp

q η σ2

Mk =

K

X

k=1

p λkφkηk

U(η, q) p(d|η, q, σ2

0)

p(η, q, σ2

0|d)

Model solve (λk, φk)k=1,K Likelihood Posterior

many KL decomposition many model solves change of coordinate Use of PC surrogate

O.P . Le Maître et.al Change of Coordinates & PC Acceleration for Bayesian Inference

Bayesian Inference Coordinate transformation for Uncertain Correlation Function PC surrogate model Inference Example

Reference Basis For any covariance parameters q, the elements of the KL expansion are solution of ˆ

Ω

C(x, x′, q)Φk(x′, q)dx′ = λk(q)Φk(x, q), (Φk, Φk)X = 1. We observe that {Φk(q)} is a CONS of L2(Ω). It suggests the introduction of a reference orthonormal basis {¯ Φk}, defined for a prescribed reference covariance function C, and to project Mk(q) onto this reference subspace. Let ˜ Φk(q) = √λk(q)Φk(q), it comes Mk(q) =

K

• k=1

˜ Φk(q)ηk =

K

• k=1

∞

• k′=1

bk,k′(q)¯ Φk

• ηk,

bk,k′(q) = (˜ Φk(q), Φk′)X . For a finite dimensional reference basis (with K modes for simplicity), it comes Mk(q) =

K

• k=1

˜ Φk(q)ηk ≈ MK =

K

• k=1

¯ Φk ¯ ηk(q), ¯ η(q) = B(q)η.

O.P . Le Maître et.al Change of Coordinates & PC Acceleration for Bayesian Inference

Bayesian Inference Coordinate transformation for Uncertain Correlation Function PC surrogate model Inference Example

Change of coordinates The effect of q are reflected by a linear change of stochastic coordinates η → ¯ η(q), such that ¯ η(q) = B(q)η ⇒ Σ2(q) = E

• η(q)ηt(q)
• = B(q)Bt(q).

We note that ¯ η(q) is Gaussian with conditional density pη(η|q) = 1

• 2π|Σ2(q)|

exp

• − ηt(Σ2(q))−1η

2

• ,

where |Σ2(q)| is the determinant of Σ2(q). We shall assume Σ2(q) non-singular a.s. Regarding the selection of the reference basis : select of particular hyper-parameter value : C = C(¯ q) use the q-averaged covariance function, ¯ C = C = ˆ C(q)pq(q)dq. The latter choice is optimal in terms of representation error (averaged over q).

O.P . Le Maître et.al Change of Coordinates & PC Acceleration for Bayesian Inference

Bayesian Inference Coordinate transformation for Uncertain Correlation Function PC surrogate model Inference Example

Example of Gaussian covariance function Consider Ω = [0, 1] and a Gaussian covariance function with uncertain correlation length : C(x, x′, q = {l}) = σ2

f exp

• − (x − x′)2

2l2

• ,

l ∼ U[0.1, 1].

−1 −0.5 0.5 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

C x − x′ l = 0.1 l = 0.2 l = 0.3 l = 0.4 l = 0.5 l = 1.0 C

1 5 9 13 17 21 25 10

−20

10

−15

10

−10

10

−5

10 λK K

l = 0.1 l = 0.2 l = 0.3 l = 0.4 l = 0.5 l = 1.0 C

FIGURE: (Left) Reference covariance functions C(l) for different values of l, as indicated. Also plotted is the q-averaged covariance C and (Right) Spectra of the covariance functions shown in the left plot.

O.P . Le Maître et.al Change of Coordinates & PC Acceleration for Bayesian Inference

Bayesian Inference Coordinate transformation for Uncertain Correlation Function PC surrogate model Inference Example

Example of Gaussian covariance function We define the approximation errors : ǫM(K, q) = M(q) − MK (q)L2 M(q)L2 , E2

M(K) =

ˆ ǫ2

M(K, q)pq(q)dq.

1 5 9 13 17 21 25 10

−8

10

−6

10

−4

10

−2

10 EM(K) K

l = 0.1 l = 0.2 l = 0.3 l = 0.4 l = 0.5 l = 1.0 C

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 10

−8

10

−6

10

−4

10

−2

10 ǫM(K, l) l

l = 0.1 l = 0.2 l = 0.3 l = 0.4 l = 0.5 l = 1.0 C

FIGURE: (Left) Error EM(K) in approximating the Gaussian Process M by MK for different reference covariance functions based on selected correlation lengths l as indicated. Also plotted are results

• btained with C. (Right) Relative error ǫM(K = 15, l) for the same cases as in the left plot.

O.P . Le Maître et.al Change of Coordinates & PC Acceleration for Bayesian Inference

Bayesian Inference Coordinate transformation for Uncertain Correlation Function PC surrogate model Inference Example

Example of Gaussian covariance function

0.2 0.4 0.6 0.8 1 0.5 1 x l K = 1 0.2 0.4 0.6 0.8 1 0.5 1 x l K = 4 0.2 0.4 0.6 0.8 1 0.5 1 x l K = 7 0.2 0.4 0.6 0.8 1 0.5 1 x l K = 10 0.2 0.4 0.6 0.8 1 0.5 1 x l K = 13 0.2 0.4 0.6 0.8 1 0.5 1 x l K = 19

FIGURE: Dependence of eigen-functions φk(q) with the length-scale hyper-parameter l and selected k as indicated.

O.P . Le Maître et.al Change of Coordinates & PC Acceleration for Bayesian Inference

Bayesian Inference Coordinate transformation for Uncertain Correlation Function PC surrogate model Inference Example

Motivation Sampling of the posterior p(η, q, σ2

• |d) involves many resolutions of the forward model

to predict the observation u(η, q). To accelerate this step, the use of polynomial surrogates (PC expansions) was proposed by Marzouk, Najm, et al : u(η, q) ≈

P

• α=0

uαΨα(η, q), where the Ψα’s are orthogonal polynomials and the PC expansion is truncated at some

• rder r.

The PC expansion is computed in an off-line stage. We propose an alternative approach, relying on coordinate transformation : u(η, q) ≈ ˆ u(ξ(η, q)) =

P

• α=0

uαΨ(ξ(η, q)), where the random vector ξ has the same dimension as η, that is K.

O.P . Le Maître et.al Change of Coordinates & PC Acceleration for Bayesian Inference

Bayesian Inference Coordinate transformation for Uncertain Correlation Function PC surrogate model Inference Example

PC surrogate Recall that the transformed coordinates ¯ η have for conditional density pη(η|q) = 1

• 2π|Σ2(q)|

exp

• − ηt(Σ2(q))−1η

2

• .

For ¯ C = C, it can be shown that ˙ pη(η|q)pq(q)dq = 1

• 2π|Λ2|

exp

• − ηt(Λ2)−1η

2

• ,

Λ2 = diag (¯ λ1 · · · ¯ λK ) It suggests approximating ¯ η → u(¯ η) using the reference Gaussian field M

PC K (ξ) = K

• k=1
• λkφkξk,

ξ → ˆ u(ξ) ≈

P

• α=0

ˆ uαΨα(ξ), where the ξk’s are independent standard Gaussian random variables. Then u(η, q) ≈

P

• α=0

ˆ uαΨα(ξ(η, q)), ξ(η, q) = ˜ B(q)η, ˜ Bkl(q) =      Bkl(q)

• λk

, λk/λ1 > κ, 0,

• therwise.

for some small κ > 0.

O.P . Le Maître et.al Change of Coordinates & PC Acceleration for Bayesian Inference

Bayesian Inference Coordinate transformation for Uncertain Correlation Function PC surrogate model Inference Example

Example : 1-D diffusion problem Consider the diffusion problem for x ∈ (0, 1) and t ∈ [0, Tf ], given by ∂U ∂t = ∂ ∂x

• ν ∂U

∂x

• ,

ν = ν0 + exp(M), with IC U = 0 and BCs U(x = 0, t) = −1, U(x = 1, t) = 1 and M is a (centered) Gaussian process with the previous uncertain Gaussian covariance function C(q = {l}).

1 3 5 7 9 11 13 15 10

−3

10

−2

10

−1

10 EU(r, K) K

l = 0.1 l = 0.2 l = 0.3 l = 0.4 l = 0.5 l = 1.0 C

2 4 6 8 10 10

−3

10

−2

10

−1

10 r EU(r, K)

l = 0.1 l = 0.2 l = 0.3 l = 0.4 l = 0.5 l = 1.0 C

FIGURE: Global error EU(r, K) of the PC approximation ˆ U of the diffusion model problem solution. The left plot shows the dependence of the error with K using a PC order r = 10, while the right plot is for different r and K = 10. The curves correspond to different definitions of the reference covariance function C : C(l) with l as indicated or the q-average covariance function C.

O.P . Le Maître et.al Change of Coordinates & PC Acceleration for Bayesian Inference

Bayesian Inference Coordinate transformation for Uncertain Correlation Function PC surrogate model Inference Example

Conditioning of the coordinate transformation The PC surrogate is constructed assuming ξ ∼ N(0, I) ; it is subsequently used for ξ(η, q) = ˜ B(q)η. Defining Σ2

ξ(q) = ˜

B(q)t ˜ B(q), the largest eigen-value βmax(q) of Σ2

ξ(q) measures the local stretching of the coordinate transformation.

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 10

−3

10

−2

10

−1

ǫU(r, K, l) l l = 0.1 l = 0.2 l = 0.3 l = 0.4 l = 0.5 l = 1.0 C 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 10 10

1

10

2

10

3

10

4

10

5

10

6

• βmax(l)

l

l = 0.1 l = 0.2 l = 0.3 l = 0.4 l = 0.5 l = 1.0 C

0.1 0.5 1 1 2 3

FIGURE: Global error EU(r, K) of the PC approximation ˆ U of the diffusion model problem solution. The left plot shows the dependence of the error with K using a PC order r = 10, while the right plot is for different r and K = 10. The curves correspond to different definitions of the reference covariance function C : C(l) with l as indicated or the q-average covariance function C.

O.P . Le Maître et.al Change of Coordinates & PC Acceleration for Bayesian Inference

Bayesian Inference Coordinate transformation for Uncertain Correlation Function PC surrogate model Inference Example

Test problem Inference for "true" log-diffusivity fields : Sinusoidal profile : Msin(x) = sin(πx), Step function : Mstep(x) =

• −1/2,

x < 0.5 1/2, x ≥ 0.5 , Random profile : Mran(x) drawn at random from GP(0, C) where C is the Gaussian covariance with length-scale l = 0.25 and variance σ2

f = 0.65.

Observations are measurements of U(x, t) at several locations in space and time, corrupted with i.i.d. ǫi ∼ N(0, σ2

ǫ = 0.01).

For prior, we use M ∼ GP(0, C(q)), with Gaussian covariance C(q) and hyper-parameter q = {l, σ2

f } :

l ∼ U[0.1, 1], σ2

f ∼ InvΓ(α, β), with mean 0.5 and variance 0.25.

O.P . Le Maître et.al Change of Coordinates & PC Acceleration for Bayesian Inference

Bayesian Inference Coordinate transformation for Uncertain Correlation Function PC surrogate model Inference Example

Sampling flow-chart

KL decomp (λk, φk)k=1,K Solve PC problem

C

M

PC K = K

X

k=1

q λk φkξk ˆ U(ξ) =

P

X

α=0

UαΨα(ξ)

Offline

KL decomp (λk, φk)k=1,K Change of coord U(η, q) ≈ X

α

UαΨα(ξ) PC surrogate ˆ p(d|η, q, σ2

0)

PC Likelihood PC Posterior ˆ p(η, q, σ2

0|d)

Uα, α = 0, . . . , P ξ = ˜ B(q)η

σ2 η q ˜ B(q)

Online FIGURE: Offline step (surrogate construction) of the accelerated MCMC sampler and Online step of the PC surrogate based evaluation of the posterior.

O.P . Le Maître et.al Change of Coordinates & PC Acceleration for Bayesian Inference

Bayesian Inference Coordinate transformation for Uncertain Correlation Function PC surrogate model Inference Example

Inference without covariance Hyper-parameters We set l = 0.5 and σ2

f = 0.5. Also K = 15 and r = 10.

−6 −4 −2 2 4 6 0.5 1 1.5 2 η1 pdf KLD = 39.6066 Prior Posterior −6 −4 −2 2 4 6 1 2 3 4 η2 pdf KLD = 63.1121 Prior Posterior −6 −4 −2 2 4 6 0.5 1 1.5 η3 pdf KLD = 40.7446 Prior Posterior −6 −4 −2 2 4 6 0.2 0.4 0.6 0.8 η4 pdf KLD = 64.5038 Prior Posterior −6 −4 −2 2 4 6 0.1 0.2 0.3 0.4 0.5 η5 pdf KLD = 0.22482 Prior Posterior −6 −4 −2 2 4 6 0.1 0.2 0.3 0.4 η6 pdf KLD = 0.034163 Prior Posterior −6 −4 −2 2 4 6 0.1 0.2 0.3 0.4 0.5 η7 pdf KLD = 0.03121 Prior Posterior −6 −4 −2 2 4 6 0.1 0.2 0.3 0.4 η8 pdf KLD = 0.011068 Prior Posterior 0.006 0.008 0.01 0.012 0.014 0.016 100 200 300 400 500 σo

2

pdf Posterior

FIGURE: Comparison of priors and marginals posterior of the first 8 KL coordinates ηk for the inference of Msin without using ovariance

hyper-parameters (a Gaussian covariance with l = 0.5 and σ2 f = 0.5 is assumed). The Kullback-Leibler Divergence (KLD) between the priors and marginal posteriors are also indicated on top of each plot. The posterior of the noise hyper-parameter σ2

• is indicated.

O.P . Le Maître et.al Change of Coordinates & PC Acceleration for Bayesian Inference

Bayesian Inference Coordinate transformation for Uncertain Correlation Function PC surrogate model Inference Example

Inference with Hyper-parameters K = 15 and r = 10.

−6 −4 −2 2 4 6 0.5 1 1.5 η1 pdf KLD = 12.1381 Prior Posterior −6 −4 −2 2 4 6 0.2 0.4 0.6 0.8 1 η2 pdf KLD = 53.6187 Prior Posterior −6 −4 −2 2 4 6 0.5 1 1.5 2 η3 pdf KLD = 21.8182 Prior Posterior −6 −4 −2 2 4 6 0.2 0.4 0.6 0.8 1 η4 pdf KLD = 36.9149 Prior Posterior −6 −4 −2 2 4 6 0.2 0.4 0.6 0.8 1 η5 pdf KLD = 15.8119 Prior Posterior −6 −4 −2 2 4 6 0.2 0.4 0.6 0.8 η6 pdf KLD = 3.8215 Prior Posterior −6 −4 −2 2 4 6 0.2 0.4 0.6 0.8 η7 pdf KLD = 3.4565 Prior Posterior −6 −4 −2 2 4 6 0.1 0.2 0.3 0.4 0.5 η8 pdf KLD = 0.23583 Prior Posterior 0.006 0.008 0.01 0.012 0.014 100 200 300 400 500 σo

2

pdf Posterior

FIGURE: Comparison of priors and marginals posterior of the first 8 KL coordinates ηk for the inference of Msin without using ovariance

hyper-parameters (a Gaussian covariance with l = 0.5 and σ2 f = 0.5 is assumed). The Kullback-Leibler Divergence (KLD) between the priors and marginal posteriors are also indicated on top of each plot. The posterior of the noise hyper-parameter σ2

• is indicated.

O.P . Le Maître et.al Change of Coordinates & PC Acceleration for Bayesian Inference

Bayesian Inference Coordinate transformation for Uncertain Correlation Function PC surrogate model Inference Example

Inference : comparison of inferred field

0.2 0.4 0.6 0.8 1 −1.5 −1 −0.5 0.5 1 1.5 x M(x) True Profile 0.50 Percentile 0.05/0.95 Percentile MAP Mean 0.2 0.4 0.6 0.8 1 −1.5 −1 −0.5 0.5 1 1.5 x M(x) True Profile 0.50 Percentile 0.05/0.95 Percentile MAP Mean 0.2 0.4 0.6 0.8 1 −1.5 −1 −0.5 0.5 1 1.5 x M(x) True Profile 0.50 Percentile 0.05/0.95 Percentile MAP Mean 0.2 0.4 0.6 0.8 1 −1.5 −1 −0.5 0.5 1 1.5 x M(x) True Profile Median Profile infered q Median Profile pre-assigned q 0.2 0.4 0.6 0.8 1 −1.5 −1 −0.5 0.5 1 1.5 x M(x) True Profile Median Profile infered q Median Profile pre-assigned q 0.2 0.4 0.6 0.8 1 −1.5 −1 −0.5 0.5 1 1.5 x M(x) True Profile Median Profile infered q Median Profile pre-assigned q

FIGURE: Comparison of inferred log-diffusivity profile : pre-assigned hyper-parameters versus uncertain hyper-parameters.

O.P . Le Maître et.al Change of Coordinates & PC Acceleration for Bayesian Inference

Bayesian Inference Coordinate transformation for Uncertain Correlation Function PC surrogate model Inference Example

Inference of Hyper-parameters

1 2 3 0.5 1 1.5 2 2.5 σ2

f

pdf Prior Posterior 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 2 4 6 8 10 l pdf Prior Posterior

FIGURE: Posterior pdfs of sinusoidal log-diffusivity profile hyper-parameters.

O.P . Le Maître et.al Change of Coordinates & PC Acceleration for Bayesian Inference

Bayesian Inference Coordinate transformation for Uncertain Correlation Function PC surrogate model Inference Example

Conclusion & Future work Effective treatment of covariance hyper-parameters Generic PC construction for the surrogate Further accelerations Projection of the change of coordinate : u(η, q) ≈

• α

ˆ uαΨα(ξ), ξ = ˜ B(q)η, with ˜ B(q) ≈

• β

˜ BβΨβ(q). Adaptive construction of the PC surrogate

O.P . Le Maître et.al Change of Coordinates & PC Acceleration for Bayesian Inference