[PPT] - Nonlinear dimensionality reduction for functional computer code PowerPoint Presentation

SLIDE 1

Nonlinear dimensionality reduction for functional computer code modelling

Benjamin Auder

CEA - UPMC

24 august 2010

Thesis since 02/2008 PhD supervisors : G´ erard Biau (UPMC) Bertrand Iooss (EDF)

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 1 / 17

SLIDE 2

Industrial context

Framework : life span of reactor vessels. → Several sequences of accidents can occur. Goal = estimate their probabilities of occurrence.

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 2 / 17

SLIDE 3

Industrial context

Framework : life span of reactor vessels. → Several sequences of accidents can occur. Goal = estimate their probabilities of occurrence.

Methodology

Modelling

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 2 / 17

SLIDE 4

Industrial context

Framework : life span of reactor vessels. → Several sequences of accidents can occur. Goal = estimate their probabilities of occurrence.

Methodology

Modelling − → Simulation

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 2 / 17

SLIDE 5

Industrial context

Framework : life span of reactor vessels. → Several sequences of accidents can occur. Goal = estimate their probabilities of occurrence.

Methodology

Modelling − → Simulation − → Computation. → Sensitivity analysis, uncertainty propagation . . .

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 2 / 17

SLIDE 6

Industrial context

Framework : life span of reactor vessels. → Several sequences of accidents can occur. Goal = estimate their probabilities of occurrence.

Methodology

Modelling − → Simulation − → Computation. → Sensitivity analysis, uncertainty propagation . . . Improve the simulation stage to allow accurate computations

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 2 / 17

SLIDE 7

Mathematical formulation

n known couples (xi, yi) = inputs-outputs of a very slow code : Inputs xi ∈ ❘p = initial state of physical system ; Outputs yi ∈ C([a, b], ❘) = evolutions of parameters. − → ❘ ❘

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 3 / 17

SLIDE 8

Mathematical formulation

n known couples (xi, yi) = inputs-outputs of a very slow code : Inputs xi ∈ ❘p = initial state of physical system ; Outputs yi ∈ C([a, b], ❘) = evolutions of parameters. Goal = prediction of functional data : ynew ≃ ϕ(xnew) . − → ❘ ❘

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 3 / 17

SLIDE 9

Mathematical formulation

n known couples (xi, yi) = inputs-outputs of a very slow code : Inputs xi ∈ ❘p = initial state of physical system ; Outputs yi ∈ C([a, b], ❘) = evolutions of parameters. Goal = prediction of functional data : ynew ≃ ϕ(xnew) . − → Statistical learning ”regression” ❘p → C([a, b], ❘)

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 3 / 17

SLIDE 10

Back to the ”simple” case of yi ∈ ❘d

❘ ❘ ❘ ❘

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 4 / 17

SLIDE 11

Back to the ”simple” case of yi ∈ ❘d

1

dimensionality reduction : r : C([a, b], ❘) → ❘d (representation) ; ❘ ❘ ❘ ❘

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 4 / 17

SLIDE 12

Back to the ”simple” case of yi ∈ ❘d

1

dimensionality reduction : r : C([a, b], ❘) → ❘d (representation) ;

2

statistical learning : f : ❘p (inputs) → ❘d (reduced outputs) ; ❘ ❘

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 4 / 17

SLIDE 13

Back to the ”simple” case of yi ∈ ❘d

1

dimensionality reduction : r : C([a, b], ❘) → ❘d (representation) ;

2

statistical learning : f : ❘p (inputs) → ❘d (reduced outputs) ;

3

utput space parametrization :

R : ❘d → C([a, b], ❘) (reconstruction).

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 4 / 17

SLIDE 14

Back to the ”simple” case of yi ∈ ❘d

1

dimensionality reduction : r : C([a, b], ❘) → ❘d (representation) ;

2

statistical learning : f : ❘p (inputs) → ❘d (reduced outputs) ;

3

utput space parametrization :

R : ❘d → C([a, b], ❘) (reconstruction).

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 4 / 17

SLIDE 15

State of art

(More or less) classical methods

Functional linear regression : Faraway, 1997 ; Ramsay & Silverman, 2005, . . .

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 5 / 17

SLIDE 16

State of art

(More or less) classical methods

Functional linear regression : Faraway, 1997 ; Ramsay & Silverman, 2005, . . . Decomposition on an orthonormal basis, then learning of d-dimensional coefficients : Chiou et al., 2004 ; Govaerts & No¨ el, 2005 ; Bayarri et al., 2007 ; Marrel, 2008 ; Monestiez & Nerini, 2009

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 5 / 17

SLIDE 17

State of art

(More or less) classical methods

Functional linear regression : Faraway, 1997 ; Ramsay & Silverman, 2005, . . . Decomposition on an orthonormal basis, then learning of d-dimensional coefficients : Chiou et al., 2004 ; Govaerts & No¨ el, 2005 ; Bayarri et al., 2007 ; Marrel, 2008 ; Monestiez & Nerini, 2009 (New) goal : minimize the representation dimension d, to simplify the model ; avoid overfitting, while keeping good performances.

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 5 / 17

SLIDE 18

1

Riemannian Manifold Learning

2

Applications

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 6 / 17

SLIDE 19

1

Riemannian Manifold Learning

2

Applications

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 7 / 17

SLIDE 20

Local steps

RML ≃ preservation of angles and geodesic distances.

1

choose an origin curve y0 among the yi, (e.g., the mean) ;

Fig.: Origin curve y0

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 8 / 17

SLIDE 21

Local steps

RML ≃ preservation of angles and geodesic distances.

1

choose an origin curve y0 among the yi, (e.g., the mean) ;

2

determine a local basis Q0 = (e1, . . . , ed) for the tangent space at y0 (PCA on the neighborhoods curves) ;

Fig.: Tangent plane at y0 + local basis Q0

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 8 / 17

SLIDE 22

Local steps

RML ≃ preservation of angles and geodesic distances.

1

choose an origin curve y0 among the yi, (e.g., the mean) ;

2

determine a local basis Q0 = (e1, . . . , ed) for the tangent space at y0 (PCA on the neighborhoods curves) ;

3

compute the reduced coordinates zi or curves yi ”close” to y0 by projecting on Q0,

Fig.: Local coordinates zi on the tangent plane

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 8 / 17

SLIDE 23

Local steps

RML ≃ preservation of angles and geodesic distances.

1

choose an origin curve y0 among the yi, (e.g., the mean) ;

2

determine a local basis Q0 = (e1, . . . , ed) for the tangent space at y0 (PCA on the neighborhoods curves) ;

3

compute the reduced coordinates zi or curves yi ”close” to y0 by projecting on Q0,

4

then normalize to verify the identity y − y0 = x − x0.

Fig.: Normalization of coordinates zi

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 8 / 17

SLIDE 24

”far” from y0

Step 4 : for y far from y0 (too far for last step to be accurate), yp = predecessor of y on a shortest path from y0 yi1, . . . , yid = neighbors of yp for which the zi coordinates are known (breadth-first)

Fig.: Data yi in dim. D

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 9 / 17

SLIDE 25

”far” from y0

Step 4 : for y far from y0 (too far for last step to be accurate), yp = predecessor of y on a shortest path from y0 yi1, . . . , yid = neighbors of yp for which the zi coordinates are known (breadth-first)

z = r(y) computed by..

..preserving angles as much as possible : zzpzij ≃ yypyij ; ..under the normalization constraint y − yp = z − zp.

Fig.: Data yi in dim. D Fig.: zi = r(yi) in dim. d ≪ D

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 9 / 17

SLIDE 26

Examples

Fig.: 3D Swissroll, 400 points Fig.: RML Representation

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 10 / 17

SLIDE 27

Examples

Fig.: 3D Swissroll, 400 points Fig.: RML Representation Fig.: 3D Gaussian, 1000 points Fig.: RML Representation

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 10 / 17

SLIDE 28

1

Riemannian Manifold Learning

2

Applications

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 11 / 17

SLIDE 29

Validation step

Data : training = {(xi, yi) , i = 1, . . . , n} ; test = {(x′

i , y ′ i ) , i = 1, . . . , m} ;

Model predictions : ˆ y ′

i = M(x′ i ), i = 1, . . . , m.

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 12 / 17

SLIDE 30

Validation step

Data : training = {(xi, yi) , i = 1, . . . , n} ; test = {(x′

i , y ′ i ) , i = 1, . . . , m} ;

Model predictions : ˆ y ′

i = M(x′ i ), i = 1, . . . , m.

”Absolute” then relative measures of the pointwise error

MSE[ j ] = 1 m

m

i=1

(ˆ y ′

i (j) − y ′ i (j))2 ,

j = 1, . . . , D (discretization).

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 12 / 17

SLIDE 31

Validation step

Data : training = {(xi, yi) , i = 1, . . . , n} ; test = {(x′

i , y ′ i ) , i = 1, . . . , m} ;

Model predictions : ˆ y ′

i = M(x′ i ), i = 1, . . . , m.

”Absolute” then relative measures of the pointwise error

MSE[ j ] = 1 m

m

i=1

(ˆ y ′

i (j) − y ′ i (j))2 ,

j = 1, . . . , D (discretization). Q2[ j ] = 1 − m.MSE[ j ] m

i=1(¯

y(j) − y ′

i (j))2 (compararison to the mean).

−∞ < Q2 ≤ 1 : ≤ 0 ⇒ (very) bad model ; ≃ 1 ⇒ perfect model.

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 12 / 17

SLIDE 32

Test I - temperature curves

100 model runs, 4 dimensions on input, 168 discretization points.

Fig.: The 100 code outputs

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 13 / 17

SLIDE 33

Test I - temperature curves

100 model runs, 4 dimensions on input, 168 discretization points. cross validation leave-10-out : MSE at l., Q2 at r. ; d = 4

Fig.: The 100 code outputs Fig.: Black : functional PCA ; blue : RML ; green : Nadaraya-Watson.

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 13 / 17

SLIDE 34

5 predicted curves

Fig.: Real curves Fig.: PCA dim. red. Fig.: RML dim. red. Fig.: Nadaraya-Watson

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 14 / 17

SLIDE 35

Test II - temperature curves

600 model runs, 11 dimensions on input, 414 discretization points.

Fig.: The 600 code outputs

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 15 / 17

SLIDE 36

Test II - temperature curves

600 model runs, 11 dimensions on input, 414 discretization points. cross validation leave-10-out : MSE at l., Q2 at r. ; d = 7

Fig.: The 600 code outputs Fig.: Black : functional PCA ; blue : RML ; green : Nadaraya-Watson.

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 15 / 17

SLIDE 37

5 predicted curves

Fig.: Real curves Fig.: PCA red. dim. Fig.: RML red. dim. Fig.: Nadaraya-Watson

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 16 / 17

SLIDE 38

Conclusion

The model is good enough regarding to the industrial context. ⇒ help to the projet DDVCV (life span of reactor vessels).

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 17 / 17

SLIDE 39

Conclusion

The model is good enough regarding to the industrial context. ⇒ help to the projet DDVCV (life span of reactor vessels).

Dimensionality reduction

Linear (PCA) and non-linear : complementary.

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 17 / 17

SLIDE 40

Conclusion

The model is good enough regarding to the industrial context. ⇒ help to the projet DDVCV (life span of reactor vessels).

Dimensionality reduction

Linear (PCA) and non-linear : complementary. Missing : proof of convergence to the right manifold . . . . . .and some parameters which should be ”optimized” automatically.

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 17 / 17

SLIDE 41

Conclusion

The model is good enough regarding to the industrial context. ⇒ help to the projet DDVCV (life span of reactor vessels).

Dimensionality reduction

Linear (PCA) and non-linear : complementary. Missing : proof of convergence to the right manifold . . . . . .and some parameters which should be ”optimized” automatically. Future research : ”functional” principal curves and surfaces. Example of a principal surface in 2D :

Benjamin Auder (CEA - UPMC) Nonlinear dimensionality reduction 24 august 2010 17 / 17