Functional Median Polish, with Climate Applications Marc G. Genton - - PowerPoint PPT Presentation

▶

May 23, 2023 513 likes •836 views

Functional Median Polish Motivation Univariate ANOVA Functional ANOVA Simulation Studies Applications Discussion Functional Median Polish, with Climate Applications Marc G. Genton Department of Statistics, Texas A&M University Program

SLIDE 1

Functional Median Polish Motivation Univariate ANOVA Functional ANOVA Simulation Studies Applications Discussion

Functional Median Polish, with Climate Applications

Marc G. Genton

Department of Statistics, Texas A&M University Program in Spatial Statistics (stat.tamu.edu/pss) Institute for Applied Mathematics and Computational Sciences (iamcs.tamu.edu) Based on joint work with Ying Sun

May 11, 2012

Marc G. Genton Functional Median Polish, with Climate Applications

SLIDE 2

Functional Median Polish Motivation Univariate ANOVA Functional ANOVA Simulation Studies Applications Discussion

Functional Median Polish

1 Motivation 2 Univariate ANOVA 3 Functional ANOVA 4 Simulation Studies 5 Applications 6 Discussion

Marc G. Genton Functional Median Polish, with Climate Applications

SLIDE 3

Functional Median Polish Motivation Univariate ANOVA Functional ANOVA Simulation Studies Applications Discussion

Observations and Climate Models

Observations:

provide a corroborating source of information about physical processes being modeled. have methodological and practical issues due to uncertainties.

Climate Models:

numerically solve systems of differential equations representing physical relationships in the climate system. have huge uncertainties and biases.

Scientific Questions:

How do we compare sources of variability in observations or climate model outputs? i.e. quantification of uncertainties?

Marc G. Genton Functional Median Polish, with Climate Applications

SLIDE 4

Spatio-Temporal Precipitation Data

Spatio-temporal precipitation data: annual total precipitation data for U.S. from 1895 to 1997 at 11,918 weather stations. Nine climatic regions for precipitation defined by National Climatic Data Center. Several areas of outliers detected by Sun and Genton (2011, 2012).

+ + + + + + + + ++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ++ + + + ++ + + + + + + + + + + + + + + + ++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ++ + + + + + + + ++ + + + + + + + + + + + + + ++

NW W WNC S SW ENC C SE NE

SLIDE 5

Functional Median Polish Motivation Univariate ANOVA Functional ANOVA Simulation Studies Applications Discussion

Analysis of Variance

Analysis of Variance (ANOVA):

An important technique for analyzing the effect of categorical factors on a response. It decomposes the variability in the response variable among the different factors. A two-way additive model: for i = 1, . . . , r, j = 1, . . . , c,

yij = µ + αi + βj + ǫij. The ANOVA model can be fitted by arithmetic means (no

utliers), or medians (robust).

Marc G. Genton Functional Median Polish, with Climate Applications

SLIDE 6

Functional Median Polish Motivation Univariate ANOVA Functional ANOVA Simulation Studies Applications Discussion

ANOVA Model Fitting

Fitted by means:

ˆ µ = ¯ y (grand effect), ˆ αi = ¯ yi· − ¯ y (row effect), ˆ βj = ¯ y·j − ¯ y (column effect).

Fitted by medians:

Median polish (Tukey, 1970, 1977). An iterative technique for extracting row and column effects in a two-way table using medians rather than means. It stops when no more changes occur in the row and column effects, or changes are sufficiently small.

Marc G. Genton Functional Median Polish, with Climate Applications

SLIDE 7

Median Polish Example

Original table: find row medians.

1st iteration: subtract row medians, find column medians. Grand median in red, row effects in blue, column effects in green.

2nd iteration: subtract column medians, find row medians, 6 3 11 6 3 2 4 3 9 → −3 5 6 −1 1 3 9 −1 1 3 → −2 4 3 9 1 −1 1 −3 −1 1 3 →

subtract new row medians, add their medians to the grand median, find column medians.

Polished table: new row and column medians are zero after two iterations. −2 4 3 8 −2 −3 −1 1 3 + 0 → −2 4 3 8 −2 −3 −1 1 3

SLIDE 8

Functional Median Polish

Observe functional data at each combination of two categorical factors. Examine their effects: functional row or column effects. yijk(x) = µ(x) + αi(x) + βj(x) + ǫijk(x), where i = 1, . . . , r, j = 1, . . . , c, k = 1, . . . , mij. Constraints: mediani{αi(x)} = 0, medianj{βj(x)} = 0 and mediani{ǫijk(x)} = medianj{ǫijk(x)} = 0 for all k. x can be time for curves or spatial index for surfaces/images. Iterative procedure sweeping out column and row medians. One-way functional ANOVA can be done in a similar way. Need to order functional data.

SLIDE 9

Functional Median Polish Motivation Univariate ANOVA Functional ANOVA Simulation Studies Applications Discussion

Multivariate Ordering

Basic ideas of depth in functional context

provides a method to order sample curves according to decreasing depth values, y[1]: the deepest (most central or median) curve, y[n]: the most outlying (least representative) curve, y[1], . . . , y[n]: start from the center outwards.

Usual order statistics: ordered from the smallest sample value to the largest.

Marc G. Genton Functional Median Polish, with Climate Applications

SLIDE 10

Band Depth for Functional Data

L´

pez-Pintado and Romo (2009) introduced the band depth

(BD) concept through a graph-based approach. Grey area: band determined by two curves, y1 and y3. Contains the curve y2, but does not contain y4.

0.0 0.2 0.4 0.6 0.8 1.0 −3 −2 −1 1 2 3 4

y1 y2 y3 y4 t y(t)

SLIDE 11

Functional Median Polish Motivation Univariate ANOVA Functional ANOVA Simulation Studies Applications Discussion

Band Depth for Functional Data

Population version of BD(2): BD(2)(y, P) = P{G(y) ⊂ B(Y1, Y2)}.

G(y): graph of the curve y, B(Y1, Y2): band delimited by 2 random curves.

The band could be delimited by more than 2 random curves, BDJ(y, P) =

J

BD(j)(y, P).

Marc G. Genton Functional Median Polish, with Climate Applications

SLIDE 12

Functional Median Polish Motivation Univariate ANOVA Functional ANOVA Simulation Studies Applications Discussion

Sample Band Depth

Population level: BD(j)(y, P) is a probability. Sample version of BD(j)(y, P) BD(j)

n (y) =

n j −1

1≤i1<i2<...<ij≤n

I{G(y) ⊆ B(yi1, . . . , yij)},

I{·}: the indicator function, fraction of the bands completely containing the curve y.

Sample BD: BDn,J(y) = J

j=2 BD(j) n (y).

Marc G. Genton Functional Median Polish, with Climate Applications

SLIDE 13

Functional Median Polish Motivation Univariate ANOVA Functional ANOVA Simulation Studies Applications Discussion

Modified Band Depth

L´

pez-Pintado and Romo (2009) also proposed a more flexible

definition, the modified band depth (MBD). BD(j)

n (y) =

n j −1

1≤i1<i2<...<ij≤n

I{G(y) ⊆ B(yi1, . . . , yij)}, MBD(j)

n (y) =

n j −1

1≤i1<i2<...<ij≤n

λr{A(y; yi1, . . . , yij)}. λr{A(y; yi1, . . . , yij)} measures the proportion of time that a curve y is in the band.

Marc G. Genton Functional Median Polish, with Climate Applications

SLIDE 14

Functional Boxplots (Sun and Genton, 2011, 2012)

2 4 6 8 10 12 18 20 22 24 26 28 30

Month Sea Surface Temperature

2 4 6 8 10 12 18 20 22 24 26 28 30

Functional Boxplot

Sea Surface Temperature 1998 1983 1997 1982

SLIDE 15

Surface Boxplot

SLIDE 16

Functional Median Polish Motivation Univariate ANOVA Functional ANOVA Simulation Studies Applications Discussion

True Model

Generate data from a true model with r = 2, c = 3, and m = 100 curves in each cell at p = 50 time points.

0.0 0.2 0.4 0.6 0.8 1.0 1 2 3 4

True Grand Effect

x Grand Effect 0.0 0.2 0.4 0.6 0.8 1.0 −1.0 −0.5 0.0 0.5 1.0

True Row Effect

x Row Effect 0.0 0.2 0.4 0.6 0.8 1.0 −4 −2 2 4

True Column Effect

x Column Effect

Introduce outliers through a Gaussian process ǫijk(t). Replications: 1,000.

Marc G. Genton Functional Median Polish, with Climate Applications

SLIDE 17

Outlier Models

Model 1: ǫijk(t) = eijk(t), where eijk(t) ∼ GP(0, γ) with γ(t1, t2) = exp{−|t2 − t1|}. Model 2: ǫijk(t) = eijk(t) + cijkK, where cijk is 1 with prob qij and 0 with prob 1 − qij, qij is different for each cell. Model 3: ǫijk(t) = eijk(t) + cijkK, if t ≥ Tijk and ǫijk(t) = eijk(t), if t < Tijk, where Tijk ∼ U(0, 1).

0.0 0.2 0.4 0.6 0.8 1.0 −5 5 10 15 20 25 t y

Model 1

0.0 0.2 0.4 0.6 0.8 1.0 −5 5 10 15 20 25 t y

Model 2

0.0 0.2 0.4 0.6 0.8 1.0 −5 5 10 15 20 25 t y

Model 3

SLIDE 18

Simulations: Model 1

0.0 0.2 0.4 0.6 0.8 1.0 −2 2 4 6 8 t Grand Effect

Model 1 (Median)

0.0 0.2 0.4 0.6 0.8 1.0 −3 −2 −1 1 2 t Row Effect 1

Model 1 (Median)

0.0 0.2 0.4 0.6 0.8 1.0 −2 −1 1 2 3 t Row Effect 2

Model 1 (Median)

0.0 0.2 0.4 0.6 0.8 1.0 −2 2 4 6 8 t Grand Effect

Model 1 (Mean)

0.0 0.2 0.4 0.6 0.8 1.0 −3 −2 −1 1 2 t Row Effect 1

Model 1 (Mean)

0.0 0.2 0.4 0.6 0.8 1.0 −3 −2 −1 1 2 3 t Row Effect 2

Model 1 (Mean)

SLIDE 19

Simulations: Model 1

0.0 0.2 0.4 0.6 0.8 1.0 −10 −8 −6 −4 −2 2 t Column Effect 1

Model 1 (Median)

0.0 0.2 0.4 0.6 0.8 1.0 −6 −4 −2 2 4 6 t Column Effect 2

Model 1 (Median)

0.0 0.2 0.4 0.6 0.8 1.0 −2 2 4 6 8 10 t Column Effect 3

Model 1 (Median)

0.0 0.2 0.4 0.6 0.8 1.0 −10 −8 −6 −4 −2 2 t Column Effect 1

Model 1 (Mean)

0.0 0.2 0.4 0.6 0.8 1.0 −6 −4 −2 2 4 6 t Column Effect 2

Model 1 (Mean)

0.0 0.2 0.4 0.6 0.8 1.0 −2 2 4 6 8 10 t Column Effect 3

Model 1 (Mean)

SLIDE 20

Simulations: Model 2

0.0 0.2 0.4 0.6 0.8 1.0 −2 2 4 6 8 t Grand Effect

Model 2 (Median)

0.0 0.2 0.4 0.6 0.8 1.0 −3 −2 −1 1 2 t Row Effect 1

Model 2 (Median)

0.0 0.2 0.4 0.6 0.8 1.0 −2 −1 1 2 3 t Row Effect 2

Model 2 (Median)

0.0 0.2 0.4 0.6 0.8 1.0 −2 2 4 6 8 t Grand Effect

Model 2 (Mean)

0.0 0.2 0.4 0.6 0.8 1.0 −3 −2 −1 1 2 t Row Effect 1

Model 2 (Mean)

0.0 0.2 0.4 0.6 0.8 1.0 −3 −2 −1 1 2 3 t Row Effect 2

Model 2 (Mean)

SLIDE 21

Simulations: Model 2

0.0 0.2 0.4 0.6 0.8 1.0 −10 −8 −6 −4 −2 2 t Column Effect 1

Model 2 (Median)

0.0 0.2 0.4 0.6 0.8 1.0 −6 −4 −2 2 4 6 t Column Effect 2

Model 2 (Median)

0.0 0.2 0.4 0.6 0.8 1.0 −2 2 4 6 8 10 t Column Effect 3

Model 2 (Median)

0.0 0.2 0.4 0.6 0.8 1.0 −10 −8 −6 −4 −2 2 t Column Effect 1

Model 2 (Mean)

0.0 0.2 0.4 0.6 0.8 1.0 −6 −4 −2 2 4 6 t Column Effect 2

Model 2 (Mean)

0.0 0.2 0.4 0.6 0.8 1.0 −2 2 4 6 8 10 t Column Effect 3

Model 2 (Mean)

SLIDE 22

Simulations: Model 3

0.0 0.2 0.4 0.6 0.8 1.0 −2 2 4 6 8 t Grand Effect

Model 3 (Median)

0.0 0.2 0.4 0.6 0.8 1.0 −3 −2 −1 1 2 t Row Effect 1

Model 3 (Median)

0.0 0.2 0.4 0.6 0.8 1.0 −2 −1 1 2 3 t Row Effect 2

Model 3 (Median)

0.0 0.2 0.4 0.6 0.8 1.0 −2 2 4 6 8 t Grand Effect

Model 3 (Mean)

0.0 0.2 0.4 0.6 0.8 1.0 −3 −2 −1 1 2 t Row Effect 1

Model 3 (Mean)

0.0 0.2 0.4 0.6 0.8 1.0 −2 −1 1 2 3 t Row Effect 2

Model 3 (Mean)

SLIDE 23

Simulations: Model 3

0.0 0.2 0.4 0.6 0.8 1.0 −10 −8 −6 −4 −2 2 t Column Effect 1

Model 3 (Median)

0.0 0.2 0.4 0.6 0.8 1.0 −6 −4 −2 2 4 6 t Column Effect 2

Model 3 (Median)

0.0 0.2 0.4 0.6 0.8 1.0 −2 2 4 6 8 10 t Column Effect 3

Model 3 (Median)

0.0 0.2 0.4 0.6 0.8 1.0 −10 −8 −6 −4 −2 2 t Column Effect 1

Model 3 (Mean)

0.0 0.2 0.4 0.6 0.8 1.0 −6 −4 −2 2 4 6 t Column Effect 2

Model 3 (Mean)

0.0 0.2 0.4 0.6 0.8 1.0 −2 2 4 6 8 10 t Column Effect 3

Model 3 (Mean)

SLIDE 24

Spatio-Temporal Precipitation Data

Spatio-temporal precipitation data: annual total precipitation data for U.S. from 1895 to 1997 at 11,918 weather stations. Nine climatic regions for precipitation defined by National Climatic Data Center. Several areas of outliers detected by Sun and Genton (2011, 2012).

+ + + + + + + + ++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ++ + + + ++ + + + + + + + + + + + + + + + ++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ++ + + + + + + + ++ + + + + + + + + + + + + + ++

NW W WNC S SW ENC C SE NE

SLIDE 25

Functional Boxplots for Nine Climatic Regions

North East (0.21% Outliers) Precipitation (mm/year)

1895 1920 1945 1970 1995 1000 2000 3000 4000

East North Central (0% Outliers)

1895 1920 1945 1970 1995 1000 2000 3000 4000

Central (0.25% Outliers)

1895 1920 1945 1970 1995 1000 2000 3000 4000

South East (2.52% Outliers) Precipitation (mm/year)

1895 1920 1945 1970 1995 1000 2000 3000 4000

West North Central (2.04% Outliers)

1895 1920 1945 1970 1995 1000 2000 3000 4000

South (0% Outliers)

1895 1920 1945 1970 1995 1000 2000 3000 4000

South West (3.03% Outliers) Year Precipitation (mm/year)

1895 1920 1945 1970 1995 1000 2000 3000 4000

North West (2.13% Outliers) Year

1895 1920 1945 1970 1995 1000 2000 3000 4000

West (4.04% Outliers) Year

1895 1920 1945 1970 1995 1000 2000 3000 4000

SLIDE 26

Climatic Region Effects

Year Grand Effect 1895 1920 1945 1970 1995 750 800 850 900

Weather Station (Median)

Year Grand Effect 1895 1920 1945 1970 1995 750 800 850 900

Weather Station (Mean)

20 40 60 80 100 −1000 −500 500 1000 Year Region Effect

Weather Station (Median)

North West West West North Central South South West East North Central Central South East North East 20 40 60 80 100 −1000 −500 500 1000 Year Region Effect

Weather Station (Mean)

North West West West North Central South South West East North Central Central South East North East

SLIDE 27

Weather Station and GCM Effects

Year Grand Effect

Weather Station and GCM (Median)

1895 1920 1945 1970 1995 600 700 800 900 1000 Year Grand Effect

Weather Station and GCM (Mean)

1895 1920 1945 1970 1995 600 700 800 900 1000 Year Data Source Effect

Weather Station and GCM (Median)

1895 1920 1945 1970 1995 −200 −100 100 200 Weather Station GCM Year Data Source Effect

Weather Station and GCM (Mean)

1895 1920 1945 1970 1995 −200 −100 100 200 Weather Station GCM

SLIDE 28

UK and Ireland Temperatures

Regional Climate Model (RCM): has higher resolution, covers a limited area of the globe. The boundaries of RCM are driven by variables output from a global climate model (GCM). Question: how much variability in the model output is from RCM and how much is due to the boundary conditions from GCM. Functional ANOVA: Kaufman and Sain (2010) proposed a mean-based Bayesian framework for spatial data. Data: PRUDENCE project (Christensen, Carter, and Giorgi 2002), consists of control runs (1961-1990). Two factors: RCM (HIRHAM and RCAO), GCM (ECHAM4 and HadAm3H).

SLIDE 29

Grand Effect and Residuals

−10 −8 −6 −4 −2 2 50 52 54 56 58 Lon Lat 11 12 13 14 15 16 17

Control Total (Median)

−10 −8 −6 −4 −2 2 50 52 54 56 58 Lon Lat 11 12 13 14 15 16 17

Control Total (Mean)

−10 −8 −6 −4 −2 2 50 52 54 56 58 Lon Lat −0.4 −0.2 0.0 0.2 0.4

Residuals (Median)

−10 −8 −6 −4 −2 2 50 52 54 56 58 Lon Lat −0.4 −0.2 0.0 0.2 0.4

Residuals (Mean)

SLIDE 30

GCM and RCM Effects

−10 −8 −6 −4 −2 2 50 52 54 56 58 Lon Lat −1.0 −0.5 0.0 0.5 1.0

Control GCM: ECHAM4 (Median)

−10 −8 −6 −4 −2 2 50 52 54 56 58 Lon Lat −1.0 −0.5 0.0 0.5 1.0

Control GCM: HadAM3H (Median)

−10 −8 −6 −4 −2 2 50 52 54 56 58 Lon Lat −0.6 −0.4 −0.2 0.0 0.2 0.4 0.6

Control RCM: HIRHAM (Median)

−10 −8 −6 −4 −2 2 50 52 54 56 58 Lon Lat −0.6 −0.4 −0.2 0.0 0.2 0.4 0.6

Control RCM: RCAO (Median)

SLIDE 31

Functional Median Polish Motivation Univariate ANOVA Functional ANOVA Simulation Studies Applications Discussion

Discussion

Functional Median Polish: robust functional ANOVA fitted by functional median. Band depth: graph-based nonparametric ordering for functional/image data (e.g. median image). The functional median polish algorithm does not guarantee to yield the least L1-norm residuals. Fink (1988) proposed a rather complex modification of the classical procedure that converges to the least L1-norm residuals.

Marc G. Genton Functional Median Polish, with Climate Applications