Scalable gapfilling in spatio- temporal remote sensing data - - PowerPoint PPT Presentation

@ReinhardFurrer, I-Math/ICS, UZH JSM, 2019/07/30

NZZ.ch

Scalable gapfilling in spatio- temporal remote sensing data

2

Joint work with Florian Gerber

and with contributions of several others URPP Global Change and Biodiversity 175529

Global Change and Biodiversity

3

◮ Species abundance plot scale measurements from the International Tundra Experiment (ITEX) Shannon biodiversity index on site and plot scale ◮ MODIS/Landsat NDVI images and ASTER elevation data characterization of the landscape heterogeneity

Source: F. Gerber

Visual example

4

Source: Gerber et al (2018), TGRS

Visual example

5

Source: Gerber et al (2018), TGRS

Spatial statistics: prediction

6

Spatial process:

• Y (s) = Z(s) + ε(s), s ∈ D ⊂ Rd

. Predict Z(s0) given y(s1), . . . , y(sn). Minimize mean squared prediction error (over all linear unbiased predictors)

Spatial statistics: prediction

6

Spatial process:

• Y (s) = Z(s) + ε(s), s ∈ D ⊂ Rd

. Predict Z(s0) given y(s1), . . . , y(sn). Minimize mean squared prediction error (over all linear unbiased predictors)

• Best Linear Unbiased Predictor:

BLUP = Cov

• Z(spredict), Y (sobs)
• Var
• Y (sobs)

−1obs

• Z(s0) = cT Σ−1 y

(one spatial process, no trend, known covariance structure;

• therwise almost the same)

Issues of basic, classical kriging

7

• Z(s0) = cT Σ−1 y = Cov(pred, obs) · Var(obs)−1 · obs

◮ “Simple” spatial interpolation . . . . . . on paper or in class!

Issues of basic, classical kriging

7

• Z(s0) = cT Σ−1 y = Cov(pred, obs) · Var(obs)−1 · obs

◮ “Simple” spatial interpolation . . . . . . on paper or in class! ◮ BUT:

• 1. Complex mean structure
• 2. Unknown parameters
• 3. Non-stationary covariances
• 4. Large spatial fields
• 5. Space-time data on the sphere

Methods for large spatial datasets

8

◮ Sparse Covariance methods: — Spatial Partitioning Heaton — Covariance Tapering Furrer ◮ Sparse Precision methods: — Lattice Kriging Nychka — Multiresolution Approximations Katzfuss — Stochastic Partial Differential Equations Lindgren — Periodic Embedding Guinness — Nearest Neighbor Processes Datta ◮ Low rank approximation: — Fixed Rank Kriging Zammit-Mangion — Predictive Processes Finley ◮ Algorithmic approaches: — Local Approximate Gaussian Processes Gramacy — Metakriging Guhaniyogi

Arctic NDVI data

9

MODIS NDIV data (satellite product MOD13A1, NDVI = NIR − R NIR + R )

Kriging is smoothing

10

Kriging is smoothing

10

Interpolation using gapfill

11

gapfill: ranking of the images

12

r = 1 2 3 4 5 6 7 8 9 10 11 12

• 161

177 193 2004 2005 2006 2007

Day of the year Year

0.2 0.4 0.6 0.8

NDVI

low high

gapfill: quantile regression

13

Date: 193 doy 2004 177 doy 2006 177 doy 2005 193 doy 2006 193 doy 2005 Score: 0.65 0.71 0.77 0.88 0.91 Rank: 8 9 10 11 12 ˆ q: NA 0.64 NA 0.12 0.77

Interpolation using gapfill

14

145 161 177 193 2004 2005 2006 2007

Day of the year

0.2 0.4 0.6 0.8

NDVI

gapfill: prediction uncertainties

15

145 161 177 193 2004 2005 2006 2007

Day of the year Year

0.2 0.4 0.6 0.8

NDVI

145 161 177 193 2004 2005 2006 2007

Day of the year Year

0.2 0.4 0.6 0.8

interval length

data and predictions uncertainties

gapfill: location

16

gapfill: comparison

17 RMSE ×103

gapfill: uncertainties

18 (l) Uncertainty contribution from the indicated four steps of the gapfill procedure. (m) Average width of the 90% prediction intervals (40% missing values). (r) Average interval widths and coverage rate per day of the year.

Underlying statistical model

19

Z[a, s, x, y] is locally g[a, s] + Y [x, y] + ε[a, s, x, y] ◮ g[a, s] “arbitrary” ◮ Y (s) “some” process ◮ ε “some” noise

Visual example II

20

Source: Gerber et al (2018), TGRS

Visual example II

21

Source: Gerber et al (2018), TGRS

Visual example II

22

Add an overall mean field to model, similar as in IMA (interpolation

• f mean anomalies) approach Militino et al., TGRS (2019).

Summary

23

Implementation:

gapfill

Intuition: conceptual Model: distribution free Uncertainties: resampling type Practicality: competitive

Brute force kriging cTΣ−1y

24

Software to exploit the sparse structure spam for : ◮ an R package for sparse matrix algebra ◮ storage economical and fast ◮ versatile, intuitive and simple

See Furrer et al. (2006) JCGS; Furrer, Sain (2010) JSS

Brute force kriging cTΣ−1y

24

Software to exploit the sparse structure spam64 for : ◮ an R package for sparse matrix algebra ◮ storage economical and fast ◮ versatile, intuitive and simple

See Furrer et al. (2006) JCGS; Furrer, Sain (2010) JSS

◮ R objects have at most 231 elements (almost) ◮ R does not ‘have’ 64-bit integers: stored as doubles ◮ 64-bit exploitation consists of type conversions between front-end R and pre-compiled code

Gerber, M¨

• singer, Furrer (2017) CaGeo

Gerber, M¨

• singer, Furrer (2018) Software X

Summary

25

Implementation:

spam64 gapfill

Intuition: statistical conceptual Model: frequentist based distribution free Uncertainties: formal resampling type Practicality: play ground competitive

References (some, alphabetical)

26 Furrer Sain (2010) spam: A sparse matrix R package with emphasis on MCMC methods for Gaussian Markov random fields JSS 36 1–25 Furrer et al (2017) spam: Sparse Matrix algebra. R package version 2.2-0 Furrer et al (2017) spam64: 64-Bit Extension of the SPArse Matrix R Package ’spam’. R package version 2.2-0 Gerber Moesinger Furrer (2017) Extending R Packages to Support 64-bit Compiled Code: An Illustration with spam64 and GIMMS NDVI3g Data Comput Geosci 104 109-119 Gerber et al (2018) Predicting Missing Values in Spatio-Temporal Remote Sensing

• Data. IEEE TGRS 56(5) 2841-2853

Gerber Moesinger Furrer (2017) dotCall64: An Efficient Interface to Compiled C/C++ and Fortran Code Supporting Long Vectors SoftwareX 7 217-221 Heaton et al (2017) A Case Study Competition among Methods for Analyzing Large Spatial Data arXiv:1710.05013 Porcu Alegria Furrer (2018) Modeling Temporally Evolving and Spatially Globally Dependent Data International Statistical Review 86 344–377 Complete list at: www.math.uzh.ch/furrer/research/publications.shtml