Fast and Accurate Inference for the Smoothing Parameter in - - PowerPoint PPT Presentation

▶
fast and accurate inference for the smoothing parameter
SMART_READER_LITE
LIVE PREVIEW

Fast and Accurate Inference for the Smoothing Parameter in - - PowerPoint PPT Presentation

Nov 14, 2022 201 likes •410 views

Fast and Accurate Inference for the Smoothing Parameter in Semiparametric Models Alex Trindade Dept. of Mathematics & Statistics, Texas Tech University Joint work with Rob Paige , Missouri University of Science and Technology Funded in part by

slide-1
SLIDE 1

Fast and Accurate Inference for the Smoothing Parameter in Semiparametric Models

Alex Trindade

  • Dept. of Mathematics & Statistics, Texas Tech University

Joint work with Rob Paige, Missouri University of Science and Technology Funded in part by the National Security Agency Grants: H98230-09-1-0071 (Paige) & H98230-08-1-0071 (Trindade)

May 2011

alex.trindade@ttu.edu (Dept. of Mathematics & Statistics, Texas Tech University Joint work with Rob SPBB Inference for Splines May 2011 1 / 19

slide-2
SLIDE 2

Outline

1

Motivation The need for a semiparametric smoothing approach... Penalized spline models Penalized splines as linear mixed models (LMMs)

2

Main Result: Inference on Smoothing Parameter Estimators as roots of quadratic estimating equations (QEEs) Saddlepoint-based bootstrap (SPBB) inference for QEEs Exact ML & REML inference in LMMs

3

Simulations: Confidence Intervals Coverages, lengths, compute times

4

Application: The Fossil Data

alex.trindade@ttu.edu (Dept. of Mathematics & Statistics, Texas Tech University Joint work with Rob SPBB Inference for Splines May 2011 2 / 19

slide-3
SLIDE 3

The LIDAR Data (Ruppert, Wand, & Carroll, 2003)

Model: y = µ(x) + error. Goal: estimate mean function µ(x), i.e. smooth data.

  • 400

450 500 550 600 650 700 −0.8 −0.6 −0.4 −0.2 0.0 range log ratio

alex.trindade@ttu.edu (Dept. of Mathematics & Statistics, Texas Tech University Joint work with Rob SPBB Inference for Splines May 2011 3 / 19

slide-4
SLIDE 4

The Fossil Data (Ruppert, Wand, & Carroll, 2003)

95 100 105 110 115 120 0.70720 0.70730 0.70740 0.70750 age (millions of years) strontium ratio

  • ●
  • ●
  • bvious dip!

dip here?

  • alex.trindade@ttu.edu

(Dept. of Mathematics & Statistics, Texas Tech University Joint work with Rob SPBB Inference for Splines May 2011 4 / 19

slide-5
SLIDE 5

Penalized Spline Model (degree p, with K knots)

µ(x) = β0 + β1x + · · · + βpxp +

K

∑

k=1

uk(x − κk)p

+

For n obs (xi, yi), write in matrix form: µ = X β + Zu ≡ Bθ. Model can allow for autocorrelation, R, in residuals (e.g. time series). Estimate θ by minimizing ˆ θPS = arg min

θ

(y − Bθ)′R−1(y − Bθ) + αu′u

  • α is a smoothing parameter controlling balance between:

fidelity to data (α = 0) smoothness of fit (α = ∞)

  • 400

450 500 550 600 650 700 −0.8 −0.6 −0.4 −0.2 0.0 LIDAR: linear spline fits with max and min smoothing (24 knots) range log ratio α = = 0 α = = ∞

alex.trindade@ttu.edu (Dept. of Mathematics & Statistics, Texas Tech University Joint work with Rob SPBB Inference for Splines May 2011 5 / 19

slide-6
SLIDE 6

Linear Mixed Model (LMM) Formulation & BLUP’s

Penalized spline can be recast as LMM with one variance component (Brumback, Ruppert, & Wand, 1999) y = X β

  • fixed effects

+ Zu + ε

random effects

BLUP of y in this context is ˜ y = B ˜ θ, where ˜ θ = arg min

θ

  • (y − Bθ)′R−1(y − Bθ) + σ2

ε

σ2

u

u′u

  • .

Implies BLUP-optimal value for α is: α = σ2

ε /σ2 u

alex.trindade@ttu.edu (Dept. of Mathematics & Statistics, Texas Tech University Joint work with Rob SPBB Inference for Splines May 2011 6 / 19

slide-7
SLIDE 7

Estimation of Smoothing Parameter

Since α is ratio of variance components in LMM, many parametric methods available. Also have several nonparametric methods.

Examples (Parametric)

Maximum Likelihood (ML) REstricted Maximum Likelihood (REML)

Examples (Nonparametric)

Akaike’s Information Criterion (AIC) Generalized Cross-Validation (GCV)

alex.trindade@ttu.edu (Dept. of Mathematics & Statistics, Texas Tech University Joint work with Rob SPBB Inference for Splines May 2011 7 / 19

slide-8
SLIDE 8

A Unified View of Smoothing Parameter Estimators (New)

Above estimators can be viewed as roots of a quadratic estimating equation (QEE) in normal random variables Q(α) = y′Aαy The n × n matrix Aα has a (complicated, but) closed form expression in each case... Theorem (Paige & Trindade, 2010): REML QEE is unbiased. Krivobokova & Kauermann (2007): REML less sensitive to misspecification of residual correlation than AIC or GCV.

alex.trindade@ttu.edu (Dept. of Mathematics & Statistics, Texas Tech University Joint work with Rob SPBB Inference for Splines May 2011 8 / 19

slide-9
SLIDE 9

Saddlepoint-Based Bootstrap (SPBB) Inference for QEEs

Pioneered by Paige, Trindade, & Fernando (2009): Relate distribution of root of QEE to that of estimator. Under normality have closed form for MGF of QEE. Use to saddlepoint approximate distribution of estimator. Now invert distribution to get CI... numerically! Leads to 2nd order accurate CIs: coverage is O(n−1). Works for: ML, REML, AIC, GCV, etc.!

alex.trindade@ttu.edu (Dept. of Mathematics & Statistics, Texas Tech University Joint work with Rob SPBB Inference for Splines May 2011 9 / 19

slide-10
SLIDE 10

SPBB: An Approximate Parametric Bootstrap

Fˆ

α(ˆ

αobs) Q(α) = 0 ˆ α solves Fˆ

α(ˆ

αobs) = FQ(ˆ

αobs)(0)

Q(α) monotone ˆ FQ(ˆ

αobs)(0)

saddlepoint approx via MGF of Q(α) (αL, αU) pivot Intractable! (And bootstrap too expensive...)

alex.trindade@ttu.edu (Dept. of Mathematics & Statistics, Texas Tech University Joint work with Rob SPBB Inference for Splines May 2011 10 / 19

slide-11
SLIDE 11

Exact ML & REML Inference for α

Exact finite sample inference for α = σ2

ε /σ2 u in LMMs with one variance

component (Crainiceanu, Ruppert, Claeskens, & Wand, 2005): Note: asymptotic χ2 dist is poor approx in finite samples due to substantial point mass at 0 (Crainiceanu & Ruppert, 2004). Invert (restricted) likelihood ratio test. Grid search needed to locate endpoints of CI (αL, αU). Only works for ML & REML...

alex.trindade@ttu.edu (Dept. of Mathematics & Statistics, Texas Tech University Joint work with Rob SPBB Inference for Splines May 2011 11 / 19

slide-12
SLIDE 12

Simulations: Mimic Extensive Study of Lee (2003)

Simulate datasets of sample size n = 200 from curves y = f (x) + ε, ε ∼ IID N(0, σ2

ε )

Vary 3 factors:

noise level (σ2

ε );

design density (number of x’s); spatial variation (type of curve).

Each factor at 3 levels (j = 1, 3.5, 6). Each scenario (factor-level combo) replicated 200 times. REML-Fit linear penalized spline: O-spline basis with 35 knots placed at empirical quantiles of x ∈ (0, 1) (Wand & Ormerod, 2008).

alex.trindade@ttu.edu (Dept. of Mathematics & Statistics, Texas Tech University Joint work with Rob SPBB Inference for Splines May 2011 12 / 19

slide-13
SLIDE 13

Results: Empirical Coverage of Nominal 95% CIs

Empirical Probabilities (Exact, SPBB) Scenario Level Underage Coverage Overage Noise Level j = 1 0.065 0.055 0.915 0.925 0.020 0.020 j = 3.5 0.035 0.025 0.950 0.945 0.015 0.030 j = 6 0.000 0.000 0.987 0.970 0.013 0.030 Design Density j = 1 0.040 0.040 0.945 0.935 0.015 0.025 j = 3.5 0.045 0.035 0.925 0.920 0.030 0.045 j = 6 0.040 0.040 0.945 0.945 0.015 0.015 Spatial Variation j = 1 0.000 0.000 0.934 0.970 0.066 0.030 j = 3.5 0.000 0.000 0.928 0.965 0.072 0.035 j = 6 0.000 0.000 0.883 0.960 0.117 0.040

alex.trindade@ttu.edu (Dept. of Mathematics & Statistics, Texas Tech University Joint work with Rob SPBB Inference for Splines May 2011 13 / 19

slide-14
SLIDE 14

CI Lengths: Trellis Boxplots of SPBB vs. Exact

Confidence Interval Lengths (degress of freedom of fit scale)

Exact SPBB

  • Scenario 1: Noise Level

Level: j=1

4 6 8

  • ●
  • Scenario 2: Design Density

Level: j=1

  • Scenario 3: Spatial Variation

Level: j=1

Exact SPBB

  • Scenario 1: Noise Level

Level: j=3.5

  • Scenario 2: Design Density

Level: j=3.5

  • Scenario 3: Spatial Variation

Level: j=3.5

Exact SPBB 4 6 8

  • Scenario 1: Noise Level

Level: j=6

  • ●
  • Scenario 2: Design Density

Level: j=6

4 6 8

  • Scenario 3: Spatial Variation

Level: j=6

alex.trindade@ttu.edu (Dept. of Mathematics & Statistics, Texas Tech University Joint work with Rob SPBB Inference for Splines May 2011 14 / 19

slide-15
SLIDE 15

Comparison: Exact, SPBB, and Bootstrap CIs

For the 200 simulated datasets with Noise Level factor at level j = 1

Method and Coverage Interval Length Statistics (minutes/CI) Probability Min Q1 Median Q3 Max SPBB-REML (15) 0.925 5.25 6.09 6.31 6.51 6.80 Exact-REML (105) 0.915 5.13 5.87 6.09 6.52 8.86 Bootstrap (2,100) 1.000 8.84 13.18 15.48 18.13 28.57

alex.trindade@ttu.edu (Dept. of Mathematics & Statistics, Texas Tech University Joint work with Rob SPBB Inference for Splines May 2011 15 / 19

slide-16
SLIDE 16

The Smoothed Fossil Data

Chaudhuri & Marron (1999): SiZer method to assess significance of small dip around 100 MY ago (NOT sig. at 95% level). Ruppert et al. (2003): fit penalized spline models with truncated polynomial bases with a variety of knots, degrees, and amounts of smoothing. Wand & Ormerod (2008): showcase “natural boundary” properties of O-splines; use judiciously chosen set of 20 interior knots. Our analysis: fit O-spline of Wand & Ormerod (2008); get 95% Exact-REML, SPBB-REML, and SPBB-GCV CIs.

alex.trindade@ttu.edu (Dept. of Mathematics & Statistics, Texas Tech University Joint work with Rob SPBB Inference for Splines May 2011 16 / 19

slide-17
SLIDE 17

Application: The Fossil Data

95 100 105 110 115 120 0.70720 0.70730 0.70740 0.70750 age (millions of years) strontium ratio

  • ●
  • ●
  • Point and 95% Interval Smoothing Estimates

REML & GCV point estimate (solid) Exact−REML lower bound (dash) Exact−REML upper bound (dot) SPBB−GCV lower bound (dot−dash) SPBB−GCV upper bound (long−dash)

alex.trindade@ttu.edu (Dept. of Mathematics & Statistics, Texas Tech University Joint work with Rob SPBB Inference for Splines May 2011 17 / 19

slide-18
SLIDE 18

Summary of SPBB Inference

Can be used under a variety of different criteria: ML, REML, GCV, and AIC. Performance: nearly exact. Computing:

1 order of magnitude faster than exact; 2 orders of magnitude faster than bootstrap.

Only computationally feasible alternative when no known exact or asymptotic methods exist, e.g. GCV and AIC. Smoothing parameter is tuning parameter; but can be used to uncover features in data...

alex.trindade@ttu.edu (Dept. of Mathematics & Statistics, Texas Tech University Joint work with Rob SPBB Inference for Splines May 2011 18 / 19

slide-19
SLIDE 19

Key References

Chaudhuri, P. & Marron J.S. (1999). SiZer for exploration of structures in

  • curves. J. Amer. Statist. Assoc. 94, 807-823.

Crainiceanu, C., Ruppert, D., Claeskens, G., and Wand, M. (2005), “Exact likelihood ratio tests for penalized splines”, Biometrika, 92, 91-103. Krivobokova, T., & Kauermann, G. (2007), “A Note on Penalized Spline Smoothing with Correlated Errors”, J. Amer. Statist. Assoc., 102, 1328-1337. Lee, T.C.M. (2003). Smoothing parameter selection for smoothing splines: a simulation study. Comp. Statist. Data Anal. 42, 139-148. Paige, R.L., Trindade, A.A. and Fernando, P.H. (2009), “Saddlepoint-based bootstrap inference for quadratic estimating equations”, Scand. J. Stat., 36, 98-111. Paige, R.L., & Trindade, A.A., “Fast and Accurate Inference for the Smoothing Parameter in Semiparametric Models”, Aust. & New

  • Zeal. J. Stat., (to appear).

Ruppert, D., Wand, M.P., & Carroll, R.J. (2003), Semiparametric Regression, London: Cambridge.

alex.trindade@ttu.edu (Dept. of Mathematics & Statistics, Texas Tech University Joint work with Rob SPBB Inference for Splines May 2011 19 / 19