Variance Parameters Recall the general mean-variance specification - - PowerPoint PPT Presentation

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

Variance Parameters

Recall the general mean-variance specification E(Y |x) = f (x, β), var(Y |x) = σ2g(β, θ, x)2. To the first order approximation, the folklore theorem states that the asymptotic distribution of ˆ βGLS is unaffected by how θ is estimated. To the second order approximation, the asymptotic distribution of ˆ βGLS does depend on how well θ is estimated. Note that estimation of σ plays no role in the properties of the GLS estimator β.

1 / 17 Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

Transformed residuals Define ǫj = Yj − f (xj, β0) σ0g (β0, θ0, xj). Without further assumptions, E (ǫj| xj) = 0, var (ǫj| xj) = E

• ǫ2

j

• xj
• = 1.

We explore estimating σ based on |ǫj|λ for various λ.

2 / 17 Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

Recall key assumption The relevant moments of ǫj are not dependent on xj and are constant for all j: E(|ǫj|λ|xj) = E(|ǫj|λ) = constant ∀j, E(|ǫj|2λ|xj) = E(|ǫj|2λ) = constant ∀j. For λ = 2, the first requirement is automatically met, and similarly for λ = 1 the second requirement is automatically met.

3 / 17 Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

More general forms of estimating equations for θ Define η by eλη = σλE

• |ǫj|λ

. Identify |Yj − f (xj, β)|λ as the “response”. For λ = 2, η = log σ and η is simply a reparameterization; but for

• ther λ it depends on the distribution of ǫj:

η = log σ + 1 λ log

• E
• |ǫj|λ

4 / 17 Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

Then θ and η may be estimated by solving the joint estimating equations

n

• j=1

    

• Yj − f
• xj, ˆ

β

• λ

− eληg

• ˆ

β, θ, xj λ g

• ˆ

β, θ, xj 2λ      g

• ˆ

β, θ, xj λ τθ

• ˆ

β, θ, xj

• = 0

where ˆ β is held fixed. We shall study the large sample distribution of (ˆ θ

T, ˆ

η)T for different choices of λ.

5 / 17 Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

Consistency This is an unbiased M-estimating equation, as long as ˆ β is a consistent estimator for β0, which we will assume. Then ˆ θ and ˆ η are consistent estimators of θ0 and η0. Also note that applying the usual M-estimator argument to deduce the properties of (ˆ θ

T, ˆ

η)T requires the summand in the estimating equations to be differentiable with respect to θ, η, and ˆ β. This is not always true, e.g., when λ = 1.

6 / 17 Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

Asymptotic distribution for λ = 2 Complicated, depends on: whether ˆ β is based on linear or quadratic estimating equations; excess kurtosis. Simplifies if either σ0 → 0; g(·) does not depend on β. Then √n

• ˆ

θ − θ0

• L

− → N

• 0, 2 + κ

4 Λθ

• .

7 / 17 Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

Asymptotic distribution for λ = 1 Complicated, with an additional technical difficulty because of non-differentiability of |x| Simplifies if either σ0 → 0; g(·) does not depend on β, and ǫj| xj has a symmetric distribution. Then √n

• ˆ

θ − θ0

• L

− → N (0, c1Λθ) where c1 = var (|ǫj|| xj) E (|ǫj|| xj)2 .

8 / 17 Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

Asymptotic distribution for general λ Equally complicated, and same technical difficulty because of non-differentiability. Simplifies if ǫj| xj has a symmetric distribution with common absolute moments up to power 2λ, and either σ0 → 0; g(·) does not depend on β. Then √n

• ˆ

θ − θ0

• L

− → N (0, cλΛθ) .

9 / 17 Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

Here, cλ = var

• |ǫj|λ
• xj
• λ2E
• |ǫj|λ
• xj

2, λ = 0, = var

• log |ǫj|2
• xj
• 4

, λ = 0. So one can compare the asymptotic efficiency of the two competing methods by comparing cλ1 and cλ2.

10 / 17 Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

Asymptotic efficiency relative to λ = 2: c2/cλ Assume ǫj is N(0, 1) contaminated with fraction α of N(0, 9); i.e., (1 − α) × N(0, 1) + α × N(0, 9). α λ = 1 λ = 2

3

λ = 1

2

λ = 1

3

λ = 0 0.000 0.876 0.772 0.693 0.606 0.405 0.001 0.948 0.841 0.756 0.662 0.440 0.002 1.016 0.906 0.816 0.715 0.480 0.010 1.439 1.334 1.216 1.075 0.720 0.050 2.035 2.100 1.996 1.823 1.220 λ = 1 performs better than λ = 2 even for tiny levels of contamination (2 × 10−3, or 2 observations per thousand).

11 / 17 Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

Assume ǫj is t-distributed with ν degrees of freedom. To match the excess kurtosis in the contaminated normal table, use ν = ∞, 35.78, 20.16, 7.68, 5.29. ν λ = 1 λ = 2

3

λ = 1

2

λ = 1

3

λ = 0 ∞ 0.876 0.765 0.693 0.610 0.405 35.78 0.921 0.813 0.741 0.655 0.439 20.16 0.965 0.861 0.787 0.698 0.471 7.68 1.270 1.191 1.111 1.002 0.695 5.29 2.016 1.994 1.897 1.739 1.234 λ = 1 performs better than λ = 2 for ν < 16.

12 / 17 Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

Bottom line Asymptotic distributions of ˆ θ are complicated in general. The “small σ0” simplification is quite useful and also relevant in practice. Using λ = 1 has good relative efficiency and requires estimating only E (|ǫj|| xj), which is not difficult.

13 / 17 Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

In some circumstances, estimation of variance parameters is of critical importance. Two common such situations are prediction and calibration. Prediction Find Y given x ˆ Y0 = f (x0, ˆ β) Calibration Find x given Y ˆ x0 = f −1(Y0, ˆ β)

14 / 17 Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

Variance of prediction Y0 − ˆ Y0 = Y0 − f (x0, ˆ β) ≈ Y0 − f (x0, β0) − f T

β (x0, β0)

• ˆ

β − β0

• 15 / 17

Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

Since ˆ β depends only on the training data, it is independent of Y0, and thus var(Y0 − ˆ Y0) ≈ var{Y0 − f (x0, β0)} + fβ(x0, β0)Tvar(ˆ β − β0)fβ(x0, β0) = σ2

0g(x0, β0, θ0)2

+ n−1σ2

0fβ(x0, β0)Tvar{n1/2(ˆ

β − β0)}fβ(x0, β0) ≈ σ2

0g(x0, β0, θ0)2

+ σ2

0fβ(x0, β0)T{n−1 ˆ

Σ}fβ(x0, β0).

16 / 17 Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

The first term in the variance reflects the uncertainty due to variation in Y0, and regardless of how much data are collected, the inherent variation in the response will always be there. The second term reflects uncertainty due to fitting the model to the training data, and it diminishes as more data are collected and used to fit the model. The first term dominates the second term, as the second term is O(n−1). So the predominant source of error in prediction is that due to inherent variation in the response. One can do Wald type inference for prediction based on this formula. The result for calibration is similar.

17 / 17 Variance Parameters