Quadratic versus Linear Estimating Equations GLS estimating - - PowerPoint PPT Presentation

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

Quadratic versus Linear Estimating Equations

GLS estimating equations

n

• j=1

  fβj 2σ2g2

j

1/σ νθj

• 

 σ2g2

j

2σ4g4

j

−1 Yj − fj (Yj − fj)2 − σ2g2

j

• = 0.

Estimating equations for β are linear in Yj. Estimating equations for β only require specification of the first two moments. GLS is optimal among all linear estimating equations.

1 / 26 Quadratic versus Linear Estimating Equations

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

Gaussian ML estimating equations

n

• j=1

  fβj 2σ2g2

j νβj

2σ2g2

j

1/σ νθj

• 

 σ2g2

j

2σ4g4

j

−1 Yj − fj (Yj − fj)2 − σ2g2

j

• = 0.

Estimating equations for β are quadratic in Yj. Estimating equations for β require specification of the third and fourth moments as well. Specifically, if we let ǫj = Yj − f (xj, β) σg (β, θ, xj) , then we need to know E

• ǫ3

j

• = ζ∗

j

and var

• ǫ2

j

• = 2 + κ∗

j .

2 / 26 Quadratic versus Linear Estimating Equations

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

Questions If we know the true values ζ∗

j and κ∗ j , how much is ˆ

β improved using the quadratic estimating equations versus using the linear estimating equations? If we use working values (for example ζj = κj = 0, corresponding to normality) that are not the true values (i.e., ζ∗

j and κ∗ j ), is there any

improvement in using the quadratic estimating equations? If we use working variance functions that are not the true variance functions, is there any improvement in using the quadratic estimating equations?

3 / 26 Quadratic versus Linear Estimating Equations

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

General form of quadratic estimating equations:

n

• j=1

   fβ,j 2σ2g 2

j νβ,j

2σ2g 2

j

• 1

σ

νθ,j

• 

  σ2g 2

j

ζjσ3g 3

j

ζjσ3g 3

j

(2 + κj) σ4g 4

j

−1 ×

• Yj − fj

(Yj − fj)2 − σ2g 2

j

• = 0.

4 / 26 Quadratic versus Linear Estimating Equations

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

Large sample distribution for all parameters jointly: √n   ˆ β − β0 ˆ σ − σ0 ˆ θ − θ0  

L

− → N

• 0, A−1BA−1

. Here A = lim

n→∞

1 n

n

• j=1

DT

0,jV−1 0,j D0,j,

B = lim

n→∞

1 n

n

• j=1

DT

0,jV−1 0,j var (s0,j| xj) V−1 0,j D0,j,

5 / 26 Quadratic versus Linear Estimating Equations

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

Also Vj = σ2g 2

j

ζjσ3g 3

j

ζjσ3g 3

j

(2 + κj) σ4g 4

j

• ,

Dj =    fβ,j 2σ2g 2

j νβ,j

2σ2g 2

j

• 1

σ

νθ,j

• 

 

T

and V0,j and D0,j are evaluated at the true β0 and θ0.

6 / 26 Quadratic versus Linear Estimating Equations

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

Also var (s0,j| xj) =

• σ2g 2

j

ζ∗

j σ3g 3 j

ζ∗

j σ3g 3 j

• 2 + κ∗

j

• σ4g 4

j

• ,

the true variance matrix. Note that if the working values for ζj and κj are the same as the true values, var (s0,j| xj) = V0,j, so B = A, and the large sample distribution simplifies to √n   ˆ β − β0 ˆ σ − σ0 ˆ θ − θ0  

L

− → N

• 0, A−1

.

7 / 26 Quadratic versus Linear Estimating Equations

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

To deduce the limiting distribution of n1/2(ˆ β − β0), it would be necessary to carry out the indicated matrix inversion and multiplications and extract the upper left p × p submatrix of the result. It is possible to show that, just as GLS is optimal among linear estimating equations, ˆ β (as well as ˆ σ and ˆ θ) are optimal among quadratic estimating equations, provided the working values for ζj and κj are the true values. Next we consider a special case to gain better ideas of comparison between linear and quadratic estimating equations.

8 / 26 Quadratic versus Linear Estimating Equations

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

If we take E

• ǫ3

j

• = 0

and var

• ǫ2

j

• = 2 + κ

for all j, while in truth E

• ǫ3

j

• = ζ∗

and var

• ǫ2

j

• = 2 + κ∗

for all j then we have n1/2 ˆ β − β0

• L

− → N

• 0, σ2

0Γ−1∆Γ−1

.

9 / 26 Quadratic versus Linear Estimating Equations

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

Here

Γ = lim

n→∞ Γn

= lim

n→∞ n−1

• XT WX + 4σ2

2 + κ RT PR

• = Σ−1

WLS + 4σ2

2 + κ Σβ

and

∆ = lim

n→∞ ∆n

= lim

n→∞

• XT WX + 4σ2

0(2 + κ∗)

(2 + κ2) RT PR + 2σ0ξ∗ 2 + κ

• XT W1/2PR + RT PW1/2X
• = Σ−1

WLS + 4σ2 0(2 + κ∗)

(2 + κ)2 Σβ + 2σ0ζ∗ 2 + κ

• Tβ + TT

β

• 10 / 26

Quadratic versus Linear Estimating Equations

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

and R =    νT

β01

. . . νT

β0n

  

n×p

Q =    τ T

θ01

. . . τ T

θ0n

  

n×(q+1)

and P = I − Q(QTQ)−1QT.

11 / 26 Quadratic versus Linear Estimating Equations

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

Recall that, if the first two moments are correctly specified, then √n

• ˆ

βGLS − β0

• L

− → N

• 0, σ2

0ΣWLS

• .

First we note that the properties of ˆ βGLS do not depend on those of ˆ σ and ˆ θ, whereas the properties of ˆ βML do. Next we compare ˆ β from the linear and quadratic equations in various scenarios under this special case.

12 / 26 Quadratic versus Linear Estimating Equations

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

When the data are really normal That is, we choose ζ = 0 and κ = 0 while ζ∗ = 0 and κ∗ = 0. Then Γ = Σ−1

WLS + 2σ2 0Σβ = Σ−1 ML

and ∆ = Σ−1

ML.

So √n

• ˆ

βML − β0

• L

− → N

• 0, σ2

0ΣML

• .

We can show that ΣGLS − ΣML is nonnegative definite, as ΣML = (Σ−1

WLS + 2σ2 0Σβ)−1

= ΣWLS − ΣWLS

• ΣWLS + σ−2

0 /2Σ−1 β

−1 ΣWLS.

13 / 26 Quadratic versus Linear Estimating Equations

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

That is, ˆ βML is more efficient compared to ˆ βGLS when the data truly are normal and we use the normal theory ML estimating equations for β. The source of improvement is from Σβ, which arises from taking advantage of the additional information β available in the variance function g(·).

14 / 26 Quadratic versus Linear Estimating Equations

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

When the data are only symmetrically distributed That is, we choose ζ = 0 and κ = 0 while ζ∗ = 0 and κ∗ > 0. Then Γ = Σ−1

WLS + 2σ2 0Σβ

and ∆ = Σ−1

WLS + (2 + κ)σ2 0Σβ.

√n

• ˆ

βML − β0

• L

− → N

• 0, σ2

0ΣQ

• ,

with ΣQ = Γ−1∆Γ We can show that ΣGLS − ΣQ is nonnegative definite, if κ∗ ≤ 2. That is, the optimality of ˆ βML no longer applies uniformly, when the data are only symmetric and we use the normal theory ML estimating equations for β.

15 / 26 Quadratic versus Linear Estimating Equations

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

When the data are symmetrically distributed ...and we correctly specify both ζ and κ. That is, we choose ζ = ζ∗ = 0 and κ = κ∗. Then Γ = Σ−1

WLS +

4σ2 2 + κ∗Σβ and ∆ = Σ−1

WLS +

4σ2 2 + κ∗Σβ and √n

• ˆ

βML − β0

• L

− → N

• 0, σ2

0ΣC

• ,

with ΣC =

• Σ−1

WLS +

4σ2 2 + κ∗Σβ −1

16 / 26 Quadratic versus Linear Estimating Equations

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

We can show that ΣGLS − ΣC is nonnegative definite. That is, ˆ βML is more efficient compared to ˆ βGLS when we know the data are symmetric and we are able to specify correctly a value for the excess kurtosis.

17 / 26 Quadratic versus Linear Estimating Equations

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

When the variance function g(·) does not depend on β In this case, Σβ = 0 and Tβ = 0. Then Γ = ∆ = Σ−1

WLS

and √n

• ˆ

βML − β0

• L

− → N

• 0, σ2

0ΣWLS

• .

That is, there is nothing to be gained by using a quadratic estimating equation over a linear one, because there is no additional information

• n β to be gained from g(·).

18 / 26 Quadratic versus Linear Estimating Equations

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

In general The large sample properties of the quadratic estimator depend on the assumed and true third and fourth moments of the data. Those of the GLS estimator do not, and are unchanged regardless of the nature of the true third and fourth moments. If the third and fourth moments are correctly specified, the linear estimator is inefficient relative to the quadratic estimator for β. If these are not correctly specified, it is no longer clear that one estimator dominates the other in terms of efficiency. Intuitively, because the performance of the quadratic estimator depends on third and fourth moment properties, it would seem to be sensitive to incorrect assumptions about them, whereas the performance of the GLS estimator does not depend on these moments at all. — We will study this next.

19 / 26 Quadratic versus Linear Estimating Equations

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

Sensitivity analysis of linear and quadratic equations to misspecification of third and fourth moments Consider an example and numerical analysis: True model: E(Yj) = β0, var(Yj) = σ2

0β2 0,

Working model: E(Yj) = β, var(Yj) = σ2β2. We can obtain: ˆ βGLS = ¯ Y

p

− → β0 ˆ βML = ( ¯ Y 2 + 4σ2 n

j=1 Y 2 j /n)1/2 − ¯

Y 2σ2

p

− → β0 as well as the explicit forms of ΣWLS and ΣML.

20 / 26 Quadratic versus Linear Estimating Equations

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

Table: ARE of ML to GLS for the simple model

True Distribution κ0 ζ0 σ0 ARE Normal 0.20 1.08 0.30 1.18 1.00 3.00 Symmetric 2 0.20 1.01 (ζ0 = 0) 2 0.30 1.02 2 1.00 1.80 4 0.20 0.94 4 0.30 0.90 4 1.00 1.29 6 0.20 0.88 6 0.30 0.81 6 1.00 1.00 8 0.20 0.83 8 0.30 0.73 8 1.00 0.82 Gamma 0.24 0.40 0.20 0.93 (ζ0 = 2σ0, κ0 = 6σ2

0)

0.54 0.60 0.30 0.88 0.96 0.80 0.40 0.82 6.00 2.00 1.00 0.69

21 / 26 Quadratic versus Linear Estimating Equations

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

If the data are truly normally distributed The quadratic estimator is uniformly more precise than the GLS estimator, as expected. Also note for σ0’s that are relatively small (≤ 0.30), the gain in efficiency for ML is not substantial and decreases with decreasing σ0. So, for “high quality” data where the “signal” dominates the “noise” (small σ0), ML and GLS appear to exhibit similar performance; for “low quality” data, where the noise dominates the signal, we see that ML performs substantially better. This makes intuitive sense – as the ML estimator exploits information about β in the variance, when the variance is large, it seems likely that we would be able to gain more information about β than when the variance is of much smaller magnitude than the mean.

22 / 26 Quadratic versus Linear Estimating Equations

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

If the data come from a symmetric but “heavy-tailed” distribution The quadratic estimator, which assumes excess kurtosis is zero, is inefficient relative to GLS, except when σ0 gets very large. The inefficiency becomes worse as κ0 increases. This shows that there is no general ordering of the relative precision

• f GLS and normal theory ML in this case.

23 / 26 Quadratic versus Linear Estimating Equations

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

If the data come from a Gamma distribution Recall that the linear estimator ˆ βGLS is the maximum likelihood estimator for β under the gamma distribution; hence, we would expect that GLS is uniformly relatively more efficient, as seen in the table. In practice, it may be difficult to distinguish between normal and gamma distributions if σ0 is “small”. Thus, if we mistakenly assume normality when the data really arise from a gamma distribution, and use ˆ βML instead of ˆ βGLS, we stand to lose efficiency.

24 / 26 Quadratic versus Linear Estimating Equations

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

Sensitivity analysis of linear and quadratic equations to misspecification of variance function Consider an example and numerical analysis: True model: E(Yj) = β0, var(Yj) = σ2

0β2+2θ0

, Working model: E(Yj) = β, var(Yj) = σ2β2. We can obtain: ˆ βGLS = ¯ Y

p

− → β0 ˆ βML = ( ¯ Y 2 + 4σ2 n

j=1 Y 2 j /n)1/2 − ¯

Y 2σ2

p

− → β0

• (1 + 4σ2

0 + 4σ4 0β2θ0 0 )1/2 − 1

2σ2

• .

25 / 26 Quadratic versus Linear Estimating Equations

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response

Misspecification of the variance function g(·) can cause ˆ βML to be inconsistent. By contrast, misspecification of the variance function g(·) can cause ˆ βGLS to be inefficient, but still consistent. Bottom line Unless we have extensive information about third and fourth moments, or about the full conditional distribution of Y (say, normal), using GLS seems safer.

26 / 26 Quadratic versus Linear Estimating Equations