Non-linear Least Squares and Durbins Problem Asymptotic Theory - - PowerPoint PPT Presentation
Non-linear Least Squares and Durbins Problem Asymptotic Theory Part V James J. Heckman University of Chicago Econ 312 This draft, April 18, 2006 This lecture consists of two parts: 1. Non-linear least squares: This looks at Non-linear
This lecture consists of two parts:
squares estimation in detail; and
ymptotic variances in the case of two stage estimators. 1
In this section, we examine in detail the Non-linear Least Squares estimator. The section is organized as follows:
the NLLS estimator (using the extremum principle);
. 2
Here we recap the derivation of the NLLS estimator as an ap- plication of the Extremum principle, from section 3.2 of the notes Asymptotic Theory II, with slight modification in no- tation. As noted there, we could also motivate NLLS as a moment estimator (refer section 3.2 of Asymptotic Theory II).
lowing model holds:
( | 0) + (1) = (; ) + [(; 0) (; )] + where is a vector of exogenous variables. Unlike in 3
the linear regression model, may not necessarily be of the same dimension as . Since (|) is a nonlinear function of and , (*) is called the nonlinear regression
so that: (; ) . Then we can write out a least square criterion function as below.
= ( (; ))2 = [(; 0) (; )]2 + 2
0 (true parameter value). If = 0 is the only such value, model is identified (wrt criterion). 4
Pick: () = 1
=1
( (; ))2 as analog to in the sample. As established in the OLS case in the notes Asymptotic Theory II (Section 3.2), we can show that plim = .
ˆ = argmin() Thus we chose ˆ to minimize (). In the next few sections, we establish consistency and asymp- totic normality for the NLLS estimator (under certain condi- tions), and discuss conditions for asymptotic eciency. 5
Assume:
) = 2 ;
0;
4. exists and is continuous in nbd of 0;
0 there exists 0 such that |(1) (2)| for 1 2 closer than (i.e. ||1 2|| ), for all 1 2 in nbd of 0 and for all ); 6
=1
(1)(2) converges uniformly in 1 2 in nbd of 0;
Then, we have that there exists a unique root b such that: b = arg min X ( ( | ))2; and that it is consistent, i.e.: b
Extremum Analogy Theorem for the class of estimators defined as b = arg min (). 7
Gallant (1975): Consider the NLLS model from (1) above: = ( | ) + Now expand in nbd of in Taylor series to get: = ( | ) + ( | ) ¯ ¯ ¯ ¯
( ) +
Rewrite the equation as: ( | ) + ( | ) ¯ ¯ ¯ ¯
= ( | )
8
Now by analogy with classical linear regression model, we have:
¯ ¯ ¯ ¯
is analogous to the
dependent variable in OLS.
trix in OLS. 9
The NLLS estimator is: b
X μ( | )
μ( | ) ¶ ¸1 (2) × μX ( | )
so that in comparison to the OLS estimator we have:
replaced by P
=1
à ˜
˜
; and
replaced by P
=1
μ
, where ˜ = (1 ) = (1 ). 10
Then analogy with OLS goes through exactly. Now, as for the OLS case, we can do hypothesis testing, etc., using derivatives in nbd of optimum. Using the analogy, we also obtain the estimator for Asy. var (b ) as: \
) = ˆ 2(˜
˜
)1 where ˜
= ( )
¯ ¯ ¯ˆ
To justify large sample normality, we need additional condi- tions on the model. The required conditions for asymptotic normality, assuming the conditions for consistency hold, are the following .
=1
¯ ¯ ¯
¯ ¯ ¯ = a positive definite matrix;
=1
3. 2 0 is continuous in in an open nbd of 0 uniformly ( · need uniform continuity of first and second partials); 12
2
=1
2 ¸ = 0 for all in an open nbd of 0; and
=1
(1) 2 ¯ ¯ ¯ ¯
2
converges to a finite matrix uniformly for all 1 2 in an open nbd of 0 Then:
0) (0 2
1)
where 2
= ().
Sketch of proof (For rigorous proof, see Amemiya, p.132-4).The intuition for this result is exactly as in Cramer’s Theorem (refer to Section 2 of notes Asymptotic Theory III). 13
Look at first order condition:
=1
( ())
get:
¯ ¯ ¯ = 1
()
¯ ¯ ¯ 14
This is asymptotically normal (i.i.d. r.v.) by Lindeberg-Levy Central Limit Theorem. Then using equation (2) we obtain:
0) = " 1
=1
μ( | )
¶#1
=1
μ( | )
() We get that this is asymptotically normal in nbd of 0, if £ 1
¢¤
trix (which is true by assumption). This completes the analogy with the Cramer’s theorem proved in earlier lecture. (See Amemiya for a rigorous derivation. Also, see the result in Gallant.) 15
Analogy of the NLLS estimator with is complete if we assume is normal. Then, we get the log likelihood function: ln $ = 2 ln 2 1 22
( ( | ))2 So that here we get b = b (FOC and asy. theory as before). 16
Thus we obtain the general result that any nonlinear regression model if we have that normal. Though the nonlinear regression is picking another criterion, the estimator is identical to the MLE estimator.
·
In general, Greene (p. 305-8) shows that (unless is normal) NLLS is not necessarily asymptotically ecient. 17
? The two commonly used methods are:
18
1.6.1 Newton-Raphson Method In the NLLS case, we wish to find a solution to the equation: ()
(all criteria in Asymptotic Theory handout III). We expand the criterion function () in nbd of an initial starting value b
approximation to get: () ' (b
¯ ¯ ¯
(b
2(b
0(b
This quadratic problem has a solution if Hessian matrix 2 is a definite matrix (pos. def. for min). 19
In equation (3), we min () wrt (by taking the FOC) and
b
2 ¸1
¯ ¯ ¯
We continue iteration until convergence occurs. The method assumes that we can approximate with a quadratic. Some the drawbacks of the method and possible fixes are discussed below. (A) Singular Hessian: There is a problem if the Hessian singular: the method fails as we are unable then to obtain 2 ¸1
20
In case the Hessian is singular, the following correction could be used : Use such that: 2 ¸
Usually we pick scalar (obviously can pick vectors). One can then fiddle with this to get out of nbd of lo- cal singularity. In applications of the Newton- Raphson method, one could use idea due to T.W. An- derson on reading list and note that asymptotically:
¶ = μ ln
¶ to arrive at an alternative estimator (sometimes called BHHH but method due to Anderson) for the Hessian. 21
(B) Algorithm Overshoots: In this case, one could scale the step back by : b
2 ¸1
¯ ¯ ¯
We choose 0 1 so that the iteration dierences get dampened, reducing chances of overshooting. 22
1.6.2 Gauss-Newton Method The motivation for the Gauss-Newton method mimics exactly the NLLS set up in section 1.3 where we drew the analogy with
get: ( | b
This set-up is analogous to OLS; the LHS and part of RHS are data once one knows (guesses) the starting value b
OLS, to get the next iteration in the algorithm: b
" 1
=1
1
=1
( | b
so that we get: b
" 1
=1
1
=1
( | b
( | b
i Revise, update, start all over again. This method has the same problems as in Newton-Raphson. (A) Singular Hessian: As in the Newton-Raphson method, to solve for optimum use:
¸ scalar. (B) Algorithm Overshoots: To avoid overshooting, use 24
Hartley modification. Form: 1 = " 1
=1
1
=1
( | b
( | b
i
(b
where () = P( (|))2. Update by setting b 2 = b 1 + 1. Then algorithm converges to a root of the equation. General Global convergence is a mess, un- resolved. 25
1.6.3 Eciency theorems for estimation methods Theorem 1 One Newton-Raphson Step toward an optimum is fully ecient if you start from an initial consistent estimator. This theorem suggests a strategy for quick convergence. Get cheap (low computational cost) estimator which is consistent but not ecient. Then iterate once — avoids computational
use unmodified Hessians (without corrections for overshooting
X
0)
It is consistent but not necessarily ecient. Now expand root 26
ln $
¯ ¯ ¯
¯ ¯ ¯ + 2 ln $ ¯ ¯ ¯ ¯
1 (b
b 1 does not necessarily set left hand side to zero. If it did, we would have an ecient estimator. As before 1 is intermediate value. 27
Now look at Newton-Raphson criterion. b
= (b
2 ln $ ¸1
ln $
= (b
2 ln $ ¸1
¯ ¯ ¯ + 2 ln $ ¯ ¯ ¯ ¯
(b
b
´ =
¸1
1
¯ ¯ ¯ + " 1
¸1
1
¯ ¯ ¯ ¯
1
#
b
´ 28
00
ln $
¯ ¯ ¯ + h£ 1
0000
¤
b
´i The second term vanishes as
29
Same result obviously holds for Gauss-Newton. One G-N step for a consistent estimator is fully ecient (or at least as ecient as NLLS). Thus starting from a consistent estimator (where possible) saves computer time, avoids problems of nonlinear optimiza- tions and also avoids local optimization problem (i.e., possibil- ity of arriving at an inconsistent local optima). 30
Durbin’s problem is question of arriving at the correct variance- covariance matrix for a set of parameters estimated in the sec-
For example, let 0 = (¯ 1 ¯ 2), where ¯ 1 ¯ 2 are “true values”, as in the case with the composite hypothesis considered in the earlier lecture (Asymptotic theory IV). 31
Suppose we use an initial consistent estimator for 2. Then if we treat likelihood as if 2 known (but it is estimated by ˜ 2), we have: ln $
¯ ¯ ¯
1 2
| {z }
=0
= 1
1 ¯ ¯ ¯ ¯ + 1
10
1
¯ ¯ ¯ ¯
b 1 1 ´ + 1
10
2
¯ ¯ ¯ ¯
˜ 2 2 ´ | {z }
“Durbin Problem”
We assume sample sizes the same in both samples. 32
which implies
b 1 ¯ 1 ´ = 1
11
1
1 ¯ ¯ ¯ ¯ 1
11 12
˜ 2 ¯ 2 ´
$ is from the likelihood with sample size used to produce ˜ 2. 33
Thus to obtain the right covariance matrix for ˆ 1, we need covariance between the two score vectors. We have:
2 ¯ 2) = Ã 1
$ 20
2
!1 Ã ˜ $ 2 ! = 1
22
à ˜ $ 2 ! which implies
1 ¯ 1) = 1
11
1
1 ¯ ¯ ¯ ¯ 1
11121 22
1
$ 2 ¯ ¯ ¯ ¯ ¯ 34
We need to compute covariance to get the right standard errors. Just form a new covariance matrix: (ˆ 1 ¯ 1) = 1
11 + 1 1112 ˜
1
22211 11
1
1112 ˜
22( ˆ 1 ˜ 2)1
11
1
11( ˆ
1 ˜ 2)˜ 22211
11
where (now we assume 2 dierent sample sizes): 35
1 = 1
1 ¯ ¯ ¯ ¯˜
1ˆ 2
2 = 1 p ˜
$ 2 ¯ ¯ ¯ ¯ ¯
2
is the sample size for samples used to get ˜ 2. In the independent sample case, we get that last two terms in (*) vanish, so that we get (ˆ 1 ¯ 1) = 1
11 + 1 1112 ˜
1
22211 11
36
This problem seems similar to, but is actually dierent from the Durbin problem. Here we have a log likelihood function which has two sets of parameters, ln $( ). In the first step here, we solve ln $ ( ) = 0 to get () We then optimize with respect to (). While this looks like the two-step estimator in Durbin’s prob- lem, here we are not using an estimate of , but rather using (). In fact here we can show that this is the same as joint maximization of . 37
Using the Envelope Theorem (i.e., utilizing the fact that () is arrived at through an optimization), we get: ln $( ())
ln $( ())
= 2 ln $( ()) +
= 2 ln $ + () 2 ln $ 0 = 0 =
2 ln $
¶1 38
Substituting into previous expression, we get: plim 1
= ( 1
)
= The asymptotic distribution is the same for if we esti- mate jointly or through the concentrated likelihood approach. (Refer to Asymptotic Theory — Lecture IV, section on compos- ite hypothesis for the distribution of sub-vector of parameters when estimation is done jointly.) 39