Statistics Point Estimation Shiu-Sheng Chen Department of Economics National Taiwan University Fall 2019 Shiu-Sheng Chen (NTU Econ) Statistics Fall 2019 1 / 38
Estimation Section 1 Estimation Shiu-Sheng Chen (NTU Econ) Statistics Fall 2019 2 / 38
Estimation Statistical Inference Given a random sample with sample size n { X 1 , X 2 , . . . , X n } ∼ i . i . d . f ( x ; θ ) where θ is an unknown population parameter. For example, suppose we are interested in θ , which is the (unknown) population proportion of NTU students, who have a significant other. { X 1 , X 2 , . . . , X n } ∼ i . i . d . Bernoulli ( θ ) Shiu-Sheng Chen (NTU Econ) Statistics Fall 2019 3 / 38
Estimation Statistical Inference We would like to find a good guess of the parameter θ : a point estimator. That is, we would like to come up with a random variable θ = δ ( X 1 , X 2 , . . . , X n ) ˆ that we expect to be close to θ . Shiu-Sheng Chen (NTU Econ) Statistics Fall 2019 4 / 38
Estimation Estimator Definition Let { X 1 , X 2 , . . . , X n } be a random sample from the joint distribution indexed by a parameter θ ∈ Θ . A function ˆ θ = δ ( X 1 , X 2 , . . . , X n ) is called a point estimator of the parameter θ . Θ is called a parameter space. When X 1 = x 1 , X 2 = x 2 ,..., X n = x n are observed , then δ ( x 1 , x 2 , . . . x n ) is called the point estimate of θ . Every estimator is also a statistic (by nature of being a function of a random sample). Shiu-Sheng Chen (NTU Econ) Statistics Fall 2019 5 / 38
Estimation Estimator There can be more than one unknown parameter: { X i } n i = 1 ∼ i . i . d . f ( x , θ 1 , θ 2 , . . . , θ k ) For example, ( θ 1 , θ 2 ) = ( µ , σ 2 ) denotes the (unknown) population mean and variance of the S&P500 stock returns. { X 1 , X 2 , . . . , X n } ∼ i . i . d . N ( µ , σ 2 ) Estimators: θ 1 = ˆ µ = δ 1 ( X 1 , X 2 , . . . , X n ) ˆ σ 2 = δ 2 ( X 1 , X 2 , . . . , X n ) θ 2 = ˆ ˆ Shiu-Sheng Chen (NTU Econ) Statistics Fall 2019 6 / 38
Estimation How to Guess? Analogy Principle ( 類 比 原則 ) Method of Moments ( 動 差 法 ) Method of Maximum Likelihood ( 最 大 概 似 法 ) Shiu-Sheng Chen (NTU Econ) Statistics Fall 2019 7 / 38
Estimation Analogy Principle To estimate the population mean µ = E ( X ) , use the sample mean X n = 1 ∑ n ¯ X i n i = 1 To estimate the population variance σ 2 = Var ( X ) = E ( X − E ( X )) 2 , use the sample variance σ 2 = 1 ( X i − ¯ X n ) 2 or S 2 n = ( X i − ¯ X n ) 2 ∑ n ∑ n 1 n − 1 ˆ n i = 1 i = 1 In general, to estimate the population moments m j = E ( X r ) , use the sample moments n ∑ 1 X j i n i = 1 Shiu-Sheng Chen (NTU Econ) Statistics Fall 2019 8 / 38
Estimation Analogy Principle To estimate distribution function F X ( x ) = P ( X ≤ x ) , use the empirical distribution function. F n ( x ) = ∑ n i = 1 I { X i ≤ x } ˆ n Shiu-Sheng Chen (NTU Econ) Statistics Fall 2019 9 / 38
Estimation Method of Moments The method of moments is simply an application of analogy principle. Suppose that { X i } n i = 1 ∼ i . i . d . f ( x , θ 1 , θ 2 , . . . , θ k ) j -th population moment, which is a function of unknown parameters E ( X j ) = m j ( θ 1 , θ 2 , . . . , θ k ) Shiu-Sheng Chen (NTU Econ) Statistics Fall 2019 10 / 38
Estimation Example For example, let X ∼ Uniform [ θ 1 , θ 2 ] , dx = θ 1 + θ 2 E ( X ) = m 1 ( θ 1 , θ 2 ) = ∫ θ 2 1 θ 2 − θ 1 x 2 θ 1 2 + θ 1 θ 2 + θ 2 E ( X 2 ) = m 2 ( θ 1 , θ 2 ) = ∫ dx = θ 2 θ 2 1 1 x 2 θ 2 − θ 1 3 θ 1 Shiu-Sheng Chen (NTU Econ) Statistics Fall 2019 11 / 38
Estimation Method of Moments We then find ˆ θ 1 , ˆ θ 2 , . . . , ˆ θ k to solve the following moment condition = m j ( ˆ θ k ) , j = 1, 2, . . . , k , ∑ n 1 X j θ 1 , ˆ θ 2 , . . . , ˆ �ÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜ�ÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜ� i n �ÜÜÜÜÜÜÜÜÜ�ÜÜÜÜÜÜÜÜÜ� i = 1 population moments sample moments The solutions: ( ˆ θ 1 , ˆ θ 2 , . . . , ˆ θ k ) are the MM estimators of ( θ 1 , θ 2 , . . . , θ k ) k unknown parameters with k moment conditions Shiu-Sheng Chen (NTU Econ) Statistics Fall 2019 12 / 38
Estimation Method of Moments The method of moments involves equating sample moments with population moments. We impose the condition so that the sample moment is equal to the population moment: the moment condition Moment Conditions 𝜄 = 1 𝑞 𝑘 𝑛 𝑘 መ 𝑜 𝑜 ∑ 𝑗=1 𝑌 𝑗 𝐹(𝑌 𝑘 ) = 𝑛 𝑘 (𝜄) 靠近 where θ = ( θ 1 , θ 2 , . . . , θ k ) , ˆ θ = ( ˆ θ k ) θ 1 , ˆ θ 2 , . . . , ˆ Shiu-Sheng Chen (NTU Econ) Statistics Fall 2019 13 / 38
Estimation Method of Moments The basic idea behind this form of the method is to: (1) Equate the first sample moment m 1 = 1 n ∑ n i = 1 X i to the first theoretical moment E ( X ) . (2) Equate the second sample moment m 2 = 1 n ∑ n i = 1 X 2 i to the second theoretical moment E ( X 2 ) . (3) Continue equating sample moments m j with the corresponding theoretical moments E ( X j ) , j = 3, 4, . . . until you have as many equations as you have parameters. (4) Solve for the parameters. Shiu-Sheng Chen (NTU Econ) Statistics Fall 2019 14 / 38
Estimation Example Let { X i } n i = 1 ∼ i . i . d . U ( θ 1 , θ 2 ) find the MMEs for θ 1 and θ 2 Recall that the moments are E ( X ) = m 1 ( θ 1 , θ 2 ) = θ 1 + θ 2 2 2 + θ 1 θ 2 + θ 2 E ( X 2 ) = m 2 ( θ 1 , θ 2 ) = θ 2 1 3 Shiu-Sheng Chen (NTU Econ) Statistics Fall 2019 15 / 38
Estimation Example The moment conditions are θ 1 + ˆ ˆ X = 1 X i = E ( X ) = ∑ n θ 2 ¯ �ÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜ�ÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜ� n 2 i = 1 m 1 ( ˆ θ 1 , ˆ θ 2 ) 2 + ˆ θ 2 + ˆ ˆ θ 1 ˆ n i = E ( X 2 ) = ∑ θ 2 θ 2 1 X 2 1 n 3 �ÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜ�ÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜÜ� i = 1 m 1 ( ˆ θ 1 , ˆ θ 2 ) Shiu-Sheng Chen (NTU Econ) Statistics Fall 2019 16 / 38
Estimation Example We can solve for ˆ θ 1 and ˆ θ 2 as � � � � 3 θ 1 = ¯ X − n ( X i − ¯ X ) 2 ∑ ˆ n i = 1 � � � � 3 θ 2 = ¯ X + ( X i − ¯ X ) 2 ∑ n ˆ n i = 1 Shiu-Sheng Chen (NTU Econ) Statistics Fall 2019 17 / 38
Estimation Method of Moments: Remarks Note that the Method of Moment Estimator is not unique. Different moment conditions may obtain different estimators. In general, we use the first few moments for simplicity. Shiu-Sheng Chen (NTU Econ) Statistics Fall 2019 18 / 38
Estimation Method of Maximum Likelihood Assume { X i } n i = 1 ∼ i . i . d . f ( x , θ ) where f ( ⋅ ) is known but θ is an unknown parameter. Joint pmf/pdf (function of random sample) f ( x 1 , . . . , x n ; θ ) = f ( x 1 ; θ ) ⋯ f ( x n ; θ ) = ∏ f ( x i ; θ ) i We can also call it a likelihood function of θ : L( θ ) = ∏ f ( x i ; θ ) i Shiu-Sheng Chen (NTU Econ) Statistics Fall 2019 19 / 38
Estimation Maximum Likelihood Estimator (MLE) The maximum likelihood estimator ˆ θ θ = a rgm a x θ ∈ Θ L( θ ) ˆ To find the value of θ such that the random sample { X 1 = x 1 , X 2 = x 2 , . . . , X n = x n } is most likely to be observed. Shiu-Sheng Chen (NTU Econ) Statistics Fall 2019 20 / 38
Estimation Example There are 5 balls in the urn. Let µ denote the portion of blue balls in the urn, and 1 − µ be the portion of green balls in the urn. µ is the unknown parameter The random sample is { X 1 , X 2 , . . . , X 1 0 } , where ⎧ ⎪ ⎪ X i = ⎨ 1 If the ball is blue , ⎪ ⎪ ⎩ If the ball is green . 0 Shiu-Sheng Chen (NTU Econ) Statistics Fall 2019 21 / 38
Estimation Example It is clear that X i ∼ i . i . d . Bernoulli ( µ ) Let Y 1 0 = X 1 + X 2 + ⋯ + X 1 0 = 1 0 ∑ X i i = 1 Y 1 0 represents the number of blue ball, and Y 1 0 ∼ Binomial ( 1 0, µ ) Shiu-Sheng Chen (NTU Econ) Statistics Fall 2019 22 / 38
Estimation Likelihood Function Consider the following two possible samples Sample 1: Y 1 0 = 7 P ( Y 1 0 = 7 ) = ( 1 0 7 ) µ 7 ( 1 − µ ) 3 µ 0 0 1/5 0.000786 2/5 0.042467 3/5 0.214991 4/5 0.201327 5/5 0 Shiu-Sheng Chen (NTU Econ) Statistics Fall 2019 23 / 38
Estimation Likelihood Function Sample 2: Y 1 0 = 2 P ( Y 1 0 = 2 ) = ( 1 0 2 ) µ 2 ( 1 − µ ) 8 µ 0 0 1/5 0.301990 2/5 0.120932 3/5 0.010617 4/5 0.000074 5/5 0 Shiu-Sheng Chen (NTU Econ) Statistics Fall 2019 24 / 38
Estimation Likelihood Function Sample 1: Y 1 0 = 7 Sample 2: Y 1 0 = 2 n = 7 ) = ( 1 0 n = 2 ) = ( 1 0 P ( S ∗ 7 ) µ 7 ( 1 − µ ) 3 P ( S ∗ 2 ) µ 2 ( 1 − µ ) 8 µ 0 0 0 1/5 0.000786 0.301990 2/5 0.042467 0.120932 3/5 0.214991 0.010617 4/5 0.201327 0.000074 5/5 0 0 Shiu-Sheng Chen (NTU Econ) Statistics Fall 2019 25 / 38
Recommend
More recommend
