Biostatistics 602 - Statistical Inference February 19th, 2013 - PowerPoint PPT Presentation

. .. .. . . .. . . . . . .. . . .. . . . .. .. Biostatistics 602 - Statistical Inference February 19th, 2013 Biostatistics 602 - Lecture 12 Hyun Min Kang February 19th, 2013 Hyun Min Kang Cramer-Rao Theorem Lecture 12 . . . Summary . Attainability Regularity Condition Recap . . . . . . . . . .. . . .. .. . . .. . . . .. . .. . . .. . . . . .. .. . . .. . . .. . . . . . .. . . .. . 1 / 24 . . . . . . . . . . . . .

. . Recap . . . . . . . . . . .. . . .. . Attainability .. . . .. . . .. . Regularity Condition . .. 3 What is the best unbiased estimator or uniformly unbiased minimium February 19th, 2013 Biostatistics 602 - Lecture 12 Hyun Min Kang UMVUE? 4 What is the Cramer-Rao bound, and how can it be useful to find . . variance estimator (UMVUE) ? . Summary . estimators? 2 What are plausible ways to compare between different point . . . . Last Lecture .. . . . .. . . .. . . .. . .. . . . .. . . .. . . . . .. .. .. . . .. . . .. . . . . .. . . .. . . 2 / 24 . . . . . . . . . . . . . 1 If you know MLE of θ , can you also know MLE of τ ( θ ) for any function τ ?

. . Recap . . . . . . . . . . .. . . .. . Attainability .. . . .. . . .. . Regularity Condition . .. 3 What is the best unbiased estimator or uniformly unbiased minimium February 19th, 2013 Biostatistics 602 - Lecture 12 Hyun Min Kang UMVUE? 4 What is the Cramer-Rao bound, and how can it be useful to find . . variance estimator (UMVUE) ? . Summary . estimators? 2 What are plausible ways to compare between different point . . . . Last Lecture .. . . . .. . . .. . . .. . .. . . . .. . . .. . . . . .. .. .. . . .. . . .. . . . . .. . . .. . . 2 / 24 . . . . . . . . . . . . . 1 If you know MLE of θ , can you also know MLE of τ ( θ ) for any function τ ?

. . Recap . . . . . . . . . . .. . . .. . Attainability .. . . .. . . .. . Regularity Condition . .. 3 What is the best unbiased estimator or uniformly unbiased minimium February 19th, 2013 Biostatistics 602 - Lecture 12 Hyun Min Kang UMVUE? 4 What is the Cramer-Rao bound, and how can it be useful to find . . variance estimator (UMVUE) ? . Summary . estimators? 2 What are plausible ways to compare between different point . . . . Last Lecture .. . . . .. . . .. . . .. . .. . . . .. . . .. . . . . .. .. .. . . .. . . .. . . . . .. . . .. . . 2 / 24 . . . . . . . . . . . . . 1 If you know MLE of θ , can you also know MLE of τ ( θ ) for any function τ ?

. . Recap . . . . . . . . . . .. . . .. . Attainability .. . . .. . . .. . Regularity Condition . .. 3 What is the best unbiased estimator or uniformly unbiased minimium February 19th, 2013 Biostatistics 602 - Lecture 12 Hyun Min Kang UMVUE? 4 What is the Cramer-Rao bound, and how can it be useful to find . . variance estimator (UMVUE) ? . Summary . estimators? 2 What are plausible ways to compare between different point . . . . Last Lecture .. . . . .. . . .. . . .. . .. . . . .. . . .. . . . . .. .. .. . . .. . . .. . . . . .. . . .. . . 2 / 24 . . . . . . . . . . . . . 1 If you know MLE of θ , can you also know MLE of τ ( θ ) for any function τ ?

. . Recap . . . . . . . . . . .. . . .. .. Attainability .. . . .. . . .. . Regularity Condition . .. . February 19th, 2013 Biostatistics 602 - Lecture 12 Hyun Min Kang E d d interchangeable, i.e. . Summary . . is an estimator satisfying . . Theorem 7.3.9 : Cramer-Rao Theorem . Recap : Cramer-Rao inequality . . . . .. . . .. . . . . .. . . . .. . . .. . . . .. 3 / 24 . . .. .. . . . . .. . .. .. .. . . .. . . . . . . . . . . . . . . . . Let X 1 , · · · , X n be a sample with joint pdf/pmf of f X ( x | θ ) . Suppose W ( X ) 1 E [ W ( X ) | θ ] = τ ( θ ) , ∀ θ ∈ Ω . 2 Var [ W ( X ) | θ ] < ∞ . For h ( x ) = 1 and h ( x ) = W ( x ) , if the differentiation and integrations are ∫ ∫ h ( x ) ∂ d θ E [ h ( x ) | θ ] = h ( x ) f X ( x | θ ) d x = ∂θ f X ( x | θ ) d x d θ x ∈X x ∈X Then, a lower bound of Var [ W ( X ) | θ ] is [ τ ′ ( θ )] 2 Var [ W ( X ) | θ ] ≥ [ { ∂ ] ∂θ log f X ( X | θ ) } 2 | θ

. .. .. . . .. . . . . . .. . .. .. . . . .. . Corollary 7.3.10 February 19th, 2013 Biostatistics 602 - Lecture 12 Hyun Min Kang nE in the above Cramer-Rao theorem hold, then the lower-bound of . . . . Recap : Cramer-Rao bound in iid case Summary . Attainability Regularity Condition Recap . . . . . . . . . .. . . .. .. .. . .. . . . . . .. . . .. . . . . .. . . . .. . . . .. . 4 / 24 .. . . .. . . .. . . . . . . . . . . . . . . If X 1 , · · · , X n are iid samples from pdf/pmf f X ( x | θ ) , and the assumptions Var [ W ( X ) | θ ] becomes [ τ ′ ( θ )] 2 Var [ W ( X ) | θ ] ≥ { ∂ [ ∂θ log f X ( X | θ ) } 2 | θ ]

. .. .. . . .. . . . . . .. . . .. . . . .. . Definition: Score or Score Function for X February 19th, 2013 Biostatistics 602 - Lecture 12 Hyun Min Kang i.i.d. . . . . Recap : Score Function Summary . Attainability Regularity Condition Recap . . . . . . . . . .. .. . .. .. . . .. . . . . . .. . . .. . . . .. .. .. . . .. . . .. . . . . . . .. . 5 / 24 . .. . . . . . . . . . . . . . X 1 , · · · , X n ∼ f X ( x | θ ) ∂ S ( X | θ ) ∂θ log f X ( X | θ ) = E [ S ( X | θ )] = 0 ∂ S n ( X | θ ) = ∂θ log f X ( X | θ )

. .. . . .. . . .. . .. . . . .. . . .. . .. . . . . . . . . . .. . February 19th, 2013 Biostatistics 602 - Lecture 12 Hyun Min Kang The bigger the information number, the more information we have about nE E E . Recap Definition: Fisher Information Number . Recap : Fisher Information Number Summary . Attainability Regularity Condition . . . .. . .. . . .. . . . .. . .. . . .. . . . . . . .. . . .. . . . .. 6 / 24 . .. .. . . . . .. . . . . . . . . . . . . . [{ ∂ } 2 ] S 2 ( X | θ ) [ ] I ( θ ) = ∂θ log f X ( X | θ ) = E [{ ∂ } 2 ] I n ( θ ) = ∂θ log f X ( X | θ ) [{ ∂ } 2 ] ∂θ log f X ( X | θ ) = = nI ( θ ) θ , the smaller bound on the variance of unbiased estimates.

. . . .. .. . .. . .. .. . . .. . . .. . . . .. . February 19th, 2013 Biostatistics 602 - Lecture 12 Hyun Min Kang which are true for exponential family, then d d . Lemma 7.3.11 . . . . . . . . . . Recap : Simplified Fisher Information Summary . Attainability Regularity Condition Recap . . . . .. . . .. . . .. . . . .. . . .. . . . 7 / 24 .. .. .. . . .. . .. . . . . . .. . . .. . . . . . . . . . . . . . . . If f X ( x | θ ) satisfies the two interchangeability conditions ∫ ∫ ∂ f X ( x | θ ) dx = ∂θ f X ( x | θ ) dx d θ x ∈X x ∈X ∂ 2 ∫ ∂ ∫ ∂θ f X ( x | θ ) dx = ∂θ 2 f X ( x | θ ) dx d θ x ∈X x ∈X [{ ∂ [ ∂ 2 } 2 ] ] I ( θ ) = E ∂θ log f X ( X | θ ) = − E ∂θ 2 log f X ( X | θ )

log f X X Var X . Therefore X . . . Attainability Regularity Condition Recap . . . . . . . . . . .. . Recap - Normal Distribution .. . . .. . . .. . Summary I i.i.d. X February 19th, 2013 Biostatistics 602 - Lecture 12 Hyun Min Kang . attains the Cramer-Rao bound and thus the best unbiased estimator for n is nI The Cramer-Rao bound for E .. X log E X exp log E E .. . . . . . .. . . .. . . .. . . .. . . .. . . .. . . .. 8 / 24 . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . . . . . . . . . . . . ∼ N ( µ, σ 2 ) , where σ 2 is known. X 1 , · · · , X n

Var X . Therefore X . .. Regularity Condition Recap . . . . . . . . . . .. . . . . . .. . . .. .. . .. Attainability Recap - Normal Distribution Summary X February 19th, 2013 Biostatistics 602 - Lecture 12 Hyun Min Kang . attains the Cramer-Rao bound and thus the best unbiased estimator for n is nI The Cramer-Rao bound for E . X log E X exp log E i.i.d. . . .. . . .. . . .. . . .. . .. .. . . .. . . .. . . . . . . . .. . . .. . . .. . . .. . .. . . . .. 8 / 24 . . . . . . . . . . . . . ∼ N ( µ, σ 2 ) , where σ 2 is known. X 1 , · · · , X n [ ∂ 2 ] I ( µ ) = − E ∂µ 2 log f X ( X | µ )

Var X . Therefore X . .. . .. . . .. . . . Recap . .. . . .. . . . . . . . . . . . Attainability Regularity Condition The Cramer-Rao bound for February 19th, 2013 Biostatistics 602 - Lecture 12 Hyun Min Kang . attains the Cramer-Rao bound and thus the best unbiased estimator for n is nI X . E X log E i.i.d. Recap - Normal Distribution Summary . .. .. . . . . .. .. . . .. . .. . . . .. . . .. . . .. 8 / 24 . .. . .. . . .. . . . . .. . . . .. . .. . . . . . . . . . . . . . . ∼ N ( µ, σ 2 ) , where σ 2 is known. X 1 , · · · , X n [ ∂ 2 ] I ( µ ) = − E ∂µ 2 log f X ( X | µ ) [ ∂ 2 − ( X − µ ) 2 { 1 ( )}] − E √ = ∂µ 2 log 2 σ 2 2 πσ 2 exp