[PPT] - Biostatistics 602 - Statistical Inference February 19th, 2013 PowerPoint Presentation

SLIDE 1

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

. .

Biostatistics 602 - Statistical Inference Lecture 12 Cramer-Rao Theorem

Hyun Min Kang February 19th, 2013

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 1 / 24

SLIDE 2

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Last Lecture

. . 1 If you know MLE of θ, can you also know MLE of τ(θ) for any

function τ?

. . 2 What are plausible ways to compare between different point

estimators?

. . 3 What is the best unbiased estimator or uniformly unbiased minimium

variance estimator (UMVUE)?

. . 4 What is the Cramer-Rao bound, and how can it be useful to find

UMVUE?

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 2 / 24

SLIDE 3

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Last Lecture

. . 1 If you know MLE of θ, can you also know MLE of τ(θ) for any

function τ?

. . 2 What are plausible ways to compare between different point

estimators?

. . 3 What is the best unbiased estimator or uniformly unbiased minimium

variance estimator (UMVUE)?

. . 4 What is the Cramer-Rao bound, and how can it be useful to find

UMVUE?

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 2 / 24

SLIDE 4

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Last Lecture

. . 1 If you know MLE of θ, can you also know MLE of τ(θ) for any

function τ?

. . 2 What are plausible ways to compare between different point

estimators?

. . 3 What is the best unbiased estimator or uniformly unbiased minimium

variance estimator (UMVUE)?

. . 4 What is the Cramer-Rao bound, and how can it be useful to find

UMVUE?

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 2 / 24

SLIDE 5

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Last Lecture

. . 1 If you know MLE of θ, can you also know MLE of τ(θ) for any

function τ?

. . 2 What are plausible ways to compare between different point

estimators?

. . 3 What is the best unbiased estimator or uniformly unbiased minimium

variance estimator (UMVUE)?

. . 4 What is the Cramer-Rao bound, and how can it be useful to find

UMVUE?

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 2 / 24

SLIDE 6

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Recap : Cramer-Rao inequality

.

Theorem 7.3.9 : Cramer-Rao Theorem

. . Let X1, · · · , Xn be a sample with joint pdf/pmf of fX(x|θ). Suppose W(X) is an estimator satisfying

. . 1 E[W(X)|θ] = τ(θ), ∀θ ∈ Ω. . . 2 Var[W(X)|θ] < ∞.

For h(x) = 1 and h(x) = W(x), if the differentiation and integrations are interchangeable, i.e. d dθE[h(x)|θ] = d dθ ∫

x∈X

h(x)fX(x|θ)dx = ∫

x∈X

h(x) ∂ ∂θfX(x|θ)dx Then, a lower bound of Var[W(X)|θ] is Var[W(X)|θ] ≥ [τ ′(θ)]2 E [ { ∂

∂θ log fX(X|θ)}2|θ

]

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 3 / 24

SLIDE 7

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Recap : Cramer-Rao bound in iid case

.

Corollary 7.3.10

. . If X1, · · · , Xn are iid samples from pdf/pmf fX(x|θ), and the assumptions in the above Cramer-Rao theorem hold, then the lower-bound of Var[W(X)|θ] becomes Var[W(X)|θ] ≥ [τ ′(θ)]2 nE [ { ∂

∂θ log fX(X|θ)}2|θ

]

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 4 / 24

SLIDE 8

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Recap : Score Function

.

Definition: Score or Score Function for X

. . X1, · · · , Xn

i.i.d.

∼

fX(x|θ) S(X|θ) = ∂ ∂θ log fX(X|θ) E [S(X|θ)] = Sn(X|θ) = ∂ ∂θ log fX(X|θ)

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 5 / 24

SLIDE 9

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Recap : Fisher Information Number

.

Definition: Fisher Information Number

. . I(θ) = E [{ ∂ ∂θ log fX(X|θ) }2] = E [ S2(X|θ) ] In(θ) = E [{ ∂ ∂θ log fX(X|θ) }2] = nE [{ ∂ ∂θ log fX(X|θ) }2] = nI(θ) The bigger the information number, the more information we have about θ, the smaller bound on the variance of unbiased estimates.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 6 / 24

SLIDE 10

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Recap : Simplified Fisher Information

.

Lemma 7.3.11

. . If fX(x|θ) satisfies the two interchangeability conditions d dθ ∫

x∈X

fX(x|θ)dx = ∫

x∈X

∂ ∂θfX(x|θ)dx d dθ ∫

x∈X

∂ ∂θfX(x|θ)dx = ∫

x∈X

∂2 ∂θ2 fX(x|θ)dx which are true for exponential family, then I(θ) = E [{ ∂ ∂θ log fX(X|θ) }2] = −E [ ∂2 ∂θ2 log fX(X|θ) ]

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 7 / 24

SLIDE 11

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Recap - Normal Distribution

X1, · · · , Xn

i.i.d.

∼ N(µ, σ2), where σ2 is known.

I E log fX X E log exp X E log X E X The Cramer-Rao bound for is nI

n

Var X . Therefore X attains the Cramer-Rao bound and thus the best unbiased estimator for .

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 8 / 24

SLIDE 12

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Recap - Normal Distribution

X1, · · · , Xn

i.i.d.

∼ N(µ, σ2), where σ2 is known.

I(µ) = −E [ ∂2 ∂µ2 log fX(X|µ) ] E log exp X E log X E X The Cramer-Rao bound for is nI

n

Var X . Therefore X attains the Cramer-Rao bound and thus the best unbiased estimator for .

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 8 / 24

SLIDE 13

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Recap - Normal Distribution

X1, · · · , Xn

i.i.d.

∼ N(µ, σ2), where σ2 is known.

I(µ) = −E [ ∂2 ∂µ2 log fX(X|µ) ] = −E [ ∂2 ∂µ2 log { 1 √ 2πσ2 exp ( −(X − µ)2 2σ2 )}] E log X E X The Cramer-Rao bound for is nI

n

Var X . Therefore X attains the Cramer-Rao bound and thus the best unbiased estimator for .

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 8 / 24

SLIDE 14

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Recap - Normal Distribution

X1, · · · , Xn

i.i.d.

∼ N(µ, σ2), where σ2 is known.

I(µ) = −E [ ∂2 ∂µ2 log fX(X|µ) ] = −E [ ∂2 ∂µ2 log { 1 √ 2πσ2 exp ( −(X − µ)2 2σ2 )}] = −E [ ∂2 ∂µ2 { −1 2 log(2πσ2) − (X − µ)2 2σ2 }] E X The Cramer-Rao bound for is nI

n

Var X . Therefore X attains the Cramer-Rao bound and thus the best unbiased estimator for .

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 8 / 24

SLIDE 15

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Recap - Normal Distribution

X1, · · · , Xn

i.i.d.

∼ N(µ, σ2), where σ2 is known.

I(µ) = −E [ ∂2 ∂µ2 log fX(X|µ) ] = −E [ ∂2 ∂µ2 log { 1 √ 2πσ2 exp ( −(X − µ)2 2σ2 )}] = −E [ ∂2 ∂µ2 { −1 2 log(2πσ2) − (X − µ)2 2σ2 }] = −E [ ∂ ∂µ {2(X − µ) 2σ2 }] = 1 σ2 The Cramer-Rao bound for µ is [nI(µ)]−1 = σ2

n

Var X . Therefore X attains the Cramer-Rao bound and thus the best unbiased estimator for .

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 8 / 24

SLIDE 16

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Recap - Normal Distribution

X1, · · · , Xn

i.i.d.

∼ N(µ, σ2), where σ2 is known.

I(µ) = −E [ ∂2 ∂µ2 log fX(X|µ) ] = −E [ ∂2 ∂µ2 log { 1 √ 2πσ2 exp ( −(X − µ)2 2σ2 )}] = −E [ ∂2 ∂µ2 { −1 2 log(2πσ2) − (X − µ)2 2σ2 }] = −E [ ∂ ∂µ {2(X − µ) 2σ2 }] = 1 σ2 The Cramer-Rao bound for µ is [nI(µ)]−1 = σ2

n = Var(X).

Therefore X attains the Cramer-Rao bound and thus the best unbiased estimator for .

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 8 / 24

SLIDE 17

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Recap - Normal Distribution

X1, · · · , Xn

i.i.d.

∼ N(µ, σ2), where σ2 is known.

I(µ) = −E [ ∂2 ∂µ2 log fX(X|µ) ] = −E [ ∂2 ∂µ2 log { 1 √ 2πσ2 exp ( −(X − µ)2 2σ2 )}] = −E [ ∂2 ∂µ2 { −1 2 log(2πσ2) − (X − µ)2 2σ2 }] = −E [ ∂ ∂µ {2(X − µ) 2σ2 }] = 1 σ2 The Cramer-Rao bound for µ is [nI(µ)]−1 = σ2

n = Var(X). Therefore X

attains the Cramer-Rao bound and thus the best unbiased estimator for µ.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 8 / 24

SLIDE 18

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Example of Cramer-Rao lower bound attainment

.

Problem

. . X1, · · · , Xn

i.i.d.

∼ Bernoulli(p). Is X the best unbiased estimator of p?

Does it attain the Cramer-Rao lower bound? .

Solution

. . . . . . . . E X p Var X nVar X p p n I p E log fX X p E log fX X p

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 9 / 24

SLIDE 19

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Example of Cramer-Rao lower bound attainment

.

Problem

. . X1, · · · , Xn

i.i.d.

∼ Bernoulli(p). Is X the best unbiased estimator of p?

Does it attain the Cramer-Rao lower bound? .

Solution

. . E(X) = p Var X nVar X p p n I p E log fX X p E log fX X p

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 9 / 24

SLIDE 20

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Example of Cramer-Rao lower bound attainment

.

Problem

. . X1, · · · , Xn

i.i.d.

∼ Bernoulli(p). Is X the best unbiased estimator of p?

Does it attain the Cramer-Rao lower bound? .

Solution

. . E(X) = p Var(X) = 1 nVar(X) = p(1 − p) n I p E log fX X p E log fX X p

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 9 / 24

SLIDE 21

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Example of Cramer-Rao lower bound attainment

.

Problem

. . X1, · · · , Xn

i.i.d.

∼ Bernoulli(p). Is X the best unbiased estimator of p?

Does it attain the Cramer-Rao lower bound? .

Solution

. . E(X) = p Var(X) = 1 nVar(X) = p(1 − p) n I(p) = E [{ ∂ ∂θ log fX(X|θ) }2

p

] E log fX X p

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 9 / 24

SLIDE 22

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Example of Cramer-Rao lower bound attainment

.

Problem

. . X1, · · · , Xn

i.i.d.

∼ Bernoulli(p). Is X the best unbiased estimator of p?

Does it attain the Cramer-Rao lower bound? .

Solution

. . E(X) = p Var(X) = 1 nVar(X) = p(1 − p) n I(p) = E [{ ∂ ∂θ log fX(X|θ) }2

p

] = −E [ ∂2 ∂θ2 log fX(X|θ)|p ]

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 9 / 24

SLIDE 23

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Solution (cont’d)

fX(x|θ) = px(1 − p)1−x log fX x x log p x log p p log fX x p x p x p p log fX x p x p x p I p E X p X p p p p p p p p p p Therefore, the Cramer-Rao bound is nI p

p p n

VarX, and X attains the Cramer-Rao lower bound, and it is the UMVUE.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 10 / 24

SLIDE 24

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Solution (cont’d)

fX(x|θ) = px(1 − p)1−x log fX(x|θ) = x log p + (1 − x) log(1 − p) p log fX x p x p x p p log fX x p x p x p I p E X p X p p p p p p p p p p Therefore, the Cramer-Rao bound is nI p

p p n

VarX, and X attains the Cramer-Rao lower bound, and it is the UMVUE.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 10 / 24

SLIDE 25

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Solution (cont’d)

fX(x|θ) = px(1 − p)1−x log fX(x|θ) = x log p + (1 − x) log(1 − p) ∂ ∂p log fX(x|p) = x p − 1 − x 1 − p p log fX x p x p x p I p E X p X p p p p p p p p p p Therefore, the Cramer-Rao bound is nI p

p p n

VarX, and X attains the Cramer-Rao lower bound, and it is the UMVUE.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 10 / 24

SLIDE 26

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Solution (cont’d)

fX(x|θ) = px(1 − p)1−x log fX(x|θ) = x log p + (1 − x) log(1 − p) ∂ ∂p log fX(x|p) = x p − 1 − x 1 − p ∂2 ∂p2 log fX(x|p) = − x p2 − 1 − x (1 − p)2 I p E X p X p p p p p p p p p p Therefore, the Cramer-Rao bound is nI p

p p n

VarX, and X attains the Cramer-Rao lower bound, and it is the UMVUE.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 10 / 24

SLIDE 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Solution (cont’d)

fX(x|θ) = px(1 − p)1−x log fX(x|θ) = x log p + (1 − x) log(1 − p) ∂ ∂p log fX(x|p) = x p − 1 − x 1 − p ∂2 ∂p2 log fX(x|p) = − x p2 − 1 − x (1 − p)2 I(p) = −E [ − X p2 − 1 − X (1 − p)2 |p ] p p p p p p p p Therefore, the Cramer-Rao bound is nI p

p p n

VarX, and X attains the Cramer-Rao lower bound, and it is the UMVUE.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 10 / 24

SLIDE 28

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Solution (cont’d)

fX(x|θ) = px(1 − p)1−x log fX(x|θ) = x log p + (1 − x) log(1 − p) ∂ ∂p log fX(x|p) = x p − 1 − x 1 − p ∂2 ∂p2 log fX(x|p) = − x p2 − 1 − x (1 − p)2 I(p) = −E [ − X p2 − 1 − X (1 − p)2 |p ] = p p2 + 1 − p (1 − p)2 = 1 p + 1 1 − p = 1 p(1 − p) Therefore, the Cramer-Rao bound is nI p

p p n

VarX, and X attains the Cramer-Rao lower bound, and it is the UMVUE.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 10 / 24

SLIDE 29

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Solution (cont’d)

fX(x|θ) = px(1 − p)1−x log fX(x|θ) = x log p + (1 − x) log(1 − p) ∂ ∂p log fX(x|p) = x p − 1 − x 1 − p ∂2 ∂p2 log fX(x|p) = − x p2 − 1 − x (1 − p)2 I(p) = −E [ − X p2 − 1 − X (1 − p)2 |p ] = p p2 + 1 − p (1 − p)2 = 1 p + 1 1 − p = 1 p(1 − p) Therefore, the Cramer-Rao bound is

1 nI(p) = p(1−p) n

= VarX, and X attains the Cramer-Rao lower bound, and it is the UMVUE.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 10 / 24

SLIDE 30

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Regularity condition for Cramer-Rao Theorem

d dθ ∫

x∈X

h(x)fX(x|θ)dx = ∫

x∈X

h(x) ∂ ∂θfX(x|θ)dx

This regularity condition holds for exponential family.
How about non-exponential family, such as

X Xn

i.i.d. Uniform

?

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 11 / 24

SLIDE 31

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Regularity condition for Cramer-Rao Theorem

d dθ ∫

x∈X

h(x)fX(x|θ)dx = ∫

x∈X

h(x) ∂ ∂θfX(x|θ)dx

This regularity condition holds for exponential family.
How about non-exponential family, such as

X Xn

i.i.d. Uniform

?

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 11 / 24

SLIDE 32

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Regularity condition for Cramer-Rao Theorem

d dθ ∫

x∈X

h(x)fX(x|θ)dx = ∫

x∈X

h(x) ∂ ∂θfX(x|θ)dx

This regularity condition holds for exponential family.
How about non-exponential family, such as

X1, · · · , Xn

i.i.d.

∼ Uniform(0, θ)?

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 11 / 24

SLIDE 33

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Using Leibnitz’s Rule

.

Leibnitz’s Rule

. . d dθ ∫ b(θ)

a(θ)

f(x|θ)dx = f(b(θ)|θ)b′(θ) − f(a(θ)|θ)a′(θ) + ∫ b(θ)

a(θ)

∂ ∂θf(x|θ)dx .

Applying to Uniform Distribution

. . . . . . . . fX x d d h x dx h d d h fX d d h x dx h x dx The interchangeability condition is not satisfied.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 12 / 24

SLIDE 34

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Using Leibnitz’s Rule

.

Leibnitz’s Rule

. . d dθ ∫ b(θ)

a(θ)

f(x|θ)dx = f(b(θ)|θ)b′(θ) − f(a(θ)|θ)a′(θ) + ∫ b(θ)

a(θ)

∂ ∂θf(x|θ)dx .

Applying to Uniform Distribution

. . fX(x|θ) = 1/θ d d h x dx h d d h fX d d h x dx h x dx The interchangeability condition is not satisfied.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 12 / 24

SLIDE 35

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Using Leibnitz’s Rule

.

Leibnitz’s Rule

. . d dθ ∫ b(θ)

a(θ)

f(x|θ)dx = f(b(θ)|θ)b′(θ) − f(a(θ)|θ)a′(θ) + ∫ b(θ)

a(θ)

∂ ∂θf(x|θ)dx .

Applying to Uniform Distribution

. . fX(x|θ) = 1/θ d dθ ∫ θ h(x) (1 θ ) dx = h(θ) θ dθ dθ − h(0)fX(0|θ)d0 dθ + ∫ θ ∂ ∂θh(x) (1 θ ) dx h x dx The interchangeability condition is not satisfied.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 12 / 24

SLIDE 36

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Using Leibnitz’s Rule

.

Leibnitz’s Rule

. . d dθ ∫ b(θ)

a(θ)

f(x|θ)dx = f(b(θ)|θ)b′(θ) − f(a(θ)|θ)a′(θ) + ∫ b(θ)

a(θ)

∂ ∂θf(x|θ)dx .

Applying to Uniform Distribution

. . fX(x|θ) = 1/θ d dθ ∫ θ h(x) (1 θ ) dx = h(θ) θ dθ dθ − h(0)fX(0|θ)d0 dθ + ∫ θ ∂ ∂θh(x) (1 θ ) dx ̸= ∫ θ ∂ ∂θh(x) (1 θ ) dx The interchangeability condition is not satisfied.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 12 / 24

SLIDE 37

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Using Leibnitz’s Rule

.

Leibnitz’s Rule

. . d dθ ∫ b(θ)

a(θ)

f(x|θ)dx = f(b(θ)|θ)b′(θ) − f(a(θ)|θ)a′(θ) + ∫ b(θ)

a(θ)

∂ ∂θf(x|θ)dx .

Applying to Uniform Distribution

. . fX(x|θ) = 1/θ d dθ ∫ θ h(x) (1 θ ) dx = h(θ) θ dθ dθ − h(0)fX(0|θ)d0 dθ + ∫ θ ∂ ∂θh(x) (1 θ ) dx ̸= ∫ θ ∂ ∂θh(x) (1 θ ) dx The interchangeability condition is not satisfied.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 12 / 24

SLIDE 38

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Solving the Uniform Distribution Example

If X1, · · · , Xn

i.i.d.

∼ Uniform(0, θ), the unbiased estimator of θ is

T(X) = n + 1 n X(n) E n n X n Var n n X n n n n The Cramer-Rao lower bound (if interchangeability condition was met) is

nI n .

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 13 / 24

SLIDE 39

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Solving the Uniform Distribution Example

If X1, · · · , Xn

i.i.d.

∼ Uniform(0, θ), the unbiased estimator of θ is

T(X) = n + 1 n X(n) E [n + 1 n X(n) ] = θ Var n n X n n n n The Cramer-Rao lower bound (if interchangeability condition was met) is

nI n .

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 13 / 24

SLIDE 40

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Solving the Uniform Distribution Example

If X1, · · · , Xn

i.i.d.

∼ Uniform(0, θ), the unbiased estimator of θ is

T(X) = n + 1 n X(n) E [n + 1 n X(n) ] = θ Var [n + 1 n X(n) ] = 1 n(n + 2)θ2 < θ2 n The Cramer-Rao lower bound (if interchangeability condition was met) is

nI n .

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 13 / 24

SLIDE 41

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Solving the Uniform Distribution Example

If X1, · · · , Xn

i.i.d.

∼ Uniform(0, θ), the unbiased estimator of θ is

T(X) = n + 1 n X(n) E [n + 1 n X(n) ] = θ Var [n + 1 n X(n) ] = 1 n(n + 2)θ2 < θ2 n The Cramer-Rao lower bound (if interchangeability condition was met) is

1 nI(θ) = θ2 n .

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 13 / 24

SLIDE 42

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

When is the Cramer-Rao Lower Bound Attainable?

It is possible that the value of Cramer-Rao bound may be strictly smaller than the variance of any unbiased estimator .

Corollary 7.3.15 : Attainment of Cramer-Rao Bound

. . . . . . . . Let X Xn be iid with pdf/pmf fX x , where fX x satisfies the assumptions of the Cramer-Rao Theorem. Let L x

n i

fX xi denote the likelihood function. If W X is unbiased for , then W X attains the Cramer-Rao lower bound if and only if log L x Sn x a W X for some function a .

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 14 / 24

SLIDE 43

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

When is the Cramer-Rao Lower Bound Attainable?

It is possible that the value of Cramer-Rao bound may be strictly smaller than the variance of any unbiased estimator .

Corollary 7.3.15 : Attainment of Cramer-Rao Bound

. . Let X1, · · · , Xn be iid with pdf/pmf fX(x|θ), where fX(x|θ) satisfies the assumptions of the Cramer-Rao Theorem. Let L x

n i

fX xi denote the likelihood function. If W X is unbiased for , then W X attains the Cramer-Rao lower bound if and only if log L x Sn x a W X for some function a .

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 14 / 24

SLIDE 44

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

When is the Cramer-Rao Lower Bound Attainable?

It is possible that the value of Cramer-Rao bound may be strictly smaller than the variance of any unbiased estimator .

Corollary 7.3.15 : Attainment of Cramer-Rao Bound

. . Let X1, · · · , Xn be iid with pdf/pmf fX(x|θ), where fX(x|θ) satisfies the assumptions of the Cramer-Rao Theorem. Let L(θ|x) = ∏n

i=1 fX(xi|θ)

denote the likelihood function. If W(X) is unbiased for τ(θ), then W(X) attains the Cramer-Rao lower bound if and only if log L x Sn x a W X for some function a .

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 14 / 24

SLIDE 45

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

When is the Cramer-Rao Lower Bound Attainable?

It is possible that the value of Cramer-Rao bound may be strictly smaller than the variance of any unbiased estimator .

Corollary 7.3.15 : Attainment of Cramer-Rao Bound

. . Let X1, · · · , Xn be iid with pdf/pmf fX(x|θ), where fX(x|θ) satisfies the assumptions of the Cramer-Rao Theorem. Let L(θ|x) = ∏n

i=1 fX(xi|θ)

denote the likelihood function. If W(X) is unbiased for τ(θ), then W(X) attains the Cramer-Rao lower bound if and only if ∂ ∂θ log L(θ|x) = Sn(x|θ) = a(θ)[W(X) − τ(θ)] for some function a(θ).

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 14 / 24

SLIDE 46

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Proof of Corollary 7.3.15

We used Cauchy-Schwarz inequality to prove that [ Cov{W(X), ∂ ∂θ log fX(X|θ)} ]2 ≤ Var[W(X)]Var [ ∂ ∂θ log fX(X|θ) ] In Cauchy-Schwarz inequality, the equality satisfies if and only if there is a linear relationship between the two variables, that is log fX x log L x a W x b

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 15 / 24

SLIDE 47

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Proof of Corollary 7.3.15

We used Cauchy-Schwarz inequality to prove that [ Cov{W(X), ∂ ∂θ log fX(X|θ)} ]2 ≤ Var[W(X)]Var [ ∂ ∂θ log fX(X|θ) ] In Cauchy-Schwarz inequality, the equality satisfies if and only if there is a linear relationship between the two variables, that is ∂ ∂θ log fX(x|θ) = ∂ ∂θ log L(θ|x) = a(θ)W(x) + b(θ)

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 15 / 24

SLIDE 48

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Proof of Corollary 7.3.15 (cont’d)

E [ ∂ ∂θ log fX(X|θ) ] = E [Sn(X|θ)] = 0 E a W X b a E W X b a b b a log L x a W x a a W x

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 16 / 24

SLIDE 49

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Proof of Corollary 7.3.15 (cont’d)

E [ ∂ ∂θ log fX(X|θ) ] = E [Sn(X|θ)] = 0 E [a(θ)W(X) + b(θ)] = a E W X b a b b a log L x a W x a a W x

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 16 / 24

SLIDE 50

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Proof of Corollary 7.3.15 (cont’d)

E [ ∂ ∂θ log fX(X|θ) ] = E [Sn(X|θ)] = 0 E [a(θ)W(X) + b(θ)] = a(θ)E [W(X)] + b(θ) = a b b a log L x a W x a a W x

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 16 / 24

SLIDE 51

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Proof of Corollary 7.3.15 (cont’d)

E [ ∂ ∂θ log fX(X|θ) ] = E [Sn(X|θ)] = 0 E [a(θ)W(X) + b(θ)] = a(θ)E [W(X)] + b(θ) = a(θ)τ(θ) + b(θ) = b a log L x a W x a a W x

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 16 / 24

SLIDE 52

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Proof of Corollary 7.3.15 (cont’d)

E [ ∂ ∂θ log fX(X|θ) ] = E [Sn(X|θ)] = 0 E [a(θ)W(X) + b(θ)] = a(θ)E [W(X)] + b(θ) = a(θ)τ(θ) + b(θ) = b(θ) = −a(θ)τ(θ) log L x a W x a a W x

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 16 / 24

SLIDE 53

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Proof of Corollary 7.3.15 (cont’d)

E [ ∂ ∂θ log fX(X|θ) ] = E [Sn(X|θ)] = 0 E [a(θ)W(X) + b(θ)] = a(θ)E [W(X)] + b(θ) = a(θ)τ(θ) + b(θ) = b(θ) = −a(θ)τ(θ) ∂ ∂θ log L(θ|x) = a(θ)W(x) − a(θ)τ(θ) = a(θ) [W(x) − τ(θ)]

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 16 / 24

SLIDE 54

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Revisiting the Bernoulli Example

.

Problem

. . X1, · · · , Xn

i.i.d.

∼ Bernoulli(p). Is X the best unbiased estimator of p?

Does it attain the Cramer-Rao lower bound?

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 17 / 24

SLIDE 55

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Method Using Corollary 7.3.15

L(p|x) =

n

∏

i=1

pxi(1 − p)1−xi log L p x log

n i

pxi p

xi n i

log pxi p

xi n i

xi log p xi log p log p

n i

xi log p n

n i

xi

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 18 / 24

SLIDE 56

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Method Using Corollary 7.3.15

L(p|x) =

n

∏

i=1

pxi(1 − p)1−xi log L(p|x) = log

n

∏

i=1

pxi(1 − p)1−xi

n i

log pxi p

xi n i

xi log p xi log p log p

n i

xi log p n

n i

xi

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 18 / 24

SLIDE 57

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Method Using Corollary 7.3.15

L(p|x) =

n

∏

i=1

pxi(1 − p)1−xi log L(p|x) = log

n

∏

i=1

pxi(1 − p)1−xi =

n

∑

i=1

log[pxi(1 − p)1−xi]

n i

xi log p xi log p log p

n i

xi log p n

n i

xi

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 18 / 24

SLIDE 58

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Method Using Corollary 7.3.15

L(p|x) =

n

∏

i=1

pxi(1 − p)1−xi log L(p|x) = log

n

∏

i=1

pxi(1 − p)1−xi =

n

∑

i=1

log[pxi(1 − p)1−xi] =

n

∑

i=1

[xi log p + (1 − xi) log(1 − p)] log p

n i

xi log p n

n i

xi

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 18 / 24

SLIDE 59

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Method Using Corollary 7.3.15

L(p|x) =

n

∏

i=1

pxi(1 − p)1−xi log L(p|x) = log

n

∏

i=1

pxi(1 − p)1−xi =

n

∑

i=1

log[pxi(1 − p)1−xi] =

n

∑

i=1

[xi log p + (1 − xi) log(1 − p)] = log p

n

∑

i=1

xi + log(1 − p)(n −

n

∑

i=1

xi)

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 18 / 24

SLIDE 60

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Method Using Corollary 7.3.15 (cont’d)

∂ ∂p log L(p|x) = ∑n

i=1 xi

p − n − ∑n

i=1 xi

1 − p nx p n x p p nx np x p p n x p p p a p W x p where a p

n p p , W x

x, p

p. Therefore, X is the best

unbiased estimator for p and attains the Cramer-Rao lower bound.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 19 / 24

SLIDE 61

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Method Using Corollary 7.3.15 (cont’d)

∂ ∂p log L(p|x) = ∑n

i=1 xi

p − n − ∑n

i=1 xi

1 − p = nx p − n(1 − x) 1 − p p nx np x p p n x p p p a p W x p where a p

n p p , W x

x, p

p. Therefore, X is the best

unbiased estimator for p and attains the Cramer-Rao lower bound.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 19 / 24

SLIDE 62

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Method Using Corollary 7.3.15 (cont’d)

∂ ∂p log L(p|x) = ∑n

i=1 xi

p − n − ∑n

i=1 xi

1 − p = nx p − n(1 − x) 1 − p = (1 − p)nx − np(1 − x) p(1 − p) n x p p p a p W x p where a p

n p p , W x

x, p

p. Therefore, X is the best

unbiased estimator for p and attains the Cramer-Rao lower bound.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 19 / 24

SLIDE 63

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Method Using Corollary 7.3.15 (cont’d)

∂ ∂p log L(p|x) = ∑n

i=1 xi

p − n − ∑n

i=1 xi

1 − p = nx p − n(1 − x) 1 − p = (1 − p)nx − np(1 − x) p(1 − p) = n(x − p) p(1 − p) a p W x p where a p

n p p , W x

x, p

p. Therefore, X is the best

unbiased estimator for p and attains the Cramer-Rao lower bound.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 19 / 24

SLIDE 64

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Method Using Corollary 7.3.15 (cont’d)

∂ ∂p log L(p|x) = ∑n

i=1 xi

p − n − ∑n

i=1 xi

1 − p = nx p − n(1 − x) 1 − p = (1 − p)nx − np(1 − x) p(1 − p) = n(x − p) p(1 − p) = a(p)[W(x) − τ(p)] where a p

n p p , W x

x, p

p. Therefore, X is the best

unbiased estimator for p and attains the Cramer-Rao lower bound.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 19 / 24

SLIDE 65

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Method Using Corollary 7.3.15 (cont’d)

∂ ∂p log L(p|x) = ∑n

i=1 xi

p − n − ∑n

i=1 xi

1 − p = nx p − n(1 − x) 1 − p = (1 − p)nx − np(1 − x) p(1 − p) = n(x − p) p(1 − p) = a(p)[W(x) − τ(p)] where a(p) =

n p(1−p), W(x) = x, τ(p) = p.

Therefore, X is the best unbiased estimator for p and attains the Cramer-Rao lower bound.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 19 / 24

SLIDE 66

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Method Using Corollary 7.3.15 (cont’d)

∂ ∂p log L(p|x) = ∑n

i=1 xi

p − n − ∑n

i=1 xi

1 − p = nx p − n(1 − x) 1 − p = (1 − p)nx − np(1 − x) p(1 − p) = n(x − p) p(1 − p) = a(p)[W(x) − τ(p)] where a(p) =

n p(1−p), W(x) = x, τ(p) = p. Therefore, X is the best

unbiased estimator for p and attains the Cramer-Rao lower bound.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 19 / 24

SLIDE 67

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Normal distribution example

.

Problem

. . X1, · · · , Xn

i.i.d.

∼ N(µ, σ2). Consider estimating σ2, assuming µ is known.

Is Cramer-Rao bound attainable? .

Solution

. . . . . . . . I E log fX X p f x exp x log f x log x log f x x

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 20 / 24

SLIDE 68

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Normal distribution example

.

Problem

. . X1, · · · , Xn

i.i.d.

∼ N(µ, σ2). Consider estimating σ2, assuming µ is known.

Is Cramer-Rao bound attainable? .

Solution

. . I(σ2) = −E [ ∂2 ∂(σ2)2 log fX(X|µ, σ)|p ] f x exp x log f x log x log f x x

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 20 / 24

SLIDE 69

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Normal distribution example

.

Problem

. . X1, · · · , Xn

i.i.d.

∼ N(µ, σ2). Consider estimating σ2, assuming µ is known.

Is Cramer-Rao bound attainable? .

Solution

. . I(σ2) = −E [ ∂2 ∂(σ2)2 log fX(X|µ, σ)|p ] f(x|µ, σ2) = 1 2πσ2 exp [ −(x − µ)2 2σ2 ] log f x log x log f x x

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 20 / 24

SLIDE 70

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Normal distribution example

.

Problem

. . X1, · · · , Xn

i.i.d.

∼ N(µ, σ2). Consider estimating σ2, assuming µ is known.

Is Cramer-Rao bound attainable? .

Solution

. . I(σ2) = −E [ ∂2 ∂(σ2)2 log fX(X|µ, σ)|p ] f(x|µ, σ2) = 1 2πσ2 exp [ −(x − µ)2 2σ2 ] log f(x|µ, σ2) = −1 2 log(2πσ2) − (x − µ)2 2σ2 log f x x

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 20 / 24

SLIDE 71

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Normal distribution example

.

Problem

. . X1, · · · , Xn

i.i.d.

∼ N(µ, σ2). Consider estimating σ2, assuming µ is known.

Is Cramer-Rao bound attainable? .

Solution

. . I(σ2) = −E [ ∂2 ∂(σ2)2 log fX(X|µ, σ)|p ] f(x|µ, σ2) = 1 2πσ2 exp [ −(x − µ)2 2σ2 ] log f(x|µ, σ2) = −1 2 log(2πσ2) − (x − µ)2 2σ2 ∂ ∂(σ2) log f(x|µ, σ2) = −1 2 1 σ2 + (x − µ)2 2(σ2)2

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 20 / 24

SLIDE 72

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Solution (cont’d)

∂2 ∂(σ2)2 log f(x|µ, σ2) = 1 2 1 (σ2)2 − 2(x − µ)2 2(σ2)3 I E x E x Cramer-Rao lower bound is nI

n . The unbiased estimator of n n i

xi x , gives Var n n So, does not attain the Cramer-Rao lower-bound.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 21 / 24

SLIDE 73

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Solution (cont’d)

∂2 ∂(σ2)2 log f(x|µ, σ2) = 1 2 1 (σ2)2 − 2(x − µ)2 2(σ2)3 I(σ2) = −E [ 1 2σ4 − 2(x − µ)2 2σ6 ] E x Cramer-Rao lower bound is nI

n . The unbiased estimator of n n i

xi x , gives Var n n So, does not attain the Cramer-Rao lower-bound.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 21 / 24

SLIDE 74

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Solution (cont’d)

∂2 ∂(σ2)2 log f(x|µ, σ2) = 1 2 1 (σ2)2 − 2(x − µ)2 2(σ2)3 I(σ2) = −E [ 1 2σ4 − 2(x − µ)2 2σ6 ] = − 1 2σ4 + 1 σ6 E[(x − µ)2] = − 1 2σ4 + 1 σ6 σ2 = 1 2σ4 Cramer-Rao lower bound is nI

n . The unbiased estimator of n n i

xi x , gives Var n n So, does not attain the Cramer-Rao lower-bound.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 21 / 24

SLIDE 75

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Solution (cont’d)

∂2 ∂(σ2)2 log f(x|µ, σ2) = 1 2 1 (σ2)2 − 2(x − µ)2 2(σ2)3 I(σ2) = −E [ 1 2σ4 − 2(x − µ)2 2σ6 ] = − 1 2σ4 + 1 σ6 E[(x − µ)2] = − 1 2σ4 + 1 σ6 σ2 = 1 2σ4 Cramer-Rao lower bound is

1 nI(σ2) = 2σ4 n .

The unbiased estimator of

n n i

xi x , gives Var n n So, does not attain the Cramer-Rao lower-bound.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 21 / 24

SLIDE 76

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Solution (cont’d)

∂2 ∂(σ2)2 log f(x|µ, σ2) = 1 2 1 (σ2)2 − 2(x − µ)2 2(σ2)3 I(σ2) = −E [ 1 2σ4 − 2(x − µ)2 2σ6 ] = − 1 2σ4 + 1 σ6 E[(x − µ)2] = − 1 2σ4 + 1 σ6 σ2 = 1 2σ4 Cramer-Rao lower bound is

1 nI(σ2) = 2σ4 n . The unbiased estimator of

ˆ σ2 =

1 n−1

∑n

i=1(xi − x)2, gives

Var n n So, does not attain the Cramer-Rao lower-bound.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 21 / 24

SLIDE 77

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Solution (cont’d)

∂2 ∂(σ2)2 log f(x|µ, σ2) = 1 2 1 (σ2)2 − 2(x − µ)2 2(σ2)3 I(σ2) = −E [ 1 2σ4 − 2(x − µ)2 2σ6 ] = − 1 2σ4 + 1 σ6 E[(x − µ)2] = − 1 2σ4 + 1 σ6 σ2 = 1 2σ4 Cramer-Rao lower bound is

1 nI(σ2) = 2σ4 n . The unbiased estimator of

ˆ σ2 =

1 n−1

∑n

i=1(xi − x)2, gives

Var( ˆ σ2) = 2σ4 n − 1 > 2σ4 n So, does not attain the Cramer-Rao lower-bound.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 21 / 24

SLIDE 78

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Solution (cont’d)

∂2 ∂(σ2)2 log f(x|µ, σ2) = 1 2 1 (σ2)2 − 2(x − µ)2 2(σ2)3 I(σ2) = −E [ 1 2σ4 − 2(x − µ)2 2σ6 ] = − 1 2σ4 + 1 σ6 E[(x − µ)2] = − 1 2σ4 + 1 σ6 σ2 = 1 2σ4 Cramer-Rao lower bound is

1 nI(σ2) = 2σ4 n . The unbiased estimator of

ˆ σ2 =

1 n−1

∑n

i=1(xi − x)2, gives

Var( ˆ σ2) = 2σ4 n − 1 > 2σ4 n So, ˆ σ2 does not attain the Cramer-Rao lower-bound.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 21 / 24

SLIDE 79

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Is Cramer-Rao lower-bound for σ2 attainable?

L(σ2|x) =

n

∏

i=1

1 √ 2πσ2 exp [ −(xi − µ)2 2σ2 ] log L x n log

n i

xi log L x n

n i

xi n

n i

xi n

n i

xi n a W x

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 22 / 24

SLIDE 80

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Is Cramer-Rao lower-bound for σ2 attainable?

L(σ2|x) =

n

∏

i=1

1 √ 2πσ2 exp [ −(xi − µ)2 2σ2 ] log L(σ2|x) = −n 2 log(2πσ2) −

n

∑

i=1

(xi − µ)2 2σ2 log L x n

n i

xi n

n i

xi n

n i

xi n a W x

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 22 / 24

SLIDE 81

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Is Cramer-Rao lower-bound for σ2 attainable?

L(σ2|x) =

n

∏

i=1

1 √ 2πσ2 exp [ −(xi − µ)2 2σ2 ] log L(σ2|x) = −n 2 log(2πσ2) −

n

∑

i=1

(xi − µ)2 2σ2 ∂ log L(σ2|x) ∂σ2 = −n 2 2π 2πσ2 +

n

∑

i=1

(xi − µ)2 2(σ2)2 n

n i

xi n

n i

xi n a W x

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 22 / 24

SLIDE 82

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Is Cramer-Rao lower-bound for σ2 attainable?

L(σ2|x) =

n

∏

i=1

1 √ 2πσ2 exp [ −(xi − µ)2 2σ2 ] log L(σ2|x) = −n 2 log(2πσ2) −

n

∑

i=1

(xi − µ)2 2σ2 ∂ log L(σ2|x) ∂σ2 = −n 2 2π 2πσ2 +

n

∑

i=1

(xi − µ)2 2(σ2)2 = − n 2σ2 +

n

∑

i=1

(xi − µ)2 2σ4 n

n i

xi n a W x

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 22 / 24

SLIDE 83

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Is Cramer-Rao lower-bound for σ2 attainable?

L(σ2|x) =

n

∏

i=1

1 √ 2πσ2 exp [ −(xi − µ)2 2σ2 ] log L(σ2|x) = −n 2 log(2πσ2) −

n

∑

i=1

(xi − µ)2 2σ2 ∂ log L(σ2|x) ∂σ2 = −n 2 2π 2πσ2 +

n

∑

i=1

(xi − µ)2 2(σ2)2 = − n 2σ2 +

n

∑

i=1

(xi − µ)2 2σ4 = n 2σ4 (∑n

i=1(xi − µ)2

n − σ2 ) a W x

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 22 / 24

SLIDE 84

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Is Cramer-Rao lower-bound for σ2 attainable?

L(σ2|x) =

n

∏

i=1

1 √ 2πσ2 exp [ −(xi − µ)2 2σ2 ] log L(σ2|x) = −n 2 log(2πσ2) −

n

∑

i=1

(xi − µ)2 2σ2 ∂ log L(σ2|x) ∂σ2 = −n 2 2π 2πσ2 +

n

∑

i=1

(xi − µ)2 2(σ2)2 = − n 2σ2 +

n

∑

i=1

(xi − µ)2 2σ4 = n 2σ4 (∑n

i=1(xi − µ)2

n − σ2 ) = a(σ2)(W(x) − σ2)

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 22 / 24

SLIDE 85

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Is Cramer-Rao lower-bound for σ2 attainable? (cont’d)

Therefore,

. . 1 If µ is known, the best unbiased estimator for σ2 is ∑n i=1(xi − µ)2/n,

and it attains the Cramer-Rao lower bound, i.e. Var

n i

Xi n n

. . 2 If

is not known, the Cramer-Rao lower-bound cannot be attained. At this point, we do not know if

n n i

xi x is the best unbiased estimator for

r not.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 23 / 24

SLIDE 86

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Is Cramer-Rao lower-bound for σ2 attainable? (cont’d)

Therefore,

. . 1 If µ is known, the best unbiased estimator for σ2 is ∑n i=1(xi − µ)2/n,

and it attains the Cramer-Rao lower bound, i.e. Var [∑n

i=1(Xi − µ)2

n ] = 2σ4 n

. 2 If

is not known, the Cramer-Rao lower-bound cannot be attained. At this point, we do not know if

n n i

xi x is the best unbiased estimator for

r not.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 23 / 24

SLIDE 87

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Is Cramer-Rao lower-bound for σ2 attainable? (cont’d)

Therefore,

. . 1 If µ is known, the best unbiased estimator for σ2 is ∑n i=1(xi − µ)2/n,

and it attains the Cramer-Rao lower bound, i.e. Var [∑n

i=1(Xi − µ)2

n ] = 2σ4 n

. . 2 If µ is not known, the Cramer-Rao lower-bound cannot be attained.

At this point, we do not know if

n n i

xi x is the best unbiased estimator for

r not.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 23 / 24

SLIDE 88

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Is Cramer-Rao lower-bound for σ2 attainable? (cont’d)

Therefore,

. . 1 If µ is known, the best unbiased estimator for σ2 is ∑n i=1(xi − µ)2/n,

and it attains the Cramer-Rao lower bound, i.e. Var [∑n

i=1(Xi − µ)2

n ] = 2σ4 n

. . 2 If µ is not known, the Cramer-Rao lower-bound cannot be attained.

At this point, we do not know if ˆ σ2 =

1 n−1

∑n

i=1(xi − x)2 is the best

unbiased estimator for σ2 or not.

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 23 / 24

SLIDE 89

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary

Summary

.

Today : Cramero-Rao Theorem

. .

Recap of Cramer-Rao Theorem and Corollary
Examples with Simple Distributions
Regularity Condition
Attainability

.

Next Lecture

. . . . . . . . • Rao-Blackwell Theorem

Hyun Min Kang Biostatistics 602 - Lecture 12 February 19th, 2013 24 / 24

SLIDE 90

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . . . . . Recap . . . Regularity Condition . . . . . . . . . . Attainability . Summary