Biostatistics 602 - Statistical Inference February 26th, 2013 - - PowerPoint PPT Presentation

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

. .

Biostatistics 602 - Statistical Inference Lecture 13 Rao-Blackwell Theorem

Hyun Min Kang February 26th, 2013

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 1 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Last Lecture

Submit your answers (after the question ID) either

• At http://pollEv.com
• By text to 22333

117261 Which family of distribution is always guaranteed to satisfy the interchangeability condition? 117322 For the rest of distributions, how can we check whether the interchangeability condition holds or not? 117325 When the become the Cramer-Rao bound attainable? HandsUp If the Cramer-Rao bound is not attainable, does it imply that the estimator cannot be UMVUE?

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 2 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Last Lecture

Submit your answers (after the question ID) either

• At http://pollEv.com
• By text to 22333

117261 Which family of distribution is always guaranteed to satisfy the interchangeability condition? 117322 For the rest of distributions, how can we check whether the interchangeability condition holds or not? 117325 When the become the Cramer-Rao bound attainable? HandsUp If the Cramer-Rao bound is not attainable, does it imply that the estimator cannot be UMVUE?

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 2 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Last Lecture

Submit your answers (after the question ID) either

• At http://pollEv.com
• By text to 22333

117261 Which family of distribution is always guaranteed to satisfy the interchangeability condition? 117322 For the rest of distributions, how can we check whether the interchangeability condition holds or not? 117325 When the become the Cramer-Rao bound attainable? HandsUp If the Cramer-Rao bound is not attainable, does it imply that the estimator cannot be UMVUE?

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 2 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Last Lecture

Submit your answers (after the question ID) either

• At http://pollEv.com
• By text to 22333

117261 Which family of distribution is always guaranteed to satisfy the interchangeability condition? 117322 For the rest of distributions, how can we check whether the interchangeability condition holds or not? 117325 When the become the Cramer-Rao bound attainable? HandsUp If the Cramer-Rao bound is not attainable, does it imply that the estimator cannot be UMVUE?

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 2 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Recap - Using Leibnitz’s Rule

.

Leibnitz’s Rule

. . d dθ ∫ b(θ)

a(θ)

f(x|θ)dx = f(b(θ)|θ)b′(θ) − f(a(θ)|θ)a′(θ) + ∫ b(θ)

a(θ)

∂ ∂θf(x|θ)dx .

Applying to Uniform Distribution

. . fX(x|θ) = 1/θ d dθ ∫ θ h(x) (1 θ ) dx = h(θ) θ dθ dθ − h(0)fX(0|θ)d0 dθ + ∫ θ ∂ ∂θh(x) (1 θ ) dx ̸= ∫ θ ∂ ∂θh(x) (1 θ ) dx The interchangeability condition is not satisfied.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 3 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Recap - When is the Cramer-Rao Lower Bound Attainable?

It is possible that the value of Cramer-Rao bound may be strictly smaller than the variance of any unbiased estimator .

Corollary 7.3.15 : Attainment of Cramer-Rao Bound

. . Let X1, · · · , Xn be iid with pdf/pmf fX(x|θ), where fX(x|θ) satisfies the assumptions of the Cramer-Rao Theorem. Let L(θ|x) = ∏n

i=1 fX(xi|θ) denote the likelihood function. If W(X) is

unbiased for τ(θ), then W(X) attains the Cramer-Rao lower bound if and

• nly if

∂ ∂θ log L(θ|x) = Sn(x|θ) = a(θ)[W(X) − t(θ)] for some function a(θ).

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 4 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Recap - Attainability of C-R bound for σ2 in N(µ, σ2)

. . 1 If µ is known, the best unbiased estimator for σ2 is ∑n i=1(xi − µ)2/n,

and it attains the Cramer-Rao lower bound, i.e. Var [∑n

i=1(Xi − µ)2

n ] = 2σ4 n

. . 2 If µ is not known, the Cramer-Rao lower-bound cannot be attained.

At this point, we do not know if ˆ σ2 =

1 n−1

∑n

i=1(xi − x)2 is the best

unbiased estimator for σ2 or not.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 5 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Fact for one-parameter exponential family

Let X1, · · · , Xn be iid from the one parameter exponential family with pdf/pmf fX(x|θ) = c(θ)h(x) exp [w(θ)t(x)]. Assume that E t X . Then n

n i

t xi , which is an unbiased estimator of , attains the Cramer-Rao lower-bound. That is, Var n

n i

t Xi In

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 6 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Fact for one-parameter exponential family

Let X1, · · · , Xn be iid from the one parameter exponential family with pdf/pmf fX(x|θ) = c(θ)h(x) exp [w(θ)t(x)]. Assume that E[t(X)] = τ(θ). Then 1

n

∑n

i=1 t(xi), which is an unbiased

estimator of τ(θ), attains the Cramer-Rao lower-bound. That is, Var n

n i

t Xi In

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 6 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Fact for one-parameter exponential family

Let X1, · · · , Xn be iid from the one parameter exponential family with pdf/pmf fX(x|θ) = c(θ)h(x) exp [w(θ)t(x)]. Assume that E[t(X)] = τ(θ). Then 1

n

∑n

i=1 t(xi), which is an unbiased

estimator of τ(θ), attains the Cramer-Rao lower-bound. That is, Var ( 1 n

n

∑

i=1

t(Xi) ) = [τ ′(θ)]2 In(θ)

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 6 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof

E [ 1 n

n

∑

i=1

t(Xi) ] = E [t(X1)] = · · · = E [t(Xn)] = τ(θ) So, n

n i

t xi is an unbiased estimator of . log L x

n i

log fX xi

n i

log c log h x w t xi

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 7 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof

E [ 1 n

n

∑

i=1

t(Xi) ] = E [t(X1)] = · · · = E [t(Xn)] = τ(θ) So, 1

n

∑n

i=1 t(xi) is an unbiased estimator of τ(θ).

log L x

n i

log fX xi

n i

log c log h x w t xi

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 7 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof

E [ 1 n

n

∑

i=1

t(Xi) ] = E [t(X1)] = · · · = E [t(Xn)] = τ(θ) So, 1

n

∑n

i=1 t(xi) is an unbiased estimator of τ(θ).

log L(θ|x) =

n

∑

i=1

log fX(xi|θ) =

n

∑

i=1

[log c(θ) + log h(x) + w(θ)t(xi)]

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 7 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof (cont’d)

∂ log L(θ|x) ∂θ =

n

∑

i=1

[c′(θ) c(θ) + 0 + w′(θ)t(xi) ] nw n

n i

t xi c c w

• n

n i

t xi is the best unbiased estimator of

c c w

• And it attains the Cramer-Rao lower bound.
• Because E

log L x ,

c c w

.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 8 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof (cont’d)

∂ log L(θ|x) ∂θ =

n

∑

i=1

[c′(θ) c(θ) + 0 + w′(θ)t(xi) ] = nw′(θ) [ 1 n

n

∑

i=1

t(xi) − { − c′(θ) c(θ)w′(θ) }]

• n

n i

t xi is the best unbiased estimator of

c c w

• And it attains the Cramer-Rao lower bound.
• Because E

log L x ,

c c w

.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 8 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof (cont’d)

∂ log L(θ|x) ∂θ =

n

∑

i=1

[c′(θ) c(θ) + 0 + w′(θ)t(xi) ] = nw′(θ) [ 1 n

n

∑

i=1

t(xi) − { − c′(θ) c(θ)w′(θ) }]

• 1

n

∑n

i=1 t(xi) is the best unbiased estimator of − c′(θ) c(θ)w′(θ)

• And it attains the Cramer-Rao lower bound.
• Because E

log L x ,

c c w

.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 8 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof (cont’d)

∂ log L(θ|x) ∂θ =

n

∑

i=1

[c′(θ) c(θ) + 0 + w′(θ)t(xi) ] = nw′(θ) [ 1 n

n

∑

i=1

t(xi) − { − c′(θ) c(θ)w′(θ) }]

• 1

n

∑n

i=1 t(xi) is the best unbiased estimator of − c′(θ) c(θ)w′(θ)

• And it attains the Cramer-Rao lower bound.
• Because E

log L x ,

c c w

.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 8 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof (cont’d)

∂ log L(θ|x) ∂θ =

n

∑

i=1

[c′(θ) c(θ) + 0 + w′(θ)t(xi) ] = nw′(θ) [ 1 n

n

∑

i=1

t(xi) − { − c′(θ) c(θ)w′(θ) }]

• 1

n

∑n

i=1 t(xi) is the best unbiased estimator of − c′(θ) c(θ)w′(θ)

• And it attains the Cramer-Rao lower bound.
• Because E

[ ∂

∂θ log L(θ|x)

] = 0, τ(θ) = −

c′(θ) c(θ)w′(θ).

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 8 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Cramer-Rao Theorem on Exponential Family

.

Fact

. . fX(x|θ) = c(θ)h(x) exp [w(θ)t(x)] If X1, · · · , Xn are iid samples from fX(x|θ), 1

n

∑n

i=1 t(Xi) is the best

unbiased estimator for its expected value. In other words, E t X Var n

n i

t Xi In

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 9 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Cramer-Rao Theorem on Exponential Family

.

Fact

. . fX(x|θ) = c(θ)h(x) exp [w(θ)t(x)] If X1, · · · , Xn are iid samples from fX(x|θ), 1

n

∑n

i=1 t(Xi) is the best

unbiased estimator for its expected value. In other words, E[t(X)] = τ(θ) Var n

n i

t Xi In

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 9 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Cramer-Rao Theorem on Exponential Family

.

Fact

. . fX(x|θ) = c(θ)h(x) exp [w(θ)t(x)] If X1, · · · , Xn are iid samples from fX(x|θ), 1

n

∑n

i=1 t(Xi) is the best

unbiased estimator for its expected value. In other words, E[t(X)] = τ(θ) Var [ 1 n

n

∑

i=1

t(Xi) ] = [τ ′(θ)]2 In(θ)

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 9 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof

∂ ∂θ log L(θ|x) = nw′(θ) [ 1 n

n

∑

i=1

t(Xi) + c′(θ) c(θ)w′(θ) ] c c w log L x a W x where a nw , W x

n n i

t xi

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 10 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof

∂ ∂θ log L(θ|x) = nw′(θ) [ 1 n

n

∑

i=1

t(Xi) + c′(θ) c(θ)w′(θ) ] τ(θ) = − c′(θ) c(θ)w′(θ) log L x a W x where a nw , W x

n n i

t xi

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 10 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof

∂ ∂θ log L(θ|x) = nw′(θ) [ 1 n

n

∑

i=1

t(Xi) + c′(θ) c(θ)w′(θ) ] τ(θ) = − c′(θ) c(θ)w′(θ) ∂ ∂θ log L(θ|x) = a(θ)[W(x) − τ(θ)] where a(θ) = nw′(θ), W(x) = 1

n

∑n

i=1 t(xi)

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 10 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Obtaining In(θ)

∂ ∂θ log L(θ|x) = nw′(θ) [ 1 n

n

∑

i=1

t(Xi) − τ(θ) ] E log L x In E nw n

n i

t Xi Var nw n

n i

t Xi n w Var n

n i

t Xi n w In

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 11 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Obtaining In(θ)

∂ ∂θ log L(θ|x) = nw′(θ) [ 1 n

n

∑

i=1

t(Xi) − τ(θ) ] E [{ ∂ ∂θ log L(θ|x) }2] = In(θ) = E  (nw′(θ))2 ( 1 n

n

∑

i=1

t(Xi) − τ(θ) )2  Var nw n

n i

t Xi n w Var n

n i

t Xi n w In

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 11 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Obtaining In(θ)

∂ ∂θ log L(θ|x) = nw′(θ) [ 1 n

n

∑

i=1

t(Xi) − τ(θ) ] E [{ ∂ ∂θ log L(θ|x) }2] = In(θ) = E  (nw′(θ))2 ( 1 n

n

∑

i=1

t(Xi) − τ(θ) )2  = Var [ nw′(θ) { 1 n

n

∑

i=1

t(Xi) − τ(θ) }] n w Var n

n i

t Xi n w In

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 11 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Obtaining In(θ)

∂ ∂θ log L(θ|x) = nw′(θ) [ 1 n

n

∑

i=1

t(Xi) − τ(θ) ] E [{ ∂ ∂θ log L(θ|x) }2] = In(θ) = E  (nw′(θ))2 ( 1 n

n

∑

i=1

t(Xi) − τ(θ) )2  = Var [ nw′(θ) { 1 n

n

∑

i=1

t(Xi) − τ(θ) }] = n2 { w′(θ) }2 Var [ 1 n

n

∑

i=1

t(Xi) ] n w In

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 11 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Obtaining In(θ)

∂ ∂θ log L(θ|x) = nw′(θ) [ 1 n

n

∑

i=1

t(Xi) − τ(θ) ] E [{ ∂ ∂θ log L(θ|x) }2] = In(θ) = E  (nw′(θ))2 ( 1 n

n

∑

i=1

t(Xi) − τ(θ) )2  = Var [ nw′(θ) { 1 n

n

∑

i=1

t(Xi) − τ(θ) }] = n2 { w′(θ) }2 Var [ 1 n

n

∑

i=1

t(Xi) ] = n2 { w′(θ) }2 [τ ′(θ)]2 In(θ)

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 11 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Obtaining In(θ)

E [{ ∂ ∂θ log L(θ|x) }2] = In(θ) n w In nw In In In In nw

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 12 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Obtaining In(θ)

E [{ ∂ ∂θ log L(θ|x) }2] = In(θ) = n2 { w′(θ) }2 [τ ′(θ)]2 In(θ) nw In In In In nw

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 12 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Obtaining In(θ)

E [{ ∂ ∂θ log L(θ|x) }2] = In(θ) = n2 { w′(θ) }2 [τ ′(θ)]2 In(θ) [ nw′(θ) ]2 = In(θ) · In(θ) [τ ′(θ)]2 In In nw

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 12 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Obtaining In(θ)

E [{ ∂ ∂θ log L(θ|x) }2] = In(θ) = n2 { w′(θ) }2 [τ ′(θ)]2 In(θ) [ nw′(θ) ]2 = In(θ) · In(θ) [τ ′(θ)]2 = (In(θ) τ ′(θ) )2 In(θ) = |nw′(θ)τ ′(θ)|

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 12 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Summary

. . 1 If ”regularity conditions” are satisfied, then we have a Cramer-Rao

bound for unbiased estimators of τ(θ).

• It helps to confirm an estimator is the best unbiased estimator of

if it happens to attain the CR-bound.

• If an unbiased estimator of

has variance greater than the CR-bound, it does NOT mean that it is not the best unbiased estimator.

. . 2 When ”regularity conditions” are not satisfied, In

is no longer a valid lower bound.

• There may be unbiased estimators of

that have variance smaller than

In

.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 13 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Summary

. . 1 If ”regularity conditions” are satisfied, then we have a Cramer-Rao

bound for unbiased estimators of τ(θ).

• It helps to confirm an estimator is the best unbiased estimator of τ(θ)

if it happens to attain the CR-bound.

• If an unbiased estimator of

has variance greater than the CR-bound, it does NOT mean that it is not the best unbiased estimator.

. . 2 When ”regularity conditions” are not satisfied, In

is no longer a valid lower bound.

• There may be unbiased estimators of

that have variance smaller than

In

.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 13 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Summary

. . 1 If ”regularity conditions” are satisfied, then we have a Cramer-Rao

bound for unbiased estimators of τ(θ).

• It helps to confirm an estimator is the best unbiased estimator of τ(θ)

if it happens to attain the CR-bound.

• If an unbiased estimator of τ(θ) has variance greater than the

CR-bound, it does NOT mean that it is not the best unbiased estimator.

. . 2 When ”regularity conditions” are not satisfied, In

is no longer a valid lower bound.

• There may be unbiased estimators of

that have variance smaller than

In

.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 13 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Summary

. . 1 If ”regularity conditions” are satisfied, then we have a Cramer-Rao

bound for unbiased estimators of τ(θ).

• It helps to confirm an estimator is the best unbiased estimator of τ(θ)

if it happens to attain the CR-bound.

• If an unbiased estimator of τ(θ) has variance greater than the

CR-bound, it does NOT mean that it is not the best unbiased estimator.

. . 2 When ”regularity conditions” are not satisfied, [τ ′(θ)]2 In(θ)

is no longer a valid lower bound.

• There may be unbiased estimators of

that have variance smaller than

In

.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 13 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Summary

. . 1 If ”regularity conditions” are satisfied, then we have a Cramer-Rao

bound for unbiased estimators of τ(θ).

• It helps to confirm an estimator is the best unbiased estimator of τ(θ)

if it happens to attain the CR-bound.

• If an unbiased estimator of τ(θ) has variance greater than the

CR-bound, it does NOT mean that it is not the best unbiased estimator.

. . 2 When ”regularity conditions” are not satisfied, [τ ′(θ)]2 In(θ)

is no longer a valid lower bound.

• There may be unbiased estimators of τ(θ) that have variance smaller

than [τ ′(θ)]2

In(θ) .

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 13 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Methods for finding best unbiased estimator

. . 1 Using Cramer-Rao bound

• How do we find the best unbiased estimator?

. . 2 Using Rao-Blackwell theorem

• Use complete and sufficient statistic.
• Find a ’better’ unbiased estimator

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 14 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Methods for finding best unbiased estimator

. . 1 Using Cramer-Rao bound

• How do we find the best unbiased estimator?

. . 2 Using Rao-Blackwell theorem

• Use complete and sufficient statistic.
• Find a ’better’ unbiased estimator

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 14 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Methods for finding best unbiased estimator

. . 1 Using Cramer-Rao bound

• How do we find the best unbiased estimator?

. . 2 Using Rao-Blackwell theorem

• Use complete and sufficient statistic.
• Find a ’better’ unbiased estimator

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 14 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Methods for finding best unbiased estimator

. . 1 Using Cramer-Rao bound

• How do we find the best unbiased estimator?

. . 2 Using Rao-Blackwell theorem

• Use complete and sufficient statistic.
• Find a ’better’ unbiased estimator

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 14 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Methods for finding best unbiased estimator

. . 1 Using Cramer-Rao bound

• How do we find the best unbiased estimator?

. . 2 Using Rao-Blackwell theorem

• Use complete and sufficient statistic.
• Find a ’better’ unbiased estimator

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 14 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Important Facts

X and Y are two random variables

• E(X) = E[E(X|Y)] (Theorem 4.4.3)
• Var X

E Var X Y Var E X Y (Theorem 4.4.7)

• E g X Y

x

g x f x Y dx is a function of Y.

• If X and Y are independent, E g X Y

E g X .

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 15 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Important Facts

X and Y are two random variables

• E(X) = E[E(X|Y)] (Theorem 4.4.3)
• Var(X) = E[Var(X|Y)] + Var[E(X|Y)] (Theorem 4.4.7)
• E g X Y

x

g x f x Y dx is a function of Y.

• If X and Y are independent, E g X Y

E g X .

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 15 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Important Facts

X and Y are two random variables

• E(X) = E[E(X|Y)] (Theorem 4.4.3)
• Var(X) = E[Var(X|Y)] + Var[E(X|Y)] (Theorem 4.4.7)
• E[g(X)|Y] =

∫

x∈X g(x)f(x|Y)dx is a function of Y.

• If X and Y are independent, E g X Y

E g X .

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 15 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Important Facts

X and Y are two random variables

• E(X) = E[E(X|Y)] (Theorem 4.4.3)
• Var(X) = E[Var(X|Y)] + Var[E(X|Y)] (Theorem 4.4.7)
• E[g(X)|Y] =

∫

x∈X g(x)f(x|Y)dx is a function of Y.

• If X and Y are independent, E[g(X)|Y] = E[g(X)].

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 15 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Seeking for a better unbiased estimator

Suppose W(X) is an unbiased estimator of τ(θ). That is, E[W(X)] = τ(θ). Suppose T X is any function of X X Xn . Consider T E W X T E T E E W X T E W X (unbiased for ) Var T Var E W T Var W E Var W T Var W (smaller variance than W)

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 16 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Seeking for a better unbiased estimator

Suppose W(X) is an unbiased estimator of τ(θ). That is, E[W(X)] = τ(θ). Suppose T(X) is any function of X = (X1, · · · , Xn). Consider T E W X T E T E E W X T E W X (unbiased for ) Var T Var E W T Var W E Var W T Var W (smaller variance than W)

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 16 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Seeking for a better unbiased estimator

Suppose W(X) is an unbiased estimator of τ(θ). That is, E[W(X)] = τ(θ). Suppose T(X) is any function of X = (X1, · · · , Xn). Consider φ(T) = E(W(X)|T) E T E E W X T E W X (unbiased for ) Var T Var E W T Var W E Var W T Var W (smaller variance than W)

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 16 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Seeking for a better unbiased estimator

Suppose W(X) is an unbiased estimator of τ(θ). That is, E[W(X)] = τ(θ). Suppose T(X) is any function of X = (X1, · · · , Xn). Consider φ(T) = E(W(X)|T) E[φ(T)] = E[E(W(X)|T)] = E[W(X)] = τ(θ) (unbiased for τ(θ)) Var T Var E W T Var W E Var W T Var W (smaller variance than W)

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 16 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Seeking for a better unbiased estimator

Suppose W(X) is an unbiased estimator of τ(θ). That is, E[W(X)] = τ(θ). Suppose T(X) is any function of X = (X1, · · · , Xn). Consider φ(T) = E(W(X)|T) E[φ(T)] = E[E(W(X)|T)] = E[W(X)] = τ(θ) (unbiased for τ(θ)) Var(φ(T)) = Var[E(W|T)] Var W E Var W T Var W (smaller variance than W)

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 16 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Seeking for a better unbiased estimator

Suppose W(X) is an unbiased estimator of τ(θ). That is, E[W(X)] = τ(θ). Suppose T(X) is any function of X = (X1, · · · , Xn). Consider φ(T) = E(W(X)|T) E[φ(T)] = E[E(W(X)|T)] = E[W(X)] = τ(θ) (unbiased for τ(θ)) Var(φ(T)) = Var[E(W|T)] = Var(W) − E[Var(W|T)] Var W (smaller variance than W)

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 16 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Seeking for a better unbiased estimator

Suppose W(X) is an unbiased estimator of τ(θ). That is, E[W(X)] = τ(θ). Suppose T(X) is any function of X = (X1, · · · , Xn). Consider φ(T) = E(W(X)|T) E[φ(T)] = E[E(W(X)|T)] = E[W(X)] = τ(θ) (unbiased for τ(θ)) Var(φ(T)) = Var[E(W|T)] = Var(W) − E[Var(W|T)] ≤ Var(W) (smaller variance than W)

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 16 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

A better unbiased estimator?

Does this mean that φ(T) is a better estimator than W(X)?

. . 1 If

T is an estimator, then T is equal or better than W X .

. . 2

T E W T E W T . T may depend on , which means that T may not be an estimator.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 17 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

A better unbiased estimator?

Does this mean that φ(T) is a better estimator than W(X)?

. . 1 If φ(T) is an estimator, then φ(T) is equal or better than W(X). . . 2

T E W T E W T . T may depend on , which means that T may not be an estimator.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 17 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

A better unbiased estimator?

Does this mean that φ(T) is a better estimator than W(X)?

. . 1 If φ(T) is an estimator, then φ(T) is equal or better than W(X). . . 2 φ(T) = E[W|T] = E[W|T, θ].

φ(T) may depend on θ, which means that φ(T) may not be an estimator.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 17 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Example 1

Let X1, · · · , Xn

i.i.d.

∼ N(θ, 1). W(X) = 1

2(X1 + X2) is an unbiased

estimator of θ. Consider conditioning it on T(X) = X1. φ(T) = E[W|T] = E [1 2(X1 + X2)|X1 ] E X X E X X X E X X

• E

T (unbiased)

• Var

T Var X X

• But

T is NOT an estimator.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 18 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Example 1

Let X1, · · · , Xn

i.i.d.

∼ N(θ, 1). W(X) = 1

2(X1 + X2) is an unbiased

estimator of θ. Consider conditioning it on T(X) = X1. φ(T) = E[W|T] = E [1 2(X1 + X2)|X1 ] = 1 2E(X1|X1) + 1 2E(X2|X1) X E X X

• E

T (unbiased)

• Var

T Var X X

• But

T is NOT an estimator.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 18 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Example 1

Let X1, · · · , Xn

i.i.d.

∼ N(θ, 1). W(X) = 1

2(X1 + X2) is an unbiased

estimator of θ. Consider conditioning it on T(X) = X1. φ(T) = E[W|T] = E [1 2(X1 + X2)|X1 ] = 1 2E(X1|X1) + 1 2E(X2|X1) = 1 2X1 + 1 2E(X2) X

• E

T (unbiased)

• Var

T Var X X

• But

T is NOT an estimator.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 18 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Example 1

Let X1, · · · , Xn

i.i.d.

∼ N(θ, 1). W(X) = 1

2(X1 + X2) is an unbiased

estimator of θ. Consider conditioning it on T(X) = X1. φ(T) = E[W|T] = E [1 2(X1 + X2)|X1 ] = 1 2E(X1|X1) + 1 2E(X2|X1) = 1 2X1 + 1 2E(X2) = 1 2X1 + 1 2θ

• E

T (unbiased)

• Var

T Var X X

• But

T is NOT an estimator.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 18 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Example 1

Let X1, · · · , Xn

i.i.d.

∼ N(θ, 1). W(X) = 1

2(X1 + X2) is an unbiased

estimator of θ. Consider conditioning it on T(X) = X1. φ(T) = E[W|T] = E [1 2(X1 + X2)|X1 ] = 1 2E(X1|X1) + 1 2E(X2|X1) = 1 2X1 + 1 2E(X2) = 1 2X1 + 1 2θ

• E[φ(T)] = 1

2θ + 1 2θ = θ (unbiased)

• Var

T Var X X

• But

T is NOT an estimator.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 18 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Example 1

Let X1, · · · , Xn

i.i.d.

∼ N(θ, 1). W(X) = 1

2(X1 + X2) is an unbiased

estimator of θ. Consider conditioning it on T(X) = X1. φ(T) = E[W|T] = E [1 2(X1 + X2)|X1 ] = 1 2E(X1|X1) + 1 2E(X2|X1) = 1 2X1 + 1 2E(X2) = 1 2X1 + 1 2θ

• E[φ(T)] = 1

2θ + 1 2θ = θ (unbiased)

• Var[φ(T)] = 1

4 < Var( 1 2(X1 + X2)) = 1 2

• But

T is NOT an estimator.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 18 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Example 1

Let X1, · · · , Xn

i.i.d.

∼ N(θ, 1). W(X) = 1

2(X1 + X2) is an unbiased

estimator of θ. Consider conditioning it on T(X) = X1. φ(T) = E[W|T] = E [1 2(X1 + X2)|X1 ] = 1 2E(X1|X1) + 1 2E(X2|X1) = 1 2X1 + 1 2E(X2) = 1 2X1 + 1 2θ

• E[φ(T)] = 1

2θ + 1 2θ = θ (unbiased)

• Var[φ(T)] = 1

4 < Var( 1 2(X1 + X2)) = 1 2

• But φ(T) is NOT an estimator.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 18 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Example 2

Let X1, · · · , Xn

i.i.d.

∼ N(θ, 1). W(X) = X1 is an unbiased estimator of θ.

Consider conditioning it on X. T E W T E X X E X X E X X E Xn X n E X Xn X n E nX X n nX n X

• E

T (unbiased)

• Var

T

Var X n n

Var W

• T is an estimator.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 19 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Example 2

Let X1, · · · , Xn

i.i.d.

∼ N(θ, 1). W(X) = X1 is an unbiased estimator of θ.

Consider conditioning it on X. φ(T) = E[W|T] = E(X1|X) E X X E X X E Xn X n E X Xn X n E nX X n nX n X

• E

T (unbiased)

• Var

T

Var X n n

Var W

• T is an estimator.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 19 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Example 2

Let X1, · · · , Xn

i.i.d.

∼ N(θ, 1). W(X) = X1 is an unbiased estimator of θ.

Consider conditioning it on X. φ(T) = E[W|T] = E(X1|X) = E(X1|X) + E(X2|X) + · · · + E(Xn|X) n E X Xn X n E nX X n nX n X

• E

T (unbiased)

• Var

T

Var X n n

Var W

• T is an estimator.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 19 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Example 2

Let X1, · · · , Xn

i.i.d.

∼ N(θ, 1). W(X) = X1 is an unbiased estimator of θ.

Consider conditioning it on X. φ(T) = E[W|T] = E(X1|X) = E(X1|X) + E(X2|X) + · · · + E(Xn|X) n = E(X1 + · · · + Xn|X) n E nX X n nX n X

• E

T (unbiased)

• Var

T

Var X n n

Var W

• T is an estimator.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 19 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Example 2

Let X1, · · · , Xn

i.i.d.

∼ N(θ, 1). W(X) = X1 is an unbiased estimator of θ.

Consider conditioning it on X. φ(T) = E[W|T] = E(X1|X) = E(X1|X) + E(X2|X) + · · · + E(Xn|X) n = E(X1 + · · · + Xn|X) n = E(nX|X) n = nX n = X

• E[φ(T)] = θ (unbiased)
• Var

T

Var X n n

Var W

• T is an estimator.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 19 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Example 2

Let X1, · · · , Xn

i.i.d.

∼ N(θ, 1). W(X) = X1 is an unbiased estimator of θ.

Consider conditioning it on X. φ(T) = E[W|T] = E(X1|X) = E(X1|X) + E(X2|X) + · · · + E(Xn|X) n = E(X1 + · · · + Xn|X) n = E(nX|X) n = nX n = X

• E[φ(T)] = θ (unbiased)
• Var[φ(T)] = Var(X)

n

= 1

n < Var(W) = 1

• T is an estimator.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 19 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Example 2

Let X1, · · · , Xn

i.i.d.

∼ N(θ, 1). W(X) = X1 is an unbiased estimator of θ.

Consider conditioning it on X. φ(T) = E[W|T] = E(X1|X) = E(X1|X) + E(X2|X) + · · · + E(Xn|X) n = E(X1 + · · · + Xn|X) n = E(nX|X) n = nX n = X

• E[φ(T)] = θ (unbiased)
• Var[φ(T)] = Var(X)

n

= 1

n < Var(W) = 1

• φ(T) is an estimator.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 19 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Rao-Blackwell Theorem

.

Theorem 7.3.17

. . Let W(X) be any unbiased estimator of τ(θ), and T be a sufficient statistic for θ. Define T E W T . Then the followings hold.

. . 1 E

T

. . 2 Var

T Var W for all . That is, T is a uniformly better unbiased estimator of .

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 20 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Rao-Blackwell Theorem

.

Theorem 7.3.17

. . Let W(X) be any unbiased estimator of τ(θ), and T be a sufficient statistic for θ. Define φ(T) = E[W|T]. Then the followings hold.

. 1 E

T

. . 2 Var

T Var W for all . That is, T is a uniformly better unbiased estimator of .

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 20 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Rao-Blackwell Theorem

.

Theorem 7.3.17

. . Let W(X) be any unbiased estimator of τ(θ), and T be a sufficient statistic for θ. Define φ(T) = E[W|T]. Then the followings hold.

. . 1 E[φ(T)|θ] = τ(θ) . 2 Var

T Var W for all . That is, T is a uniformly better unbiased estimator of .

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 20 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Rao-Blackwell Theorem

.

Theorem 7.3.17

. . Let W(X) be any unbiased estimator of τ(θ), and T be a sufficient statistic for θ. Define φ(T) = E[W|T]. Then the followings hold.

. . 1 E[φ(T)|θ] = τ(θ) . . 2 Var[φ(T)|θ] ≤ Var(W|θ) for all θ.

That is, T is a uniformly better unbiased estimator of .

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 20 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Rao-Blackwell Theorem

.

Theorem 7.3.17

. . Let W(X) be any unbiased estimator of τ(θ), and T be a sufficient statistic for θ. Define φ(T) = E[W|T]. Then the followings hold.

. . 1 E[φ(T)|θ] = τ(θ) . . 2 Var[φ(T)|θ] ≤ Var(W|θ) for all θ.

That is, φ(T) is a uniformly better unbiased estimator of τ(θ).

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 20 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof of Rao-Blackwell Theorem

. . 1 E[φ(T)] = E[E(W|T)] = E(W) = τ(θ) (unbiased) . . 2 Var

T Var E W T Var W E Var W T Var W (better than W).

. . 3 Need to show

T is indeed an estimator. T E W T E W X T

x

W x f x T dx Because T is a sufficient statistic, f x T does not depend on . Therefore, T

x

W x f x T dx does not depend on , and T is indeed an estimator of .

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 21 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof of Rao-Blackwell Theorem

. . 1 E[φ(T)] = E[E(W|T)] = E(W) = τ(θ) (unbiased) . . 2 Var[φ(T)] = Var[E(W|T)] = Var(W) − E[Var(W|T)] ≤ Var(W)

(better than W).

. 3 Need to show

T is indeed an estimator. T E W T E W X T

x

W x f x T dx Because T is a sufficient statistic, f x T does not depend on . Therefore, T

x

W x f x T dx does not depend on , and T is indeed an estimator of .

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 21 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof of Rao-Blackwell Theorem

. . 1 E[φ(T)] = E[E(W|T)] = E(W) = τ(θ) (unbiased) . . 2 Var[φ(T)] = Var[E(W|T)] = Var(W) − E[Var(W|T)] ≤ Var(W)

(better than W).

. . 3 Need to show φ(T) is indeed an estimator.

T E W T E W X T

x

W x f x T dx Because T is a sufficient statistic, f x T does not depend on . Therefore, T

x

W x f x T dx does not depend on , and T is indeed an estimator of .

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 21 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof of Rao-Blackwell Theorem

. . 1 E[φ(T)] = E[E(W|T)] = E(W) = τ(θ) (unbiased) . . 2 Var[φ(T)] = Var[E(W|T)] = Var(W) − E[Var(W|T)] ≤ Var(W)

(better than W).

. . 3 Need to show φ(T) is indeed an estimator.

φ(T) = E(W|T) = E[W(X)|T] = ∫

x∈X

W(x)f(x|T)dx Because T is a sufficient statistic, f x T does not depend on . Therefore, T

x

W x f x T dx does not depend on , and T is indeed an estimator of .

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 21 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof of Rao-Blackwell Theorem

. . 1 E[φ(T)] = E[E(W|T)] = E(W) = τ(θ) (unbiased) . . 2 Var[φ(T)] = Var[E(W|T)] = Var(W) − E[Var(W|T)] ≤ Var(W)

(better than W).

. . 3 Need to show φ(T) is indeed an estimator.

φ(T) = E(W|T) = E[W(X)|T] = ∫

x∈X

W(x)f(x|T)dx Because T is a sufficient statistic, f(x|T) does not depend on θ. Therefore, T

x

W x f x T dx does not depend on , and T is indeed an estimator of .

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 21 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof of Rao-Blackwell Theorem

. . 1 E[φ(T)] = E[E(W|T)] = E(W) = τ(θ) (unbiased) . . 2 Var[φ(T)] = Var[E(W|T)] = Var(W) − E[Var(W|T)] ≤ Var(W)

(better than W).

. . 3 Need to show φ(T) is indeed an estimator.

φ(T) = E(W|T) = E[W(X)|T] = ∫

x∈X

W(x)f(x|T)dx Because T is a sufficient statistic, f(x|T) does not depend on θ. Therefore, φ(T) = ∫

x∈X W(x)f(x|T)dx does not depend on θ,

and T is indeed an estimator of .

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 21 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof of Rao-Blackwell Theorem

. . 1 E[φ(T)] = E[E(W|T)] = E(W) = τ(θ) (unbiased) . . 2 Var[φ(T)] = Var[E(W|T)] = Var(W) − E[Var(W|T)] ≤ Var(W)

(better than W).

. . 3 Need to show φ(T) is indeed an estimator.

φ(T) = E(W|T) = E[W(X)|T] = ∫

x∈X

W(x)f(x|T)dx Because T is a sufficient statistic, f(x|T) does not depend on θ. Therefore, φ(T) = ∫

x∈X W(x)f(x|T)dx does not depend on θ, and φ(T) is indeed an

estimator of θ.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 21 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Uniqueness of UMVUE

.

Theorem 7.3.19

. . If W is a best unbiased estimator of τ(θ), then W is unique. .

Proof

. . . . . . . . Suppose W and W are two best unbiased estimators of . Consider estimator W W W . E W E W W Var W Var W W Var W Var W Cov W W Var W Var W Var W Var W Var W Var W Therefore W is better or equal to W and W .

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 22 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Uniqueness of UMVUE

.

Theorem 7.3.19

. . If W is a best unbiased estimator of τ(θ), then W is unique. .

Proof

. . Suppose W1 and W2 are two best unbiased estimators of τ(θ). Consider estimator W3 = 1

2(W1 + W2).

E W E W W Var W Var W W Var W Var W Cov W W Var W Var W Var W Var W Var W Var W Therefore W is better or equal to W and W .

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 22 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Uniqueness of UMVUE

.

Theorem 7.3.19

. . If W is a best unbiased estimator of τ(θ), then W is unique. .

Proof

. . Suppose W1 and W2 are two best unbiased estimators of τ(θ). Consider estimator W3 = 1

2(W1 + W2).

E(W3) = E (1 2W1 + 1 2W2 ) = 1 2τ(θ) + 1 2τ(θ) = τ(θ) Var W Var W W Var W Var W Cov W W Var W Var W Var W Var W Var W Var W Therefore W is better or equal to W and W .

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 22 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Uniqueness of UMVUE

.

Theorem 7.3.19

. . If W is a best unbiased estimator of τ(θ), then W is unique. .

Proof

. . Suppose W1 and W2 are two best unbiased estimators of τ(θ). Consider estimator W3 = 1

2(W1 + W2).

E(W3) = E (1 2W1 + 1 2W2 ) = 1 2τ(θ) + 1 2τ(θ) = τ(θ) Var(W3) = Var (1 2W1 + 1 2W2 ) Var W Var W Cov W W Var W Var W Var W Var W Var W Var W Therefore W is better or equal to W and W .

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 22 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Uniqueness of UMVUE

.

Theorem 7.3.19

. . If W is a best unbiased estimator of τ(θ), then W is unique. .

Proof

. . Suppose W1 and W2 are two best unbiased estimators of τ(θ). Consider estimator W3 = 1

2(W1 + W2).

E(W3) = E (1 2W1 + 1 2W2 ) = 1 2τ(θ) + 1 2τ(θ) = τ(θ) Var(W3) = Var (1 2W1 + 1 2W2 ) = 1 4Var(W1) + 1 4Var(W2) + 1 2Cov(W1, W2) Var W Var W Var W Var W Var W Var W Therefore W is better or equal to W and W .

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 22 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Uniqueness of UMVUE

.

Theorem 7.3.19

. . If W is a best unbiased estimator of τ(θ), then W is unique. .

Proof

. . Suppose W1 and W2 are two best unbiased estimators of τ(θ). Consider estimator W3 = 1

2(W1 + W2).

E(W3) = E (1 2W1 + 1 2W2 ) = 1 2τ(θ) + 1 2τ(θ) = τ(θ) Var(W3) = Var (1 2W1 + 1 2W2 ) = 1 4Var(W1) + 1 4Var(W2) + 1 2Cov(W1, W2) ≤ 1 4Var(W1) + 1 4Var(W2) + 1 2 √ Var(W1)Var(W2) Var W Var W Therefore W is better or equal to W and W .

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 22 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Uniqueness of UMVUE

.

Theorem 7.3.19

. . If W is a best unbiased estimator of τ(θ), then W is unique. .

Proof

. . Suppose W1 and W2 are two best unbiased estimators of τ(θ). Consider estimator W3 = 1

2(W1 + W2).

E(W3) = E (1 2W1 + 1 2W2 ) = 1 2τ(θ) + 1 2τ(θ) = τ(θ) Var(W3) = Var (1 2W1 + 1 2W2 ) = 1 4Var(W1) + 1 4Var(W2) + 1 2Cov(W1, W2) ≤ 1 4Var(W1) + 1 4Var(W2) + 1 2 √ Var(W1)Var(W2) = Var(W1) = Var(W2) Therefore W is better or equal to W and W .

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 22 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Uniqueness of UMVUE

.

Theorem 7.3.19

. . If W is a best unbiased estimator of τ(θ), then W is unique. .

Proof

. . Suppose W1 and W2 are two best unbiased estimators of τ(θ). Consider estimator W3 = 1

2(W1 + W2).

E(W3) = E (1 2W1 + 1 2W2 ) = 1 2τ(θ) + 1 2τ(θ) = τ(θ) Var(W3) = Var (1 2W1 + 1 2W2 ) = 1 4Var(W1) + 1 4Var(W2) + 1 2Cov(W1, W2) ≤ 1 4Var(W1) + 1 4Var(W2) + 1 2 √ Var(W1)Var(W2) = Var(W1) = Var(W2) Therefore W is better or equal to W and W .

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 22 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof of Theorem 7.3.19 (cont’d)

Var(W3) ≤ Var(W1) = Var(W2). If strict inequality holds, W is better than W and W , which is contradictory to the assumption. Therefore, the equality must hold, requiring Cov W W Var W Var W By Cauchy-Schwarz inequality, this is true if and only if W aW b Cov W W Cov W aW b aVar W Var W Var W Var W E W a b a b must hold, and W W . Therefore, the best unbiased estimator is unique.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 23 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof of Theorem 7.3.19 (cont’d)

Var(W3) ≤ Var(W1) = Var(W2). If strict inequality holds, W3 is better than W1 and W2, which is contradictory to the assumption. Therefore, the equality must hold, requiring Cov W W Var W Var W By Cauchy-Schwarz inequality, this is true if and only if W aW b Cov W W Cov W aW b aVar W Var W Var W Var W E W a b a b must hold, and W W . Therefore, the best unbiased estimator is unique.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 23 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof of Theorem 7.3.19 (cont’d)

Var(W3) ≤ Var(W1) = Var(W2). If strict inequality holds, W3 is better than W1 and W2, which is contradictory to the assumption. Therefore, the equality must hold, requiring 1 2Cov(W1, W2) = 1 2 √ Var(W1)Var(W2) By Cauchy-Schwarz inequality, this is true if and only if W aW b Cov W W Cov W aW b aVar W Var W Var W Var W E W a b a b must hold, and W W . Therefore, the best unbiased estimator is unique.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 23 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof of Theorem 7.3.19 (cont’d)

Var(W3) ≤ Var(W1) = Var(W2). If strict inequality holds, W3 is better than W1 and W2, which is contradictory to the assumption. Therefore, the equality must hold, requiring 1 2Cov(W1, W2) = 1 2 √ Var(W1)Var(W2) By Cauchy-Schwarz inequality, this is true if and only if W2 = aW1 + b Cov W W Cov W aW b aVar W Var W Var W Var W E W a b a b must hold, and W W . Therefore, the best unbiased estimator is unique.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 23 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof of Theorem 7.3.19 (cont’d)

Var(W3) ≤ Var(W1) = Var(W2). If strict inequality holds, W3 is better than W1 and W2, which is contradictory to the assumption. Therefore, the equality must hold, requiring 1 2Cov(W1, W2) = 1 2 √ Var(W1)Var(W2) By Cauchy-Schwarz inequality, this is true if and only if W2 = aW1 + b Cov(W1, W2) = Cov(W1, aW1 + b) = aVar(W1) Var W Var W Var W E W a b a b must hold, and W W . Therefore, the best unbiased estimator is unique.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 23 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof of Theorem 7.3.19 (cont’d)

Var(W3) ≤ Var(W1) = Var(W2). If strict inequality holds, W3 is better than W1 and W2, which is contradictory to the assumption. Therefore, the equality must hold, requiring 1 2Cov(W1, W2) = 1 2 √ Var(W1)Var(W2) By Cauchy-Schwarz inequality, this is true if and only if W2 = aW1 + b Cov(W1, W2) = Cov(W1, aW1 + b) = aVar(W1) = Var(W1)Var(W2) = Var(W1) E W a b a b must hold, and W W . Therefore, the best unbiased estimator is unique.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 23 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof of Theorem 7.3.19 (cont’d)

Var(W3) ≤ Var(W1) = Var(W2). If strict inequality holds, W3 is better than W1 and W2, which is contradictory to the assumption. Therefore, the equality must hold, requiring 1 2Cov(W1, W2) = 1 2 √ Var(W1)Var(W2) By Cauchy-Schwarz inequality, this is true if and only if W2 = aW1 + b Cov(W1, W2) = Cov(W1, aW1 + b) = aVar(W1) = Var(W1)Var(W2) = Var(W1) E(W2) = aτ(θ) + b a b must hold, and W W . Therefore, the best unbiased estimator is unique.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 23 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof of Theorem 7.3.19 (cont’d)

Var(W3) ≤ Var(W1) = Var(W2). If strict inequality holds, W3 is better than W1 and W2, which is contradictory to the assumption. Therefore, the equality must hold, requiring 1 2Cov(W1, W2) = 1 2 √ Var(W1)Var(W2) By Cauchy-Schwarz inequality, this is true if and only if W2 = aW1 + b Cov(W1, W2) = Cov(W1, aW1 + b) = aVar(W1) = Var(W1)Var(W2) = Var(W1) E(W2) = aτ(θ) + b = τ(θ) a = 1, b = 0 must hold, and W2 = W1. Therefore, the best unbiased estimator is unique.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 23 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Unbiased estimator of zero

.

Definition

. . If U(X) satisfies E(U) = 0. Then we call U an unbiased estimator of 0. .

Theorem 7.3.20

. . . . . . . . If E W X . W is the best unbiased estimator of if an only if W is uncorrelated with all unbiased estimator of .

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 24 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Unbiased estimator of zero

.

Definition

. . If U(X) satisfies E(U) = 0. Then we call U an unbiased estimator of 0. .

Theorem 7.3.20

. . If E[W(X)] = τ(θ). W is the best unbiased estimator of τ(θ) if an only if W is uncorrelated with all unbiased estimator of 0.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 24 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof of Theorem 7.3.20

Let W be an unbiased estimator of τ(θ). Let V = W + U and U ∈ U, which is the class of unbiased estimators of 0. By construction, V is an unbiased estimator of . Consider Va W aU where a is a constant. E Va E W aU E W aE U a Var Va Var W aU a Var U aCov W U Var W

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 25 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof of Theorem 7.3.20

Let W be an unbiased estimator of τ(θ). Let V = W + U and U ∈ U, which is the class of unbiased estimators of 0. By construction, V is an unbiased estimator of τ(θ). Consider Va W aU where a is a constant. E Va E W aU E W aE U a Var Va Var W aU a Var U aCov W U Var W

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 25 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof of Theorem 7.3.20

Let W be an unbiased estimator of τ(θ). Let V = W + U and U ∈ U, which is the class of unbiased estimators of 0. By construction, V is an unbiased estimator of τ(θ). Consider V = {Va = W + aU} where a is a constant. E Va E W aU E W aE U a Var Va Var W aU a Var U aCov W U Var W

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 25 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof of Theorem 7.3.20

Let W be an unbiased estimator of τ(θ). Let V = W + U and U ∈ U, which is the class of unbiased estimators of 0. By construction, V is an unbiased estimator of τ(θ). Consider V = {Va = W + aU} where a is a constant. E(Va) = E(W + aU) = E(W) + aE(U) a Var Va Var W aU a Var U aCov W U Var W

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 25 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof of Theorem 7.3.20

Let W be an unbiased estimator of τ(θ). Let V = W + U and U ∈ U, which is the class of unbiased estimators of 0. By construction, V is an unbiased estimator of τ(θ). Consider V = {Va = W + aU} where a is a constant. E(Va) = E(W + aU) = E(W) + aE(U) = τ(θ) + a · 0 = τ(θ) Var Va Var W aU a Var U aCov W U Var W

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 25 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof of Theorem 7.3.20

Let W be an unbiased estimator of τ(θ). Let V = W + U and U ∈ U, which is the class of unbiased estimators of 0. By construction, V is an unbiased estimator of τ(θ). Consider V = {Va = W + aU} where a is a constant. E(Va) = E(W + aU) = E(W) + aE(U) = τ(θ) + a · 0 = τ(θ) Var(Va) = Var(W + aU) a Var U aCov W U Var W

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 25 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof of Theorem 7.3.20

Let W be an unbiased estimator of τ(θ). Let V = W + U and U ∈ U, which is the class of unbiased estimators of 0. By construction, V is an unbiased estimator of τ(θ). Consider V = {Va = W + aU} where a is a constant. E(Va) = E(W + aU) = E(W) + aE(U) = τ(θ) + a · 0 = τ(θ) Var(Va) = Var(W + aU) = a2Var(U) + 2aCov(W, U) + Var(W)

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 25 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof of Theorem 7.3.20 (cont’d)

The variance is minimized when a = −2Cov(W, U) 2Var(U) = −Cov(W, U) Var(U) The best unbiased estimator in this class is W Cov W U Var U U W is the best unbiased estimator in this class if and only if Cov W U . Therefore for W is the best among all unbiased estimators of if and only if Cov W U for every U .

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 26 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof of Theorem 7.3.20 (cont’d)

The variance is minimized when a = −2Cov(W, U) 2Var(U) = −Cov(W, U) Var(U) The best unbiased estimator in this class is W − Cov(W, U) Var(U) U W is the best unbiased estimator in this class if and only if Cov W U . Therefore for W is the best among all unbiased estimators of if and only if Cov W U for every U .

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 26 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof of Theorem 7.3.20 (cont’d)

The variance is minimized when a = −2Cov(W, U) 2Var(U) = −Cov(W, U) Var(U) The best unbiased estimator in this class is W − Cov(W, U) Var(U) U W is the best unbiased estimator in this class if and only if Cov(W, U) = 0. Therefore for W is the best among all unbiased estimators of if and only if Cov W U for every U .

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 26 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Proof of Theorem 7.3.20 (cont’d)

The variance is minimized when a = −2Cov(W, U) 2Var(U) = −Cov(W, U) Var(U) The best unbiased estimator in this class is W − Cov(W, U) Var(U) U W is the best unbiased estimator in this class if and only if Cov(W, U) = 0. Therefore for W is the best among all unbiased estimators of τ(θ) if and only if Cov(W, U) = 0 for every U ∈ U.

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 26 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Summary

.

Today

. .

• Cramer-Rao Theorem with single parameter exponential family.
• Rao-Blackwell Theorem

.

Next Lecture

. . . . . . . . • More Rao-Blackwell Theorem

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 27 / 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . Recap . . . . . . . . Exponential Family . . . . . . . . . . . . . Rao-Blackwell . Summary

Summary

.

Today

. .

• Cramer-Rao Theorem with single parameter exponential family.
• Rao-Blackwell Theorem

.

Next Lecture

. . • More Rao-Blackwell Theorem

Hyun Min Kang Biostatistics 602 - Lecture 13 February 26th, 2013 27 / 27