[PPT] - Biostatistics 602 - Statistical Inference April 16th, 2013 PowerPoint Presentation

SLIDE 1

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

. .

Biostatistics 602 - Statistical Inference Lecture 24 E-M Algorithm & Practice Examples

Hyun Min Kang April 16th, 2013

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 1 / 33

SLIDE 2

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Last Lecture

What is an interval estimator?
What is the coverage probability, confidence coefficient, and

confidence interval?

How can a

confidence interval typically be constructed?

To obtain a lower-bounded (upper-tail) CI, whose acceptance region
f a test should be inverted?

(a) H vs H (b) H vs H

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 2 / 33

SLIDE 3

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Last Lecture

What is an interval estimator?
What is the coverage probability, confidence coefficient, and

confidence interval?

How can a

confidence interval typically be constructed?

To obtain a lower-bounded (upper-tail) CI, whose acceptance region
f a test should be inverted?

(a) H vs H (b) H vs H

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 2 / 33

SLIDE 4

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Last Lecture

What is an interval estimator?
What is the coverage probability, confidence coefficient, and

confidence interval?

How can a

confidence interval typically be constructed?

To obtain a lower-bounded (upper-tail) CI, whose acceptance region
f a test should be inverted?

(a) H vs H (b) H vs H

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 2 / 33

SLIDE 5

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Last Lecture

What is an interval estimator?
What is the coverage probability, confidence coefficient, and

confidence interval?

How can a 1 − α confidence interval typically be constructed?
To obtain a lower-bounded (upper-tail) CI, whose acceptance region
f a test should be inverted?

(a) H vs H (b) H vs H

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 2 / 33

SLIDE 6

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Last Lecture

What is an interval estimator?
What is the coverage probability, confidence coefficient, and

confidence interval?

How can a 1 − α confidence interval typically be constructed?
To obtain a lower-bounded (upper-tail) CI, whose acceptance region
f a test should be inverted?

(a) H0 : θ = θ0 vs H1 : θ > θ0 (b) H0 : θ = θ0 vs H1 : θ < θ0

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 2 / 33

SLIDE 7

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Interval Estimation

ˆ θ(X) is usually represented as a point estimator .

Interval Estimator

. . . . . . . . Let L X U X , where L X and U X are functions of sample X and L X U X . Based on the observed sample x, we can make an inference that L X U X Then we call L X U X an interval estimator of . Three types of intervals

Two-sided interval L X

U X

One-sided (with lower-bound) interval L X
One-sided (with upper-bound) interval

U X

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 3 / 33

SLIDE 8

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Interval Estimation

ˆ θ(X) is usually represented as a point estimator .

Interval Estimator

. . Let [L(X), U(X)], where L(X) and U(X) are functions of sample X and L(X) ≤ U(X). Based on the observed sample x, we can make an inference that L X U X Then we call L X U X an interval estimator of . Three types of intervals

Two-sided interval L X

U X

One-sided (with lower-bound) interval L X
One-sided (with upper-bound) interval

U X

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 3 / 33

SLIDE 9

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Interval Estimation

ˆ θ(X) is usually represented as a point estimator .

Interval Estimator

. . Let [L(X), U(X)], where L(X) and U(X) are functions of sample X and L(X) ≤ U(X). Based on the observed sample x, we can make an inference that θ ∈ [L(X), U(X)] Then we call L X U X an interval estimator of . Three types of intervals

Two-sided interval L X

U X

One-sided (with lower-bound) interval L X
One-sided (with upper-bound) interval

U X

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 3 / 33

SLIDE 10

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Interval Estimation

ˆ θ(X) is usually represented as a point estimator .

Interval Estimator

. . Let [L(X), U(X)], where L(X) and U(X) are functions of sample X and L(X) ≤ U(X). Based on the observed sample x, we can make an inference that θ ∈ [L(X), U(X)] Then we call [L(X), U(X)] an interval estimator of θ. Three types of intervals

Two-sided interval L X

U X

One-sided (with lower-bound) interval L X
One-sided (with upper-bound) interval

U X

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 3 / 33

SLIDE 11

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Interval Estimation

ˆ θ(X) is usually represented as a point estimator .

Interval Estimator

. . Let [L(X), U(X)], where L(X) and U(X) are functions of sample X and L(X) ≤ U(X). Based on the observed sample x, we can make an inference that θ ∈ [L(X), U(X)] Then we call [L(X), U(X)] an interval estimator of θ. Three types of intervals

Two-sided interval L X

U X

One-sided (with lower-bound) interval L X
One-sided (with upper-bound) interval

U X

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 3 / 33

SLIDE 12

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Interval Estimation

ˆ θ(X) is usually represented as a point estimator .

Interval Estimator

. . Let [L(X), U(X)], where L(X) and U(X) are functions of sample X and L(X) ≤ U(X). Based on the observed sample x, we can make an inference that θ ∈ [L(X), U(X)] Then we call [L(X), U(X)] an interval estimator of θ. Three types of intervals

Two-sided interval [L(X), U(X)]
One-sided (with lower-bound) interval L X
One-sided (with upper-bound) interval

U X

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 3 / 33

SLIDE 13

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Interval Estimation

ˆ θ(X) is usually represented as a point estimator .

Interval Estimator

. . Let [L(X), U(X)], where L(X) and U(X) are functions of sample X and L(X) ≤ U(X). Based on the observed sample x, we can make an inference that θ ∈ [L(X), U(X)] Then we call [L(X), U(X)] an interval estimator of θ. Three types of intervals

Two-sided interval [L(X), U(X)]
One-sided (with lower-bound) interval [L(X), ∞)
One-sided (with upper-bound) interval

U X

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 3 / 33

SLIDE 14

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Interval Estimation

ˆ θ(X) is usually represented as a point estimator .

Interval Estimator

. . Let [L(X), U(X)], where L(X) and U(X) are functions of sample X and L(X) ≤ U(X). Based on the observed sample x, we can make an inference that θ ∈ [L(X), U(X)] Then we call [L(X), U(X)] an interval estimator of θ. Three types of intervals

Two-sided interval [L(X), U(X)]
One-sided (with lower-bound) interval [L(X), ∞)
One-sided (with upper-bound) interval (−∞, U(X)]

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 3 / 33

SLIDE 15

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Definitions

.

Definition : Coverage Probability

. . Given an interval estimator [L(X), U(X)] of θ, its coverage probability is defined as Pr L X U X In other words, the probability of a random variable in interval L X U X covers the parameter . .

Definition: Confidence Coefficient

. . . . . . . . Confidence coefficient is defined as inf Pr L X U X

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 4 / 33

SLIDE 16

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Definitions

.

Definition : Coverage Probability

. . Given an interval estimator [L(X), U(X)] of θ, its coverage probability is defined as Pr(θ ∈ [L(X), U(X)]) In other words, the probability of a random variable in interval L X U X covers the parameter . .

Definition: Confidence Coefficient

. . . . . . . . Confidence coefficient is defined as inf Pr L X U X

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 4 / 33

SLIDE 17

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Definitions

.

Definition : Coverage Probability

. . Given an interval estimator [L(X), U(X)] of θ, its coverage probability is defined as Pr(θ ∈ [L(X), U(X)]) In other words, the probability of a random variable in interval [L(X), U(X)] covers the parameter θ. .

Definition: Confidence Coefficient

. . . . . . . . Confidence coefficient is defined as inf Pr L X U X

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 4 / 33

SLIDE 18

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Definitions

.

Definition : Coverage Probability

. . Given an interval estimator [L(X), U(X)] of θ, its coverage probability is defined as Pr(θ ∈ [L(X), U(X)]) In other words, the probability of a random variable in interval [L(X), U(X)] covers the parameter θ. .

Definition: Confidence Coefficient

. . Confidence coefficient is defined as inf Pr L X U X

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 4 / 33

SLIDE 19

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Definitions

.

Definition : Coverage Probability

. . Given an interval estimator [L(X), U(X)] of θ, its coverage probability is defined as Pr(θ ∈ [L(X), U(X)]) In other words, the probability of a random variable in interval [L(X), U(X)] covers the parameter θ. .

Definition: Confidence Coefficient

. . Confidence coefficient is defined as inf

θ∈Ω Pr(θ ∈ [L(X), U(X)])

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 4 / 33

SLIDE 20

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Definitions

.

Definition : Confidence Interval

. . Given an interval estimator [L(X), U(X)] of θ, if its confidence coefficient is 1 − α, we call it a (1 − α) confidence interval .

Definition: Expected Length

. . . . . . . . Given an interval estimator L X U X

f

, its expected length is defined as E U X L X where X are random samples from fX x . In other words, it is the average length of the interval estimator.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 5 / 33

SLIDE 21

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Definitions

.

Definition : Confidence Interval

. . Given an interval estimator [L(X), U(X)] of θ, if its confidence coefficient is 1 − α, we call it a (1 − α) confidence interval .

Definition: Expected Length

. . Given an interval estimator [L(X), U(X)] of θ, its expected length is defined as E U X L X where X are random samples from fX x . In other words, it is the average length of the interval estimator.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 5 / 33

SLIDE 22

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Definitions

.

Definition : Confidence Interval

. . Given an interval estimator [L(X), U(X)] of θ, if its confidence coefficient is 1 − α, we call it a (1 − α) confidence interval .

Definition: Expected Length

. . Given an interval estimator [L(X), U(X)] of θ, its expected length is defined as E[U(X) − L(X)] where X are random samples from fX x . In other words, it is the average length of the interval estimator.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 5 / 33

SLIDE 23

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Definitions

.

Definition : Confidence Interval

. . Given an interval estimator [L(X), U(X)] of θ, if its confidence coefficient is 1 − α, we call it a (1 − α) confidence interval .

Definition: Expected Length

. . Given an interval estimator [L(X), U(X)] of θ, its expected length is defined as E[U(X) − L(X)] where X are random samples from fX(x|θ). In other words, it is the average length of the interval estimator.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 5 / 33

SLIDE 24

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Confidence set and confidence interval

There is no guarantee that the confidence set obtained from Theorem 9.2.2 is an interval, but quite often

. . 1 To obtain

two-sided CI L X U X , we invert the acceptance region of a level test for H

vs. H

. . 2 To obtain a lower-bounded CI L X

, then we invert the acceptance region of a test for H

vs. H

, where .

. . 3 To obtain a upper-bounded CI

U X , then we invert the acceptance region of a test for H

vs. H

, where .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 6 / 33

SLIDE 25

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Confidence set and confidence interval

There is no guarantee that the confidence set obtained from Theorem 9.2.2 is an interval, but quite often

. . 1 To obtain (1 − α) two-sided CI [L(X), U(X)], we invert the

acceptance region of a level α test for H0 : θ = θ0 vs. H1 : θ ̸= θ0

. . 2 To obtain a lower-bounded CI L X

, then we invert the acceptance region of a test for H

vs. H

, where .

. . 3 To obtain a upper-bounded CI

U X , then we invert the acceptance region of a test for H

vs. H

, where .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 6 / 33

SLIDE 26

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Confidence set and confidence interval

There is no guarantee that the confidence set obtained from Theorem 9.2.2 is an interval, but quite often

. . 1 To obtain (1 − α) two-sided CI [L(X), U(X)], we invert the

acceptance region of a level α test for H0 : θ = θ0 vs. H1 : θ ̸= θ0

. . 2 To obtain a lower-bounded CI [L(X), ∞), then we invert the

acceptance region of a test for H0 : θ = θ0 vs. H1 : θ > θ0, where Ω = {θ : θ ≥ θ0}.

. 3 To obtain a upper-bounded CI

U X , then we invert the acceptance region of a test for H

vs. H

, where .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 6 / 33

SLIDE 27

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Confidence set and confidence interval

There is no guarantee that the confidence set obtained from Theorem 9.2.2 is an interval, but quite often

. . 1 To obtain (1 − α) two-sided CI [L(X), U(X)], we invert the

acceptance region of a level α test for H0 : θ = θ0 vs. H1 : θ ̸= θ0

. . 2 To obtain a lower-bounded CI [L(X), ∞), then we invert the

acceptance region of a test for H0 : θ = θ0 vs. H1 : θ > θ0, where Ω = {θ : θ ≥ θ0}.

. . 3 To obtain a upper-bounded CI (−∞, U(X)], then we invert the

acceptance region of a test for H0 : θ = θ0 vs. H1 : θ < θ0, where Ω = {θ : θ ≤ θ0}.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 6 / 33

SLIDE 28

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Typical strategies for finding MLEs

. . 1 Write the joint (log-)likelihood function, L(θ|x) = fX(x|θ). . . 2 Find candidates that makes first order derivative to be zero . . 3 Check second-order derivative to check local maximum.

For one-dimensional parameter, negative second order derivative

implies local maximum.

. . 4 Check boundary points to see whether boundary gives global

maximum.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 7 / 33

SLIDE 29

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Typical strategies for finding MLEs

. . 1 Write the joint (log-)likelihood function, L(θ|x) = fX(x|θ). . . 2 Find candidates that makes first order derivative to be zero . . 3 Check second-order derivative to check local maximum.

For one-dimensional parameter, negative second order derivative

implies local maximum.

. . 4 Check boundary points to see whether boundary gives global

maximum.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 7 / 33

SLIDE 30

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Typical strategies for finding MLEs

. . 1 Write the joint (log-)likelihood function, L(θ|x) = fX(x|θ). . . 2 Find candidates that makes first order derivative to be zero . . 3 Check second-order derivative to check local maximum.

For one-dimensional parameter, negative second order derivative

implies local maximum.

. . 4 Check boundary points to see whether boundary gives global

maximum.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 7 / 33

SLIDE 31

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Typical strategies for finding MLEs

. . 1 Write the joint (log-)likelihood function, L(θ|x) = fX(x|θ). . . 2 Find candidates that makes first order derivative to be zero . . 3 Check second-order derivative to check local maximum.

For one-dimensional parameter, negative second order derivative

implies local maximum.

. . 4 Check boundary points to see whether boundary gives global

maximum.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 7 / 33

SLIDE 32

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Example: A mixture distribution

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 8 / 33

SLIDE 33

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

A general mixture distribution

f(x|π, φ, η) =

k

∑

i=1

πif(x|φi, η) x observed data mixture proportion of each component f the probability density function parameters specific to each component parameters shared among components k number of mixture components

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 9 / 33

SLIDE 34

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

A general mixture distribution

f(x|π, φ, η) =

k

∑

i=1

πif(x|φi, η) x observed data mixture proportion of each component f the probability density function parameters specific to each component parameters shared among components k number of mixture components

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 9 / 33

SLIDE 35

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

A general mixture distribution

f(x|π, φ, η) =

k

∑

i=1

πif(x|φi, η) x observed data π mixture proportion of each component f the probability density function parameters specific to each component parameters shared among components k number of mixture components

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 9 / 33

SLIDE 36

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

A general mixture distribution

f(x|π, φ, η) =

k

∑

i=1

πif(x|φi, η) x observed data π mixture proportion of each component f the probability density function parameters specific to each component parameters shared among components k number of mixture components

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 9 / 33

SLIDE 37

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

A general mixture distribution

f(x|π, φ, η) =

k

∑

i=1

πif(x|φi, η) x observed data π mixture proportion of each component f the probability density function φ parameters specific to each component parameters shared among components k number of mixture components

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 9 / 33

SLIDE 38

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

A general mixture distribution

f(x|π, φ, η) =

k

∑

i=1

πif(x|φi, η) x observed data π mixture proportion of each component f the probability density function φ parameters specific to each component η parameters shared among components k number of mixture components

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 9 / 33

SLIDE 39

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

A general mixture distribution

f(x|π, φ, η) =

k

∑

i=1

πif(x|φi, η) x observed data π mixture proportion of each component f the probability density function φ parameters specific to each component η parameters shared among components k number of mixture components

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 9 / 33

SLIDE 40

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

MLE Problem for mixture of normals

.

Problem

. . f(x|θ = (π, µ, σ2)) =

k

∑

i=1

πifi(x|µi, σ2

i )

fi x

i i i

exp x

i i n i i

Find MLEs for .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 10 / 33

SLIDE 41

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

MLE Problem for mixture of normals

.

Problem

. . f(x|θ = (π, µ, σ2)) =

k

∑

i=1

πifi(x|µi, σ2

i )

fi(x|µi, σ2

i )

= 1 √ 2πσ2

i

exp [ −(x − µi)2 2σ2

i

]

n i i

Find MLEs for .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 10 / 33

SLIDE 42

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

MLE Problem for mixture of normals

.

Problem

. . f(x|θ = (π, µ, σ2)) =

k

∑

i=1

πifi(x|µi, σ2

i )

fi(x|µi, σ2

i )

= 1 √ 2πσ2

i

exp [ −(x − µi)2 2σ2

i

]

n

∑

i=1

πi = 1 Find MLEs for .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 10 / 33

SLIDE 43

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

MLE Problem for mixture of normals

.

Problem

. . f(x|θ = (π, µ, σ2)) =

k

∑

i=1

πifi(x|µi, σ2

i )

fi(x|µi, σ2

i )

= 1 √ 2πσ2

i

exp [ −(x − µi)2 2σ2

i

]

n

∑

i=1

πi = 1 Find MLEs for θ = (π, µ, σ2).

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 10 / 33

SLIDE 44

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution when k = 1

f(x|θ) =

k

∑

i=1

πifi(x|µi, σ2

i )

x
n

i

xi x n

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 11 / 33

SLIDE 45

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution when k = 1

f(x|θ) =

k

∑

i=1

πifi(x|µi, σ2

i )

π = π1 = 1
µ = µ1 = x
σ2 = σ2

1 = ∑n i=1(xi − x)2/n

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 11 / 33

SLIDE 46

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Incomplete data problem when k > 1

f(x|θ) =

n

∏

i=1

 

k

∑

j=1

πifi(xi|µj, σ2

j )

  The MLE solution is not analytically tractable, because it involves multiple sums of exponential functions.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 12 / 33

SLIDE 47

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Incomplete data problem when k > 1

f(x|θ) =

n

∏

i=1

 

k

∑

j=1

πifi(xi|µj, σ2

j )

  The MLE solution is not analytically tractable, because it involves multiple sums of exponential functions.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 12 / 33

SLIDE 48

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Converting to a complete data problem

Let zi ∈ {1, · · · , k} denote the source distribution where each xi was sampled from. f x z

n i k j

I zi j fi xi

j j n i

fi xi

zi zi i n i

I zi i n

i n i

I zi i xi

n i

I zi i

i n i

I zi i xi

i n i

I zi i The MLE solution is analytically tractable, if z is known.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 13 / 33

SLIDE 49

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Converting to a complete data problem

Let zi ∈ {1, · · · , k} denote the source distribution where each xi was sampled from. f(x|z, θ) =

n

∏

i=1

 

k

∑

j=1

I(zi = j)fi(xi|µj, σ2

j )

 

n i

fi xi

zi zi i n i

I zi i n

i n i

I zi i xi

n i

I zi i

i n i

I zi i xi

i n i

I zi i The MLE solution is analytically tractable, if z is known.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 13 / 33

SLIDE 50

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Converting to a complete data problem

Let zi ∈ {1, · · · , k} denote the source distribution where each xi was sampled from. f(x|z, θ) =

n

∏

i=1

 

k

∑

j=1

I(zi = j)fi(xi|µj, σ2

j )

  =

n

∏

i=1

fi(xi|µzi, σ2

zi) i n i

I zi i n

i n i

I zi i xi

n i

I zi i

i n i

I zi i xi

i n i

I zi i The MLE solution is analytically tractable, if z is known.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 13 / 33

SLIDE 51

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Converting to a complete data problem

Let zi ∈ {1, · · · , k} denote the source distribution where each xi was sampled from. f(x|z, θ) =

n

∏

i=1

 

k

∑

j=1

I(zi = j)fi(xi|µj, σ2

j )

  =

n

∏

i=1

fi(xi|µzi, σ2

zi)

ˆ πi = ∑n

i=1 I(zi = i)

n

i n i

I zi i xi

n i

I zi i

i n i

I zi i xi

i n i

I zi i The MLE solution is analytically tractable, if z is known.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 13 / 33

SLIDE 52

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Converting to a complete data problem

Let zi ∈ {1, · · · , k} denote the source distribution where each xi was sampled from. f(x|z, θ) =

n

∏

i=1

 

k

∑

j=1

I(zi = j)fi(xi|µj, σ2

j )

  =

n

∏

i=1

fi(xi|µzi, σ2

zi)

ˆ πi = ∑n

i=1 I(zi = i)

n ˆ µi = ∑n

i=1 I(zi = i)xi

∑n

i=1 I(zi = i) i n i

I zi i xi

i n i

I zi i The MLE solution is analytically tractable, if z is known.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 13 / 33

SLIDE 53

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Converting to a complete data problem

Let zi ∈ {1, · · · , k} denote the source distribution where each xi was sampled from. f(x|z, θ) =

n

∏

i=1

 

k

∑

j=1

I(zi = j)fi(xi|µj, σ2

j )

  =

n

∏

i=1

fi(xi|µzi, σ2

zi)

ˆ πi = ∑n

i=1 I(zi = i)

n ˆ µi = ∑n

i=1 I(zi = i)xi

∑n

i=1 I(zi = i)

ˆ σ2

i

= ∑n

i=1 I(zi = i)(xi − ˆ

µi)2 ∑n

i=1 I(zi = i)

The MLE solution is analytically tractable, if z is known.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 13 / 33

SLIDE 54

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Converting to a complete data problem

Let zi ∈ {1, · · · , k} denote the source distribution where each xi was sampled from. f(x|z, θ) =

n

∏

i=1

 

k

∑

j=1

I(zi = j)fi(xi|µj, σ2

j )

  =

n

∏

i=1

fi(xi|µzi, σ2

zi)

ˆ πi = ∑n

i=1 I(zi = i)

n ˆ µi = ∑n

i=1 I(zi = i)xi

∑n

i=1 I(zi = i)

ˆ σ2

i

= ∑n

i=1 I(zi = i)(xi − ˆ

µi)2 ∑n

i=1 I(zi = i)

The MLE solution is analytically tractable, if z is known.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 13 / 33

SLIDE 55

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

E-M Algorithm

E-M (Expectation-Maximization) algorithm is

A procedure for typically solving for the MLE.
Guaranteed to converge the MLE (!)
Particularly suited to the ”missing data” problems where analytic

solution of MLE is not tractable The algorithm was derived and used in various special cases by a number

f authors, but it was not identified as a general algorithm until the

seminal paper by Dempster, Laird, and Rubin in Journal of Royal Statistical Society Series B (1977).

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 14 / 33

SLIDE 56

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

E-M Algorithm

E-M (Expectation-Maximization) algorithm is

A procedure for typically solving for the MLE.
Guaranteed to converge the MLE (!)
Particularly suited to the ”missing data” problems where analytic

solution of MLE is not tractable The algorithm was derived and used in various special cases by a number

f authors, but it was not identified as a general algorithm until the

seminal paper by Dempster, Laird, and Rubin in Journal of Royal Statistical Society Series B (1977).

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 14 / 33

SLIDE 57

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

E-M Algorithm

E-M (Expectation-Maximization) algorithm is

A procedure for typically solving for the MLE.
Guaranteed to converge the MLE (!)
Particularly suited to the ”missing data” problems where analytic

solution of MLE is not tractable The algorithm was derived and used in various special cases by a number

f authors, but it was not identified as a general algorithm until the

seminal paper by Dempster, Laird, and Rubin in Journal of Royal Statistical Society Series B (1977).

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 14 / 33

SLIDE 58

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

E-M Algorithm

E-M (Expectation-Maximization) algorithm is

A procedure for typically solving for the MLE.
Guaranteed to converge the MLE (!)
Particularly suited to the ”missing data” problems where analytic

solution of MLE is not tractable The algorithm was derived and used in various special cases by a number

f authors, but it was not identified as a general algorithm until the

seminal paper by Dempster, Laird, and Rubin in Journal of Royal Statistical Society Series B (1977).

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 14 / 33

SLIDE 59

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

E-M Algorithm

E-M (Expectation-Maximization) algorithm is

A procedure for typically solving for the MLE.
Guaranteed to converge the MLE (!)
Particularly suited to the ”missing data” problems where analytic

solution of MLE is not tractable The algorithm was derived and used in various special cases by a number

f authors, but it was not identified as a general algorithm until the

seminal paper by Dempster, Laird, and Rubin in Journal of Royal Statistical Society Series B (1977).

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 14 / 33

SLIDE 60

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Overview of E-M Algorithm

.

Basic Structure

. .

y is observed (or incomplete) data
z is missing (or augmented) data
x = (y, z) is complete data

.

Complete and incomplete data likelihood

. . . . . . . .

Complete data likelihood : f x

f y z

Incomplete data likelihood : g y

f y z dz We are interested in MLE for L y g y .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 15 / 33

SLIDE 61

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Overview of E-M Algorithm

.

Basic Structure

. .

y is observed (or incomplete) data
z is missing (or augmented) data
x = (y, z) is complete data

.

Complete and incomplete data likelihood

. .

Complete data likelihood : f(x|θ) = f(y, z|θ)
Incomplete data likelihood : g y

f y z dz We are interested in MLE for L y g y .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 15 / 33

SLIDE 62

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Overview of E-M Algorithm

.

Basic Structure

. .

y is observed (or incomplete) data
z is missing (or augmented) data
x = (y, z) is complete data

.

Complete and incomplete data likelihood

. .

Complete data likelihood : f(x|θ) = f(y, z|θ)
Incomplete data likelihood : g(y|θ) =

∫ f(y, z|θ)dz We are interested in MLE for L y g y .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 15 / 33

SLIDE 63

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Overview of E-M Algorithm

.

Basic Structure

. .

y is observed (or incomplete) data
z is missing (or augmented) data
x = (y, z) is complete data

.

Complete and incomplete data likelihood

. .

Complete data likelihood : f(x|θ) = f(y, z|θ)
Incomplete data likelihood : g(y|θ) =

∫ f(y, z|θ)dz We are interested in MLE for L(θ|y) = g(y|θ).

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 15 / 33

SLIDE 64

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Maximizing incomplete data likelihood

L(θ|y, z) = f(y, z|θ) L y g y k z y f y z g y log L y log L y z log k z y Because z is missing data, we replace the right side with its expectation under k z y , creating the new identity log L y E log L y Z y E log k Z y y Iteratively maximizing the first term in the right-hand side results in E-M algorithm.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 16 / 33

SLIDE 65

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Maximizing incomplete data likelihood

L(θ|y, z) = f(y, z|θ) L(θ|y) = g(y|θ) k z y f y z g y log L y log L y z log k z y Because z is missing data, we replace the right side with its expectation under k z y , creating the new identity log L y E log L y Z y E log k Z y y Iteratively maximizing the first term in the right-hand side results in E-M algorithm.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 16 / 33

SLIDE 66

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Maximizing incomplete data likelihood

L(θ|y, z) = f(y, z|θ) L(θ|y) = g(y|θ) k(z|θ, y) = f(y, z|θ) g(y|θ) log L y log L y z log k z y Because z is missing data, we replace the right side with its expectation under k z y , creating the new identity log L y E log L y Z y E log k Z y y Iteratively maximizing the first term in the right-hand side results in E-M algorithm.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 16 / 33

SLIDE 67

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Maximizing incomplete data likelihood

L(θ|y, z) = f(y, z|θ) L(θ|y) = g(y|θ) k(z|θ, y) = f(y, z|θ) g(y|θ) log L(θ|y) = log L(θ|y, z) − log k(z|θ, y) Because z is missing data, we replace the right side with its expectation under k z y , creating the new identity log L y E log L y Z y E log k Z y y Iteratively maximizing the first term in the right-hand side results in E-M algorithm.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 16 / 33

SLIDE 68

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Maximizing incomplete data likelihood

L(θ|y, z) = f(y, z|θ) L(θ|y) = g(y|θ) k(z|θ, y) = f(y, z|θ) g(y|θ) log L(θ|y) = log L(θ|y, z) − log k(z|θ, y) Because z is missing data, we replace the right side with its expectation under k(z|θ′, y), creating the new identity log L y E log L y Z y E log k Z y y Iteratively maximizing the first term in the right-hand side results in E-M algorithm.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 16 / 33

SLIDE 69

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Maximizing incomplete data likelihood

L(θ|y, z) = f(y, z|θ) L(θ|y) = g(y|θ) k(z|θ, y) = f(y, z|θ) g(y|θ) log L(θ|y) = log L(θ|y, z) − log k(z|θ, y) Because z is missing data, we replace the right side with its expectation under k(z|θ′, y), creating the new identity log L(θ|y) = E [ log L(θ|y, Z)|θ′, y ] − E [ log k(Z|θ, y)|θ′, y ] Iteratively maximizing the first term in the right-hand side results in E-M algorithm.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 16 / 33

SLIDE 70

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Maximizing incomplete data likelihood

L(θ|y, z) = f(y, z|θ) L(θ|y) = g(y|θ) k(z|θ, y) = f(y, z|θ) g(y|θ) log L(θ|y) = log L(θ|y, z) − log k(z|θ, y) Because z is missing data, we replace the right side with its expectation under k(z|θ′, y), creating the new identity log L(θ|y) = E [ log L(θ|y, Z)|θ′, y ] − E [ log k(Z|θ, y)|θ′, y ] Iteratively maximizing the first term in the right-hand side results in E-M algorithm.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 16 / 33

SLIDE 71

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Overview of E-M Algorithm (cont’d)

.

Objective

. .

Maximize L(θ|y) or l(θ|y).
Let f y z

denotes the pdf of complete data. In E-M algorithm, rather than working with l y directly, we work with the surrogate function Q

r

E log f y Z y

r

where

r is the estimation of

in r-th iteration.

Q

r

is the expected log-likelihood of complete data, conditioning

n the observed data and

r .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 17 / 33

SLIDE 72

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Overview of E-M Algorithm (cont’d)

.

Objective

. .

Maximize L(θ|y) or l(θ|y).
Let f(y, z|θ) denotes the pdf of complete data. In E-M algorithm,

rather than working with l(θ|y) directly, we work with the surrogate function Q

r

E log f y Z y

r

where

r is the estimation of

in r-th iteration.

Q

r

is the expected log-likelihood of complete data, conditioning

n the observed data and

r .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 17 / 33

SLIDE 73

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Overview of E-M Algorithm (cont’d)

.

Objective

. .

Maximize L(θ|y) or l(θ|y).
Let f(y, z|θ) denotes the pdf of complete data. In E-M algorithm,

rather than working with l(θ|y) directly, we work with the surrogate function Q(θ|θ(r)) = E [ log f(y, Z|θ)|y, θ(r)] where

r is the estimation of

in r-th iteration.

Q

r

is the expected log-likelihood of complete data, conditioning

n the observed data and

r .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 17 / 33

SLIDE 74

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Overview of E-M Algorithm (cont’d)

.

Objective

. .

Maximize L(θ|y) or l(θ|y).
Let f(y, z|θ) denotes the pdf of complete data. In E-M algorithm,

rather than working with l(θ|y) directly, we work with the surrogate function Q(θ|θ(r)) = E [ log f(y, Z|θ)|y, θ(r)] where θ(r) is the estimation of θ in r-th iteration.

Q

r

is the expected log-likelihood of complete data, conditioning

n the observed data and

r .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 17 / 33

SLIDE 75

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Overview of E-M Algorithm (cont’d)

.

Objective

. .

Maximize L(θ|y) or l(θ|y).
Let f(y, z|θ) denotes the pdf of complete data. In E-M algorithm,

rather than working with l(θ|y) directly, we work with the surrogate function Q(θ|θ(r)) = E [ log f(y, Z|θ)|y, θ(r)] where θ(r) is the estimation of θ in r-th iteration.

Q(θ|θ(r)) is the expected log-likelihood of complete data, conditioning
n the observed data and θ(r).

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 17 / 33

SLIDE 76

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Key Steps of E-M algorithm

.

Expectation Step

. .

Compute Q(θ|θ(r)).
This typically involves in estimating the conditional distribution Z|Y,

assuming θ = θ(r).

After computing Q(θ|θ(r)), move to the M-step

.

Maximization Step

. . . . . . . .

Maximize Q

r

with respect to .

The arg max Q

r

will be the r

th

to be fed into the E-step.

Repeat E-step until convergence

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 18 / 33

SLIDE 77

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Key Steps of E-M algorithm

.

Expectation Step

. .

Compute Q(θ|θ(r)).
This typically involves in estimating the conditional distribution Z|Y,

assuming θ = θ(r).

After computing Q(θ|θ(r)), move to the M-step

.

Maximization Step

. .

Maximize Q(θ|θ(r)) with respect to θ.
The arg maxθ Q(θ|θ(r)) will be the (r + 1)-th θ to be fed into the

E-step.

Repeat E-step until convergence

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 18 / 33

SLIDE 78

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

E-M algorithm for mixture of normals

.

E-step

. . Q(θ|θ(r)) = E [ log f(y, Z|θ)|y, θ(r)]

z

k z

r y log f y z n i k zi

k zi

r yi log f yi zi n i k zi

f yi zi

r

g yi

r

log f yi zi f yi zi

zi zi

g yi

k j if yi zi

j

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 19 / 33

SLIDE 79

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

E-M algorithm for mixture of normals

.

E-step

. . Q(θ|θ(r)) = E [ log f(y, Z|θ)|y, θ(r)] = ∑

z

k(z|θ(r), y) log f(y, z|θ)

n i k zi

k zi

r yi log f yi zi n i k zi

f yi zi

r

g yi

r

log f yi zi f yi zi

zi zi

g yi

k j if yi zi

j

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 19 / 33

SLIDE 80

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

E-M algorithm for mixture of normals

.

E-step

. . Q(θ|θ(r)) = E [ log f(y, Z|θ)|y, θ(r)] = ∑

z

k(z|θ(r), y) log f(y, z|θ) =

n

∑

i=1 k

∑

zi=1

k(zi|θ(r), yi) log f(yi, zi|θ)

n i k zi

f yi zi

r

g yi

r

log f yi zi f yi zi

zi zi

g yi

k j if yi zi

j

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 19 / 33

SLIDE 81

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

E-M algorithm for mixture of normals

.

E-step

. . Q(θ|θ(r)) = E [ log f(y, Z|θ)|y, θ(r)] = ∑

z

k(z|θ(r), y) log f(y, z|θ) =

n

∑

i=1 k

∑

zi=1

k(zi|θ(r), yi) log f(yi, zi|θ) =

n

∑

i=1 k

∑

zi=1

f(yi, zi|θ(r)) g(yi|θ(r)) log f(yi, zi|θ) f yi zi

zi zi

g yi

k j if yi zi

j

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 19 / 33

SLIDE 82

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

E-M algorithm for mixture of normals

.

E-step

. . Q(θ|θ(r)) = E [ log f(y, Z|θ)|y, θ(r)] = ∑

z

k(z|θ(r), y) log f(y, z|θ) =

n

∑

i=1 k

∑

zi=1

k(zi|θ(r), yi) log f(yi, zi|θ) =

n

∑

i=1 k

∑

zi=1

f(yi, zi|θ(r)) g(yi|θ(r)) log f(yi, zi|θ) f(yi, zi|θ) ∼ N(µzi, σ2

zi)

g yi

k j if yi zi

j

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 19 / 33

SLIDE 83

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

E-M algorithm for mixture of normals

.

E-step

. . Q(θ|θ(r)) = E [ log f(y, Z|θ)|y, θ(r)] = ∑

z

k(z|θ(r), y) log f(y, z|θ) =

n

∑

i=1 k

∑

zi=1

k(zi|θ(r), yi) log f(yi, zi|θ) =

n

∑

i=1 k

∑

zi=1

f(yi, zi|θ(r)) g(yi|θ(r)) log f(yi, zi|θ) f(yi, zi|θ) ∼ N(µzi, σ2

zi)

g(yi|θ) =

k

∑

j=1

πif(yi, zi = j|θ)

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 19 / 33

SLIDE 84

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

E-M algorithm for mixture of normals (cont’d)

.

M-step

. . Q(θ|θ(r)) =

n

∑

i=1 k

∑

zi=1

f(yi, zi|θ(r)) g(yi|θ(r)) log f(yi, zi|θ)

r j

n

n i

k zi j yi

r

n

n i

f yi zi j

r

g yi

r r j n i

xik zi j yi

r n i

k zi j yi

r n i

xik zi j yi

r

n

r j r j n i

xi

r j

k zi j yi

r n i

k zi j yi

r n i

xi

r j

k zi j yi

r

n

r j

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 20 / 33

SLIDE 85

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

E-M algorithm for mixture of normals (cont’d)

.

M-step

. . Q(θ|θ(r)) =

n

∑

i=1 k

∑

zi=1

f(yi, zi|θ(r)) g(yi|θ(r)) log f(yi, zi|θ) π(r+1)

j

= 1 n

n

∑

i=1

k(zi = j|yi, θ(r)) = 1 n

n

∑

i=1

f(yi, zi = j|θ(r)) g(yi|θ(r))

r j n i

xik zi j yi

r n i

k zi j yi

r n i

xik zi j yi

r

n

r j r j n i

xi

r j

k zi j yi

r n i

k zi j yi

r n i

xi

r j

k zi j yi

r

n

r j

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 20 / 33

SLIDE 86

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

E-M algorithm for mixture of normals (cont’d)

.

M-step

. . Q(θ|θ(r)) =

n

∑

i=1 k

∑

zi=1

f(yi, zi|θ(r)) g(yi|θ(r)) log f(yi, zi|θ) π(r+1)

j

= 1 n

n

∑

i=1

k(zi = j|yi, θ(r)) = 1 n

n

∑

i=1

f(yi, zi = j|θ(r)) g(yi|θ(r)) µ(r+1)

j

= ∑n

i=1 xik(zi = j|yi, θ(r))

∑n

i=1 k(zi = j|yi, θ(r)) =

∑n

i=1 xik(zi = j|yi, θ(r))

nπ(r+1)

j r j n i

xi

r j

k zi j yi

r n i

k zi j yi

r n i

xi

r j

k zi j yi

r

n

r j

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 20 / 33

SLIDE 87

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

E-M algorithm for mixture of normals (cont’d)

.

M-step

. . Q(θ|θ(r)) =

n

∑

i=1 k

∑

zi=1

f(yi, zi|θ(r)) g(yi|θ(r)) log f(yi, zi|θ) π(r+1)

j

= 1 n

n

∑

i=1

k(zi = j|yi, θ(r)) = 1 n

n

∑

i=1

f(yi, zi = j|θ(r)) g(yi|θ(r)) µ(r+1)

j

= ∑n

i=1 xik(zi = j|yi, θ(r))

∑n

i=1 k(zi = j|yi, θ(r)) =

∑n

i=1 xik(zi = j|yi, θ(r))

nπ(r+1)

j

σ2,(r+1)

j

= ∑n

i=1(xi − µ(r+1) j

)2k(zi = j|yi, θ(r)) ∑n

i=1 k(zi = j|yi, θ(r)) n i

xi

r j

k zi j yi

r

n

r j

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 20 / 33

SLIDE 88

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

E-M algorithm for mixture of normals (cont’d)

.

M-step

. . Q(θ|θ(r)) =

n

∑

i=1 k

∑

zi=1

f(yi, zi|θ(r)) g(yi|θ(r)) log f(yi, zi|θ) π(r+1)

j

= 1 n

n

∑

i=1

k(zi = j|yi, θ(r)) = 1 n

n

∑

i=1

f(yi, zi = j|θ(r)) g(yi|θ(r)) µ(r+1)

j

= ∑n

i=1 xik(zi = j|yi, θ(r))

∑n

i=1 k(zi = j|yi, θ(r)) =

∑n

i=1 xik(zi = j|yi, θ(r))

nπ(r+1)

j

σ2,(r+1)

j

= ∑n

i=1(xi − µ(r+1) j

)2k(zi = j|yi, θ(r)) ∑n

i=1 k(zi = j|yi, θ(r))

= ∑n

i=1(xi − µ(r+1) j

)2k(zi = j|yi, θ(r)) nπ(r+1)

j

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 20 / 33

SLIDE 89

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Does E-M iteration converge to MLE?

.

Theorem 7.2.20 - Monotonic EM sequence

. . The sequence {ˆ θ(r)} defined by the E-M procedure satisfies L

r

y L

r y

with equality holding if and only if successive iterations yield the same value of the maximized expected complete-data log likelihood, that is E log L

r

y Z

r y

E log L

r y Z r y

Theorem 7.5.2 further guarantees that L

r y converges monotonically

to L y for some stationary point .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 21 / 33

SLIDE 90

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Does E-M iteration converge to MLE?

.

Theorem 7.2.20 - Monotonic EM sequence

. . The sequence {ˆ θ(r)} defined by the E-M procedure satisfies L ( ˆ θ(r+1)|y ) ≥ L ( ˆ θ(r)|y ) with equality holding if and only if successive iterations yield the same value of the maximized expected complete-data log likelihood, that is E log L

r

y Z

r y

E log L

r y Z r y

Theorem 7.5.2 further guarantees that L

r y converges monotonically

to L y for some stationary point .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 21 / 33

SLIDE 91

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Does E-M iteration converge to MLE?

.

Theorem 7.2.20 - Monotonic EM sequence

. . The sequence {ˆ θ(r)} defined by the E-M procedure satisfies L ( ˆ θ(r+1)|y ) ≥ L ( ˆ θ(r)|y ) with equality holding if and only if successive iterations yield the same value of the maximized expected complete-data log likelihood, that is E log L

r

y Z

r y

E log L

r y Z r y

Theorem 7.5.2 further guarantees that L

r y converges monotonically

to L y for some stationary point .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 21 / 33

SLIDE 92

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Does E-M iteration converge to MLE?

.

Theorem 7.2.20 - Monotonic EM sequence

. . The sequence {ˆ θ(r)} defined by the E-M procedure satisfies L ( ˆ θ(r+1)|y ) ≥ L ( ˆ θ(r)|y ) with equality holding if and only if successive iterations yield the same value of the maximized expected complete-data log likelihood, that is E [ log L ( ˆ θ(r+1)|y, Z ) |ˆ θ(r), y ] = E [ log L ( ˆ θ(r)|y, Z ) |ˆ θ(r), y ] Theorem 7.5.2 further guarantees that L

r y converges monotonically

to L y for some stationary point .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 21 / 33

SLIDE 93

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Does E-M iteration converge to MLE?

.

Theorem 7.2.20 - Monotonic EM sequence

. . The sequence {ˆ θ(r)} defined by the E-M procedure satisfies L ( ˆ θ(r+1)|y ) ≥ L ( ˆ θ(r)|y ) with equality holding if and only if successive iterations yield the same value of the maximized expected complete-data log likelihood, that is E [ log L ( ˆ θ(r+1)|y, Z ) |ˆ θ(r), y ] = E [ log L ( ˆ θ(r)|y, Z ) |ˆ θ(r), y ] Theorem 7.5.2 further guarantees that L(ˆ θ(r)|y) converges monotonically to L(ˆ θ|y) for some stationary point ˆ θ.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 21 / 33

SLIDE 94

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

A working example (from BIOSTAT615/815 Fall 2012)

.

Example Data (n=1,500)

. . .

Running example of implemented software

. . . . . . . .

user@host~/> ./mixEM ./mix.dat Maximum log-likelihood = 3043.46, at pi = (0.667842,0.332158) between N(-0.0299457,1.00791) and N(5.0128,0.913825)

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 22 / 33

SLIDE 95

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

A working example (from BIOSTAT615/815 Fall 2012)

.

Example Data (n=1,500)

. . .

Running example of implemented software

. .

user@host~/> ./mixEM ./mix.dat Maximum log-likelihood = 3043.46, at pi = (0.667842,0.332158) between N(-0.0299457,1.00791) and N(5.0128,0.913825)

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 22 / 33

SLIDE 96

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Practice Problem 1

.

Problem

. . Let X1, · · · , Xn be a random sample from a population with pdf f(x|θ) = 1 2θ − θ < x < θ, θ > 0 Find, if one exists, a best unbiased estimator of θ. .

Strategy to solve the problem

. . . . . . . .

Can we use the Cramer-Rao bound? No, because the

interchangeability condition does not hold

Then, can we use complete sufficient statistics?

. . 1 Find a complete sufficient statistic T. . . 2 For a trivial unbiased estimator W for

, and compute T E W T

. . 3 or Make a function

T such that E T .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 23 / 33

SLIDE 97

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Practice Problem 1

.

Problem

. . Let X1, · · · , Xn be a random sample from a population with pdf f(x|θ) = 1 2θ − θ < x < θ, θ > 0 Find, if one exists, a best unbiased estimator of θ. .

Strategy to solve the problem

. .

Can we use the Cramer-Rao bound?

No, because the interchangeability condition does not hold

Then, can we use complete sufficient statistics?

. . 1 Find a complete sufficient statistic T. . . 2 For a trivial unbiased estimator W for

, and compute T E W T

. . 3 or Make a function

T such that E T .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 23 / 33

SLIDE 98

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Practice Problem 1

.

Problem

. . Let X1, · · · , Xn be a random sample from a population with pdf f(x|θ) = 1 2θ − θ < x < θ, θ > 0 Find, if one exists, a best unbiased estimator of θ. .

Strategy to solve the problem

. .

Can we use the Cramer-Rao bound? No, because the

interchangeability condition does not hold

Then, can we use complete sufficient statistics?

. . 1 Find a complete sufficient statistic T. . . 2 For a trivial unbiased estimator W for

, and compute T E W T

. . 3 or Make a function

T such that E T .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 23 / 33

SLIDE 99

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Practice Problem 1

.

Problem

. . Let X1, · · · , Xn be a random sample from a population with pdf f(x|θ) = 1 2θ − θ < x < θ, θ > 0 Find, if one exists, a best unbiased estimator of θ. .

Strategy to solve the problem

. .

Can we use the Cramer-Rao bound? No, because the

interchangeability condition does not hold

Then, can we use complete sufficient statistics?

. . 1 Find a complete sufficient statistic T. . . 2 For a trivial unbiased estimator W for

, and compute T E W T

. . 3 or Make a function

T such that E T .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 23 / 33

SLIDE 100

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Practice Problem 1

.

Problem

. . Let X1, · · · , Xn be a random sample from a population with pdf f(x|θ) = 1 2θ − θ < x < θ, θ > 0 Find, if one exists, a best unbiased estimator of θ. .

Strategy to solve the problem

. .

Can we use the Cramer-Rao bound? No, because the

interchangeability condition does not hold

Then, can we use complete sufficient statistics?

. . 1 Find a complete sufficient statistic T. . . 2 For a trivial unbiased estimator W for

, and compute T E W T

. . 3 or Make a function

T such that E T .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 23 / 33

SLIDE 101

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Practice Problem 1

.

Problem

. . Let X1, · · · , Xn be a random sample from a population with pdf f(x|θ) = 1 2θ − θ < x < θ, θ > 0 Find, if one exists, a best unbiased estimator of θ. .

Strategy to solve the problem

. .

Can we use the Cramer-Rao bound? No, because the

interchangeability condition does not hold

Then, can we use complete sufficient statistics?

. . 1 Find a complete sufficient statistic T. . . 2 For a trivial unbiased estimator W for θ, and compute φ(T) = E[W|T] . . 3 or Make a function

T such that E T .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 23 / 33

SLIDE 102

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Practice Problem 1

.

Problem

. . Let X1, · · · , Xn be a random sample from a population with pdf f(x|θ) = 1 2θ − θ < x < θ, θ > 0 Find, if one exists, a best unbiased estimator of θ. .

Strategy to solve the problem

. .

Can we use the Cramer-Rao bound? No, because the

interchangeability condition does not hold

Then, can we use complete sufficient statistics?

. . 1 Find a complete sufficient statistic T. . . 2 For a trivial unbiased estimator W for θ, and compute φ(T) = E[W|T] . . 3 or Make a function φ(T) such that E[φ(T)] = θ.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 23 / 33

SLIDE 103

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution

First, we need to find a complete sufficient statistic. fX x I x fX x

n I max i

xi Let T X maxi Xi , then fT t

ntn

n I

t E g T ntn g t

n

dt tn g t dt

n

g g Therefore the family of T is complete.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 24 / 33

SLIDE 104

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution

First, we need to find a complete sufficient statistic. fX(x|θ) = 1 2θI(|x| < θ) fX(x|θ) = 1 (2θ)n I(max

i

|xi| < θ) Let T X maxi Xi , then fT t

ntn

n I

t E g T ntn g t

n

dt tn g t dt

n

g g Therefore the family of T is complete.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 24 / 33

SLIDE 105

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution

First, we need to find a complete sufficient statistic. fX(x|θ) = 1 2θI(|x| < θ) fX(x|θ) = 1 (2θ)n I(max

i

|xi| < θ) Let T(X) = maxi |Xi|, then fT(t|θ) = ntn−1

θn I(0 ≤ t < θ)

E g T ntn g t

n

dt tn g t dt

n

g g Therefore the family of T is complete.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 24 / 33

SLIDE 106

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution

First, we need to find a complete sufficient statistic. fX(x|θ) = 1 2θI(|x| < θ) fX(x|θ) = 1 (2θ)n I(max

i

|xi| < θ) Let T(X) = maxi |Xi|, then fT(t|θ) = ntn−1

θn I(0 ≤ t < θ)

E[g(T)] = ∫ θ ntn−1g(t) θn dt = 0 tn g t dt

n

g g Therefore the family of T is complete.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 24 / 33

SLIDE 107

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution

First, we need to find a complete sufficient statistic. fX(x|θ) = 1 2θI(|x| < θ) fX(x|θ) = 1 (2θ)n I(max

i

|xi| < θ) Let T(X) = maxi |Xi|, then fT(t|θ) = ntn−1

θn I(0 ≤ t < θ)

E[g(T)] = ∫ θ ntn−1g(t) θn dt = 0 ∫ θ tn−1g(t)dt =

n

g g Therefore the family of T is complete.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 24 / 33

SLIDE 108

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution

First, we need to find a complete sufficient statistic. fX(x|θ) = 1 2θI(|x| < θ) fX(x|θ) = 1 (2θ)n I(max

i

|xi| < θ) Let T(X) = maxi |Xi|, then fT(t|θ) = ntn−1

θn I(0 ≤ t < θ)

E[g(T)] = ∫ θ ntn−1g(t) θn dt = 0 ∫ θ tn−1g(t)dt = θn−1g(θ) = g Therefore the family of T is complete.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 24 / 33

SLIDE 109

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution

First, we need to find a complete sufficient statistic. fX(x|θ) = 1 2θI(|x| < θ) fX(x|θ) = 1 (2θ)n I(max

i

|xi| < θ) Let T(X) = maxi |Xi|, then fT(t|θ) = ntn−1

θn I(0 ≤ t < θ)

E[g(T)] = ∫ θ ntn−1g(t) θn dt = 0 ∫ θ tn−1g(t)dt = θn−1g(θ) = g(θ) = Therefore the family of T is complete.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 24 / 33

SLIDE 110

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution

We need to make a φ(T) such that E[φ(T)] = θ. First, let’s see what the expectation of T is E T tntn

n

dt ntn

n dt

n n T

n n T is an unbiased estimator and a function of a complete

sufficient statistic. Therefore, T is the best unbiased estimator by Theorem 7.3.23.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 25 / 33

SLIDE 111

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution

We need to make a φ(T) such that E[φ(T)] = θ. First, let’s see what the expectation of T is E T tntn

n

dt ntn

n dt

n n T

n n T is an unbiased estimator and a function of a complete

sufficient statistic. Therefore, T is the best unbiased estimator by Theorem 7.3.23.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 25 / 33

SLIDE 112

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution

We need to make a φ(T) such that E[φ(T)] = θ. First, let’s see what the expectation of T is E[T] = ∫ θ tntn−1 θn dt ntn

n dt

n n T

n n T is an unbiased estimator and a function of a complete

sufficient statistic. Therefore, T is the best unbiased estimator by Theorem 7.3.23.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 25 / 33

SLIDE 113

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution

We need to make a φ(T) such that E[φ(T)] = θ. First, let’s see what the expectation of T is E[T] = ∫ θ tntn−1 θn dt = ∫ θ ntn θn dt n n T

n n T is an unbiased estimator and a function of a complete

sufficient statistic. Therefore, T is the best unbiased estimator by Theorem 7.3.23.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 25 / 33

SLIDE 114

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution

We need to make a φ(T) such that E[φ(T)] = θ. First, let’s see what the expectation of T is E[T] = ∫ θ tntn−1 θn dt = ∫ θ ntn θn dt = n n + 1θ T

n n T is an unbiased estimator and a function of a complete

sufficient statistic. Therefore, T is the best unbiased estimator by Theorem 7.3.23.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 25 / 33

SLIDE 115

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution

We need to make a φ(T) such that E[φ(T)] = θ. First, let’s see what the expectation of T is E[T] = ∫ θ tntn−1 θn dt = ∫ θ ntn θn dt = n n + 1θ φ(T) = n+1

n T is an unbiased estimator and a function of a complete

sufficient statistic. Therefore, T is the best unbiased estimator by Theorem 7.3.23.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 25 / 33

SLIDE 116

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution

We need to make a φ(T) such that E[φ(T)] = θ. First, let’s see what the expectation of T is E[T] = ∫ θ tntn−1 θn dt = ∫ θ ntn θn dt = n n + 1θ φ(T) = n+1

n T is an unbiased estimator and a function of a complete

sufficient statistic. Therefore, φ(T) is the best unbiased estimator by Theorem 7.3.23.

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 25 / 33

SLIDE 117

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Practice Problem 2

.

Problem

. . Let X1, · · · , Xn+1 be the iid Bernoulli(p), and define the function h(p) by h p Pr

n i

Xi Xn p the probability that the first n observations exceed the n

st.

. . 1 Show that

W X Xn I

n i

Xi Xn is an unbiased estimator of h p .

. . 2 Find the best unbiased estimator of h p .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 26 / 33

SLIDE 118

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Practice Problem 2

.

Problem

. . Let X1, · · · , Xn+1 be the iid Bernoulli(p), and define the function h(p) by h(p) = Pr (

n

∑

i=1

Xi > Xn+1

p

) the probability that the first n observations exceed the n

st.

. . 1 Show that

W X Xn I

n i

Xi Xn is an unbiased estimator of h p .

. . 2 Find the best unbiased estimator of h p .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 26 / 33

SLIDE 119

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Practice Problem 2

.

Problem

. . Let X1, · · · , Xn+1 be the iid Bernoulli(p), and define the function h(p) by h(p) = Pr (

n

∑

i=1

Xi > Xn+1

p

) the probability that the first n observations exceed the (n + 1)-st.

. . 1 Show that

W X Xn I

n i

Xi Xn is an unbiased estimator of h p .

. . 2 Find the best unbiased estimator of h p .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 26 / 33

SLIDE 120

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Practice Problem 2

.

Problem

. . Let X1, · · · , Xn+1 be the iid Bernoulli(p), and define the function h(p) by h(p) = Pr (

n

∑

i=1

Xi > Xn+1

p

) the probability that the first n observations exceed the (n + 1)-st.

. . 1 Show that

W(X1, · · · , Xn+1) = I ( n ∑

i=1

Xi > Xn+1 ) is an unbiased estimator of h(p).

. . 2 Find the best unbiased estimator of h p .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 26 / 33

SLIDE 121

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Practice Problem 2

.

Problem

. . Let X1, · · · , Xn+1 be the iid Bernoulli(p), and define the function h(p) by h(p) = Pr (

n

∑

i=1

Xi > Xn+1

p

) the probability that the first n observations exceed the (n + 1)-st.

. . 1 Show that

W(X1, · · · , Xn+1) = I ( n ∑

i=1

Xi > Xn+1 ) is an unbiased estimator of h(p).

. . 2 Find the best unbiased estimator of h(p).

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 26 / 33

SLIDE 122

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution for (a)

E[W] = ∑

X

W(X) Pr(X)

X

I

n i

Xi Xn Pr X

n i

Xi Xn

Pr X Pr

n i

Xi Xn h p Therefore T is an unbiased estimator of h p .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 27 / 33

SLIDE 123

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution for (a)

E[W] = ∑

X

W(X) Pr(X) = ∑

X

I ( n ∑

i=1

Xi > Xn+1 ) Pr(X)

n i

Xi Xn

Pr X Pr

n i

Xi Xn h p Therefore T is an unbiased estimator of h p .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 27 / 33

SLIDE 124

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution for (a)

E[W] = ∑

X

W(X) Pr(X) = ∑

X

I ( n ∑

i=1

Xi > Xn+1 ) Pr(X) = ∑

∑n

i=1 Xi>Xn+1

Pr(X) Pr

n i

Xi Xn h p Therefore T is an unbiased estimator of h p .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 27 / 33

SLIDE 125

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution for (a)

E[W] = ∑

X

W(X) Pr(X) = ∑

X

I ( n ∑

i=1

Xi > Xn+1 ) Pr(X) = ∑

∑n

i=1 Xi>Xn+1

Pr(X) = Pr ( n ∑

i=1

Xi > Xn+1 ) = h(p) Therefore T is an unbiased estimator of h p .

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 27 / 33

SLIDE 126

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution for (a)

E[W] = ∑

X

W(X) Pr(X) = ∑

X

I ( n ∑

i=1

Xi > Xn+1 ) Pr(X) = ∑

∑n

i=1 Xi>Xn+1

Pr(X) = Pr ( n ∑

i=1

Xi > Xn+1 ) = h(p) Therefore T is an unbiased estimator of h(p).

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 27 / 33

SLIDE 127

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution for (b)

T = ∑n+1

i=1 Xi is complete sufficient statistic for p.

T E W T Pr W T Pr

n i

Xi Xn T

If T

, then

n i

Xi Xn

If T

, then

Pr

n i

Xi Xn n n

Pr

n i

Xi Xn n

If T

then

Pr

n i

Xi Xn

n n

Pr

n i

Xi Xn n

If T

, then

n i

Xi Xn

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 28 / 33

SLIDE 128

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution for (b)

T = ∑n+1

i=1 Xi is complete sufficient statistic for p.

φ(T) = E[W|T] = Pr(W = 1|T) Pr

n i

Xi Xn T

If T

, then

n i

Xi Xn

If T

, then

Pr

n i

Xi Xn n n

Pr

n i

Xi Xn n

If T

then

Pr

n i

Xi Xn

n n

Pr

n i

Xi Xn n

If T

, then

n i

Xi Xn

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 28 / 33

SLIDE 129

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution for (b)

T = ∑n+1

i=1 Xi is complete sufficient statistic for p.

φ(T) = E[W|T] = Pr(W = 1|T) = Pr ( n ∑

i=1

Xi > Xn+1|T )

If T

, then

n i

Xi Xn

If T

, then

Pr

n i

Xi Xn n n

Pr

n i

Xi Xn n

If T

then

Pr

n i

Xi Xn

n n

Pr

n i

Xi Xn n

If T

, then

n i

Xi Xn

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 28 / 33

SLIDE 130

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution for (b)

T = ∑n+1

i=1 Xi is complete sufficient statistic for p.

φ(T) = E[W|T] = Pr(W = 1|T) = Pr ( n ∑

i=1

Xi > Xn+1|T )

If T = 0, then ∑n

i=1 Xi = Xn+1

If T

, then

Pr

n i

Xi Xn n n

Pr

n i

Xi Xn n

If T

then

Pr

n i

Xi Xn

n n

Pr

n i

Xi Xn n

If T

, then

n i

Xi Xn

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 28 / 33

SLIDE 131

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution for (b)

T = ∑n+1

i=1 Xi is complete sufficient statistic for p.

φ(T) = E[W|T] = Pr(W = 1|T) = Pr ( n ∑

i=1

Xi > Xn+1|T )

If T = 0, then ∑n

i=1 Xi = Xn+1

If T = 1, then
Pr(∑n

i=1 Xi = 1 > Xn+1 = 0) = n/(n + 1)

Pr(∑n

i=1 Xi = 0 < Xn+1 = 1) = 1/(n + 1)

If T

then

Pr

n i

Xi Xn

n n

Pr

n i

Xi Xn n

If T

, then

n i

Xi Xn

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 28 / 33

SLIDE 132

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution for (b)

T = ∑n+1

i=1 Xi is complete sufficient statistic for p.

φ(T) = E[W|T] = Pr(W = 1|T) = Pr ( n ∑

i=1

Xi > Xn+1|T )

If T = 0, then ∑n

i=1 Xi = Xn+1

If T = 1, then
Pr(∑n

i=1 Xi = 1 > Xn+1 = 0) = n/(n + 1)

Pr(∑n

i=1 Xi = 0 < Xn+1 = 1) = 1/(n + 1)

If T = 2 then
Pr(∑n

i=1 Xi = 2 > Xn+1 = 0) =

(n

2

) / (n+1

2

) = (n − 1)/(n + 1)

Pr(∑n

i=1 Xi = 1 = Xn+1 = 1) = 2/(n + 1)

If T

, then

n i

Xi Xn

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 28 / 33

SLIDE 133

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution for (b)

T = ∑n+1

i=1 Xi is complete sufficient statistic for p.

φ(T) = E[W|T] = Pr(W = 1|T) = Pr ( n ∑

i=1

Xi > Xn+1|T )

If T = 0, then ∑n

i=1 Xi = Xn+1

If T = 1, then
Pr(∑n

i=1 Xi = 1 > Xn+1 = 0) = n/(n + 1)

Pr(∑n

i=1 Xi = 0 < Xn+1 = 1) = 1/(n + 1)

If T = 2 then
Pr(∑n

i=1 Xi = 2 > Xn+1 = 0) =

(n

2

) / (n+1

2

) = (n − 1)/(n + 1)

Pr(∑n

i=1 Xi = 1 = Xn+1 = 1) = 2/(n + 1)

If T > 2, then ∑n

i=1 Xi ≥ 2 > 1 ≥ Xn+1

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 28 / 33

SLIDE 134

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Solution for (b) (cont’d)

Therefore, the best unbiased estimator is φ(T) = Pr ( n ∑

i=1

Xi > Xn+1|T ) =        T = 0 n/(n + 1) T = 1 (n − 1)/(n + 1) T = 2 1 T ≥ 3

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 29 / 33

SLIDE 135

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Practice Problem 3

.

Problem

. . Suppose X1, · · · , Xn are iid samples from f(x|θ) = θ exp(−θx). Suppose the prior distribution of θ is π(θ) = 1 Γ(α)βα θα−1e−θ/β where α, β are known. (a) Derive the posterior distribution of . (b) If we use the loss function L a a , what is the Bayes rule estimator for ?

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 30 / 33

SLIDE 136

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Practice Problem 3

.

Problem

. . Suppose X1, · · · , Xn are iid samples from f(x|θ) = θ exp(−θx). Suppose the prior distribution of θ is π(θ) = 1 Γ(α)βα θα−1e−θ/β where α, β are known. (a) Derive the posterior distribution of θ. (b) If we use the loss function L a a , what is the Bayes rule estimator for ?

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 30 / 33

SLIDE 137

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Practice Problem 3

.

Problem

. . Suppose X1, · · · , Xn are iid samples from f(x|θ) = θ exp(−θx). Suppose the prior distribution of θ is π(θ) = 1 Γ(α)βα θα−1e−θ/β where α, β are known. (a) Derive the posterior distribution of θ. (b) If we use the loss function L(θ, a) = (a − θ)2, what is the Bayes rule estimator for θ?

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 30 / 33

SLIDE 138

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

(a) Posterior distribution of θ

f(x, θ) = π(θ)f(x|θ)π(θ) = 1 Γ(α)βα θα−1e−θ/β

n

∏

i=1

[θ exp (−θxi)] e

n exp n i

xi

n

exp

n i

xi Gamma n

n i

xi x Gamma n

n i

xi

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 31 / 33

SLIDE 139

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

(a) Posterior distribution of θ

f(x, θ) = π(θ)f(x|θ)π(θ) = 1 Γ(α)βα θα−1e−θ/β

n

∏

i=1

[θ exp (−θxi)] = 1 Γ(α)βα θα−1e−θ/βθn exp ( −θ

n

∑

i=1

xi )

n

exp

n i

xi Gamma n

n i

xi x Gamma n

n i

xi

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 31 / 33

SLIDE 140

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

(a) Posterior distribution of θ

f(x, θ) = π(θ)f(x|θ)π(θ) = 1 Γ(α)βα θα−1e−θ/β

n

∏

i=1

[θ exp (−θxi)] = 1 Γ(α)βα θα−1e−θ/βθn exp ( −θ

n

∑

i=1

xi ) = 1 Γ(α)βα θα+n−1 exp [ −θ ( 1/β +

n

∑

i=1

xi )] Gamma n

n i

xi x Gamma n

n i

xi

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 31 / 33

SLIDE 141

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

(a) Posterior distribution of θ

f(x, θ) = π(θ)f(x|θ)π(θ) = 1 Γ(α)βα θα−1e−θ/β

n

∏

i=1

[θ exp (−θxi)] = 1 Γ(α)βα θα−1e−θ/βθn exp ( −θ

n

∑

i=1

xi ) = 1 Γ(α)βα θα+n−1 exp [ −θ ( 1/β +

n

∑

i=1

xi )] ∝ Gamma ( α + n − 1, 1 β−1 + ∑n

i=1 xi

) x Gamma n

n i

xi

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 31 / 33

SLIDE 142

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

(a) Posterior distribution of θ

f(x, θ) = π(θ)f(x|θ)π(θ) = 1 Γ(α)βα θα−1e−θ/β

n

∏

i=1

[θ exp (−θxi)] = 1 Γ(α)βα θα−1e−θ/βθn exp ( −θ

n

∑

i=1

xi ) = 1 Γ(α)βα θα+n−1 exp [ −θ ( 1/β +

n

∑

i=1

xi )] ∝ Gamma ( α + n − 1, 1 β−1 + ∑n

i=1 xi

) π(θ|x) = Gamma ( α + n − 1, 1 β−1 + ∑n

i=1 xi

)

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 31 / 33

SLIDE 143

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

(b) Bayes’ rule estimator with squared error loss

Bayes’ rule estimator with squared error loss is posterior mean. Note that the mean of Gamma(α, β) is αβ. x Gamma n

n i

xi E x E x n

n i

xi

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 32 / 33

SLIDE 144

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

(b) Bayes’ rule estimator with squared error loss

Bayes’ rule estimator with squared error loss is posterior mean. Note that the mean of Gamma(α, β) is αβ. π(θ|x) = Gamma ( α + n − 1, 1 β−1 + ∑n

i=1 xi

) E x E x n

n i

xi

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 32 / 33

SLIDE 145

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

(b) Bayes’ rule estimator with squared error loss

Bayes’ rule estimator with squared error loss is posterior mean. Note that the mean of Gamma(α, β) is αβ. π(θ|x) = Gamma ( α + n − 1, 1 β−1 + ∑n

i=1 xi

) E[θ|x] = E[π(θ|x)] = α + n − 1 β−1 + ∑n

i=1 xi

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 32 / 33

SLIDE 146

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary

Summary

.

Today

. .

E-M Algorithm
Practice Problems for the Final Exam

.

Next Lectures

. . . . . . . .

Bayesian Tests
Bayesian Intervals
More practice problems

Hyun Min Kang Biostatistics 602 - Lecture 24 April 16th, 2013 33 / 33

SLIDE 147

. .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. .

. . . . . Recap . . . . . . . . . . . . . . . . E-M . . . P1 . . . . P2 . . . P3 . Summary