Bayes Estimator Lecture 15 Biostatistics 602 - Statistical - - PowerPoint PPT Presentation

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

. .

Biostatistics 602 - Statistical Inference Lecture 15 Bayes Estimator

Hyun Min Kang March 12th, 2013

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 1 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Last Lecture

• Can Cramer-Rao bound be used to find the best unbiased estimator

for any distribution? If not, in which cases?

• When Cramer-Rao bound is attainable, can Cramer-Rao bound be

used for find best unbiased estimator for any ? If not, what is the restriction on ?

• What is another way to find the best unbiased estimator?
• Describe two strategies to obtain the best unbiased estimators for

, using complete sufficient statistics.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 2 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Last Lecture

• Can Cramer-Rao bound be used to find the best unbiased estimator

for any distribution? If not, in which cases?

• When Cramer-Rao bound is attainable, can Cramer-Rao bound be

used for find best unbiased estimator for any ? If not, what is the restriction on ?

• What is another way to find the best unbiased estimator?
• Describe two strategies to obtain the best unbiased estimators for

, using complete sufficient statistics.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 2 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Last Lecture

• Can Cramer-Rao bound be used to find the best unbiased estimator

for any distribution? If not, in which cases?

• When Cramer-Rao bound is attainable, can Cramer-Rao bound be

used for find best unbiased estimator for any τ(θ)? If not, what is the restriction on τ(θ)?

• What is another way to find the best unbiased estimator?
• Describe two strategies to obtain the best unbiased estimators for

, using complete sufficient statistics.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 2 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Last Lecture

• Can Cramer-Rao bound be used to find the best unbiased estimator

for any distribution? If not, in which cases?

• When Cramer-Rao bound is attainable, can Cramer-Rao bound be

used for find best unbiased estimator for any τ(θ)? If not, what is the restriction on τ(θ)?

• What is another way to find the best unbiased estimator?
• Describe two strategies to obtain the best unbiased estimators for

, using complete sufficient statistics.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 2 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Last Lecture

• Can Cramer-Rao bound be used to find the best unbiased estimator

for any distribution? If not, in which cases?

• When Cramer-Rao bound is attainable, can Cramer-Rao bound be

used for find best unbiased estimator for any τ(θ)? If not, what is the restriction on τ(θ)?

• What is another way to find the best unbiased estimator?
• Describe two strategies to obtain the best unbiased estimators for

τ(θ), using complete sufficient statistics.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 2 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Recap - The power of complete sufficient statistics

.

Theorem 7.3.23

. . Let T be a complete sufficient statistic for parameter θ. Let φ(T) be any estimator based on T. Then φ(T) is the unique best unbiased estimator of its expected value.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 3 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Finding UMUVE - Method 1

. . Use Cramer-Rao bound to find the best unbiased estimator for τ(θ).

. . 1 If ”regularity conditions” are satisfied, then we have a Cramer-Rao

bound for unbiased estimators of τ(θ).

• It helps to confirm an estimator is the best unbiased estimator of

if it happens to attain the CR-bound.

• If an unbiased estimator of

has variance greater than the CR-bound, it does NOT mean that it is not the best unbiased estimator.

. . 2 When ”regularity conditions” are not satisfied, In

is no longer a valid lower bound.

• There may be unbiased estimators of

that have variance smaller than

In

.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 4 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Finding UMUVE - Method 1

. . Use Cramer-Rao bound to find the best unbiased estimator for τ(θ).

. . 1 If ”regularity conditions” are satisfied, then we have a Cramer-Rao

bound for unbiased estimators of τ(θ).

• It helps to confirm an estimator is the best unbiased estimator of τ(θ)

if it happens to attain the CR-bound.

• If an unbiased estimator of

has variance greater than the CR-bound, it does NOT mean that it is not the best unbiased estimator.

. . 2 When ”regularity conditions” are not satisfied, In

is no longer a valid lower bound.

• There may be unbiased estimators of

that have variance smaller than

In

.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 4 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Finding UMUVE - Method 1

. . Use Cramer-Rao bound to find the best unbiased estimator for τ(θ).

. . 1 If ”regularity conditions” are satisfied, then we have a Cramer-Rao

bound for unbiased estimators of τ(θ).

• It helps to confirm an estimator is the best unbiased estimator of τ(θ)

if it happens to attain the CR-bound.

• If an unbiased estimator of τ(θ) has variance greater than the

CR-bound, it does NOT mean that it is not the best unbiased estimator.

. . 2 When ”regularity conditions” are not satisfied, In

is no longer a valid lower bound.

• There may be unbiased estimators of

that have variance smaller than

In

.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 4 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Finding UMUVE - Method 1

. . Use Cramer-Rao bound to find the best unbiased estimator for τ(θ).

. . 1 If ”regularity conditions” are satisfied, then we have a Cramer-Rao

bound for unbiased estimators of τ(θ).

• It helps to confirm an estimator is the best unbiased estimator of τ(θ)

if it happens to attain the CR-bound.

• If an unbiased estimator of τ(θ) has variance greater than the

CR-bound, it does NOT mean that it is not the best unbiased estimator.

. . 2 When ”regularity conditions” are not satisfied, [τ ′(θ)]2 In(θ)

is no longer a valid lower bound.

• There may be unbiased estimators of

that have variance smaller than

In

.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 4 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Finding UMUVE - Method 1

. . Use Cramer-Rao bound to find the best unbiased estimator for τ(θ).

. . 1 If ”regularity conditions” are satisfied, then we have a Cramer-Rao

bound for unbiased estimators of τ(θ).

• It helps to confirm an estimator is the best unbiased estimator of τ(θ)

if it happens to attain the CR-bound.

• If an unbiased estimator of τ(θ) has variance greater than the

CR-bound, it does NOT mean that it is not the best unbiased estimator.

. . 2 When ”regularity conditions” are not satisfied, [τ ′(θ)]2 In(θ)

is no longer a valid lower bound.

• There may be unbiased estimators of τ(θ) that have variance smaller

than [τ ′(θ)]2

In(θ) .

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 4 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Finding UMVUE - Method 2

. . Use complete sufficient statistic to find the best unbiased estimator for τ(θ).

. . 1 Find complete sufficient statistic T for

.

. . 2 Obtain

T , an unbiased estimator of using either of the following two ways

• Guess a function

T such that E T .

• Guess an unbiased estimator h X of

. Construct T E h X T , then E T E h X .

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 5 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Finding UMVUE - Method 2

. . Use complete sufficient statistic to find the best unbiased estimator for τ(θ).

. . 1 Find complete sufficient statistic T for θ. . . 2 Obtain

T , an unbiased estimator of using either of the following two ways

• Guess a function

T such that E T .

• Guess an unbiased estimator h X of

. Construct T E h X T , then E T E h X .

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 5 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Finding UMVUE - Method 2

. . Use complete sufficient statistic to find the best unbiased estimator for τ(θ).

. . 1 Find complete sufficient statistic T for θ. . . 2 Obtain φ(T), an unbiased estimator of τ(θ) using either of the

following two ways

• Guess a function

T such that E T .

• Guess an unbiased estimator h X of

. Construct T E h X T , then E T E h X .

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 5 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Finding UMVUE - Method 2

. . Use complete sufficient statistic to find the best unbiased estimator for τ(θ).

. . 1 Find complete sufficient statistic T for θ. . . 2 Obtain φ(T), an unbiased estimator of τ(θ) using either of the

following two ways

• Guess a function φ(T) such that E[φ(T)] = τ(θ).
• Guess an unbiased estimator h X of

. Construct T E h X T , then E T E h X .

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 5 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Finding UMVUE - Method 2

. . Use complete sufficient statistic to find the best unbiased estimator for τ(θ).

. . 1 Find complete sufficient statistic T for θ. . . 2 Obtain φ(T), an unbiased estimator of τ(θ) using either of the

following two ways

• Guess a function φ(T) such that E[φ(T)] = τ(θ).
• Guess an unbiased estimator h(X) of τ(θ). Construct

φ(T) = E[h(X)|T], then E[φ(T)] = E[h(X)] = τ(θ).

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 5 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Frequentists vs. Bayesians

A biased view in favor of Bayesians at http://xkcd.com/1132/

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 6 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Bayesian Statistic

.

Frequentist’s Framework

. . P = {X ∼ fX(x|θ), θ ∈ Ω} .

Bayesian Statistic

. . . . . . . .

• Parameter

is considered as a random quantity

• Distribution of

can be described by probability distribution, referred to as prior distribution

• A sample is taken from a population indexed by

, and the prior distribution is updated using information from the sample to get posterior distribution of given the sample.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 7 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Bayesian Statistic

.

Frequentist’s Framework

. . P = {X ∼ fX(x|θ), θ ∈ Ω} .

Bayesian Statistic

. .

• Parameter θ is considered as a random quantity
• Distribution of

can be described by probability distribution, referred to as prior distribution

• A sample is taken from a population indexed by

, and the prior distribution is updated using information from the sample to get posterior distribution of given the sample.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 7 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Bayesian Statistic

.

Frequentist’s Framework

. . P = {X ∼ fX(x|θ), θ ∈ Ω} .

Bayesian Statistic

. .

• Parameter θ is considered as a random quantity
• Distribution of θ can be described by probability distribution, referred

to as prior distribution

• A sample is taken from a population indexed by

, and the prior distribution is updated using information from the sample to get posterior distribution of given the sample.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 7 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Bayesian Statistic

.

Frequentist’s Framework

. . P = {X ∼ fX(x|θ), θ ∈ Ω} .

Bayesian Statistic

. .

• Parameter θ is considered as a random quantity
• Distribution of θ can be described by probability distribution, referred

to as prior distribution

• A sample is taken from a population indexed by θ, and the prior

distribution is updated using information from the sample to get posterior distribution of θ given the sample.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 7 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Bayesian Framework

• Prior distribution of θ : θ ∼ π(θ).
• Sample distribution of X given

. X f x

• Joint distribution X and

f x f x

• Marginal distribution of X.

m x f x d f x d

• Posterior distribution of

(conditional distribution of given X) x f x m x f x m x (Bayes’ Rule)

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 8 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Bayesian Framework

• Prior distribution of θ : θ ∼ π(θ).
• Sample distribution of X given θ.

X|θ ∼ f(x|θ)

• Joint distribution X and

f x f x

• Marginal distribution of X.

m x f x d f x d

• Posterior distribution of

(conditional distribution of given X) x f x m x f x m x (Bayes’ Rule)

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 8 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Bayesian Framework

• Prior distribution of θ : θ ∼ π(θ).
• Sample distribution of X given θ.

X|θ ∼ f(x|θ)

• Joint distribution X and θ

f(x, θ) = π(θ)f(x|θ)

• Marginal distribution of X.

m x f x d f x d

• Posterior distribution of

(conditional distribution of given X) x f x m x f x m x (Bayes’ Rule)

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 8 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Bayesian Framework

• Prior distribution of θ : θ ∼ π(θ).
• Sample distribution of X given θ.

X|θ ∼ f(x|θ)

• Joint distribution X and θ

f(x, θ) = π(θ)f(x|θ)

• Marginal distribution of X.

m(x) = ∫

θ∈Ω

f(x, θ)dθ = ∫

θ∈Ω

f(x|θ)π(θ)dθ

• Posterior distribution of

(conditional distribution of given X) x f x m x f x m x (Bayes’ Rule)

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 8 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Bayesian Framework

• Prior distribution of θ : θ ∼ π(θ).
• Sample distribution of X given θ.

X|θ ∼ f(x|θ)

• Joint distribution X and θ

f(x, θ) = π(θ)f(x|θ)

• Marginal distribution of X.

m(x) = ∫

θ∈Ω

f(x, θ)dθ = ∫

θ∈Ω

f(x|θ)π(θ)dθ

• Posterior distribution of θ (conditional distribution of θ given X)

π(θ|x) = f(x, θ) m(x) = f(x|θ)π(θ) m(x) (Bayes’ Rule)

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 8 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Example

Burglary (θ) Pr(Alarm|Burglary) = Pr(X = 1|θ) True (θ = 1) 0.95 False (θ = 0) 0.01 Suppose that Burglary is an unobserved parameter (θ ∈ {0, 1}), and Alarm is an observed outcome (X = {0, 1}).

• Under Frequentist’s Framework,
• If there was no burglary, there is

% of chance of alarm ringing.

• If there was a burglary, there is

% of chance of alarm ringing.

• One can come up with an estimator on

, such as MLE

• However, given that alarm already rang, one cannot calculate the

probability of burglary.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 9 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Example

Burglary (θ) Pr(Alarm|Burglary) = Pr(X = 1|θ) True (θ = 1) 0.95 False (θ = 0) 0.01 Suppose that Burglary is an unobserved parameter (θ ∈ {0, 1}), and Alarm is an observed outcome (X = {0, 1}).

• Under Frequentist’s Framework,
• If there was no burglary, there is 1% of chance of alarm ringing.
• If there was a burglary, there is

% of chance of alarm ringing.

• One can come up with an estimator on

, such as MLE

• However, given that alarm already rang, one cannot calculate the

probability of burglary.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 9 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Example

Burglary (θ) Pr(Alarm|Burglary) = Pr(X = 1|θ) True (θ = 1) 0.95 False (θ = 0) 0.01 Suppose that Burglary is an unobserved parameter (θ ∈ {0, 1}), and Alarm is an observed outcome (X = {0, 1}).

• Under Frequentist’s Framework,
• If there was no burglary, there is 1% of chance of alarm ringing.
• If there was a burglary, there is 95% of chance of alarm ringing.
• One can come up with an estimator on

, such as MLE

• However, given that alarm already rang, one cannot calculate the

probability of burglary.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 9 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Example

Burglary (θ) Pr(Alarm|Burglary) = Pr(X = 1|θ) True (θ = 1) 0.95 False (θ = 0) 0.01 Suppose that Burglary is an unobserved parameter (θ ∈ {0, 1}), and Alarm is an observed outcome (X = {0, 1}).

• Under Frequentist’s Framework,
• If there was no burglary, there is 1% of chance of alarm ringing.
• If there was a burglary, there is 95% of chance of alarm ringing.
• One can come up with an estimator on θ, such as MLE
• However, given that alarm already rang, one cannot calculate the

probability of burglary.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 9 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Example

Burglary (θ) Pr(Alarm|Burglary) = Pr(X = 1|θ) True (θ = 1) 0.95 False (θ = 0) 0.01 Suppose that Burglary is an unobserved parameter (θ ∈ {0, 1}), and Alarm is an observed outcome (X = {0, 1}).

• Under Frequentist’s Framework,
• If there was no burglary, there is 1% of chance of alarm ringing.
• If there was a burglary, there is 95% of chance of alarm ringing.
• One can come up with an estimator on θ, such as MLE
• However, given that alarm already rang, one cannot calculate the

probability of burglary.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 9 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Inference Under Bayesian’s Framework

.

Leveraging Prior Information

. . Suppose that we know that the chance of Burglary per household per night is 10−7. Pr(θ = 1|X = 1) = Pr(X = 1|θ = 1) Pr(θ = 1) Pr(X = 1) (Bayes’ rule) Pr X Pr Pr X Pr X Pr X Pr Pr X Pr Pr X Pr So, even if alarm rang, one can conclude that the burglary is unlikely to happen.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 10 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Inference Under Bayesian’s Framework

.

Leveraging Prior Information

. . Suppose that we know that the chance of Burglary per household per night is 10−7. Pr(θ = 1|X = 1) = Pr(X = 1|θ = 1) Pr(θ = 1) Pr(X = 1) (Bayes’ rule) = Pr(X = 1|θ = 1) Pr(θ = 1) Pr(θ = 1, X = 1) + Pr(θ = 0, X = 1) Pr X Pr Pr X Pr Pr X Pr So, even if alarm rang, one can conclude that the burglary is unlikely to happen.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 10 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Inference Under Bayesian’s Framework

.

Leveraging Prior Information

. . Suppose that we know that the chance of Burglary per household per night is 10−7. Pr(θ = 1|X = 1) = Pr(X = 1|θ = 1) Pr(θ = 1) Pr(X = 1) (Bayes’ rule) = Pr(X = 1|θ = 1) Pr(θ = 1) Pr(θ = 1, X = 1) + Pr(θ = 0, X = 1) = Pr(X = 1|θ = 1) Pr(θ = 1) Pr(X = 1|θ = 1) Pr(θ = 1) + Pr(X = 1|θ = 0) Pr(θ = 0) So, even if alarm rang, one can conclude that the burglary is unlikely to happen.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 10 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Inference Under Bayesian’s Framework

.

Leveraging Prior Information

. . Suppose that we know that the chance of Burglary per household per night is 10−7. Pr(θ = 1|X = 1) = Pr(X = 1|θ = 1) Pr(θ = 1) Pr(X = 1) (Bayes’ rule) = Pr(X = 1|θ = 1) Pr(θ = 1) Pr(θ = 1, X = 1) + Pr(θ = 0, X = 1) = Pr(X = 1|θ = 1) Pr(θ = 1) Pr(X = 1|θ = 1) Pr(θ = 1) + Pr(X = 1|θ = 0) Pr(θ = 0) = 0.95 × 10−7 0.95 × 10−7 + 0.01 × (1 − 10−7) ≈ 9.5 × 10−6 So, even if alarm rang, one can conclude that the burglary is unlikely to happen.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 10 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Inference Under Bayesian’s Framework

.

Leveraging Prior Information

. . Suppose that we know that the chance of Burglary per household per night is 10−7. Pr(θ = 1|X = 1) = Pr(X = 1|θ = 1) Pr(θ = 1) Pr(X = 1) (Bayes’ rule) = Pr(X = 1|θ = 1) Pr(θ = 1) Pr(θ = 1, X = 1) + Pr(θ = 0, X = 1) = Pr(X = 1|θ = 1) Pr(θ = 1) Pr(X = 1|θ = 1) Pr(θ = 1) + Pr(X = 1|θ = 0) Pr(θ = 0) = 0.95 × 10−7 0.95 × 10−7 + 0.01 × (1 − 10−7) ≈ 9.5 × 10−6 So, even if alarm rang, one can conclude that the burglary is unlikely to happen.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 10 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

What if the prior information is misleading?

.

Over-fitting to Prior Information

. . Suppose that, in fact, a thief found a security breach in my place and planning to break-in either tonight or tomorrow night for sure (with the same probability). Then the correct prior Pr(θ = 1) = 0.5. Pr X Pr X Pr Pr X Pr Pr X Pr However, if we relied on the inference based on the incorrect prior, we may end up concluding that there are % chance that this is a false alarm, and ignore it, resulting an exchange of one night of good sleep with quite a bit of fortune.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 11 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

What if the prior information is misleading?

.

Over-fitting to Prior Information

. . Suppose that, in fact, a thief found a security breach in my place and planning to break-in either tonight or tomorrow night for sure (with the same probability). Then the correct prior Pr(θ = 1) = 0.5. Pr(θ = 1|X = 1) = Pr(X = 1|θ = 1) Pr(θ = 1) Pr(X = 1|θ = 1) Pr(θ = 1) + Pr(X = 1|θ = 0) Pr(θ = 0) = 0.95 × 0.5 0.95 × 0.5 + 0.01 × (1 − 0.5) ≈ 0.99 However, if we relied on the inference based on the incorrect prior, we may end up concluding that there are % chance that this is a false alarm, and ignore it, resulting an exchange of one night of good sleep with quite a bit of fortune.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 11 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

What if the prior information is misleading?

.

Over-fitting to Prior Information

. . Suppose that, in fact, a thief found a security breach in my place and planning to break-in either tonight or tomorrow night for sure (with the same probability). Then the correct prior Pr(θ = 1) = 0.5. Pr(θ = 1|X = 1) = Pr(X = 1|θ = 1) Pr(θ = 1) Pr(X = 1|θ = 1) Pr(θ = 1) + Pr(X = 1|θ = 0) Pr(θ = 0) = 0.95 × 0.5 0.95 × 0.5 + 0.01 × (1 − 0.5) ≈ 0.99 However, if we relied on the inference based on the incorrect prior, we may end up concluding that there are > 99.9% chance that this is a false alarm, and ignore it, resulting an exchange of one night of good sleep with quite a bit of fortune.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 11 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Advantages and Drawbacks of Bayesian Inference

.

Advantages over Frequentist’s Framework

. .

• Allows making inference on the distribution of θ given data.
• Available information about

can be utilized.

• Uncertainty and information can be quantified probabilistically.

.

Drawbacks of Bayesian Inference

. . . . . . . .

• Misleading prior can result in misleading inference.
• Bayesian inference is often (but not always) prone to be ”subjective”
• See : Larry Wasserman ”Frequentist Bayes is Objective” (2006)

Bayesian Analysis 3:451-456.

• Bayesian inference could be sometimes unnecessarily complicated to

interpret, compared to Frequentist’s inference.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 12 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Advantages and Drawbacks of Bayesian Inference

.

Advantages over Frequentist’s Framework

. .

• Allows making inference on the distribution of θ given data.
• Available information about θ can be utilized.
• Uncertainty and information can be quantified probabilistically.

.

Drawbacks of Bayesian Inference

. . . . . . . .

• Misleading prior can result in misleading inference.
• Bayesian inference is often (but not always) prone to be ”subjective”
• See : Larry Wasserman ”Frequentist Bayes is Objective” (2006)

Bayesian Analysis 3:451-456.

• Bayesian inference could be sometimes unnecessarily complicated to

interpret, compared to Frequentist’s inference.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 12 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Advantages and Drawbacks of Bayesian Inference

.

Advantages over Frequentist’s Framework

. .

• Allows making inference on the distribution of θ given data.
• Available information about θ can be utilized.
• Uncertainty and information can be quantified probabilistically.

.

Drawbacks of Bayesian Inference

. . . . . . . .

• Misleading prior can result in misleading inference.
• Bayesian inference is often (but not always) prone to be ”subjective”
• See : Larry Wasserman ”Frequentist Bayes is Objective” (2006)

Bayesian Analysis 3:451-456.

• Bayesian inference could be sometimes unnecessarily complicated to

interpret, compared to Frequentist’s inference.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 12 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Advantages and Drawbacks of Bayesian Inference

.

Advantages over Frequentist’s Framework

. .

• Allows making inference on the distribution of θ given data.
• Available information about θ can be utilized.
• Uncertainty and information can be quantified probabilistically.

.

Drawbacks of Bayesian Inference

. .

• Misleading prior can result in misleading inference.
• Bayesian inference is often (but not always) prone to be ”subjective”
• See : Larry Wasserman ”Frequentist Bayes is Objective” (2006)

Bayesian Analysis 3:451-456.

• Bayesian inference could be sometimes unnecessarily complicated to

interpret, compared to Frequentist’s inference.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 12 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Advantages and Drawbacks of Bayesian Inference

.

Advantages over Frequentist’s Framework

. .

• Allows making inference on the distribution of θ given data.
• Available information about θ can be utilized.
• Uncertainty and information can be quantified probabilistically.

.

Drawbacks of Bayesian Inference

. .

• Misleading prior can result in misleading inference.
• Bayesian inference is often (but not always) prone to be ”subjective”
• See : Larry Wasserman ”Frequentist Bayes is Objective” (2006)

Bayesian Analysis 3:451-456.

• Bayesian inference could be sometimes unnecessarily complicated to

interpret, compared to Frequentist’s inference.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 12 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Advantages and Drawbacks of Bayesian Inference

.

Advantages over Frequentist’s Framework

. .

• Allows making inference on the distribution of θ given data.
• Available information about θ can be utilized.
• Uncertainty and information can be quantified probabilistically.

.

Drawbacks of Bayesian Inference

. .

• Misleading prior can result in misleading inference.
• Bayesian inference is often (but not always) prone to be ”subjective”
• See : Larry Wasserman ”Frequentist Bayes is Objective” (2006)

Bayesian Analysis 3:451-456.

• Bayesian inference could be sometimes unnecessarily complicated to

interpret, compared to Frequentist’s inference.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 12 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Bayes Estimator

.

Definition

. . Bayes Estimator of θ is defined as the posterior mean of θ. E x x d .

Example Problem

. . . . . . . . X Xn

i.i.d. Bernoulli p where

p . Assume that the prior distribution of p is Beta . Find the posterior distribution of p and the Bayes estimator of p, assuming and are known.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 13 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Bayes Estimator

.

Definition

. . Bayes Estimator of θ is defined as the posterior mean of θ. E(θ|x) = ∫

θ∈Ω

θπ(θ|x)dθ .

Example Problem

. . . . . . . . X Xn

i.i.d. Bernoulli p where

p . Assume that the prior distribution of p is Beta . Find the posterior distribution of p and the Bayes estimator of p, assuming and are known.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 13 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Bayes Estimator

.

Definition

. . Bayes Estimator of θ is defined as the posterior mean of θ. E(θ|x) = ∫

θ∈Ω

θπ(θ|x)dθ .

Example Problem

. . X1, · · · , Xn

i.i.d.

∼ Bernoulli(p) where 0 ≤ p ≤ 1. Assume that the prior

distribution of p is Beta(α, β). Find the posterior distribution of p and the Bayes estimator of p, assuming α and β are known.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 13 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Solution (1/4)

Prior distribution of p is π(p) = Γ(α + β) Γ(α)Γ(β)pα−1(1 − p)β−1 Sampling distribution of X given p is fX x p

n i

pxi p

xi

Joint distribution of X and p is fX x p fX x p p

n i

pxi p

xi

p p

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 14 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Solution (1/4)

Prior distribution of p is π(p) = Γ(α + β) Γ(α)Γ(β)pα−1(1 − p)β−1 Sampling distribution of X given p is fX(x|p) =

n

∏

i=1

{ pxi(1 − p)1−xi} Joint distribution of X and p is fX x p fX x p p

n i

pxi p

xi

p p

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 14 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Solution (1/4)

Prior distribution of p is π(p) = Γ(α + β) Γ(α)Γ(β)pα−1(1 − p)β−1 Sampling distribution of X given p is fX(x|p) =

n

∏

i=1

{ pxi(1 − p)1−xi} Joint distribution of X and p is fX(x, p) = fX(x|p)π(p) =

n

∏

i=1

{ pxi(1 − p)1−xi} Γ(α + β) Γ(α)Γ(β)pα−1(1 − p)β−1

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 14 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Solution (2/4)

The marginal distribution of X is m(x) = ∫ f(x, p)dp = ∫ 1 Γ(α + β) Γ(α)Γ(β)p

∑n

i=1 xi+α−1(1 − p)n−∑n i=1 xi+β−1dp

xi n xi n xi n xi xi n xi p

xi

p n

xi

dp

n i

xi n

n i

xi n fBeta

xi n xi

p dp

n i

xi n

n i

xi n

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 15 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Solution (2/4)

The marginal distribution of X is m(x) = ∫ f(x, p)dp = ∫ 1 Γ(α + β) Γ(α)Γ(β)p

∑n

i=1 xi+α−1(1 − p)n−∑n i=1 xi+β−1dp

= ∫ 1 Γ(α + β) Γ(α)Γ(β) Γ(∑ xi + α)Γ(n − ∑ xi + β) Γ(α + β + n) × Γ(∑ xi + α + n − ∑ xi + β) Γ(∑ xi + α)Γ(n − ∑ xi + β)p

∑ xi+α−1(1 − p)n−∑ xi+β−1dp n i

xi n

n i

xi n fBeta

xi n xi

p dp

n i

xi n

n i

xi n

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 15 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Solution (2/4)

The marginal distribution of X is m(x) = ∫ f(x, p)dp = ∫ 1 Γ(α + β) Γ(α)Γ(β)p

∑n

i=1 xi+α−1(1 − p)n−∑n i=1 xi+β−1dp

= ∫ 1 Γ(α + β) Γ(α)Γ(β) Γ(∑ xi + α)Γ(n − ∑ xi + β) Γ(α + β + n) × Γ(∑ xi + α + n − ∑ xi + β) Γ(∑ xi + α)Γ(n − ∑ xi + β)p

∑ xi+α−1(1 − p)n−∑ xi+β−1dp

= Γ(α + β) Γ(α)Γ(β) Γ(∑n

i=1 xi + α)Γ(n − ∑n i=1 xi + β)

Γ(α + β + n) × ∫ 1 fBeta(∑ xi+α,n−∑ xi+β)(p)dp

n i

xi n

n i

xi n

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 15 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Solution (2/4)

The marginal distribution of X is m(x) = ∫ f(x, p)dp = ∫ 1 Γ(α + β) Γ(α)Γ(β)p

∑n

i=1 xi+α−1(1 − p)n−∑n i=1 xi+β−1dp

= ∫ 1 Γ(α + β) Γ(α)Γ(β) Γ(∑ xi + α)Γ(n − ∑ xi + β) Γ(α + β + n) × Γ(∑ xi + α + n − ∑ xi + β) Γ(∑ xi + α)Γ(n − ∑ xi + β)p

∑ xi+α−1(1 − p)n−∑ xi+β−1dp

= Γ(α + β) Γ(α)Γ(β) Γ(∑n

i=1 xi + α)Γ(n − ∑n i=1 xi + β)

Γ(α + β + n) × ∫ 1 fBeta(∑ xi+α,n−∑ xi+β)(p)dp = Γ(α + β) Γ(α)Γ(β) Γ(∑n

i=1 xi + α)Γ(n − ∑n i=1 xi + β)

Γ(α + β + n)

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 15 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Solution (3/4)

The posterior distribution of θ|x : π(θ|x) = f(x, p) m(x) p

xi

p n

xi

xi n xi n n xi n xi p

xi

p n

xi

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 16 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Solution (3/4)

The posterior distribution of θ|x : π(θ|x) = f(x, p) m(x) = [ Γ(α + β) Γ(α)Γ(β)p

∑ xi+α−1(1 − p)n−∑ xi+β−1

] [ Γ(α + β) Γ(α)Γ(β) Γ(∑ xi + α)Γ(n − ∑ xi + β) Γ(α + β + n) ] n xi n xi p

xi

p n

xi

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 16 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Solution (3/4)

The posterior distribution of θ|x : π(θ|x) = f(x, p) m(x) = [ Γ(α + β) Γ(α)Γ(β)p

∑ xi+α−1(1 − p)n−∑ xi+β−1

] [ Γ(α + β) Γ(α)Γ(β) Γ(∑ xi + α)Γ(n − ∑ xi + β) Γ(α + β + n) ] = Γ(α + β + n) Γ(∑ xi + α)Γ(n − ∑ xi + β)p

∑ xi+α−1(1 − p)n−∑ xi+β−1

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 16 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Solution (4/4)

The Bayes estimator of p is ˆ p = ∑n

i=1 xi + α

∑n

i=1 xi + α + n − ∑n i=1 xi + β =

∑n

i=1 xi + α

α + β + n

n i

xi n n n n Guess about p from data weight Guess about p from prior weight As n increase, weight

n n

n

becomes bigger and bigger and approaches to 1. In other words, influence of data is increasing, and the influence of prior knowledge is decreasing.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 17 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Solution (4/4)

The Bayes estimator of p is ˆ p = ∑n

i=1 xi + α

∑n

i=1 xi + α + n − ∑n i=1 xi + β =

∑n

i=1 xi + α

α + β + n = ∑n

i=1 xi

n n α + β + n + α α + β α + β α + β + n Guess about p from data weight Guess about p from prior weight As n increase, weight

n n

n

becomes bigger and bigger and approaches to 1. In other words, influence of data is increasing, and the influence of prior knowledge is decreasing.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 17 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Solution (4/4)

The Bayes estimator of p is ˆ p = ∑n

i=1 xi + α

∑n

i=1 xi + α + n − ∑n i=1 xi + β =

∑n

i=1 xi + α

α + β + n = ∑n

i=1 xi

n n α + β + n + α α + β α + β α + β + n = [Guess about p from data] · weight1 + [Guess about p from prior] · weight2 As n increase, weight

n n

n

becomes bigger and bigger and approaches to 1. In other words, influence of data is increasing, and the influence of prior knowledge is decreasing.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 17 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Solution (4/4)

The Bayes estimator of p is ˆ p = ∑n

i=1 xi + α

∑n

i=1 xi + α + n − ∑n i=1 xi + β =

∑n

i=1 xi + α

α + β + n = ∑n

i=1 xi

n n α + β + n + α α + β α + β α + β + n = [Guess about p from data] · weight1 + [Guess about p from prior] · weight2 As n increase, weight1 =

n α+β+n = 1

α+β n

+1 becomes bigger and bigger and

approaches to 1. In other words, influence of data is increasing, and the influence of prior knowledge is decreasing.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 17 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Is the Bayes estimator unbiased?

E [ ∑n

i=1 +α

α + β + n ] = np + α α + β + n ̸= p Unless

α α+β = p.

Bias np n p p n As n increases, the bias approaches to zero.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 18 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Is the Bayes estimator unbiased?

E [ ∑n

i=1 +α

α + β + n ] = np + α α + β + n ̸= p Unless

α α+β = p.

Bias = np + α α + β + n − p = α − (α + β)p α + β + n As n increases, the bias approaches to zero.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 18 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Sufficient statistic and posterior distribution

.

Posterior conditioning on sufficient statistics

. . If T(X) is a sufficient statistic, then the posterior distribution of θ given X is the same to the posterior distribution given T(X). In other words, x T x

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 19 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Sufficient statistic and posterior distribution

.

Posterior conditioning on sufficient statistics

. . If T(X) is a sufficient statistic, then the posterior distribution of θ given X is the same to the posterior distribution given T(X). In other words, π(θ|x) = π(θ|T(x))

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 19 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Conjugate family

.

Definition 7.2.15

. . Let F denote the class of pdfs or pmfs for f(x|θ). A class Π of prior distributions is a conjugate family of F, if the posterior distribution is the class Π for all f ∈ F, and all priors in Π, and all x ∈ X.

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 20 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Example: Beta-Binomial conjugate

Let

• X1, · · · , Xn|p ∼ Binomial(m, p)
• p

Beta where m is known. The posterior distribution is p x Beta

n i

xi mn

n i

xi

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 21 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Example: Beta-Binomial conjugate

Let

• X1, · · · , Xn|p ∼ Binomial(m, p)
• π(p) ∼ Beta(α, β)

where m, α, β is known. The posterior distribution is p x Beta

n i

xi mn

n i

xi

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 21 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Example: Beta-Binomial conjugate

Let

• X1, · · · , Xn|p ∼ Binomial(m, p)
• π(p) ∼ Beta(α, β)

where m, α, β is known. The posterior distribution is π(p|x) ∼ Beta ( n ∑

i=1

xi + α, mn −

n

∑

i=1

xi + β )

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 21 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Example: Gamma-Poisson conjugate

• X1, · · · , Xn|λ ∼ Poisson(λ)
• Gamma
• Prior:

e

• Sampling distribution

X

i.i.d.

e

x

x fX x

n i

e

xi

xi

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 22 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Example: Gamma-Poisson conjugate

• X1, · · · , Xn|λ ∼ Poisson(λ)
• π(λ) ∼ Gamma(α, β)
• Prior:

e

• Sampling distribution

X

i.i.d.

e

x

x fX x

n i

e

xi

xi

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 22 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Example: Gamma-Poisson conjugate

• X1, · · · , Xn|λ ∼ Poisson(λ)
• π(λ) ∼ Gamma(α, β)
• Prior:

π(λ) = 1 Γ(α)βα λα−1e−λ/β

• Sampling distribution

X

i.i.d.

e

x

x fX x

n i

e

xi

xi

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 22 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Example: Gamma-Poisson conjugate

• X1, · · · , Xn|λ ∼ Poisson(λ)
• π(λ) ∼ Gamma(α, β)
• Prior:

π(λ) = 1 Γ(α)βα λα−1e−λ/β

• Sampling distribution

X|λ

i.i.d.

∼

e−λλx x! fX(x|λ) =

n

∏

i=1

e−λλxi xi!

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 22 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Gamma-Poisson conjugate (cont’d)

• Joint distribution of X and λ.

f(x|λ)π(λ) = [ n ∏

i=1

e−λλxi xi! ] 1 Γ(α)βα λα−1e−λ/β = e−nλ−λ/βλ

∑ xi+α−1

1 ∏n

i=1 xi!

1 Γ(α)βα

• Marginal distribution

m x f x d

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 23 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Gamma-Poisson conjugate (cont’d)

• Joint distribution of X and λ.

f(x|λ)π(λ) = [ n ∏

i=1

e−λλxi xi! ] 1 Γ(α)βα λα−1e−λ/β = e−nλ−λ/βλ

∑ xi+α−1

1 ∏n

i=1 xi!

1 Γ(α)βα

• Marginal distribution

m(x) = ∫ f(x|λ)π(λ)dλ

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 23 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Gamma-Poisson conjugate (cont’d)

• Posterior distribution (proportional to the joint distribution)

π(λ|x) = f(x|λ)π(λ) m(x) = e−nλ−λ/βλ

∑ xi+α−1

1 Γ(∑ xi + α) (

1 n+ 1

β

)∑ xi+α So, the posterior distribution is Gamma xi n .

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 24 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Gamma-Poisson conjugate (cont’d)

• Posterior distribution (proportional to the joint distribution)

π(λ|x) = f(x|λ)π(λ) m(x) = e−nλ−λ/βλ

∑ xi+α−1

1 Γ(∑ xi + α) (

1 n+ 1

β

)∑ xi+α So, the posterior distribution is Gamma (∑ xi + α, ( n + 1

β

)−1) .

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 24 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Example: Normal Bayes Estimators

Let X ∼ N(θ, σ2) and suppose that the prior distribution of θ is N(µ, τ 2). Assuming that σ2, µ2, τ 2 are all known, the posterior distribution of θ also becomes normal, with mean and variance given by E x x Var x

• The normal family is its own conjugate family.
• The Bayes estimator for

is a linear combination of the prior and sample means

• As the prior variance

approaches to infinity, the Bayes estimator tends toward to sample mean

• As the prior information becomes more vague, the Bayes estimator

tends to give more weight to the sample information

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 25 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Example: Normal Bayes Estimators

Let X ∼ N(θ, σ2) and suppose that the prior distribution of θ is N(µ, τ 2). Assuming that σ2, µ2, τ 2 are all known, the posterior distribution of θ also becomes normal, with mean and variance given by E[θ|x] = τ 2 τ 2 + σ2 x + σ2 σ2 + τ 2 µ Var x

• The normal family is its own conjugate family.
• The Bayes estimator for

is a linear combination of the prior and sample means

• As the prior variance

approaches to infinity, the Bayes estimator tends toward to sample mean

• As the prior information becomes more vague, the Bayes estimator

tends to give more weight to the sample information

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 25 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Example: Normal Bayes Estimators

Let X ∼ N(θ, σ2) and suppose that the prior distribution of θ is N(µ, τ 2). Assuming that σ2, µ2, τ 2 are all known, the posterior distribution of θ also becomes normal, with mean and variance given by E[θ|x] = τ 2 τ 2 + σ2 x + σ2 σ2 + τ 2 µ Var(θ|x) = σ2τ 2 σ2 + τ 2

• The normal family is its own conjugate family.
• The Bayes estimator for

is a linear combination of the prior and sample means

• As the prior variance

approaches to infinity, the Bayes estimator tends toward to sample mean

• As the prior information becomes more vague, the Bayes estimator

tends to give more weight to the sample information

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 25 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Example: Normal Bayes Estimators

Let X ∼ N(θ, σ2) and suppose that the prior distribution of θ is N(µ, τ 2). Assuming that σ2, µ2, τ 2 are all known, the posterior distribution of θ also becomes normal, with mean and variance given by E[θ|x] = τ 2 τ 2 + σ2 x + σ2 σ2 + τ 2 µ Var(θ|x) = σ2τ 2 σ2 + τ 2

• The normal family is its own conjugate family.
• The Bayes estimator for

is a linear combination of the prior and sample means

• As the prior variance

approaches to infinity, the Bayes estimator tends toward to sample mean

• As the prior information becomes more vague, the Bayes estimator

tends to give more weight to the sample information

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 25 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Example: Normal Bayes Estimators

Let X ∼ N(θ, σ2) and suppose that the prior distribution of θ is N(µ, τ 2). Assuming that σ2, µ2, τ 2 are all known, the posterior distribution of θ also becomes normal, with mean and variance given by E[θ|x] = τ 2 τ 2 + σ2 x + σ2 σ2 + τ 2 µ Var(θ|x) = σ2τ 2 σ2 + τ 2

• The normal family is its own conjugate family.
• The Bayes estimator for θ is a linear combination of the prior and

sample means

• As the prior variance

approaches to infinity, the Bayes estimator tends toward to sample mean

• As the prior information becomes more vague, the Bayes estimator

tends to give more weight to the sample information

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 25 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Example: Normal Bayes Estimators

Let X ∼ N(θ, σ2) and suppose that the prior distribution of θ is N(µ, τ 2). Assuming that σ2, µ2, τ 2 are all known, the posterior distribution of θ also becomes normal, with mean and variance given by E[θ|x] = τ 2 τ 2 + σ2 x + σ2 σ2 + τ 2 µ Var(θ|x) = σ2τ 2 σ2 + τ 2

• The normal family is its own conjugate family.
• The Bayes estimator for θ is a linear combination of the prior and

sample means

• As the prior variance τ 2 approaches to infinity, the Bayes estimator

tends toward to sample mean

• As the prior information becomes more vague, the Bayes estimator

tends to give more weight to the sample information

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 25 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Example: Normal Bayes Estimators

Let X ∼ N(θ, σ2) and suppose that the prior distribution of θ is N(µ, τ 2). Assuming that σ2, µ2, τ 2 are all known, the posterior distribution of θ also becomes normal, with mean and variance given by E[θ|x] = τ 2 τ 2 + σ2 x + σ2 σ2 + τ 2 µ Var(θ|x) = σ2τ 2 σ2 + τ 2

• The normal family is its own conjugate family.
• The Bayes estimator for θ is a linear combination of the prior and

sample means

• As the prior variance τ 2 approaches to infinity, the Bayes estimator

tends toward to sample mean

• As the prior information becomes more vague, the Bayes estimator

tends to give more weight to the sample information

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 25 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Summary

.

Today

. .

• Bayesian Statistics
• Bayes Estimator
• Conjugate family

.

Next Lecture

. . . . . . . .

• Bayesian Risk Functions
• Consistency

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 26 / 26

. . . . . .

. . . . Recap . . . . . . . Bayesian Statistics . . . . . . . Bayes Estimator . . . . . . Conjugate Family . Summary

Summary

.

Today

. .

• Bayesian Statistics
• Bayes Estimator
• Conjugate family

.

Next Lecture

. .

• Bayesian Risk Functions
• Consistency

Hyun Min Kang Biostatistics 602 - Lecture 15 March 12th, 2013 26 / 26