[PPT] - Introduction to Bayesian Statistics Lecture 3: Single Parameter (II) PowerPoint Presentation

SLIDE 1

Introduction to Bayesian Statistics

Lecture 3: Single Parameter (II)

Rung-Ching Tsai

Department of Mathematics National Taiwan Normal University

March 11, 2015

SLIDE 2

Conjugate Prior Distributions

Definition of Conjugacy: If F is a class of sampling distributions p(y|θ), and P is a class of prior distributions for θ, then the class P is conjugate for F if p(θ|y) ∈ P for all p(·|θ) ∈ F and p(θ) ∈ P.

Advantages of using conjugate priors:
computational convenience
being interpretable as additional data
Example: Beta is conjugate for binomial with θ ∼ Beta(α, β) and

θ|y ∼ Beta(α + y, β + n − y).

Exercise: What is the conjugate prior for Poisson(λ)?

2 of 14

SLIDE 3

Conjugate Prior Distributions for exponential families

Definition: The class F is an exponential family if all its members have the form p(yi|θ) = f (yi)g(θ)eφ(θ)T u(yi), where φ(θ): the “natural parameter” of the family F. Exercise: Show that the binomial(n, θ) is an exponential family with natural parameter logit(θ), and the conjugate prior on θ are Beta distributions.

3 of 14

SLIDE 4

Conjugate Prior Distributions for exponential families

Likelihood of θ:

p(y|θ) = n

i=1

f (yi)

g(θ)nexp
φ(θ)T

n

i=1

u(yi))

∝

g(θ)nexp

φ(θ)Tt(y)
,

where t(y) = n

i=1 u(yi): sufficient statistic for θ

(Conjugate) Prior:

p(θ) ∝ g(θ)ηexp

φ(θ)Tν
Posterior:

p(θ|y) ∝ g(θ)η+nexp

φ(θ)T(ν + t(y)
.
Known fact: Exponential families are, in general, the only classes
f distributions that have natural conjugate priors.

4 of 14

SLIDE 5

Single Parameter θ: Continuous y

y ∼ normal(θ, σ2), σ2 known, use Bayesian approach to estimate

θ.

choose a conjugate prior for θ, p(θ) = eAθ2+Bθ+C, such that

p(θ) ∝ exp

− 1

2τ 2 (θ − µ0)2

likelihood of θ: p(y|θ) =

1 √ 2πσexp

− 1

2σ2 (y − θ)2

find the posterior distribution of θ:

p(θ|y) ∝ p(θ)p(y|θ) ∝ exp

−1

2 (y − θ)2 σ2 + (θ − µ0)2 τ 2

∝

exp

− 1

2τ 2

1

(θ − µ1)2

,

that is, θ|y ∼ normal(µ1, τ 2

1 ), where

µ1 =

1 τ2

µ0+ 1

σ2 y 1 τ2

+ 1

σ2

and

1 τ 2

1 =

1 τ 2

0 + 1

σ2 .

5 of 14

SLIDE 6

Single Parameter θ: Continuous y

θ ∼ normal(µ0, τ 2

0 ), y ∼ normal(θ, σ2) ⇒ θ|y ∼ normal(µ1, τ 2 1 )

posterior precision

1 τ 2

1

Definition of precision: the inverse of variance
1

τ 2

1 =

1 τ 2

0 + 1

σ2 , i.e., the posterior precision equals the prior precision

plus the data precision.

posterior mean µ1
µ1 =

1 τ2

µ0+ 1

σ2 y 1 τ2

+ 1

σ2

, i.e., the posterior mean is a weighted average of the prior mean and the observed value y, with weights proportional to the precision.

the prior mean adjusted toward the observed y:

µ1 = µ0 + (y − µ0)

τ 2 σ2+τ 2

0 .

a compromise between the prior mean and the observed data y, with

data shrunk toward the prior mean: µ1 = y − (y − µ0)

σ2 σ2+τ 2

6 of 14

SLIDE 7

Single Parameter θ: Continuous y

Posterior predictive distribution p(˜ y|y) p(˜ y|y) =

p(˜

y|θ)p(θ|y)dθ ∝

exp
− 1

2σ2 (˜ y − θ)2

exp
− 1

2τ 2

1

(θ − µ1)2

dθ
˜

y|y ∼ normal(?, ?)

E(˜

y|y) = E(E(˜ y|θ, y)|y) = E(θ|y) = µ1

var(˜

y|y) = E(var(˜ y|θ, y)|y) + var(E(˜ y|θ, y)|y) = E(σ2|y) + var(θ|y) = σ2 + τ 2

1 .

Note. E(˜

y|θ) = θ, var(˜ y|θ) = σ2

7 of 14

SLIDE 8

Single Parameter θ: Continuous y = (y1, · · · , yn)

y1, · · · , yn

iid

∼ normal(θ, σ2), σ2 known, use Bayesian approach to estimate θ.

choose a conjugate prior for θ, p(θ) ∝ exp
− 1

2τ 2

0 (θ − µ0)2

likelihood of θ: p(y|θ) = n

i=1 1 √ 2πσexp

− 1

2σ2 (yi − θ)2

find the posterior distribution of θ:

p(θ|y) ∝ p(θ)p(y|θ) ∝ exp

−1

2 n

i=1(yi − θ)2

σ2 + (θ − µ0)2 τ 2

∝

exp

− 1

2τ 2

n

(θ − µn)2

,

that is, θ|y ∼ normal(µn, τ 2

n ), where

µn =

1 τ2

µ0+ n

σ2 ¯

y

1 τ2

+ n

σ2

and

1 τ 2

n =

1 τ 2

0 + n

σ2 .

8 of 14

SLIDE 9

Single Parameter θ: Continuous y = (y1, · · · , yn)

y1, · · · , yn

iid

∼ normal(θ, σ2), σ2 known, θ ∼ normal(µ0, τ 2

0 )

⇒ θ|y ∼ normal(µn, τ 2

n)

posterior precision

1 τ 2

n = 1

τ 2

0 + n

σ2 ; posterior mean µn =

1 τ2

µ0+ n

σ2 ¯

y

1 τ2

+ n

σ2

If n is large, the posterior distribution is largely determined by σ2 and

the sample value ¯ y.

As τ0 → ∞ with n fixed, or as n → ∞ with τ 2

0 fixed, we have

p(θ|y) ≈ normal(θ|¯ y, σ2 n ).

Compare the well-known result of classical statistics:

¯ y|θ, σ2 ∼ normal(θ, σ2

n ) leads to the use of ¯

y ± 1.96 σ

√n as a 95%

confidence interval for θ.

Bayesian approach gives the same result for noninformative prior.

9 of 14

SLIDE 10

Exercise

A random sample of n students is drawn from a large population, and their weights are measured. The average weight of the n sampled students is ¯ y = 150 pounds. Assume the weights in the population are normally distributed with unknown mean θ and known standard deviation 20 pounds. Suppose your prior distribution for θ is normal with mean 180 and standard deviation 40. (a) Give your posterior distribution for θ. (b) A new student is sampled at random from the same population and has a weight of ˜ y pounds. Give a posterior predictive distribution for ˜ y. (c) For n = 10, give a 95% posterior interval for θ and a 95% posterior predictive interval for ˜ y. (d) Do the same for n = 100.

10 of 14

SLIDE 11

Single Parameter σ2: Continuous y = (y1, · · · , yn)

y1, · · · , yn

iid

∼ normal(θ, σ2), θ known, use Bayesian approach to estimate σ2.

likelihood of σ2:

p(y|σ2) =

n

i=1

1 √ 2πσ exp

− 1

2σ2 (yi − θ)2

∝

σ−nexp

− 1

2σ2 (yi − θ)2

=

(σ2)− n

2 exp(− n

2σ2 v) where v = 1

n

i=1(yi − θ)2

choose a conjugate prior for σ2 (inverse-gamma):

p(σ2) ∝ (σ2)−(α+1)e− β

σ2

11 of 14

SLIDE 12

Single Parameter σ2: Continuous y = (y1, · · · , yn)

y1, · · · , yn

iid

∼ normal(θ, σ2), θ known, estimate σ2.

likelihood of σ2: p(y|σ2) = (σ2)− n

2 exp(− n

2σ2 v)2

choose a conjugate prior for σ2 (inverse-gamma):

p(σ2) ∝ (σ2)−(α+1)e− β

σ2 , i.e., σ2 ∼ Inv-χ2(ν0, σ2

0)

Note. A scaled inverse-χ2 distribution with scale σ2

0 and ν0 degrees of

freedom:

σ2

0ν0

X

∼ χ2

ν0, i.e., X ∼ Inv-χ2(ν0, σ2 0)

find the posterior distribution of σ2:

p(σ2) ∝ p(σ2)p(y|σ2) ∝ σ2 σ2 ν0/2+1 exp

−σ2

0ν0

2σ2

· (σ2)− n

2 exp(−n

2 v σ2 ) ∝ (σ2)−((n+ν0)/2+1)exp

− 1

2σ2 (ν0σ2

0 + nv)

.

that is, σ2|y ∼ Inv-χ2 ν0 + n, ν0σ2

0+nv

ν0+n

.

12 of 14

SLIDE 13

Homework II

1. The following Table gives the number of fatal accidents and deaths
n scheduled airline flights per year over a ten-year period.

Year Fatal Passenger Death Year Fatal Passenger Death accidents death rate accidents death rate 1976 24 734 0.19 1981 21 362 0.06 1977 25 516 0.12 1982 26 764 0.13 1978 31 754 0.15 1983 20 809 0.13 1979 31 877 0.16 1984 16 223 0.03 1980 22 814 0.14 1985 22 1066 0.15

(a) Assume that the number of fatal accidents in each year are independent with a Poisson(θ) distribution. Set a prior distribution for θ and determine the posterior distribution based on the data from 1976 through 1985. Under this model, give a 95% predictive interval for the number of fatal accident in 1986. You can use normal approximation to the gamma and Poisson or compute using simulation. (b) Repeat (a) above, replacing ‘fatal accidents’ with ‘passenger deaths’.

13 of 14

SLIDE 14

Homework II

2. Censored and uncensored data in the exponential model:

(a) Suppose y|θ is exponentially distributed with rate θ, and the marginal (prior) distribution of θ is Gamma(α, β). Suppose we observe that y ≥ 100, but do not observe the exact value of y. What is the posterior distribution, p(θ|y ≥ 100), as a function of α and β? Write down the posterior mean and variance of θ. (b) In the above problem, suppose that we are now told that y is exactly

100. Now what are the posterior mean and variance of θ?

14 of 14