Single-parameter models: Gaussian (normal) data Applied Bayesian - - PowerPoint PPT Presentation

Single-parameter models: Gaussian (normal) data

Applied Bayesian Statistics

• Dr. Earvin Balderama

Department of Mathematics & Statistics Loyola University Chicago

September 19, 2017

1

The Gaussian (normal) model Last edited September 8, 2017 by Earvin Balderama <ebalderama@luc.edu>

One-parameter models

The normal model

The Gaussian (normal) distribution is possibly the most useful (or most utilized) model in data analyses. Y ∼ Normal(µ, σ2) Y ∈ (−∞, ∞) E(Y) = µ V(Y) = σ2 If we choose a normal model for our likelihood function, there are two parameters to estimate. However, we can break up the task into 2

• ne-parameter models by:

1

Estimating the mean, assuming the variance is known.

2

Estimating the variance, assuming the mean is known.

2

The Gaussian (normal) model Last edited September 8, 2017 by Earvin Balderama <ebalderama@luc.edu>

One-parameter models

Estimating µ - Likelihood

Suppose we have n independent and identically distributed Gaussian

• bservations Y1, . . . , Yn. Given a mean µ and (known) variance σ2, the

distribution of each Yi is Yi |µ, σ2 iid ∼ Normal(µ, σ2) Thus, the likelihood function for Y1 = y1, . . . , Yn = yn is f(y1, . . . , yn |µ, σ2) =

n

• i=1

1 √ 2πσ exp

• −1

2 yi − µ σ 2 =

• 1

√ 2πσ n exp

• − 1

2σ2

n

• i=1

(yi − µ)2

• 3

The Gaussian (normal) model Last edited September 8, 2017 by Earvin Balderama <ebalderama@luc.edu>

One-parameter models

Estimating µ - Prior

µ is the parameter of interest, and is continuous over the entire real line. A natural prior distribution to select would then be µ |σ2 ∼ Normal(θ, τ 2). To make the math easier to interpret later on, simply let τ 2 = σ2

m .

µ |σ2 ∼ Normal

• θ, σ2

m

• ,

where the prior mean θ is the best guess before we observe data, and the prior variance σ2

m (via m > 0) controls the strength of the prior.

Note: How to choose m?

4

The Gaussian (normal) model Last edited September 8, 2017 by Earvin Balderama <ebalderama@luc.edu>

One-parameter models

Estimating µ - Posterior

We can now derive the posterior distribution, which happens to be: µ |y1, . . . , yn, σ2 ∼ Normal n¯ y + mθ n + m , σ2 n + m

• 5

The Gaussian (normal) model Last edited September 8, 2017 by Earvin Balderama <ebalderama@luc.edu>

One-parameter models

Estimating µ - Shrinkage

The prior mean was E(µ |σ2) = θ. The posterior mean is ˆ µB = E(µ |y1, . . . , yn, σ2) = n¯ y + mθ n + m =

• n

n + m

• ¯

y +

• m

n + m

• θ

The posterior mean is between the sample mean ¯ y and the prior mean θ.

1

When is ˆ µB close to the sample mean ¯ y? small m

2

When is ˆ µB shrunk towards the prior mean θ? large m

6

The Gaussian (normal) model Last edited September 8, 2017 by Earvin Balderama <ebalderama@luc.edu>

One-parameter models

Estimating µ - Shrinkage

The prior variance was V(µ |σ2) = σ2 m . The posterior variance is V(µ |y1, . . . , yn, σ2) = σ2 n + m Note: Recall the sampling variance of ¯ Y is σ2

n .

m can thus be loosely interpreted as the “prior number of observations.”

7

The Gaussian (normal) model Last edited September 8, 2017 by Earvin Balderama <ebalderama@luc.edu>

One-parameter models

Estimating σ2 - Likelihood

Suppose we have n independent and identically distributed Gaussian

• bservations Y1, . . . , Yn. Given a variance σ2 and (known) mean µ, the

distribution of each Yi is Yi |µ, σ2 iid ∼ Normal(µ, σ2) Thus, the likelihood function for Y1 = y1, . . . , Yn = yn is f(y1, . . . , yn |µ, σ2) =

n

• i=1

1 √ 2πσ exp

• −1

2 yi − µ σ 2 =

• 1

√ 2πσ n exp

• − 1

2σ2

n

• i=1

(yi − µ)2

• 8

The Gaussian (normal) model Last edited September 8, 2017 by Earvin Balderama <ebalderama@luc.edu>

One-parameter models

Estimating σ2 - Prior

σ2 is the parameter of interest, and is continuous over (0, ∞). So, naturally, we would want to select the prior distribution σ2 |µ ∼ Gamma(a, b).

9

The Gaussian (normal) model Last edited September 8, 2017 by Earvin Balderama <ebalderama@luc.edu>

One-parameter models

Estimating σ2 - Prior

σ2 is the parameter of interest, and is continuous over (0, ∞). So, naturally, we would want to select the prior distribution σ2 |µ ∼ Gamma(a, b). However, the gamma prior is not conjugate for the normal variance. The gamma prior is conjugate for the precision,

1 σ2 .

Thus, the math is easier if we use:

1 σ2

• µ ∼ Gamma(a, b),

which implies σ2 |µ ∼ InverseGamma(a, b), and the PDF for inverse gamma is f(σ2 |µ) = ba Γ(a)

• σ2−a−1e−b/σ2

9

The Gaussian (normal) model Last edited September 8, 2017 by Earvin Balderama <ebalderama@luc.edu>

One-parameter models

Estimating σ2 - Posterior

We can now derive the posterior distribution, which happens to be: σ2

• y1, . . . , yn, µ ∼ InverseGamma

n 2 + a, SSE 2 + b

• ,

where the sum of squared errors SSE =

n

• i=1

(yi − µ)2.

10

The Gaussian (normal) model Last edited September 8, 2017 by Earvin Balderama <ebalderama@luc.edu>

One-parameter models

Estimating σ2 - Shrinkage

The prior mean (if it exists, i.e., for a > 1) was E(σ2 |µ) =

b a−1.

The posterior mean is E(σ2 |y1, . . . , yn, µ) =

SSE 2

+ b

n 2 + a − 1

= SSE + b n + 2a − 2 It is common to take a and b to be small to give an uninformative prior,

= ⇒ so that the posterior mean approximates the sample variance SSE

n−1.

11

The Gaussian (normal) model Last edited September 8, 2017 by Earvin Balderama <ebalderama@luc.edu>