Chapter 8: Estimation In this chapter we will cover: 1. The - - PDF document

Chapter 8: Estimation In this chapter we will cover:

• 1. The likelihood and maximum likelihood estimation (§8.3, 8.5 Rice)
• 2. Introduction to decision theory(§15.1–15.3)

The likelihood function

• We have seen many cases where we have assume that the data comes from some distribution with some parameters

with unknown values

• For example data might come from a Binomial distribution with unknown p.
• In a simple regression analysis there are three unknown parameters β0, β1 and σ2.
• Let us write the density (or frueqency) function of this distibution as p(X|θ) where X is the data and θ the param-

eters to be estimated The likelihood function

• Suppose we observe data x1, x2, · · · , xn where these observation of n independent random variables each with a

probability density (or frequency ) function f(X|θ)

• The likelihood function is defined to be

lik(θ) =

n

• i=1

f(xi|θ) this is a function of θ since x1, x2, · · · , xn is the observed data

• The log likelihood function is the (natural) logarithm of this and is given by

ℓ(θ) =

n

• i=1

log [f(xi|θ)] 1

Example: Poisson distribution

• The number of asbestos fibres in a number of grid squares were counted. It is assumed that the number will follow

a Poisson distribution.

• Suppose we have observed the asbestos data given on page 247 Rice: i.e the counts are: 31, 34, 26, 28, 29, 27, 27,

24, 19, 34,27, 21,18, 30, 18, 17, 31, 16, 24, 24, 28, 18,22.

• We would like the estimate the value of the rate parameter λ for this Poisson distribution
• The likelihood function for the Poisson is given by

n=23

• i=1

λx

i exp(−λ)

xi!

• The log likelihood is the given by

ℓ(λ) = log(λ)

n

• i=1

xi − nλ −

n

• i=1

log xi! Example: Log-likelihood function

20 22 24 26 28 30 1258 1260 1262 1264 1266 1268 lambda Log likelihood

The plot shows the log likelihood function for the asbestos data. It is approximately quadratic 2

The maximum likelihood estimate

• One way of estimating the true parameter value θ given a set of data is to use the maximum likelihood estimate

(mle)

• This is defined to be ˆ

θ and is the value of θ as which the log likelihood function is maximised

• You can think of it as the parameter value which makes the observed data “most likely”.
• The mle is a random variable since it depends on the data

Example A: Poisson distribution

• From Rice page 254 the MLE for a Poisson distribution is given by

ˆ λ = ¯ X the sample mean of the data

Example B: Normal distribution

• If X1, X2, . . . , Xn are independent and come from a N(µ, σ2) distribution
• The likelihood function is given by

f(x1, . . . , xn|µ, σ) =

m

• i=1

1 σ √ 2π exp

• −1

2 xi − µ σ 2

• The MLE’s for µ and σ are

ˆ µ = ¯ X, ˆ σ =

• 1

n

n

• i=1

(Xi − ¯ X)2 3

Confidence intervals

• Just as in Chapter 7 when we were sampling from finite populations when we are estimating parameters we can

think of the idea of a confidence interval

• Here we want to estimate θ the parameters of a given distribution
• The interval should captures the uncertainty in the point estimate ˆ

θ.

• A confidence interval is an interval based on sample values. It has the statistical property that it contains the true

value some fixed percentage (say 95%) of the times it is used

• For the MLE the confidence intervals are typically calculated using limit theorems.

Normal distribution

• For the sample mean ¯

x, which is the MLE for µ, the 100(1 − α)% confidence interval is given by

• ¯

x − s √ntn−1(α 2 ), ¯ x + s √ntn−1(α 2 )

• where n is the sample size, S is the sample variance

s2 = 1 n − 1

n

• i=1

(xi − ¯ x)

• The term tn−1( α

2 ) denotes the point beyond which the t-distribution with n − 1 degrees of freedom has probability

α/2

• For a 95% confidence interval, Table 4 gives the following

n tn−1( α

2 )

5 2.571 10 2.228 20 2.086 ∞ 1.96 Normal distribution

• Suppose that the sample mean for a sample of size 25 is −1.23, the sample variance is 25.1
• The 95% confidence interval for µ is given by
• −1.23 −
• 25.1

25 tα, −1.23 +

• 25.1

25 tα

• From table 4 we see that we need the value for a t-distribution with 25 − 1 = 24 degrees of freedom, i.e. 2.064
• Thus the 95% confidence interval is

(−1.23 − 1.002 × 2.064, −1.23 + 1.002 × 2.064) 4

Normal distribution

• Let χ2

n−1 denote the chi-squared distribution with n − 1 degrees of freedom. This is tabulated in Table 3, Page A8

• f Rice.
• Then let

χ2

n−1(α)

be the point beyond which the χ2 distribution has probability α

• The 100(1 − α)% confidence interval for σ2 in a Normal sample is
• nˆ

σ2 χ2

n−1(α/2),

nˆ σ2 χ2

n−1(1 − α/2),

• If we want a 95% confidence interval with n = 25 then you use the α = 0.05, so

χ2

24(0.025) = 12.40, χ2 24(0.975) = 39.36

Poisson distrubution

• The mle of a Poisson(λ) distribution is the sample mean ¯

x.

• For large samples approximate confidence 100(1 − α) confidence intervals for λ are given by
• ¯

x − z(α/2)

• ¯

x n, ¯ x + z(α/2)

• ¯

x n

• where z(α) comes from the standard normal table
• If we want a 95% confidence interval z(α/2) = 1.96

Recommended Questions From §8.9 Rice please look at questions 3(a, b) , 4(a, c), 5(b), 22, 26, 27, 28, 29. 5

Bayesian methods and decision theory

• This material is from §15.1-15.2 of Rice
• Decision theory views statistics as a mathematical tool for making decisions in the face of uncertainty
• The decision maker has to choose an action a from a set A of all possible actions
• The data on which the decision is made is X and comes from a distribution function with parameters θ. These

represent states of nature

• The decision is made by a statistical decision function d such that

a = d(X)

• The risk in using the decision rule d is evaluated by a loss function l(θ, d(X)) which depends on the actual state of

nature θ Example A

• Suppose a manufacturer produces a lot of N items, n are randomly sampled and checked to see if they are defective
• Let ˆ

p be the sample proportion which are defective

• If the lot is sold for $M with a penalty $P if the number of defective items in the lot exceeds a proportion p0
• The action space is

{sell, junk}

• The data is X = ˆ

p and the state of nature is θ = p the proportion in the lot

• The loss function is given by

State of nature Sell Junk p < p0 −$M $C p ≥ p0 $P $C Risk

• The loss function l(θ, d(X)) depends on the random variable X
• The risk is called the expected loss i.e.

R(θ, d) = E [l(θ, d(X)]

• We want to find decision rules d which have ‘small’ risk
• One problem is that the risk R(θ, d) depends on θ which not known to us

6

Minimax rule

• It is possible to have two decision rules d1 and d2 such that

R(θ1, d1) < R(θ1, d2) R(θ2, d1) > R(θ2, d2)

• Thus d1 is preferred if θ1 was the state of nature, but d2 would be preferred is θ2 was
• The minimax rule takes a very conservative approach and considers the worst possible risk i.e.

max

θ∈Θ [R(θ, d)]

• It then chooses the decision function d∗ that minimizes this maximum risk i.e.

min

d

• max

θ∈Θ [R(θ, d)]

• If such a rule exists it is called the minimax rule

Bayes rule

• An alternative to the conservative minimax rule is to assume some distribution p(θ) called the prior distribution
• Then look at the average risk i.e.

B(d) = E [R(θ, d)] where this expectation is over the prior distribution p(θ)

• The decision function d∗∗ which minimizes the Bayes risk is called the Bayes rule

Example

• Suppose that the lot contains 21 items and one of them is randomly sampled
• The remaining 20 units can be sold for $1 each but if they are defective it costs $2.
• To junk the whole lot costs $1
• Suppose that there are two decisions:

d1: sell if the sample is good, junk if sample is defective d2: Always sell

• We can evaluate these rules by the minimax and bayes criterion

7

Example: Minimax

• Let X = 1 if the tested item is good and 0 if defective, and let k the the actual number of defective items (i.e. the

state of nature)

• The loss functions are then

l(k, d1(X)) = −20 + 2k if X = 1 1 if X = 0 l(k, d2(X)) =

• −20 + 2k

if X = 1 −20 + 2(k − 1) if X = 0

• The risk function then is the expected value of X for which we need the distribution of X i.e.

P(X = 0) = k 21, P(X = 1) = 1 − k 21

• So the risk functions are

R(k, d1) = −20 + 3k − 2k2 21 R(k, d2) = −20 + 40k 21 8

Example: Minimax

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 5 10 15 20 −10 10 20 k Risk 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Example: Bayes Rule

• Suppose we want to use Bayes rule. we need to assume a prior distribution for k, the number of defectives
• Let us assume that this prior distribution is Binomial(p, 20) i.e. we assume that each item is defective independently
• f the others with a fixed probability p.
• Suppose, from previous experience, we choose p = 0.1
• The Bayes risks of d1 and d2 are then (see page 577 Rice)

B(d1) = −20 + 61p − 40p2 = −14.3 B(d2) = −20 + 40p = −19.6

• Hence if p = 0.1 the decision rule d2 has a smaller Bayes risk than d1

9

Recommended Questions From §15.5 Rice please look at questions 1 (a), (b), (c), 2, 5 10