[PPT] - Likelihood and Point Estimation Lecture 09 Biostatistics 602 - PowerPoint Presentation

SLIDE 1

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

. .

Biostatistics 602 - Statistical Inference Lecture 09 Likelihood and Point Estimation

Hyun Min Kang February 7th, 2013

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 1 / 24

SLIDE 2

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Likelihood Function

.

Definition

. . X1, · · · , Xn

i.i.d.

∼ fX(x|θ). The join distribution of X = (X1, · · · , Xn) is

fX(x|θ) =

n

∏

i=1

fX(xi|θ) Given that X = x is observed, the function of θ defined by L(θ|x) = f(x|θ) is called the likelihood function.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 2 / 24

SLIDE 3

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Examples of Likelihood Function - 1/3

X1, X2, X3, X4

i.i.d.

∼ Bernoulli(p), 0 < p < 1.

x

T

Intuitively, it is more likely that p is larger than smaller.
L p x

f x p

i

pxi p

x

p .

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 3 / 24

SLIDE 4

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Examples of Likelihood Function - 1/3

X1, X2, X3, X4

i.i.d.

∼ Bernoulli(p), 0 < p < 1.

x = (1, 1, 1, 1)T
Intuitively, it is more likely that p is larger than smaller.
L p x

f x p

i

pxi p

x

p .

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 3 / 24

SLIDE 5

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Examples of Likelihood Function - 1/3

X1, X2, X3, X4

i.i.d.

∼ Bernoulli(p), 0 < p < 1.

x = (1, 1, 1, 1)T
Intuitively, it is more likely that p is larger than smaller.
L p x

f x p

i

pxi p

x

p .

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 3 / 24

SLIDE 6

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Examples of Likelihood Function - 1/3

X1, X2, X3, X4

i.i.d.

∼ Bernoulli(p), 0 < p < 1.

x = (1, 1, 1, 1)T
Intuitively, it is more likely that p is larger than smaller.
L(p|x) = f(x|p) = ∏4

i=1 pxi(1 − p)1−x1 = p4.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 3 / 24

SLIDE 7

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Examples of Likelihood Function - 1/3

X1, X2, X3, X4

i.i.d.

∼ Bernoulli(p), 0 < p < 1.

x = (1, 1, 1, 1)T
Intuitively, it is more likely that p is larger than smaller.
L(p|x) = f(x|p) = ∏4

i=1 pxi(1 − p)1−x1 = p4.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 3 / 24

SLIDE 8

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Examples of Likelihood Function - 2/3

X1, X2, X3, X4

i.i.d.

∼ Bernoulli(p), 0 < p < 1.

x

T

Intuitively, it is more likely that p is smaller than larger.
L p x

f x p

i

pxi p

x

p .

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 4 / 24

SLIDE 9

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Examples of Likelihood Function - 2/3

X1, X2, X3, X4

i.i.d.

∼ Bernoulli(p), 0 < p < 1.

x = (0, 0, 0, 0)T
Intuitively, it is more likely that p is smaller than larger.
L p x

f x p

i

pxi p

x

p .

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 4 / 24

SLIDE 10

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Examples of Likelihood Function - 2/3

X1, X2, X3, X4

i.i.d.

∼ Bernoulli(p), 0 < p < 1.

x = (0, 0, 0, 0)T
Intuitively, it is more likely that p is smaller than larger.
L p x

f x p

i

pxi p

x

p .

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 4 / 24

SLIDE 11

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Examples of Likelihood Function - 2/3

X1, X2, X3, X4

i.i.d.

∼ Bernoulli(p), 0 < p < 1.

x = (0, 0, 0, 0)T
Intuitively, it is more likely that p is smaller than larger.
L(p|x) = f(x|p) = ∏4

i=1 pxi(1 − p)1−x1 = (1 − p)4.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 4 / 24

SLIDE 12

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Examples of Likelihood Function - 2/3

X1, X2, X3, X4

i.i.d.

∼ Bernoulli(p), 0 < p < 1.

x = (0, 0, 0, 0)T
Intuitively, it is more likely that p is smaller than larger.
L(p|x) = f(x|p) = ∏4

i=1 pxi(1 − p)1−x1 = (1 − p)4.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 4 / 24

SLIDE 13

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Examples of Likelihood Function - 3/3

X1, X2, X3, X4

i.i.d.

∼ Bernoulli(p), 0 < p < 1.

x

T

Intuitively, it is more likely that p is somewhere in the middle than in

the extremes.

L p x

f x p

i

pxi p

x

p p .

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 5 / 24

SLIDE 14

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Examples of Likelihood Function - 3/3

X1, X2, X3, X4

i.i.d.

∼ Bernoulli(p), 0 < p < 1.

x = (1, 1, 0, 0)T
Intuitively, it is more likely that p is somewhere in the middle than in

the extremes.

L p x

f x p

i

pxi p

x

p p .

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 5 / 24

SLIDE 15

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Examples of Likelihood Function - 3/3

X1, X2, X3, X4

i.i.d.

∼ Bernoulli(p), 0 < p < 1.

x = (1, 1, 0, 0)T
Intuitively, it is more likely that p is somewhere in the middle than in

the extremes.

L p x

f x p

i

pxi p

x

p p .

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 5 / 24

SLIDE 16

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Examples of Likelihood Function - 3/3

X1, X2, X3, X4

i.i.d.

∼ Bernoulli(p), 0 < p < 1.

x = (1, 1, 0, 0)T
Intuitively, it is more likely that p is somewhere in the middle than in

the extremes.

L(p|x) = f(x|p) = ∏4

i=1 pxi(1 − p)1−x1 = p2(1 − p)2.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 5 / 24

SLIDE 17

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Examples of Likelihood Function - 3/3

X1, X2, X3, X4

i.i.d.

∼ Bernoulli(p), 0 < p < 1.

x = (1, 1, 0, 0)T
Intuitively, it is more likely that p is somewhere in the middle than in

the extremes.

L(p|x) = f(x|p) = ∏4

i=1 pxi(1 − p)1−x1 = p2(1 − p)2.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 5 / 24

SLIDE 18

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Point Estimation : Ingredients

Data: x = (x1, · · · , xn) - realizations of random variables

(X1, · · · , Xn).

X

Xn

i.i.d. fX x

.

Assume a model

fX x

p

where the functional form of fX x is known, but is unknown.

Task is to use data x to make inference on

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 6 / 24

SLIDE 19

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Point Estimation : Ingredients

Data: x = (x1, · · · , xn) - realizations of random variables

(X1, · · · , Xn).

X1, · · · , Xn

i.i.d.

∼ fX(x|θ).

Assume a model

fX x

p

where the functional form of fX x is known, but is unknown.

Task is to use data x to make inference on

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 6 / 24

SLIDE 20

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Point Estimation : Ingredients

Data: x = (x1, · · · , xn) - realizations of random variables

(X1, · · · , Xn).

X1, · · · , Xn

i.i.d.

∼ fX(x|θ).

Assume a model P = {fX(x|θ) : θ ∈ Ω ⊂ Rp} where the functional

form of fX(x|θ) is known, but θ is unknown.

Task is to use data x to make inference on

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 6 / 24

SLIDE 21

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Point Estimation : Ingredients

Data: x = (x1, · · · , xn) - realizations of random variables

(X1, · · · , Xn).

X1, · · · , Xn

i.i.d.

∼ fX(x|θ).

Assume a model P = {fX(x|θ) : θ ∈ Ω ⊂ Rp} where the functional

form of fX(x|θ) is known, but θ is unknown.

Task is to use data x to make inference on θ

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 6 / 24

SLIDE 22

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Point Estimation

.

Definition

. . If we use a function of sample w(X1, · · · , Xn) as a ”guess” of τ(θ), where τ(θ) is a function of true parameter θ. Then w X w X Xn is called a point estimator of . The realization of the estimation, w x w x xn is called the estimate of . .

Example

. . . . . . . .

X

Xn

i.i.d.

, where .

Suppose n

, and x x .

Define w

X Xn

n n i

Xi X .

Define w

X Xn X .

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 7 / 24

SLIDE 23

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Point Estimation

.

Definition

. . If we use a function of sample w(X1, · · · , Xn) as a ”guess” of τ(θ), where τ(θ) is a function of true parameter θ. Then w(X) = w(X1, · · · , Xn) is called a point estimator of τ(θ). The realization of the estimation, w x w x xn is called the estimate of . .

Example

. . . . . . . .

X

Xn

i.i.d.

, where .

Suppose n

, and x x .

Define w

X Xn

n n i

Xi X .

Define w

X Xn X .

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 7 / 24

SLIDE 24

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Point Estimation

.

Definition

. . If we use a function of sample w(X1, · · · , Xn) as a ”guess” of τ(θ), where τ(θ) is a function of true parameter θ. Then w(X) = w(X1, · · · , Xn) is called a point estimator of τ(θ). The realization of the estimation, w(x) = w(x1, · · · , xn) is called the estimate of τ(θ). .

Example

. .

X1, · · · , Xn

i.i.d.

∼ N(θ, 1), where θ ∈ Ω ∈ R.

Suppose n

, and x x .

Define w

X Xn

n n i

Xi X .

Define w

X Xn X .

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 7 / 24

SLIDE 25

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Point Estimation

.

Definition

. . If we use a function of sample w(X1, · · · , Xn) as a ”guess” of τ(θ), where τ(θ) is a function of true parameter θ. Then w(X) = w(X1, · · · , Xn) is called a point estimator of τ(θ). The realization of the estimation, w(x) = w(x1, · · · , xn) is called the estimate of τ(θ). .

Example

. .

X1, · · · , Xn

i.i.d.

∼ N(θ, 1), where θ ∈ Ω ∈ R.

Suppose n = 6, and (x1, · · · , x6) = (2.0, 2.1, 2.9, 2.6, 1.2, 1.8).
Define w

X Xn

n n i

Xi X .

Define w

X Xn X .

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 7 / 24

SLIDE 26

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Point Estimation

.

Definition

. . If we use a function of sample w(X1, · · · , Xn) as a ”guess” of τ(θ), where τ(θ) is a function of true parameter θ. Then w(X) = w(X1, · · · , Xn) is called a point estimator of τ(θ). The realization of the estimation, w(x) = w(x1, · · · , xn) is called the estimate of τ(θ). .

Example

. .

X1, · · · , Xn

i.i.d.

∼ N(θ, 1), where θ ∈ Ω ∈ R.

Suppose n = 6, and (x1, · · · , x6) = (2.0, 2.1, 2.9, 2.6, 1.2, 1.8).
Define w1(X1, · · · , Xn) = 1

n

∑n

i=1 Xi = X = 2.1.

Define w

X Xn X .

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 7 / 24

SLIDE 27

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Point Estimation

.

Definition

. . If we use a function of sample w(X1, · · · , Xn) as a ”guess” of τ(θ), where τ(θ) is a function of true parameter θ. Then w(X) = w(X1, · · · , Xn) is called a point estimator of τ(θ). The realization of the estimation, w(x) = w(x1, · · · , xn) is called the estimate of τ(θ). .

Example

. .

X1, · · · , Xn

i.i.d.

∼ N(θ, 1), where θ ∈ Ω ∈ R.

Suppose n = 6, and (x1, · · · , x6) = (2.0, 2.1, 2.9, 2.6, 1.2, 1.8).
Define w1(X1, · · · , Xn) = 1

n

∑n

i=1 Xi = X = 2.1.

Define w2(X1, · · · , Xn) = X(1) = 1.2.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 7 / 24

SLIDE 28

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Method of Moments

A method to equate sample moments to population moments and solve equations. Sample moments Population moments m

n n i

Xi E X m

n n i

Xi E X m

n n i

Xi E X . . . . . . Point estimator of T is obtained by solving equations like this. m m . . . . . . mk

k

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 8 / 24

SLIDE 29

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Method of Moments

A method to equate sample moments to population moments and solve equations. Sample moments Population moments m1 = 1

n

∑n

i=1 Xi

µ′

1 = E[X|θ] = µ′ 1(θ)

m2 = 1

n

∑n

i=1 X2 i

µ′

2 = E[X|θ] = µ′ 2(θ)

m3 = 1

n

∑n

i=1 X3 i

µ′

3 = E[X|θ] = µ′ 3(θ)

. . . . . . Point estimator of T is obtained by solving equations like this. m m . . . . . . mk

k

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 8 / 24

SLIDE 30

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Method of Moments

A method to equate sample moments to population moments and solve equations. Sample moments Population moments m1 = 1

n

∑n

i=1 Xi

µ′

1 = E[X|θ] = µ′ 1(θ)

m2 = 1

n

∑n

i=1 X2 i

µ′

2 = E[X|θ] = µ′ 2(θ)

m3 = 1

n

∑n

i=1 X3 i

µ′

3 = E[X|θ] = µ′ 3(θ)

. . . . . . Point estimator of T(θ) is obtained by solving equations like this. m m . . . . . . mk

k

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 8 / 24

SLIDE 31

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Method of Moments

A method to equate sample moments to population moments and solve equations. Sample moments Population moments m1 = 1

n

∑n

i=1 Xi

µ′

1 = E[X|θ] = µ′ 1(θ)

m2 = 1

n

∑n

i=1 X2 i

µ′

2 = E[X|θ] = µ′ 2(θ)

m3 = 1

n

∑n

i=1 X3 i

µ′

3 = E[X|θ] = µ′ 3(θ)

. . . . . . Point estimator of T(θ) is obtained by solving equations like this. m1 = µ′

1(θ)

m2 = µ′

2(θ)

. . . . . . mk = µ′

k(θ)

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 8 / 24

SLIDE 32

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Examples of method of moments estimator

.

Problem

. . X1, · · · , Xn

i.i.d.

∼ N(µ, σ2). Find estimator for µ, σ2.

.

Solution

. . . . . . . . EX X EX EX Var X n

n i

Xi X

n n i

Xi Solving the two equations above, X,

n i

Xi X n.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 9 / 24

SLIDE 33

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Examples of method of moments estimator

.

Problem

. . X1, · · · , Xn

i.i.d.

∼ N(µ, σ2). Find estimator for µ, σ2.

.

Solution

. . µ′

1

= EX = µ = X EX EX Var X n

n i

Xi X

n n i

Xi Solving the two equations above, X,

n i

Xi X n.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 9 / 24

SLIDE 34

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Examples of method of moments estimator

.

Problem

. . X1, · · · , Xn

i.i.d.

∼ N(µ, σ2). Find estimator for µ, σ2.

.

Solution

. . µ′

1

= EX = µ = X µ′

2

= EX2 = [EX]2 + Var(X) = µ2 + σ2 = 1 n

n

∑

i=1

X2

i

X

n n i

Xi Solving the two equations above, X,

n i

Xi X n.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 9 / 24

SLIDE 35

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Examples of method of moments estimator

.

Problem

. . X1, · · · , Xn

i.i.d.

∼ N(µ, σ2). Find estimator for µ, σ2.

.

Solution

. . µ′

1

= EX = µ = X µ′

2

= EX2 = [EX]2 + Var(X) = µ2 + σ2 = 1 n

n

∑

i=1

X2

i

{ ˆ µ = X

n n i

Xi Solving the two equations above, X,

n i

Xi X n.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 9 / 24

SLIDE 36

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Examples of method of moments estimator

.

Problem

. . X1, · · · , Xn

i.i.d.

∼ N(µ, σ2). Find estimator for µ, σ2.

.

Solution

. . µ′

1

= EX = µ = X µ′

2

= EX2 = [EX]2 + Var(X) = µ2 + σ2 = 1 n

n

∑

i=1

X2

i

{ ˆ µ = X ˆ µ2 + ˆ σ2 = 1

n

∑n

i=1 X2 i

Solving the two equations above, X,

n i

Xi X n.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 9 / 24

SLIDE 37

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Examples of method of moments estimator

.

Problem

. . X1, · · · , Xn

i.i.d.

∼ N(µ, σ2). Find estimator for µ, σ2.

.

Solution

. . µ′

1

= EX = µ = X µ′

2

= EX2 = [EX]2 + Var(X) = µ2 + σ2 = 1 n

n

∑

i=1

X2

i

{ ˆ µ = X ˆ µ2 + ˆ σ2 = 1

n

∑n

i=1 X2 i

Solving the two equations above, ˆ µ = X, ˆ σ2 = ∑n

i=1(Xi − X)2/n.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 9 / 24

SLIDE 38

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Method of moments estimator - Binomial

.

Problem

. . X1, · · · , Xn

i.i.d.

∼ Binomial(k, p). Find an estimator for k, p.

.

Solution

. . . . . . . . fX x k p k x px p k

x

x k Equating first two sample moments, n

n i

Xi x EX kp n

n i

Xi E X EX Var X k p kp p

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 10 / 24

SLIDE 39

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Method of moments estimator - Binomial

.

Problem

. . X1, · · · , Xn

i.i.d.

∼ Binomial(k, p). Find an estimator for k, p.

.

Solution

. . fX(x|k, p) = (k x ) px(1 − p)k−x x ∈ {0, 1, · · · , k} Equating first two sample moments, n

n i

Xi x EX kp n

n i

Xi E X EX Var X k p kp p

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 10 / 24

SLIDE 40

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Method of moments estimator - Binomial

.

Problem

. . X1, · · · , Xn

i.i.d.

∼ Binomial(k, p). Find an estimator for k, p.

.

Solution

. . fX(x|k, p) = (k x ) px(1 − p)k−x x ∈ {0, 1, · · · , k} Equating first two sample moments, 1 n

n

∑

i=1

Xi = x = µ′

1 = EX = kp

n

n i

Xi E X EX Var X k p kp p

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 10 / 24

SLIDE 41

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Method of moments estimator - Binomial

.

Problem

. . X1, · · · , Xn

i.i.d.

∼ Binomial(k, p). Find an estimator for k, p.

.

Solution

. . fX(x|k, p) = (k x ) px(1 − p)k−x x ∈ {0, 1, · · · , k} Equating first two sample moments, 1 n

n

∑

i=1

Xi = x = µ′

Examples of MoM estimator - Negative Binomial

.

Problem

. . X1, · · · , Xn

i.i.d.

∼ Negative Binomial(r, p). Find estimator for (r, p).

.

Solution

. . . . . . . . m n

n i

Xi EX r p p m n

n i

Xi EX r p p r p p p m m m X

n n i

Xi X r m p p Xp p

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 12 / 24

SLIDE 46

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Examples of MoM estimator - Negative Binomial

.

Problem

. . X1, · · · , Xn

i.i.d.

∼ Negative Binomial(r, p). Find estimator for (r, p).

.

Solution

. . m1 = 1 n

n

∑

i=1

Xi = EX = r(1 − p) p m n

n i

Xi EX r p p r p p p m m m X

n n i

Xi X r m p p Xp p

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 12 / 24

SLIDE 47

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Examples of MoM estimator - Negative Binomial

.

Problem

. . X1, · · · , Xn

i.i.d.

∼ Negative Binomial(r, p). Find estimator for (r, p).

.

Solution

. . m1 = 1 n

n

∑

i=1

Xi = EX = r(1 − p) p m2 = 1 n

n

∑

i=1

X2

i = EX2 =

(r(1 − p) p )2 + r(1 − p) p2 p m m m X

n n i

Xi X r m p p Xp p

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 12 / 24

SLIDE 48

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Examples of MoM estimator - Negative Binomial

.

Problem

. . X1, · · · , Xn

i.i.d.

∼ Negative Binomial(r, p). Find estimator for (r, p).

.

Solution

. . m1 = 1 n

n

∑

i=1

Xi = EX = r(1 − p) p m2 = 1 n

n

∑

i=1

X2

i = EX2 =

(r(1 − p) p )2 + r(1 − p) p2 ˆ p = m1 m2 − m2

1

= X

1 n

∑n

i=1 X2 i − X 2

r m p p Xp p

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 12 / 24

SLIDE 49

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Examples of MoM estimator - Negative Binomial

.

Problem

. . X1, · · · , Xn

i.i.d.

∼ Negative Binomial(r, p). Find estimator for (r, p).

.

Solution

. . m1 = 1 n

n

∑

i=1

Xi = EX = r(1 − p) p m2 = 1 n

n

∑

i=1

X2

i = EX2 =

(r(1 − p) p )2 + r(1 − p) p2 ˆ p = m1 m2 − m2

1

= X

1 n

∑n

i=1 X2 i − X 2

ˆ r = m1ˆ p 1 − ˆ p = Xˆ p 1 − ˆ p

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 12 / 24

SLIDE 50

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Satterthwaite Approximation

.

Problem

. . Let Y1, · · · , Yk are independently (but not identically) distributed random variables from χ2

r1, · · · , χ2 rk, respectively. We know that the distribution

∑n

i=1 Yi is also chi-squared with degrees of freedom equal to ∑k i=1 ri.

However, the distribution of

k i

aiYi, where ais are known constants with

n i

airi , in general, the distribution is hard to obtain. It is often reasonable to assume that the distribution of

k i

aiYi follows

approximately. Find a moment-based estimator of

.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 13 / 24

SLIDE 51

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Satterthwaite Approximation

.

Problem

. . Let Y1, · · · , Yk are independently (but not identically) distributed random variables from χ2

r1, · · · , χ2 rk, respectively. We know that the distribution

∑n

i=1 Yi is also chi-squared with degrees of freedom equal to ∑k i=1 ri.

However, the distribution of ∑k

i=1 aiYi, where ais are known constants

with ∑n

i=1 airi = 1, in general, the distribution is hard to obtain.

It is often reasonable to assume that the distribution of

k i

aiYi follows

approximately. Find a moment-based estimator of

.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 13 / 24

SLIDE 52

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Satterthwaite Approximation

.

Problem

. . Let Y1, · · · , Yk are independently (but not identically) distributed random variables from χ2

r1, · · · , χ2 rk, respectively. We know that the distribution

∑n

i=1 Yi is also chi-squared with degrees of freedom equal to ∑k i=1 ri.

However, the distribution of ∑k

i=1 aiYi, where ais are known constants

with ∑n

i=1 airi = 1, in general, the distribution is hard to obtain.

It is often reasonable to assume that the distribution of ∑k

i=1 aiYi follows 1 ν χ2 ν approximately. Find a moment-based estimator of ν.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 13 / 24

SLIDE 53

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

A Naive Solution

To match the first moment, let X ∼ χ2

ν/ν. Then E(X) = 1, and

Var(X) = 2/ν. E

k i

aiYi

k i

aiEYi

k i

airi E X To match the second moments, E

k i

aiYi E X Therefore, the method of moment estimator of is

k i

aiYi Note that can be negative, which is not desirable.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 14 / 24

SLIDE 54

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

A Naive Solution

To match the first moment, let X ∼ χ2

ν/ν. Then E(X) = 1, and

Var(X) = 2/ν. E ( k ∑

i=1

aiYi ) =

k

∑

i=1

aiEYi =

k

∑

i=1

airi = 1 = E(X) To match the second moments, E

k i

aiYi E X Therefore, the method of moment estimator of is

k i

aiYi Note that can be negative, which is not desirable.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 14 / 24

SLIDE 55

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

A Naive Solution

To match the first moment, let X ∼ χ2

ν/ν. Then E(X) = 1, and

Var(X) = 2/ν. E ( k ∑

i=1

aiYi ) =

k

∑

i=1

aiEYi =

k

∑

i=1

airi = 1 = E(X) To match the second moments, E ( k ∑

i=1

aiYi )2 = E ( X2) = 2 ν + 1 Therefore, the method of moment estimator of is

k i

aiYi Note that can be negative, which is not desirable.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 14 / 24

SLIDE 56

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

A Naive Solution

To match the first moment, let X ∼ χ2

ν/ν. Then E(X) = 1, and

Var(X) = 2/ν. E ( k ∑

i=1

aiYi ) =

k

∑

i=1

aiEYi =

k

∑

i=1

airi = 1 = E(X) To match the second moments, E ( k ∑

i=1

aiYi )2 = E ( X2) = 2 ν + 1 Therefore, the method of moment estimator of ν is ˆ ν = 2 (∑k

i=1 aiYi)2 − 1

Note that ν can be negative, which is not desirable.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 14 / 24

SLIDE 57

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

An alternative Solution

To match the second moments, E ( k ∑

i=1

aiYi )2 = Var ( k ∑

i=1

aiYi ) + [ E(

k

∑

i=1

aiYi) ]2 E

k i

aiYi Var

k i

aiYi E

k i

aiYi Var

k i

aiYi E

k i

aiYi E

k i

aiYi Var

k i

aiYi

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 15 / 24

SLIDE 58

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

An alternative Solution

To match the second moments, E ( k ∑

i=1

aiYi )2 = Var ( k ∑

i=1

aiYi ) + [ E(

k

∑

i=1

aiYi) ]2 = [ E(

k

∑

i=1

aiYi) ]2    Var(∑k

i=1 aiYi)

[ E(∑k

i=1 aiYi)

]2 + 1    Var

k i

aiYi E

k i

aiYi E

k i

aiYi Var

k i

aiYi

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 15 / 24

SLIDE 59

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

An alternative Solution

To match the second moments, E ( k ∑

i=1

aiYi )2 = Var ( k ∑

i=1

aiYi ) + [ E(

k

∑

i=1

aiYi) ]2 = [ E(

k

∑

i=1

aiYi) ]2    Var(∑k

i=1 aiYi)

[ E(∑k

i=1 aiYi)

]2 + 1    =    Var(∑k

i=1 aiYi)

[ E(∑k

i=1 aiYi)

]2 + 1    = 2 ν + 1 E

k i

aiYi Var

k i

aiYi

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 15 / 24

SLIDE 60

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

An alternative Solution

To match the second moments, E ( k ∑

i=1

aiYi )2 = Var ( k ∑

i=1

aiYi ) + [ E(

k

∑

i=1

aiYi) ]2 = [ E(

k

∑

i=1

aiYi) ]2    Var(∑k

i=1 aiYi)

[ E(∑k

i=1 aiYi)

]2 + 1    =    Var(∑k

i=1 aiYi)

[ E(∑k

i=1 aiYi)

]2 + 1    = 2 ν + 1 ν = 2 [ E(∑k

i=1 aiYi)

]2 Var(∑k

i=1 aiYi)

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 15 / 24

SLIDE 61

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Alternative Solution (cont’d)

To match the second moments, Finally, use the fact that Y1, · · · , Yk are independent chi-squared random variables. Var

n i

aiYi

k i

aiVar Yi

n i

ai EYi ri Substituting this expression for the variance and removing expectations, we obtain Satterthwaite’s estimator

n i

aiYi

n i ai ri Yi

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 16 / 24

SLIDE 62

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Alternative Solution (cont’d)

To match the second moments, Finally, use the fact that Y1, · · · , Yk are independent chi-squared random variables. Var(

n

∑

i=1

aiYi) =

k

∑

i=1

aiVar(Yi)

n i

ai EYi ri Substituting this expression for the variance and removing expectations, we obtain Satterthwaite’s estimator

n i

aiYi

n i ai ri Yi

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 16 / 24

SLIDE 63

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Alternative Solution (cont’d)

To match the second moments, Finally, use the fact that Y1, · · · , Yk are independent chi-squared random variables. Var(

n

∑

i=1

aiYi) =

k

∑

i=1

aiVar(Yi) = 2

n

∑

i=1

a2

i (EYi)2

ri Substituting this expression for the variance and removing expectations, we obtain Satterthwaite’s estimator

n i

aiYi

n i ai ri Yi

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 16 / 24

SLIDE 64

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Alternative Solution (cont’d)

To match the second moments, Finally, use the fact that Y1, · · · , Yk are independent chi-squared random variables. Var(

n

∑

i=1

aiYi) =

k

∑

i=1

aiVar(Yi) = 2

n

∑

i=1

a2

i (EYi)2

ri Substituting this expression for the variance and removing expectations, we obtain Satterthwaite’s estimator

n i

aiYi

n i ai ri Yi

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 16 / 24

SLIDE 65

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Alternative Solution (cont’d)

To match the second moments, Finally, use the fact that Y1, · · · , Yk are independent chi-squared random variables. Var(

n

∑

i=1

aiYi) =

k

∑

i=1

aiVar(Yi) = 2

n

∑

i=1

a2

i (EYi)2

ri Substituting this expression for the variance and removing expectations, we obtain Satterthwaite’s estimator ˆ ν = ∑n

i=1 aiYi

∑n

i=1 a2

i

ri Y2 i

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 16 / 24

SLIDE 66

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Maximum Likelihood Estimator

.

Definition

. .

For a given sample point x = (x1, · · · , xn),
let

x be the value such that

L

x attains its maximum.

More formally, L

x x L x where x .

x is called the maximum likelihood estimate of

based on data x,

and

X is the maximum likelihood estimator (MLE) of .

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 17 / 24

SLIDE 67

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Maximum Likelihood Estimator

.

Definition

. .

For a given sample point x = (x1, · · · , xn),
let ˆ

θ(x) be the value such that

L

x attains its maximum.

More formally, L

x x L x where x .

x is called the maximum likelihood estimate of

based on data x,

and

X is the maximum likelihood estimator (MLE) of .

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 17 / 24

SLIDE 68

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Maximum Likelihood Estimator

.

Definition

. .

For a given sample point x = (x1, · · · , xn),
let ˆ

θ(x) be the value such that

L(θ|x) attains its maximum.
More formally, L

x x L x where x .

x is called the maximum likelihood estimate of

based on data x,

and

X is the maximum likelihood estimator (MLE) of .

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 17 / 24

SLIDE 69

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Maximum Likelihood Estimator

.

Definition

. .

For a given sample point x = (x1, · · · , xn),
let ˆ

θ(x) be the value such that

L(θ|x) attains its maximum.
More formally, L(ˆ

θ(x)|x) ≥ L(θ|x) ∀θ ∈ Ω where ˆ θ(x) ∈ Ω.

x is called the maximum likelihood estimate of

based on data x,

and

X is the maximum likelihood estimator (MLE) of .

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 17 / 24

SLIDE 70

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Maximum Likelihood Estimator

.

Definition

. .

For a given sample point x = (x1, · · · , xn),
let ˆ

θ(x) be the value such that

L(θ|x) attains its maximum.
More formally, L(ˆ

θ(x)|x) ≥ L(θ|x) ∀θ ∈ Ω where ˆ θ(x) ∈ Ω.

ˆ

θ(x) is called the maximum likelihood estimate of θ based on data x,

and

X is the maximum likelihood estimator (MLE) of .

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 17 / 24

SLIDE 71

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Maximum Likelihood Estimator

.

Definition

. .

For a given sample point x = (x1, · · · , xn),
let ˆ

θ(x) be the value such that

L(θ|x) attains its maximum.
More formally, L(ˆ

θ(x)|x) ≥ L(θ|x) ∀θ ∈ Ω where ˆ θ(x) ∈ Ω.

ˆ

θ(x) is called the maximum likelihood estimate of θ based on data x,

and ˆ

θ(X) is the maximum likelihood estimator (MLE) of θ.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 17 / 24

SLIDE 72

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Example of MLE - Exponential Distribution

.

Problem

. . Let X1, · · · , Xn

i.i.d.

∼ Exponential(β). Find MLE of β.

.

Solution

. . . . . . . . L x fX x

n i

fX xi

n i

e

xi n exp n i

xi where .

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 18 / 24

SLIDE 73

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Example of MLE - Exponential Distribution

.

Problem

. . Let X1, · · · , Xn

i.i.d.

∼ Exponential(β). Find MLE of β.

.

Solution

. . L(β|x) = fX(x|θ) =

n

∏

i=1

fX(xi|θ)

n i

e

xi n exp n i

xi where .

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 18 / 24

SLIDE 74

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Example of MLE - Exponential Distribution

.

Problem

. . Let X1, · · · , Xn

i.i.d.

∼ Exponential(β). Find MLE of β.

.

Solution

. . L(β|x) = fX(x|θ) =

n

∏

i=1

fX(xi|θ) =

n

∏

i=1

[ 1 β e−xi/β ] = 1 βn exp ( −

n

∑

i=1

xi β ) where β > 0.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 18 / 24

SLIDE 75

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Use the derivative to find potential MLE

To maximize the likelihood function L(β|x) is equivalent to maximize the log-likelihood function l x log L x log

n exp n i

xi

n i

xi n log l

n i

xi n

n i

xi n

n i

xi n x

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 19 / 24

SLIDE 76

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Use the derivative to find potential MLE

To maximize the likelihood function L(β|x) is equivalent to maximize the log-likelihood function l(β|x) = log L(β|x) = log [ 1 βn exp ( −

n

∑

i=1

xi β )]

n i

xi n log l

n i

xi n

n i

xi n

n i

xi n x

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 19 / 24

SLIDE 77

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Use the derivative to find potential MLE

To maximize the likelihood function L(β|x) is equivalent to maximize the log-likelihood function l(β|x) = log L(β|x) = log [ 1 βn exp ( −

n

∑

i=1

xi β )] = − ∑n

i=1 xi

β − n log β l

n i

xi n

n i

xi n

n i

xi n x

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 19 / 24

SLIDE 78

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Use the derivative to find potential MLE

To maximize the likelihood function L(β|x) is equivalent to maximize the log-likelihood function l(β|x) = log L(β|x) = log [ 1 βn exp ( −

n

∑

i=1

xi β )] = − ∑n

i=1 xi

β − n log β ∂l ∂β = ∑n

i=1 xi

β2 − n β = 0

n i

xi n

n i

xi n x

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 19 / 24

SLIDE 79

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Use the derivative to find potential MLE

To maximize the likelihood function L(β|x) is equivalent to maximize the log-likelihood function l(β|x) = log L(β|x) = log [ 1 βn exp ( −

n

∑

i=1

xi β )] = − ∑n

i=1 xi

β − n log β ∂l ∂β = ∑n

i=1 xi

β2 − n β = 0

n

∑

i=1

xi = nβ

n i

xi n x

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 19 / 24

SLIDE 80

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Use the derivative to find potential MLE

To maximize the likelihood function L(β|x) is equivalent to maximize the log-likelihood function l(β|x) = log L(β|x) = log [ 1 βn exp ( −

n

∑

i=1

xi β )] = − ∑n

i=1 xi

β − n log β ∂l ∂β = ∑n

i=1 xi

β2 − n β = 0

n

∑

i=1

xi = nβ ˆ β = ∑n

i=1 xi

n = x

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 19 / 24

SLIDE 81

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Use the double derivative to confirm local maximum

∂2l ∂β2

β=x

= −2 ∑n

i=1 xi

β3 + n β2

β=x

n i

xi n

x

x nx x n x n Therefore, we can conclude that X X is unique local maximum on the interval

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 20 / 24

SLIDE 82

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Use the double derivative to confirm local maximum

∂2l ∂β2

β=x

= −2 ∑n

i=1 xi

β3 + n β2

β=x

= 1 β2 ( −2 ∑n

i=1 xi

β + n )

β=x

x nx x n x n Therefore, we can conclude that X X is unique local maximum on the interval

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 20 / 24

SLIDE 83

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Use the double derivative to confirm local maximum

∂2l ∂β2

β=x

= −2 ∑n

i=1 xi

β3 + n β2

β=x

= 1 β2 ( −2 ∑n

i=1 xi

β + n )

β=x

= 1 x2 ( −2nx x + n ) x n Therefore, we can conclude that X X is unique local maximum on the interval

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 20 / 24

SLIDE 84

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Use the double derivative to confirm local maximum

∂2l ∂β2

β=x

= −2 ∑n

i=1 xi

β3 + n β2

β=x

= 1 β2 ( −2 ∑n

i=1 xi

β + n )

β=x

= 1 x2 ( −2nx x + n ) = 1 x2 (−n) < 0 Therefore, we can conclude that X X is unique local maximum on the interval

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 20 / 24

SLIDE 85

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Use the double derivative to confirm local maximum

∂2l ∂β2

β=x

= −2 ∑n

i=1 xi

β3 + n β2

β=x

= 1 β2 ( −2 ∑n

i=1 xi

β + n )

β=x

xi n

n i

xi n L x

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 21 / 24

SLIDE 90

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Check boundary and confirm global maximum

β ∈ (0, ∞). If β → ∞ l(β|x) = − ∑n

i=1 xi

β − n log β → −∞ L(β|x) → If β → 0, use log(x) = limβ→0 1

β(xβ − 1)

l(β|x) = − ∑n

i=1 xi

β − n log β

n i

xi n

n i

xi n L x

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 21 / 24

SLIDE 91

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Check boundary and confirm global maximum

β ∈ (0, ∞). If β → ∞ l(β|x) = − ∑n

i=1 xi

β − n log β → −∞ L(β|x) → If β → 0, use log(x) = limβ→0 1

β(xβ − 1)

l(β|x) = − ∑n

i=1 xi

β − n log β = − ∑n

i=1 xi

β − n ( 1 β ββ − 1 )

n i

xi n L x

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 21 / 24

SLIDE 92

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Check boundary and confirm global maximum

β ∈ (0, ∞). If β → ∞ l(β|x) = − ∑n

i=1 xi

β − n log β → −∞ L(β|x) → If β → 0, use log(x) = limβ→0 1

β(xβ − 1)

l(β|x) = − ∑n

i=1 xi

β − n log β = − ∑n

i=1 xi

β − n ( 1 β ββ − 1 ) = − ∑n

i=1 xi − n(ββ − 1)

β → −∞ L x

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 21 / 24

SLIDE 93

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Check boundary and confirm global maximum

β ∈ (0, ∞). If β → ∞ l(β|x) = − ∑n

i=1 xi

β − n log β → −∞ L(β|x) → If β → 0, use log(x) = limβ→0 1

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

How do we find MLE?

If the function is differentiable with respect to θ.

. . 1 Find candidates that makes first order derivative to be zero . . 2 Check second-order derivative to check local maximum.

For one-dimensional parameter, negative second order derivative

implies local maximum.

For two-dimensional parameter, suppose L

is the likelihood

function. Then we need to show

(a) L

r

L . (b) Determinant of second-order derivative is positive

Check boundary points to see whether boundary gives global maximum.

If the function is NOT differentiable with respect to .

Use numerical methods
Or perform directly maximization, using inequalities, or properties of

the function.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 23 / 24

SLIDE 99

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

How do we find MLE?

If the function is differentiable with respect to θ.

. . 1 Find candidates that makes first order derivative to be zero . . 2 Check second-order derivative to check local maximum.

For one-dimensional parameter, negative second order derivative

implies local maximum.

For two-dimensional parameter, suppose L

is the likelihood

function. Then we need to show

(a) L

r

L . (b) Determinant of second-order derivative is positive

Check boundary points to see whether boundary gives global maximum.

If the function is NOT differentiable with respect to .

Use numerical methods
Or perform directly maximization, using inequalities, or properties of

the function.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 23 / 24

SLIDE 100

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

How do we find MLE?

If the function is differentiable with respect to θ.

. . 1 Find candidates that makes first order derivative to be zero . . 2 Check second-order derivative to check local maximum.

For one-dimensional parameter, negative second order derivative

implies local maximum.

For two-dimensional parameter, suppose L

is the likelihood

function. Then we need to show

(a) L

r

L . (b) Determinant of second-order derivative is positive

Check boundary points to see whether boundary gives global maximum.

If the function is NOT differentiable with respect to .

Use numerical methods
Or perform directly maximization, using inequalities, or properties of

the function.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 23 / 24

SLIDE 101

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

How do we find MLE?

If the function is differentiable with respect to θ.

. . 1 Find candidates that makes first order derivative to be zero . . 2 Check second-order derivative to check local maximum.

For one-dimensional parameter, negative second order derivative

implies local maximum.

For two-dimensional parameter, suppose L

is the likelihood

function. Then we need to show

(a) L

r

L . (b) Determinant of second-order derivative is positive

Check boundary points to see whether boundary gives global maximum.

If the function is NOT differentiable with respect to .

Use numerical methods
Or perform directly maximization, using inequalities, or properties of

the function.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 23 / 24

SLIDE 102

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

How do we find MLE?

If the function is differentiable with respect to θ.

. . 1 Find candidates that makes first order derivative to be zero . . 2 Check second-order derivative to check local maximum.

For one-dimensional parameter, negative second order derivative

implies local maximum.

For two-dimensional parameter, suppose L(θ1, θ2) is the likelihood
function. Then we need to show

(a) L

r

L . (b) Determinant of second-order derivative is positive

Check boundary points to see whether boundary gives global maximum.

If the function is NOT differentiable with respect to .

Use numerical methods
Or perform directly maximization, using inequalities, or properties of

the function.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 23 / 24

SLIDE 103

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

How do we find MLE?

If the function is differentiable with respect to θ.

. . 1 Find candidates that makes first order derivative to be zero . . 2 Check second-order derivative to check local maximum.

For one-dimensional parameter, negative second order derivative

implies local maximum.

For two-dimensional parameter, suppose L(θ1, θ2) is the likelihood
function. Then we need to show

(a) ∂2L(θ1, θ2)2/∂θ2

1 < 0 or ∂2L(θ1, θ2)2/∂θ2 2 < 0.

(b) Determinant of second-order derivative is positive

Check boundary points to see whether boundary gives global maximum.

If the function is NOT differentiable with respect to .

Use numerical methods
Or perform directly maximization, using inequalities, or properties of

the function.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 23 / 24

SLIDE 104

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

How do we find MLE?

If the function is differentiable with respect to θ.

. . 1 Find candidates that makes first order derivative to be zero . . 2 Check second-order derivative to check local maximum.

For one-dimensional parameter, negative second order derivative

implies local maximum.

For two-dimensional parameter, suppose L(θ1, θ2) is the likelihood
function. Then we need to show

(a) ∂2L(θ1, θ2)2/∂θ2

1 < 0 or ∂2L(θ1, θ2)2/∂θ2 2 < 0.

(b) Determinant of second-order derivative is positive

Check boundary points to see whether boundary gives global maximum.

If the function is NOT differentiable with respect to .

Use numerical methods
Or perform directly maximization, using inequalities, or properties of

the function.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 23 / 24

SLIDE 105

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

How do we find MLE?

If the function is differentiable with respect to θ.

. . 1 Find candidates that makes first order derivative to be zero . . 2 Check second-order derivative to check local maximum.

For one-dimensional parameter, negative second order derivative

implies local maximum.

For two-dimensional parameter, suppose L(θ1, θ2) is the likelihood
function. Then we need to show

(a) ∂2L(θ1, θ2)2/∂θ2

1 < 0 or ∂2L(θ1, θ2)2/∂θ2 2 < 0.

(b) Determinant of second-order derivative is positive

Check boundary points to see whether boundary gives global maximum.

If the function is NOT differentiable with respect to .

Use numerical methods
Or perform directly maximization, using inequalities, or properties of

the function.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 23 / 24

SLIDE 106

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

How do we find MLE?

If the function is differentiable with respect to θ.

. . 1 Find candidates that makes first order derivative to be zero . . 2 Check second-order derivative to check local maximum.

For one-dimensional parameter, negative second order derivative

implies local maximum.

For two-dimensional parameter, suppose L(θ1, θ2) is the likelihood
function. Then we need to show

(a) ∂2L(θ1, θ2)2/∂θ2

1 < 0 or ∂2L(θ1, θ2)2/∂θ2 2 < 0.

(b) Determinant of second-order derivative is positive

Check boundary points to see whether boundary gives global maximum.

If the function is NOT differentiable with respect to θ.

Use numerical methods
Or perform directly maximization, using inequalities, or properties of

the function.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 23 / 24

SLIDE 107

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

How do we find MLE?

If the function is differentiable with respect to θ.

. . 1 Find candidates that makes first order derivative to be zero . . 2 Check second-order derivative to check local maximum.

For one-dimensional parameter, negative second order derivative

implies local maximum.

For two-dimensional parameter, suppose L(θ1, θ2) is the likelihood
function. Then we need to show

(a) ∂2L(θ1, θ2)2/∂θ2

1 < 0 or ∂2L(θ1, θ2)2/∂θ2 2 < 0.

(b) Determinant of second-order derivative is positive

Check boundary points to see whether boundary gives global maximum.

If the function is NOT differentiable with respect to θ.

Use numerical methods
Or perform directly maximization, using inequalities, or properties of

the function.

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 23 / 24

SLIDE 108

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary

Summary

.

Today

. .

Likelihood Function
Point Estimator
Method of Moments Estimator
Maximum Likelihood Estimator

.

Next Lecture

. . . . . . . . • Maximum Likelihood Estimator

Hyun Min Kang Biostatistics 602 - Lecture 09 February 7th, 2013 24 / 24

SLIDE 109

. . . . . .

. . . . Likelihood Function . . . . . . . . . . . Method of Moments . . . . . . . MLE . Summary