[PPT] - Estimation Theory Overview Introduction Up until now we have PowerPoint Presentation

SLIDE 1

Terminology

f(x; θ) where θ ∈ Rp×1

given the pdf and θ

“likeliness” of θ

we wish to estimate

Estimation Theory Overview

er-Rao lower bound

Estimators as Random Variables

θ

θ to be as close to the true θ as possible

Introduction

variables and processes

autocorrelation) and derived other related properties (e.g. PSD)

SLIDE 2

Bias Bias of an estimator ˆ θ of a parameter θ is defined as B(ˆ θ) E[ˆ θ] − θ

θ) = 0

best (more later)

Natural Estimators ˆ µx = ˆ θ

N

xi

ˆ σ2

θ

N

(xi − ˆ µx)2

Variance Variance of an estimator ˆ θ of a parameter θ is defined as var(ˆ θ) = σ2

θ − E

θ

θ about its mean

Good Estimators

θ(ˆ

the time being

– Distribution of ˆ θ should be centered at the true value – Want the distribution to be as narrow as possible

SLIDE 3

Bias, Variance, and Modeling y(x) = g(x) + ε ˆ y(x) = ˆ g(x)

function

apply to estimating functional relationships

Bias-Variance Tradeoff

θ(ˆ

θ(ˆ

θ 0 has zero variance, but is generally biased

Notation and Prediction Error y = g(x) + ε g = g(x) ˆ g = ˆ g(x) ˆ ge = E[ˆ g(x)]

construct ˆ g(x) and the distribution of the process noise f(ε)

The prediction error for a new, given input is defined as PE(x) = E[(y − ˆ g)2] = E[((g − ˆ g) + ε)2] = E[(g − ˆ g)2] + 2 E[(g − ˆ g)ε] + E[ε2] = MSE(x) + σ2

The Bias-Variance Tradeoff

θ(ˆ

θ(ˆ

biased

SLIDE 4

Bias-Variance Tradeoff Comments MSE(x) = (g − E[ˆ g])2 + E

g − E[ˆ g])2 = Bias2 + Variance

data set

large number of data sets, the expected value of the model ˆ g(x) would not be close to the true function g(x)

model may have smaller error (MSE) than an unbiased model

accurate models!

The Bias-Variance Tradeoff Derivation y = g(x) + ε g = g(x) ˆ g = ˆ g(x) ˆ ge = E[ˆ g(x)]

g is a random function

MSE(x) = E[(g − ˆ g)2] = E[{(g − ˆ ge) − (ˆ g − ˆ ge)}2] = E ⎡ ⎣(g − ˆ ge)2 − 2(g − ˆ ge)(ˆ g − ˆ ge)

+ (ˆ g − ˆ ge)2

⎤ ⎦

Bias-Variance Tradeoff Comments Continued MSE(x) = (g − E[ˆ g])2 + E

g − E[ˆ g])2 = Bias2 + Variance

– If the model is too flexible, it can overfit the data – The model will change dramatically from one data set to another – In this case it has high variance, but potentially low variance

– If the model is not very flexible, it may not capture the true relationship between the inputs and the output – It will not vary as much from one data set to another – In this case the model has low variance, but potentially high bias

Bias-Variance Tradeoff Derivation Continued 1 ① = E[(g − ˆ ge)2 − 2(g − ˆ ge)(ˆ g − ˆ ge)] = E[g2 − 2gˆ ge + ˆ g2

g − ˆ ge)] + 2ˆ g2

g2

= E[g2 − 2gˆ ge + ˆ g2

g + 2gˆ ge] = E[g2 − 2gˆ g + ˆ g2

= g2 − 2g E[ˆ g] + ˆ g2

= g2 − 2gˆ ge + ˆ g2

= (g − ˆ ge)2 Thus MSE(x) = ① + ② = (g − ˆ ge)2 + E[(ˆ g − ˆ ge)2] = (g − E[ˆ g])2 + E

g − E[ˆ g])2

SLIDE 5

Cram´ er-Rao Lower Bound Comments var(ˆ θ) ≥ 1 E

2 = − 1 E

CRLB with equality

∂ ln fx;θ(x; θ) ∂θ = 0 where the pdf is evaluated at the observed outcome x(ζ)

satisfies the equation above

Mean Square Error Mean Square Error of an estimator ˆ θ of a parameter θ is defined as MSE(θ) E[|ˆ θ − θ|2] = σ2

θ)|2

performance

different bias and variance

– Picking MSE results in a simple bias/variance decomposition – Other error measures generally do not have such a decomposition

Consistency

lim

θ) = 0

– The estimator becomes unbiased – The variance approaches zero – The distribution fˆ

Cram´ er-Rao Lower Bound var(ˆ θ) ≥ − 1 E

2 = − 1 E[ ∂2 ln fx;θ(x;θ)

]

unbiased and have the smallest variance of all possible estimators

er-Rao Lower Bound (CRLB) shown above is a lower bound on unbiased estimators

(stochastic process), not the parameter

distribution

SLIDE 6

Sample Mean Confidence Intervals fˆ

µx) = 1 √ 2π(σx/ √ N) exp

2 ˆ µx − µx σx/ √ N 2 Pr

√ N < ˆ µx < µx + k σx √ N

Pr

µx − k σx √ N < µx < ˆ µx + k σx √ N

1 − α

pdf (sampling distribution) of the estimator

distribution of ˆ µx is Gaussian by the Central Limit Theorem (CLT)

Confidence Intervals

probability of covering the unknown true parameter value Pr {a < θ ≤ b} = 1 − α

1 − α

level

Sample Mean Confidence Intervals Comments Pr

µx − k σx √ N < µx < ˆ µx + k σx √ N

are only approximate

Properties of the Sample Mean ˆ µx 1 N

x(n) E[ˆ µx] = µx var(ˆ µx) = σ2

N

– Has minimum variance – If Gaussian, is the maximum likelihood estimator – If Gaussian, attains the Cram´ er-Rao Lower Bound

SLIDE 7

Sample Mean Variance when Gaussian and IID Pr

µx − k σx √ N < µx < ˆ µx + k σx √ N

ˆ σ2

1 N − 1

[x(n) − ˆ µx]2

ˆ µx − µx ˆ σx/ √ N has a Students’ t distribution with v = N − 1 degrees of freedom

Sample Mean Variance when Gaussian E[ˆ µx] = µx var(ˆ µx) = 1 N

N

mean µ

Pr

µx − k

µx) < µx < ˆ µx − k

µx)