16. Mean Square Estimation Given some information that is related to - - PowerPoint PPT Presentation

1

• 16. Mean Square Estimation

Given some information that is related to an unknown quantity of interest, the problem is to obtain a good estimate for the unknown in terms of the observed data. Suppose represent a sequence of random variables about whom one set of observations are available, and Y represents an unknown random variable. The problem is to obtain a good estimate for Y in terms of the observations Let represent such an estimate for Y. Note that can be a linear or a nonlinear function of the observation Clearly represents the error in the above estimate, and the square of

n

X X X , , ,

2 1

• .

, , ,

2 1 n

X X X

• )

( ) , , , ( ˆ

2 1

X X X X Y

n

ϕ ϕ = =

• (16-1)

) (⋅ ϕ . , , ,

2 1 n

X X X

• )

( ˆ ) ( X Y Y Y X ϕ ε − = − = (16-2)

2

| |ε

PILLAI

2

the error. Since is a random variable, represents the mean square error. One strategy to obtain a good estimator would be to minimize the mean square error by varying over all possible forms

• f and this procedure gives rise to the Minimization of the

Mean Square Error (MMSE) criterion for estimation. Thus under MMSE criterion,the estimator is chosen such that the mean square error is at its minimum. Next we show that the conditional mean of Y given X is the best estimator in the above sense. Theorem1: Under MMSE criterion, the best estimator for the unknown Y in terms of is given by the conditional mean of Y gives X. Thus Proof : Let represent an estimate of Y in terms of Then the error and the mean square error is given by ε } | | {

2

ε E ), (⋅ ϕ ) (⋅ ϕ

n

X X X , , ,

2 1

• }.

| { ) ( ˆ X Y E X Y = = ϕ (16-3) ) ( ˆ X Y ϕ = ). , , , (

2 1 n

X X X X

• =

, ˆ Y Y − = ε } | ) ( | { } | ˆ | { } | | {

2 2 2 2

X Y E Y Y E E ϕ ε σ ε − = − = = (16-4) PILLAI } | | {

2

ε E

3

Since we can rewrite (16-4) as where the inner expectation is with respect to Y, and the outer one is with respect to Thus To obtain the best estimator we need to minimize in (16-6) with respect to In (16-6), since and the variable appears only in the integrand term, minimization

• f the mean square error in (16-6) with respect to is

equivalent to minimization of with respect to }] | { [ ] [ X z E E z E

z X

= }] | ) ( | { [ } | ) ( | {

z 2 z 2 2

X X Y E E X Y E

Y X

• ϕ

ϕ σ ε − = − = . X

∫

∞ + ∞ −

− = − = . ) ( } | ) ( | { }] | ) ( | { [

2 2 2

dx X f X X Y E X X Y E E

X

ϕ ϕ σ ε (16-6) (16-5) , ϕ

2 ε

σ . ϕ , ) ( ≥ X f X , } | ) ( | {

2

≥ − X X Y E ϕ ϕ ϕ

2 ε

σ } | ) ( | {

2 X

X Y E ϕ − . ϕ

PILLAI

4

Since X is fixed at some value, is no longer random, and hence minimization of is equivalent to This gives

• r

But since when is a fixed number Using (16-9) ) (X ϕ } | ) ( | {

2 X

X Y E ϕ − . } | ) ( | {

2

= − ∂ ∂ X X Y E ϕ ϕ (16-7) } | ) ( {| = − X X Y E ϕ (16-8) ), ( } | ) ( { X X X E ϕ ϕ = (16-9) ) ( , X x X ϕ = ). (x ϕ

PILLAI

. } | ) ( { } | { = − X X E X Y E ϕ

5

in (16-8) we get the desired estimator to be Thus the conditional mean of Y given represents the best estimator for Y that minimizes the mean square error. The minimum value of the mean square error is given by As an example, suppose is the unknown. Then the best MMSE estimator is given by Clearly if then indeed is the best estimator for Y }. , , , | { } | { ) ( ˆ

2 1 n

X X X Y E X Y E X Y

• =

= = ϕ

n

X X X , , ,

2 1

• .

)} | {var( ] } | ) | ( | { [ } | ) | ( | {

) var( 2 2 2 min

≥ = − = − = X Y E X X Y E Y E E X Y E Y E

X Y

• σ

(16-11) (16-10)

3

X Y = ,

3

X Y =

3

ˆ X Y = . } | { } | { ˆ

3 3

X X X E X Y E Y = = = (16-12)

PILLAI

6

in terms of X. Thus the best estimator can be nonlinear. Next, we will consider a less trivial example. Example : Let where k > 0 is a suitable normalization constant. To determine the best estimate for Y in terms of X, we need Thus Hence the best MMSE estimator is given by    < < < =

• therwise,

1 , ) , (

,

y x kxy y x f

Y X

). | (

|

x y f

X Y

1. x , 2 ) 1 ( 2 ) , ( ) (

2 1 2 1 1

,

< < − = = = = ∫

∫

x kx kxy kxydy dy y x f x f

x x x

Y X X

y

x

1 1

. 1 ; 1 2 2 / ) 1 ( ) ( ) , ( ) | (

2 2

,

< < < − = − = = y x x y x kx kxy x f y x f x y f

X Y X X Y

PILLAI

(16-13)

7

Once again the best estimator is nonlinear. In general the best estimator is difficult to evaluate, and hence next we will examine the special subclass of best linear estimators.

Best Linear Estimator

In this case the estimator is a linear function of the

• bservations Thus

where are unknown quantities to be determined. The mean square error is given by . 1 ) 1 ( 3 2 1 1 3 2 1 3 2 ) | ( } | { ) ( ˆ

2 2 2 3 1 2 3 1 2 1 2 1 1 2 1

2 2 |

x x x x x x y dy y dy y dy x y f y X Y E X Y

x x x x x y x

X Y

− + + = − − = − = = = = = =

∫ ∫ ∫

− −

ϕ } | { X Y E (16-14) Y ˆ . , , ,

2 1 n

X X X

• n

a a a , , ,

2 1

• ∑

=

= + + + =

n i i i n n l

X a X a X a X a Y

1 2 2 1 1

. ˆ

• (16-15)

) ˆ (

l

Y Y − = ε

PILLAI

8

and under the MMSE criterion should be chosen so that the mean square error is at its minimum possible

• value. Let represent that minimum possible value. Then

To minimize (16-16), we can equate This gives But

n

a a a , , ,

2 1

• }

| | { } | ˆ {| } | | {

2 2 2

∑

− = − =

i i l

X a Y E Y Y E E ε } | | {

2

ε E

2 n

σ

∑

=

− =

n i i i a a a n

X a Y E

n

1 2 , , , 2

}. | {| min

2 1

• σ

(16-17) ,n. , , k E ak

• 2

1 , } | {|

2

= = ∂ ∂ ε (16-18) . 2 | | } | {|

* 2 2

=               ∂ ∂ =       ∂ ∂ = ∂ ∂

k k k

a E a E E a ε ε ε ε (16-19)

PILLAI

(16-16)

9

Substituting (16-19) in to (16-18), we get

• r the best linear estimator must satisfy

Notice that in (16-21), represents the estimation error and represents the data. Thus from (16-21), the error is orthogonal to the data for the best linear estimator. This is the orthogonality principle. In other words, in the linear estimator (16-15), the unknown constants must be selected such that the error . ) ( ) (

1 1 k k n i i i k k n i i i k

X a X a a Y a X a Y a − = ∂ ∂ − ∂ ∂ = ∂ − ∂ = ∂ ∂

∑ ∑

= =

ε , } { 2 } | {|

* 2

= − = ∂ ∂

k k

X E a E ε ε . , , 2 , 1 , } {

*

n k X E

k

• =

= ε (16-20) (16-21)

∑ =

−

n i i iX

a Y

1

), ( ε ε n k X k → =1 , n k X k → =1 ,

n

a a a , , ,

2 1

• PILLAI

10

is orthogonal to every data for the best linear estimator that minimizes the mean square error. Interestingly a general form of the orthogonality principle holds good in the case of nonlinear estimators also. Nonlinear Orthogonality Rule: Let represent any functional form of the data and the best estimator for Y given With we shall show that implying that This follows since

∑ =

− =

n i i iX

a Y

1

ε

n

X X X , , ,

2 1

• )

(X h } | { X Y E . X } | { X Y E Y e − = ). ( } | { X h X Y E Y e ⊥ − = , )} ( { = X eh E . )} ( { )} ( { ]} | ) ( [ { )} ( { } ) ( ] | [ { )} ( { )} ( ]) | [ {( )} ( { = − = − = − = − = X Yh E X Yh E X X Yh E E X Yh E X h X Y E E X Yh E X h X Y E Y E X eh E

PILLAI

(16-22)

11

Thus in the nonlinear version of the orthogonality rule the error is

• rthogonal to any functional form of the data.

The orthogonality principle in (16-20) can be used to obtain the unknowns in the linear case. For example suppose n = 2, and we need to estimate Y in terms of linearly. Thus From (16-20), the orthogonality rule gives Thus

• r

n

a a a , , ,

2 1

• 2

1

and X X

2 2 1 1

ˆ X a X a Yl + = } ) {( } X { } ) {( } X {

* 2 2 2 1 1 * 2 * 1 2 2 1 1 * 1

= − − = = − − = X X a X a Y E E X X a X a Y E E ε ε } { } | {| } { } { } { } | {|

* 2 2 2 2 1 * 2 1 * 1 2 * 1 2 1 2 1

YX E a X E a X X E YX E a X X E a X E = + = +

PILLAI

12

(16-23) can be solved to obtain in terms of the cross- correlations. The minimum value of the mean square error in (16-17) is given by But using (16-21), the second term in (16-24) is zero, since the error is

• rthogonal to the data where are chosen to be
• ptimum. Thus the minimum value of the mean square error is given

by

2 1

and a a         =               } { } { } | {| } { } { } | {|

* 2 * 1 2 1 2 2 * 2 1 * 1 2 2 1

YX E YX E a a X E X X E X X E X E (16-23)

2 n

σ

n

a a a , , ,

2 1

• ,

i

X }. { min } { min } ) ( { min } { min } | {| min

* 1 , , , * , , , 1 * , , , * , , , 2 , , , 2

2 1 2 1 2 1 2 1 2 1

l n i i a a a a a a n i i i a a a a a a a a a n

X E a Y E X a Y E E E

n n n n n

ε ε ε εε ε σ

∑ ∑

= =

− = − = = =

• (16-24)

PILLAI

13

where are the optimum values from (16-21). Since the linear estimate in (16-15) is only a special case of the general estimator in (16-1), the best linear estimator that satisfies (16-20) cannot be superior to the best nonlinear estimator Often the best linear estimator will be inferior to the best estimator in (16-3). This raises the following question. Are there situations in which the best estimator in (16-3) also turns out to be linear ? In those situations it is enough to use (16-21) and obtain the best linear estimators, since they also represent the best global estimators. Such is the case if Y and are distributed as jointly Gaussian We summarize this in the next theorem and prove that result. Theorem2: If and Y are jointly Gaussian zero

n

a a a , , ,

2 1

• )

(X ϕ }. | { X Y E } { } | {| } ) {( } {

1 * 2 1 * * 2

∑ ∑

= =

− = − = =

n i i i n i i i n

Y X E a Y E Y X a Y E Y E ε σ (16-25)

n

X X X , , ,

2 1

• n

X X X , , ,

2 1

• PILLAI

14

mean random variables, then the best estimate for Y in terms of is always linear. Proof : Let represent the best (possibly nonlinear) estimate of Y, and the best linear estimate of Y. Then from (16-21) is orthogonal to the data Thus Also from (16-28),

n

X X X , , ,

2 1

• }

| { ) , , , ( ˆ

2 1

X Y E X X X Y

n =

=

• ϕ

(16-26)

∑

=

=

n i i i l

X a Y

1

ˆ (16-27) . 1 , n k X k → = (16-28) . 1 , } X {

* k

n k E → = = ε (16-29) . } { } { } {

1

= − =

∑

= n i i i

X E a Y E E ε (16-30)

PILLAI

1 n l i i i

Y Y Y a X ε

=

= − = −∑

∆

15

Using (16-29)-(16-30), we get From (16-31), we obtain that and are zero mean uncorrelated random variables for But itself represents a Gaussian random variable, since from (16-28) it represents a linear combination

• f a set of jointly Gaussian random variables. Thus and X are

jointly Gaussian and uncorrelated random variables. As a result, and X are independent random variables. Thus from their independence But from (16-30), and hence from (16-32) Substituting (16-28) into (16-33), we get . 1 n k → = ε

k

X ε ε . 1 , } { } { } {

* *

n k X E E X E

k k

→ = = = ε ε (16-31) ε }. { } | { ε ε E X E = (16-32) , } { = ε E . } | {

=

X E ε (16-33) } | { } | {

1

= − =

∑

=

X X a Y E X E

n i i i

ε

PILLAI

16

• r

From (16-26), represents the best possible estimator, and from (16-28), represents the best linear estimator. Thus the best linear estimator is also the best possible overall estimator in the Gaussian case. Next we turn our attention to prediction problems using linear estimators.

Linear Prediction

Suppose are known and is unknown. Thus and this represents a one-step prediction problem. If the unknown is then it represents a k-step ahead prediction

• problem. Returning back to the one-step predictor, let

represent the best linear predictor. Then . } | { } | {

1 1 l n i i i n i i i

Y X a X X a E X Y E = = =

∑ ∑

= =

(16-34) ) ( } | { x X Y E ϕ =

∑ =

n i i i X

a

1 n

X X X , , ,

2 1

• 1

+ n

X ,

1 +

=

n

X Y ,

k n

X +

1

ˆ

+ n

X

PILLAI

17

where the error is orthogonal to the data, i.e., Using (16-36) in (16-37), we get Suppose represents the sample of a wide sense stationary

i

X (16-35) , 1 , ˆ

1 1 1 1 2 2 1 1 1 1 1 1

= = + + + + = + = − =

+ + = + = + + +

∑ ∑

n n i i i n n n n i i i n n n n

a X a X X a X a X a X a X X X

• ε

(16-36) . 1 , } {

*

n k X E

k n

→ = = ε (16-37)

∑

+ =

→ = = =

1 1 * *

. 1 , } { } {

n i k i i k n

n k X X E a X E ε (16-38)

PILLAI

1 1

ˆ = ,

n n i i i

X a X

+ =

−∑

∆

18

stochastic process so that Thus (16-38) becomes Expanding (16-40) for we get the following set of linear equations. Similarly using (16-25), the minimum mean square error is given by

* *

) ( } {

i k k i k i

r r k i R X X E

− − =

= − = ) (t X (16-39) . 1 , 1 , } {

1 1 1 *

n k a r a X E

n n i k i i k n

→ = = = =

+ + = −

∑

ε (16-40) , , , 2 , 1 n k

• =

. 2 1

1 * 3 3 * 2 2 * 1 1 1 2 1 3 2 * 1 1 1 2 3 1 2 1

n k r r a r a r a r a k r r a r a r a r a k r r a r a r a r a

n n n n n n n n n n

= ← = + + + + + = ← = + + + + + = ← = + + + + +

− − − − − −

• (16-41)

PILLAI

19

The n equations in (16-41) together with (16-42) can be represented as Let . } ) {( } { } { } | {|

1 * 2 3 * 1 2 * 1 1 1 * 1 1 1 * 1 * 1 * 2 2

r r a r a r a r a r a X X a E X E Y E E

n n n n n i i n i n i n i i n n n n

+ + + + + = = = = = =

− − + = − + + = + +

∑ ∑

• ε

ε ε σ (16-42) . 1

2 n 3 2 1 * 1 * 1 * 1 * 2 * 1 2 * 1 * 2 1 1 * 1 2 1

                    =                                        

− − − − −

σ

• n

n n n n n n n

a a a a r r r r r r r r r r r r r r r r r r r r (16-43)

PILLAI

20

Notice that is Hermitian Toeplitz and positive definite. Using (16-44), the unknowns in (16-43) can be represented as Let

n

T .

* 1 * 1 * 1 1 * 1 2 1

              =

− −

r r r r r r r r r r r r T

n n n n n

• (16-44)

              =                     =                    

− − 1 2 2 n 1 3 2 1

• f

column Last 1

n n n n

T T a a a a σ σ

• (16-45)

PILLAI

21

Then from (16-45), Thus .

1 , 1 2 , 1 1 , 1 1 , 2 22 21 1 , 1 12 11 1

              =

+ + + + + + − n n n n n n n n n n n n n n n n

T T T T T T T T T T

• .

1

1 , 1 1 , 2 1 , 1 2 2 1

              =                

+ + + + n n n n n n n n n

T T T a a a

• σ

, 1

1 , 1 2

> =

+ + n n n n

T σ (16-46) (16-47) (16-48)

PILLAI

22

and

• Eq. (16-49) represents the best linear predictor coefficients, and they

can be evaluated from the last column of in (16-45). Using these, The best one-step ahead predictor in (16-35) taken the form and from (16-48), the minimum mean square error is given by the (n +1, n +1) entry of From (16-36), since the one-step linear prediction error . 1

1 , 1 1 , 2 1 , 1 1 , 1 2 1

              =              

+ + + + + + n n n n n n n n n n n

T T T T a a a

• (16-49)

n

T .

1 − n

T . ) ( 1 ˆ

1 1 , 1 , 1 1

∑

= + + + +

        − =

n i i n i n n n n n

X T T X (16-50) ,

1 1 1 1 1

X a X a X a X

n n n n n n

+ + + + =

− − +

• ε

(16-51)

PILLAI

23

we can represent (16-51) formally as follows Thus, let them from the above figure, we also have the representation The filter represents an AR(n) filter, and this shows that linear prediction leads to an auto regressive (AR) model.

n n n n n

z a z a z a X ε → + + + + →

− − − − +

1

1 2 1 1 1

• ,

1 ) (

1 2 1 1 n n n n

z a z a z a z A

− − − −

+ + + + =

• (16-52)

. ) ( 1

1 +

→ →

n n n

X z A ε

n n n n

z a z a z a z A z H

− − − −

+ + + + = =

1 2 1 1

1 1 ) ( 1 ) (

• (16-53)

PILLAI

24

The polynomial in (16-52)-(16-53) can be simplified using (16-43)-(16-44). To see this, we rewrite as To simplify (16-54), we can make use of the following matrix identity ) (z An ) (z An                 =                 = + + + + + =

− − − − − − − − − − − − − − − 2 n 1 1 ) 1 ( 2 1 1 ) 1 ( 1 2 1 ) 1 ( 2 1

] 1 , , , , [ 1 ] 1 , , , , [ 1 ) ( σ

• n

n n n n n n n n n n

T z z z a a a z z z z a z a z a z a z A (16-54)

PILLAI

.

1 

     − =       −      

− B

CA D C A I AB I D C B A (16-55)

25

Taking determinants, we get In particular if we get Using (16-57) in (16-54), with .

1B

CA D A D C B A

−

− = (16-56) , ≡ D . ) 1 (

1

C B A A B CA

n

− =

−

(16-57)           = = =

− − − − 2 1 ) 1 (

, ], 1 , , , , [

n n n n

B T A z z z C σ

• PILLAI

26

we get Referring back to (16-43), using Cramer’s rule to solve for we get . 1 | | 1 , , , | | ) 1 ( ) (

1 ) 1 ( 1 * 2 * 1 1 1 * 1 2 1 2 1 2 − − − − − − − − −

= − = z z z r r r r r r r r r r r r T z z T T z A

n n n n n n n n n n n n n n

• σ

σ (16-58) ), 1 (

1 = + n

a

PILLAI

1 | | | | | |

1 2 1 1 2 1

= = =

− − − + n n n n n n n n

T T T r r r r a σ σ

27

• r

Thus the polynomial (16-58) reduces to The polynomial in (16-53) can be alternatively represented as in (16-60), and in fact represents a stable . | | | |

1 2

> =

− n n n

T T σ . 1 1 | | 1 ) (

1 2 1 1 1 ) 1 ( 1 * 2 * 1 1 1 * 1 2 1 1 n n n n n n n n n n n

z a z a z a z z z r r r r r r r r r r r r T z A

− − − − − − − − − − − −

+ + + + = =

• (16-59)

(16-60)

PILLAI

) (z An ) ( ~ ) ( 1 ) ( n AR z A z H

n

=

28

AR filter of order n, whose input error signal is white noise of constant spectral height equal to and output is It can be shown that has all its zeros in provided thus establishing stability.

Linear prediction Error

From (16-59), the mean square error using n samples is given by Suppose one more sample from the past is available to evaluate ( i.e., are available). Proceeding as above the new coefficients and the mean square error can be determined. From (16-59)-(16-61),

n

ε | | / | |

1 − n n

T T .

1 + n

X

1 + n

X . | | | |

1 2

> =

− n n n

T T σ (16-61)

1 1

, , , , X X X X

n n

• −

2 1 + n

σ

PILLAI

. | | | |

1 2 1 n n n

T T +

+ =

σ (16-62) ) (z An 1 | | > z | | >

n

T

29

Using another matrix identity it is easy to show that Since we must have or for every n. From (16-63), we have

• r

since Thus the mean square error decreases as more and more samples are used from the past in the linear predictor. In general from (16-64), the mean square errors for the one-step predictor form a monotonic nonincreasing sequence ). | | 1 ( | | | | | |

2 1 1 2 1 + − +

− =

n n n n

s T T T (16-63) , | | >

k

T ) | | 1 (

2 1

> −

+ n

s 1 | |

1 < + n

s ) | | 1 ( | | | | | | | |

2 1 1 1

2 2 1

+ − +

− =

+

n n n n n

s T T T T

n n

• σ

σ

, ) | | 1 (

2 2 1 2 2 1 n n n n

s σ σ σ < − =

+ +

(16-64)

PILLAI

. 1 ) | | 1 (

2 1

< −

+ n

s

30

whose limiting value Clearly, corresponds to the irreducible error in linear prediction using the entire past samples, and it is related to the power spectrum of the underlying process through the relation where represents the power spectrum of For any finite power process, we have and since Thus

2 2 2 1 2 ∞ +

→ > ≥ ≥ σ σ σ σ

• k

n n

(16-65) .

2 ≥ ∞

σ ) (nT X ). (nT X ( )

XX

S ω ≥

2

1 exp ln ( ) 0. 2

XX

S d

π π

σ ω ω π

+ ∞ −

  = ≥    

∫

(16-66) ( ) ,

XX

S d

π π

ω ω

+ −

< ∞

∫

PILLAI

2 ≥ ∞

σ ln ( ) ( ) .

XX XX

S d S d

π π π π

ω ω ω ω

+ + − −

≤ < ∞

∫ ∫

( ( ) 0), ln ( ) ( ).

XX XX XX

S S S ω ω ω ≥ ≤ (16-67)

31

Moreover, if the power spectrum is strictly positive at every Frequency, i.e., then from (16-66) and hence i.e., For processes that satisfy the strict positivity condition in (16-68) almost everywhere in the interval the final minimum mean square error is strictly positive (see (16-70)). i.e., Such processes are not completely predictable even using their entire set of past samples, or they are inherently stochastic, ( ) 0, in - ,

XX

S ω π ω π > < < ln ( ) .

XX

S d

π π

ω ω

+ −

> −∞

∫

2

1 exp ln ( ) 2

XX

S d

e

π π

σ ω ω π

+ −∞ ∞ −

  = > =    

∫

(16-68) (16-69) (16-70) ), , ( π π −

PILLAI

32

since the next output contains information that is not contained in the past samples. Such processes are known as regular stochastic processes, and their power spectrum is strictly positive.

) (ω

XX

S

ω π π − Power Spectrum of a regular stochastic Process PILLAI

Conversely, if a process has the following power spectrum, such that in then from (16-70), ( )

XX

S ω =

2 1

ω ω ω < < .

2 = ∞

σ

ω

2

ω

1

ω π − π

) (ω

XX

S

33

Such processes are completely predictable from their past data

• samples. In particular

is completely predictable from its past samples, since consists

• f line spectrum.

in (16-71) is a shape deterministic stochastic process.

∑

+ =

k k k k

t a nT X ) cos( ) ( φ ω (16-71) ( )

XX

S ω ) (nT X

• 1

ω

k

ω ω

) (ω

XX

S

PILLAI