OPTIMUM OPTIMUM ADAPTIVE ALGORITHMS ADAPTIVE ALGORITHMS for for - - PowerPoint PPT Presentation

OPTIMUM OPTIMUM ADAPTIVE ALGORITHMS ADAPTIVE ALGORITHMS

for for

SYSTEM IDENTIFICATION SYSTEM IDENTIFICATION

• Dept. of Computer Engineering & Informatics

University of Patras, GREECE George V. Moustakides

Definition of the problem

wn Physical System Physical System xn yn rn

+

Model h0 h1 … hN-1 Model h0 h1 … hN-1 r'n

• en

Common Model Transversal Filter: r'n=h0xn+ h1xn-1+…+ hN-1xn-N+1

Applications

• Echo Cancellation
• Filtering
• Equalization
• Control
• Seismology
• Array Processing

Echo Cancellation in Audio-Conferencing

Room-1 Room-2 xn xn yn yn wn wn

Multipath xn yn wn Parametric Model en

• We desire: en ≅ wn

We desire: en ≅ wn

Features of Adaptive Algorithms

• Simplicity & Low Computational Complexity
• Fast Convergence
• Fast Tracking
• Robustness in Finite Precision

Mathematical Setting

We are given sequentially two sets of data xn: Input sequence yn: Measured sequence We would like to express yn as

yn=h0xn+ h1xn-1+ …+ hN-1xn-N+1+en yn=h0xn+ h1xn-1+ …+ hN-1xn-N+1+en

and identify the filter coefficients hi adaptively using algorithms

• f the form

Hn=Hn-1 + µF(Hn-1, Yn , Xn) Hn=Hn-1 + µF(Hn-1, Yn , Xn)

where Hn=[ h0 h1 h2 … hN-1 ]T

, Xn=[ xn xn-1 … ]T , Yn=[ yn yn-1 … ]T

An Important Algorithmic Class

Xn = [ xn xn-1 … xn-N+1]T Zn: Regression Vector function of the input data xn xn-1 … Well known algorithms in the class: LMS: Zn = Xn RLS: Zn = Qn

• 1Xn with Qn = (1-µ) Qn-1 + µXnXn

T

(exponentially windowed sample covariance matrix) FNTF: Zn = Qn

• 1Xn with Qn the covariance matrix of the xn

sequence assuming that it is AR(M).

1 1 T n n n n n n n n

e y H X H H e Z µ

− −

= − = +

Generalization

Xn = [ xn xn-1 … xn-N+1]T Zn,i : vector functions of the input data xn xn-1 …

, 1 1 1 , ,

, 0,..., 1

T n i n i n n i p n n n i n i i

e y H X i p H H e Z µ

− − − − − =

= − = − = + ∑

Well known algorithms in the class: SWRLS: Zn,i = Qn

• 1Xn-i with Qn=XnXn

T +…+ Xn-p+1XT n-p+1

UDRLS: …

, 1 1 1 , ,

, 0,..., 1

T n i n i n n i p n n n i n i i

e y H X i p H H e Z µ

− − − − − =

= − = − = + ∑

By selecting different Regression Vectors Zn,i we obtain different adaptive algorithms. We need to compare them in order to select the optimum! For simplicity we assume EXACT MODEL!! and wn white noise For simplicity we assume EXACT MODEL!! For simplicity we assume EXACT MODEL!! and and w wn

n white noise

white noise

n n T n

w X H y + =

*

Comparing the Prediction Error

Classically the transient phase refers to stationary data. Let us assume that the noise wn has power 20db and we have two competing algorithms A1 and A2. We observe their prediction errors en:

1000 2000 3000 4000 5000

• 25
• 20
• 15
• 10
• 5

5 10

Prediction Error Power in db Number of Iterations

A1 A2 A2

en = wn - (Hn-1-H*)TXn εn = (Hn-1-H*)TXn

Excess Error

en = wn - (Hn-1-H*)TXn εn = (Hn-1-H*)TXn

Excess Error Excess Error

1000 2000 3000 4000 5000

• 60
• 50
• 40
• 30
• 20
• 10

10

Excess Error Power in db Number of Iterations

A1 A2 A2

Comparing the Excess Error

A Fair Comparison Method

To fairly compare adaptive algorithms

• We first select the step size µ in each algorithm so that all

algorithms under comparison have the same steady state excess error power (Excess Mean Square Error EMSE).

• The algorithm that converges faster is considered as the

“best”. Can we select the step size µ analytically in order to achieve a predefined value for the EMSE at steady state? Can we characterize the speed of convergence analytically during the transient phase?

Analysis of the EMSE

EMSE = E{[ (Hn-1-H*)TXn]2} = γn+πn EMSE = E{[ (Hn-1-H*)TXn]2} = γn+πn

γn: Starts from an O(1) value and tends

exponentially fast to zero for stable algorithms (due to initial conditions).

πn: Starts from an O(µ2) value and tends to

an O(µ) value at steady state (due to the additive noise). µ << 1 µ << 1

Assumptions

• The input vector process {Xn} is stationary.
• The Regression vector processes {Zn,i} are stationary.
• The noise process {wn} is zero mean white stationary

and independent of the process {Xn}.

• The input vector process {Xn} is stationary.
• The Regression vector processes {Zn,i} are stationary.
• The noise process {wn} is zero mean white stationary

and independent of the process {Xn}.

, 1 1 1 , ,

, 0,..., 1

T n i n i n n i p n n n i n i i

e y H X i p H H e Z µ

− − − − − =

= − = − = + ∑

RLS: Zn = Qn

• 1Xn with Qn = (1-µ) Qn-1 + µXnXn

T

Exponential Convergence

( ) { } { } { }

( )

{ }

1 min 1 1 , , 1 , min

log lim 2 ( ) ( ) ( ) min Re

n n p p T T T n i n i n i i n n n i i p n n i i i i i

• n

E Z X E Z X E Z X Z Z γ µ λ µ λ λ

− →∞ − − − + = = − + =

= + = = = = =

∑ ∑ ∑

A A A

λi : eigenvalues of the matrix A

Steady State EMSE

{ }

{ } { } { } { } { } { }

2 2 2 1 1 , , 1 ,

lim ( )

n w n w n T n n T p p T T T n i n i n i i n n n i i p n n i i i T n n

trace

• E

w E X X E Z X E Z X E Z X Z Z E Z Z π µσ µ σ

→ ∞ − − − + = = − + =

= + = = + = = = = = =

∑ ∑ ∑

QP Q AP PA R A R

Analytic Local Measure: Efficacy

{ } { } { } { }

2 2 m in m in 2 m in

4 ( ) 2 ( ) 2 ( ) 2

w w w

tra ce tra ce R a te tra ce E F F tra ce π µ σ π µ σ π λ µ λ σ λ ≈ ≈ ≈ ≈ = Q P Q P A A Q P A Q P

An algorithm A1 is better than an algorithm A2 if

EFF1>EFF2

Goal: Maximization of the Efficacy with respect to the regression vectors Zn,i. An algorithm A1 is better than an algorithm A2 if

EFF EFF1

1>EFF

>EFF2

2

Goal: Maximization of the Efficacy with respect to the Maximization of the Efficacy with respect to the regression vectors regression vectors Z Zn,i

n,i.

.

Theorem: The maximum value of the Efficacy is 1/N. The maximum is attained if and only if: where Q=E{XnXnT}, α is any positive scalar. The algorithm is called: LMS-Newton (LMS-N) Theorem: Theorem: The maximum value of the Efficacy is 1/N. The maximum is attained if and only if: where Q=E{XnXnT}, α is any positive scalar. The algorithm is called: LMS LMS-

• Newton

Newton (LMS-N)

n n

X Z

1 −

= Q α

Optimum Algorithm

Corollary: Corollary: If Q=E{XnXnT}=σx2I then the optimum algorithm in the class is the LMS.

Practically Optimum Algorithms

RLS: RLS:

Zn = Qn

• 1Xn

Qn = (1-µ) Qn-1 + µXnXn

T

Under steady state Qn is a good approximation to Q=E{XnXn

T}

So RLS is expected to match the optimum performance.

SWRLS: SWRLS:

Zn,i = Qn

• 1Xn-i

Qn = XnXn

T+ Xn-1XT n-1+…+ Xn-p+1XT n-p+1

If the window p is small then Qndoes approximate well Q. If however p is large then due to the LLN the approximation can be good.

Simulation

Xn=[xn xn-1 … xn-19], H*=[h0 h1 ... h19] xn=0.9xn-1+vn, Gaussian AR process wn: Gaussian additive white noise 20db N=20 We select µ such that the steady state EMSE=35db At time n=5000 abrupt change H*= - [h0 h1 ... h19] LMS-N (Optimum) RLS SWRLS-30 (Window size p=30) SWRLS-100 (Window size p=100)

END END