Adaptive Filters Algorithms (Part 1) Gerhard Schmidt - - PowerPoint PPT Presentation

Slide 1

Gerhard Schmidt

Christian-Albrechts-Universität zu Kiel Faculty of Engineering Electrical Engineering and Information Technology Digital Signal Processing and System Theory

Adaptive Filters – Algorithms (Part 1)

Slide 2 Slide 2 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Today:

Contents of the Lecture

Exercises:

 Topics for the Talks

Adaptive Algorithms:

 Introductory Remarks  Recursive Least Squares (RLS) Algorithm  Least Mean Square Algorithm (LMS Algorithm) – Part 1  Least Mean Square Algorithm (LMS Algorithm) – Part 2  Affine Projection Algorithm (AP Algorithm)

Slide 3 Slide 3 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Possible Topics

Adaptive Filters – Talks

Suggestions:

 Hearing aids  GSM (source) coding  Localization and tracking  Active noise control (anti-noise)  Noise suppression  Bandwidth extension  Audio upmix of stereo signals  Adaptive beamforming  MPEG audio coding  Non-linear echo cancellation  Adaptation of neural networks  Feedback suppression  …

Your own topic suggestions are welcome …

Slide 4 Slide 4 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Contents

Adaptive Filters – Algorithms

Exercises:

 Topics for the Talks

Adaptive Algorithms:

 Introductory Remarks  Recursive Least Squares (RLS) Algorithm  Least Mean Square Algorithm (LMS Algorithm) – Part 1  Least Mean Square Algorithm (LMS Algorithm) – Part 2  Affine Projection Algorithm (AP Algorithm)

Slide 5 Slide 5 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Motivation

Introductory Remarks

Why adaptive filters?

 Signal properties are not known in advance or are time variant.  System properties are not known in advance or time variant.

Examples:

 Speech signals  Mobile telephone channels

Slide 6 Slide 6 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Literature

Introductory Remarks

 E. Hänsler, G. Schmidt: Acoustic Echo and Noise Control, Wiley, 2004  S. Haykin: Adaptive Filter Theory, Prentice Hall, 2002  A. Sayed: Fundamentals of Adaptive Filtering, Wiley, 2004  E. Hänsler: Statistische Signale: Grundlagen und Anwendungen, Springer, 2001

(in German)

Books:

Slide 7 Slide 7 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Two Hook-Ups of Adaptive Filters

Introductory Remarks

Transmission channel Adaptive filter System Adaptive filter

Adaptive filter for channel equalization: Adaptive filter for system identification:

Slide 8 Slide 8 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Application Examples – Part 1

Introductory Remarks

Adaptive filter for cancellation of hybrid echoes:

Adaptive filter Hybrid

Slide 9 Slide 9 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Application Examples – Part 2

Introductory Remarks

Adaptive filter Signal source Noise source Transmission path 2 Transmission path 1

Signal model

Noisy signal Reference signal

Adaptive filter for noise reduction with reference signal:

Slide 10 Slide 10 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Application Examples – Part 3

Introductory Remarks

Antenna array:

Adaptive filter 1 Adaptive filter 2 Adaptive filter N

Slide 11 Slide 11 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Application Examples – Part 4

Introductory Remarks

Adaptive equalization without reference signal

Adaptive filter Decision circuit

Assumptions:

Slide 12 Slide 12 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Generic Setup

Introductory Remarks

Adaptive algorithm Adaptive filter Desired

• utput

signal

Slide 13 Slide 13 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Structure of an Adaptive FIR Filter

Introductory Remarks

Slide 14 Slide 14 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Error Measures – Part 1

Introductory Remarks

Mean square (signal) error: System distance:

+

No local noise

Slide 15 Slide 15 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Mean Square Error and System Distance

Introductory Remarks

Relation of the normalized mean square (signal) error power and the system distance:

Let be white noise:

Slide 16 Slide 16 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Adaptation

Introductory Remarks

Basic principle: New = old + correction

 „Correction“ depends on the input signal

and the error signal .

 Procedures differ by the functions and :

+ +

Local noise

Step size

Properties:

Slide 17 Slide 17 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Error Measures

Introductory Remarks

Three error measures control the adaptation:

 Coefficient error  A priori error  A posteriori error:

Old data New filter

+ +

Local noise

Slide 18 Slide 18 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Contents

Adaptive Filters - Algorithms

Exercises:

 Topics for the Talks

Adaptive Algorithms:

 Introductory Remarks  Recursive Least Squares (RLS) Algorithm  Least Mean Squares Algorithm (LMS Algorithm) – Part 1  Least Mean Squares Algorithm (LMS Algorithm) – Part 2  Affine projection Algorithm (AP Algorithm)

Slide 19 Slide 19 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Algorithmic Properties

Recursive Least Squares (RLS) Algorithm

Attributes of the RLS algorithm:

 No a priori knowledge of signal statistics is required.  Optimization criterion is the (weighted) sum of squared errors.

Slide 20 Slide 20 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Error Criterion

Recursive Least Squares (RLS) Algorithm

Signal Filter Forgetting factor

error: error inserted:

adaptive algorithm filter

Alternative:

Signal at time l Filter at time n Adaptation algorithm Filter

Slide 21 Slide 21 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Derivation – Part 1

Recursive Least Squares (RLS) Algorithm

Differentiate with respect to the complex filter coefficients and setting the result to zero: Definitions:

… Estimate for the auto correlation matrix … Estimate for the cross correlation vector

Cost function:

Slide 22 Slide 22 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Derivation – Part 2

Recursive Least Squares (RLS) Algorithm

From Simon Haykin, „Adaptive Filter Theory“, Prentice Hall, 2002: The Matrix Cookbook [ http://matrixcookbook.com ]

• r:

Slide 23 Slide 23 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Derivation – Part 3

Recursive Least Squares (RLS) Algorithm

Inserting the results leads to:

„Wiener solution“

… assuming that the auto correlation matrix is invertible

adaptive algorithm filter

Adaptation algorithm Filter

Slide 24 Slide 24 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Recursion – Part 1

Recursive Least Squares (RLS) Algorithm

Recursion of the auto correlation matrix over time: Recursion of the cross correlation vector over time:

Slide 25 Slide 25 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Recursion – Part 2

Recursive Least Squares (RLS) Algorithm

Matrix Inversion Lemma: Inserting the Lemma in the recursion: Recursion for the auto correlation matrix:

Slide 26 Slide 26 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Recursion – Part 3

Recursive Least Squares (RLS) Algorithm

Definition of a gain vector: Inserting this definition leads to: Recursion for the auto correlation matrix:

Slide 27 Slide 27 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Recursion – Part 4

Recursive Least Squares (RLS) Algorithm

Definition of a gain factor: Rewriting leads to: Multiplication by the denominator on the right hand side leads to:

Slide 28 Slide 28 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Recursion – Part 5

Recursive Least Squares (RLS) Algorithm

Recursion of the filter coefficient vector: Step from n to n+1: Reducing the right hand side: Inserting the recursion of the cross correlation vector leads to:

Slide 29 Slide 29 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Recursion – Part 6

Recursive Least Squares (RLS) Algorithm

we obtain: What we have so far: If we insert the recursive computation of the inverse auto correlation matrix

Slide 30 Slide 30 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Recursion – Part 7

Recursive Least Squares (RLS) Algorithm

Gain factor Error: old filter with new data

What we have so far: Inserting according to results in

Slide 31 Slide 31 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Adaptation Rule – Part 1

Recursive Least Squares (RLS) Algorithm

Adaptation rule for the filter coefficients according to the RLS algorithm: Inserting previous results:

Slide 32 Slide 32 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Summary

Recursive Least Squares (RLS) Algorithm

Computing a preliminary gain vector (complexity prop. N²): Update of the inverse auto correlation matrix (complexity prop. N²): Computing the error signal (complexity prop. N): Update of the filter vector (complexity prop. N):

Step size (0 … 1), will be treated later …

Slide 33 Slide 33 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Contents

Adaptive Filters – Algorithms

Exercises:

 Topics for the Talks

Adaptive Algorithms:

 Introductory Remarks  Recursive Least Squares (RLS) Algorithm  Least Mean Square Algorithm (LMS Algorithm) – Part 1  Least Mean Square Algorithm (LMS Algorithm) – Part 2  Affine Projection Algorithm (AP Algorithm)

Slide 34 Slide 34 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Basics – Part 1

Least Mean Square (LMS) Algorithm

Optimization criterion:

 Minimizing the mean square error

Assumptions:

 Real, stationary random processes

Structure:

Unknown system Adaptive filter

Slide 35 Slide 35 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Basics – Part 2

Least Mean Square (LMS) Algorithm

Output signal of the adaptive filter: Error signal: Mean square error:

Unknown system Adaptive filter

Slide 36 Slide 36 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Basics – Part 3

Least Mean Square (LMS) Algorithm

Mean square error: The filter coefficients are adjusted optimally in case of orthogonality: Abbreviations:

(auto correlation matrix) (cross correlation vector)

Solution (according to Wiener):

(assuming that the inverse exists)

Slide 37 Slide 37 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Basics – Part 4

Least Mean Square (LMS) Algorithm

Mean square error: Optimal filter vector: Minimum mean square error:

Slide 38 Slide 38 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Basics – Part 5

Least Mean Square (LMS) Algorithm

Quadratic form unique minimum Mean square error: Minimum mean square error : Mean square error:

Slide 39 Slide 39 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Derivation – Part 1

Least Mean Square (LMS) Algorithm

Derivation with respect to the coefficients of the adaptive filter: Inserting , results in:

… needed later on …

Slide 40 Slide 40 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Derivation – Part 2

Least Mean Square (LMS) Algorithm

Method according to Newton

What we have so far: Resolving it to leads to: With the introduction of a step size , the following adaptation rule can be formulated:

Slide 41 Slide 41 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Derivation – Part 3

Least Mean Square (LMS) Algorithm

Method according to Newton: Method of steepest descent:

LMS algorithm

For practical approaches the expectation value is replaced by its instantaneous

• value. This leads to the so-called least mean square algorithm:

Slide 42 Slide 42 Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1

Summary and Outlook

Adaptive Filters – Algorithms

This week:

 Topics for the Talks  Introductory Remarks  Recursive Least Squares (RLS) Algorithm  Least Mean Square Algorithm (LMS Algorithm) – Part 1

Next week:

 Least Mean Square Algorithm (LMS Algorithm) – Part 2  Affine Projection Algorithm (AP Algorithm)