Practical Modeling Overview Introduction Like most engineering, - - PowerPoint PPT Presentation

Parametric Models x(n) = −

P

• k=1

akx(n − k) + w(n) +

Q

• k=0

bkw(n − k)

• The entire sequence has focused on parametric models
• Defined by linear difference equations
• Three types: AZ(Q), AP(P), PZ(Q,P)
• By convention a0 = b0 = 1
• Completely specified by the parameters

{a1, . . . , aP , b1, . . . , bQ, σ2

w}

• x(n) is a stationary process if

– The parameters are constants – The system is stable (excludes random walks)

• J. McNames

Portland State University ECE 539/639 Practical Modeling

• Ver. 1.00

3

Practical Modeling Overview

• Parametric models
• Preprocessing
• Model selection
• Model estimation
• Model validation
• Inverse whiteness tests
• J. McNames

Portland State University ECE 539/639 Practical Modeling

• Ver. 1.00

1

Signal Modeling x(n) = −

P

• k=1

akx(n − k) + w(n) +

Q

• k=0

bkw(n − k)

• Signal modeling is the process of obtaining a statistical model of

a signal

• If the signal is ARMA, this is reduced to estimating the parameters

(or equivalent) of the model above: {a1, . . . , aP , b1, . . . , bQ, σ2

w}

• Just one observed signal {x(n)}N−1

n=0

• Unlike least squares filtering, in which we are trying to estimate

the relationship between two signals

• Many applications

– Signal mining (exploratory analysis, hypothesis generation) – Compression – Prediction – Spectral estimation

• J. McNames

Portland State University ECE 539/639 Practical Modeling

• Ver. 1.00

4

Introduction

• Like most engineering, statistical signal processing is a

combination of art and practice

• Becoming skilled in the art requires

– Strong theoretical background – Good grasp of the tradeoffs between possible techniques – Many tools for identifying structure in data – Experience

• J. McNames

Portland State University ECE 539/639 Practical Modeling

• Ver. 1.00

2

Model Building Process

• Essentially three key steps
• 1. Model selection
• 2. Parameter estimation
• 3. Validation
• Good judgement and experience are necessary for all three
• Difficult (impossible?) to fully automate
• J. McNames

Portland State University ECE 539/639 Practical Modeling

• Ver. 1.00

7

Parametric Spectral Estimation x(n) = −

P

• k=1

akx(n − k) + w(n) +

Q

• k=0

bkw(n − k) Rx(ejω) = σ2

w

• 1 + Q

k=1 bke−jωk

1 + P

k=1 ake−jωk

• 2
• If we have a good estimate of the model parameters, we can plug

them in to the PSD equation for ARMA processes

• If the parameters are close to the true values, we would expect the

PSD estimate to be accurate

• However, it is in no way an optimal estimate of the PSD
• There are some methods that directly attempt to estimate the

PSD in the frequency domain – Not discussed in this class

• In practice, most people use plug-in estimates
• J. McNames

Portland State University ECE 539/639 Practical Modeling

• Ver. 1.00

5

Model Selection

• Goal is to select a parametric model to fit the data
• Ideally would like the model to

– Be as simple as possible – Fully explain the variation in the data – Have parameters that have physical meaning for the application – Be in a form that is mathematically tractable (e.g., linear in the parameters)

• When possible, domain knowledge of the problem should guide

this decision

• In many instances, have to resort to data analysis methods
• J. McNames

Portland State University ECE 539/639 Practical Modeling

• Ver. 1.00

8

Preprocessing

• In practice some preprocessing is often necessary before the

techniques can be applied

• Essentially slowly varying components must be eliminated

– DC components – Linear or polynomial trends – Seasonal variations – Unit poles (random walk effects)

• Can often be achieved with highpass of difference filters
• If using a moving window, the window should span at least several

periods of the lowest frequency component of interest

• J. McNames

Portland State University ECE 539/639 Practical Modeling

• Ver. 1.00

6

Model Validation

• Goals are

– Determine whether the model sufficiently “agrees” with the

• bserved data

– Describes the true source of the data – Solves the application problem at hand

• If not, the first two steps are repeated
• Data analysis techniques can help determine how well the model

fits the observed data

• J. McNames

Portland State University ECE 539/639 Practical Modeling

• Ver. 1.00

11

Model Selection Through Data Analysis

• If considering only ARMA, AR, and MA processes can use the

autocorrelation and partial autocorrelation functions – MA processes have an autocorrelation of zero for ℓ > Q – AR processes have a partial autocorrelation of zero for ℓ > P – ARMA processes don’t have either

• Thus ACF and PACF are often used to select both

– Model structure: MA, AR, ARMA – Model order: P and Q

• Keep in mind that a high order AR or MA process can serve as a

good model of other types of processes

• Filter structure (e.g., direct or lattice) is not critical at this stage
• J. McNames

Portland State University ECE 539/639 Practical Modeling

• Ver. 1.00

9

Model Validation Through Data Analysis

• If the model is accurate, the residual or prediction error signal

should be indistinguishable from white noise

• We can test for various properties of a white noise process to help

identify statistical “structure”

• There are many techniques for this

– Autocorrelation function should be an impulse – Partial autocorrelation function should be an impulse – Periodogram should be flat – The performance criterion doesn’t decrease too fast as the model order is increased – Cross validation (rarely used in signal processing, not sure why)

• Specifics are in the text
• J. McNames

Portland State University ECE 539/639 Practical Modeling

• Ver. 1.00

12

Parameter Estimation

• Goal is to estimate the model parameters given the observed data

{x(n)}N−1

n=0

• Also called model fitting
• Many different approaches used in the literature

– Least squares – Maximum likelihood – Spectral matching (frequency domain estimation) – Robust error measures – Moment matching – Bayesean approaches

• Book focuses on least squares
• If model structure is AP, leads to a closed form solution
• J. McNames

Portland State University ECE 539/639 Practical Modeling

• Ver. 1.00

10