[PPT] - Partial and Autocorrelation Functions Overview Autocorrelation PowerPoint Presentation

SLIDE 1

Autocorrelation Function Properties and Examples ρx(ℓ) = γx(ℓ) γx(0) = γx(ℓ) σ2

The ACF has a number of useful properties

signals

– If ˆ ρx(ℓ) ≈ δ(ℓ), we can conclude that the process consists of nearly uncorrelated samples

Partial and Autocorrelation Functions Overview

Example 1: 1st Order Moving Average Find the autocorrelation function of a 1st order moving average process, MA(1): x(n) = w(n) + b1w(n − 1) where w(n) ∼ WN(0, σ2

Autocorrelation Function Defined Normalized Autocorrelation, also known as the Autocorrelation Function (ACF) is defined for a WSS signal as ρx(ℓ) = γx(ℓ) γx(0) = γx(ℓ) σ2

where γx(ℓ) is the autocovariance of x(n), γxx(ℓ) = E [[x(n + ℓ) − µx][x(n) − µx]∗] = rx(ℓ) − |µx|2

SLIDE 2

All-Pole Models H(z) = b0 A(z) = b0 1 + P

estimated by solving a set of linear equations

context of all-pole models (my motivation)

model if the AZ(Q) model is minimum phase

be well approximated by an AP(P) model

Example 2: 1st Order Autoregressive Find the autocorrelation function of a 1st order autoregressive process, AR(1): x(n) = −a1x(n − 1) + w(n) where w(n) ∼ WN(0, σ2

← →

ROC of |z| < |α|.

AP Equations Let us consider a causal AP(P) model: H(z) +

akH(z)z−k = b0 h(n) +

akh(n − k) = b0δ(n)

akh(n − k)h∗(n − ℓ) = b0h∗(n − ℓ)δ(n)

akh(n − k)h∗(n − ℓ) =

b0h∗(n − ℓ)δ(n)

akrh(ℓ − k) = b0h∗(−ℓ)

Autocorrelation Function Properties ρx(ℓ) = γx(ℓ) γx(0) = γx(ℓ) σ2

damped exponentials (infinite extent)

by solving a set of linear equations

model may be appropriate

exponentials (infinite extent)

SLIDE 3

Solving the AP Equations If we know the autocorrelation, we can solve these equations for a and b0 ⎡ ⎢ ⎢ ⎢ ⎣ rh(0) r∗

· · · r∗

rh(1) rh(0) · · · r∗

. . . . . . ... . . . rh(P) rh(P − 1) · · · rh(0) ⎤ ⎥ ⎥ ⎥ ⎦ ⎡ ⎢ ⎢ ⎢ ⎣ 1 a1 . . . aP ⎤ ⎥ ⎥ ⎥ ⎦ = ⎡ ⎢ ⎢ ⎢ ⎣ |b0|2 . . . ⎤ ⎥ ⎥ ⎥ ⎦

AP Equations Continued Since AP(P) is causal, h(0) = b0, h∗(0) = b∗

akrh(−k) = |b0|2 ℓ = 0

akrh(ℓ − k) = ℓ > 0 This has several important consequences. One is that the autocorrelation can be expressed as a recursive relation for ℓ > 0, since a0 = 1:

akrh(ℓ − k) = rh(ℓ) = −

akrh(ℓ − k) ℓ > 0

Solving for a ⎡ ⎢ ⎣ rh(1) rh(0) · · · r∗

. . . . . . ... . . . rh(P) rh(P − 1) · · · rh(0) ⎤ ⎥ ⎦ ⎡ ⎢ ⎢ ⎢ ⎣ 1 a1 . . . aP ⎤ ⎥ ⎥ ⎥ ⎦ = ⎡ ⎢ ⎣ . . . ⎤ ⎥ ⎦ ⎡ ⎢ ⎣ rh(1) . . . rh(P) ⎤ ⎥ ⎦ + ⎡ ⎢ ⎣ rh(0) · · · r∗

. . . ... . . . rh(P − 1) · · · rh(0) ⎤ ⎥ ⎦ ⎡ ⎢ ⎣ a1 . . . aP ⎤ ⎥ ⎦ = ⎡ ⎢ ⎣ . . . ⎤ ⎥ ⎦ rh + Rha = a = −R−1

These are called the Yule-Walker equations

AP Equations in Matrix Form We can collect the first P + 1 of these terms in a matrix ⎡ ⎢ ⎢ ⎢ ⎣ rh(0) rh(−1) · · · rh(−P) rh(1) rh(0) · · · rh(−P + 1) . . . . . . ... . . . rh(P) rh(P − 1) · · · rh(0) ⎤ ⎥ ⎥ ⎥ ⎦ ⎡ ⎢ ⎢ ⎢ ⎣ 1 a1 . . . aP ⎤ ⎥ ⎥ ⎥ ⎦ = ⎡ ⎢ ⎢ ⎢ ⎣ |b0|2 . . . ⎤ ⎥ ⎥ ⎥ ⎦ ⎡ ⎢ ⎢ ⎢ ⎣ rh(0) r∗

· · · r∗

rh(1) rh(0) · · · r∗

. . . . . . ... . . . rh(P) rh(P − 1) · · · rh(0) ⎤ ⎥ ⎥ ⎥ ⎦ ⎡ ⎢ ⎢ ⎢ ⎣ 1 a1 . . . aP ⎤ ⎥ ⎥ ⎥ ⎦ = ⎡ ⎢ ⎢ ⎢ ⎣ |b0|2 . . . ⎤ ⎥ ⎥ ⎥ ⎦

definite.

SLIDE 4

Yule-Walker Equation Comments Continued a = R−1

b0 =

characterizations of the model {rh(0), . . . , rh(P)} ↔ {b0, a1, . . . , aP }

and the recursive relation given earlier rh(ℓ) = −

akrh(ℓ − k) ℓ > 0 rh(−ℓ) = r∗

Solving for b0 ⎡ ⎢ ⎢ ⎢ ⎣ rh(0) r∗

· · · r∗

rh(1) rh(0) · · · r∗

. . . . . . ... . . . rh(P) rh(P − 1) · · · rh(0) ⎤ ⎥ ⎥ ⎥ ⎦ ⎡ ⎢ ⎢ ⎢ ⎣ 1 a1 . . . aP ⎤ ⎥ ⎥ ⎥ ⎦ = ⎡ ⎢ ⎢ ⎢ ⎣ |b0|2 . . . ⎤ ⎥ ⎥ ⎥ ⎦ rh(0) r∗

· · · r∗

⎡ ⎢ ⎢ ⎢ ⎣ 1 a1 . . . aP ⎤ ⎥ ⎥ ⎥ ⎦ = |b0|2 b0 = ±

akrh(k) = ±

AR Processes versus AP Models Concisely, we can write the Yule-Walker Equations as Rha = −rh If we have an AR(P) process, then we know rx(ℓ) = σ2

can equivalently write Rxa = −rx

– Find the parameters of an AR process, {a1, . . . , aP , σ2

rx(ℓ) – Find the parameters of an AP model, {a1, . . . , aP , b0}, given rh(ℓ)

Yule-Walker equations as simply Ra = −r

Yule-Walker Equation Comments a = R−1

b0 = ±

definite

determine the model parameters

autocorrelation sequence in terms the model parameters by solving a set of linear equations (Problem 4.6)

SLIDE 5

Partial Autocorrelation: Alternative Definition Define P [x(n)|x(1), . . . , x(n − 1)] as the minimum mean square error linear predictor of x(n) given {x(1), . . . , x(n − 1)} ˆ x(n) = P [x(n)|x(n − 1), . . . , x(1)] =

ckx(n − k) where ck = argmin

E

x(n))2 Similarly define P [x(0)|x(1), . . . , x(n − 1)] as the minimum mean square error linear predictor of x(0) given {x(1), . . . , x(n − 1)}, ˆ x(0) = P [x(0)|x(n − 1), . . . , x(1)] =

dkx(n − k)

Solving for a Recursively We can write the Yule-Walker equations as ⎡ ⎢ ⎢ ⎢ ⎣ r(0) r∗(1) · · · r∗(ℓ − 1) r(1) r(0) · · · r∗(ℓ − 2) . . . . . . ... . . . r(ℓ − 1) r(ℓ − 2) · · · r(0) ⎤ ⎥ ⎥ ⎥ ⎦ ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ a(ℓ)

a(ℓ)

. . . a(ℓ)

⎤ ⎥ ⎥ ⎥ ⎥ ⎦ = − ⎡ ⎢ ⎢ ⎢ ⎣ r(1) r(2) . . . r(ℓ) ⎤ ⎥ ⎥ ⎥ ⎦ Ra = −r a = −R−1r

a = [a(ℓ)

Partial Autocorrelation: Alternative Definition & Properties Then the PACF can be defined as the correlation between the residuals ˜ xn(n)

x1:n−1(n) = x(n) − P [x(n)|x(n − 1), . . . , x(1)] ˜ xn(0)

x1:n−1(0) = x(0) − P [x(0)|x(n − 1), . . . , x(1)] α(ℓ)

xn(ℓ)) (x(0) − ˆ xn(0))] E

xn(0))2 = E [(x(ℓ) − ˆ xn(ℓ)] (x(0) − ˆ xn(0))] E

xn(n))2

what has not already been explained (the residuals)

Partial Autocorrelation Partial Autocorrelation Function (PACF) also known as, the partial autocorrelation sequence (PACS), is defined as α(ℓ) ⎧ ⎪ ⎨ ⎪ ⎩ 1 ℓ = 0 a(ℓ)

ℓ > 0 α∗(−ℓ) ℓ < 0 where a(ℓ)

is the last element of a ∈ Rℓ and is given by the Yule-Walker equations R

a

= − r

complimentary properties

SLIDE 6

Example 3: MA(1) ACF

1 2 3 4 5 6 7 8 9 10 −1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1 ρ(l) Lag (l)

Partial Autocorrelation Properties & Intuiting α(ℓ) ⎧ ⎪ ⎨ ⎪ ⎩ 1 ℓ = 0 a(ℓ)

ℓ > 0 α∗(−ℓ) ℓ < 0

general

model may be appropriate

processes and ARMA(P, Q) processes

Example 3: MATLAB Code

Example 3: MA(1) ACF and PACF Plot the ACF and PACF of a MA(1) model with b1 = 0.9. Hint: the true PACF is given by α(ℓ) = (−b1)ℓ(1 − b2

1 − b2(ℓ+1)

SLIDE 7

Example 3: Relevant MATLAB Code Continued

Example 3: MA(1) PACF

1 2 3 4 5 6 7 8 9 10 −1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1 α(l) Lag (l)

Example 4: AR(1) ACF and PACF Plot the ACF and PACF of a AR(1) process with a = [1 0.9].

Example 3: Relevant MATLAB Code

SLIDE 8

Example 4: AR(1) PACF

1 2 3 4 5 6 7 8 9 10 −1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1 α(l) Lag (l)

Example 4: AR(1) ACF

1 2 3 4 5 6 7 8 9 10 −1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1 ρ(l) Lag (l)

Autocovariance Estimation

incomplete, characterization of WSS stochastic processes

– Single signal: γx(ℓ), rx(ℓ), αx(ℓ), Rx(ejω) – Two or more signals: γyx(ℓ), ryx(ℓ), Ryx(ℓ), G2

Example 4: Relevant MATLAB Code

SLIDE 9

Unbiased Autocovariance Estimation ˆ γu(ℓ) 1 N − |ℓ|

[x(n + |ℓ|) − ˆ µx] [x(n) − ˆ µx]

µx, ˆ γu(ℓ) would be unbiased

µx the estimate is asymptotically unbiased

Autocovariance Estimation Options In practical applications, we only have a real finite data record {x(n)}N−1 . There are two popular estimators of autocovariance worth considering: “unbiased” and biased. “Unbiased” ˆ γu(ℓ) 1 N − |ℓ|

[x(n + |ℓ|) − ˆ µx] [x(n) − ˆ µx] |ℓ| < N and ˆ γu(ℓ) = 0 for |ℓ| ≥ N. Here ˆ µx is the sample average of the sequence defined as ˆ µx 1 N

x(n)

Biased Autocovariance Estimation ˆ γb(ℓ)

N

[x(n + |ℓ|) − ˆ µx] [x(n) − ˆ µx] |ℓ| < N = N − |ℓ| N ˆ γu(ℓ)

E [ˆ γ(ℓ)] = N − |ℓ| N γ(ℓ)

MSE

Unbiased Autocovariance Estimation ˆ γu(ℓ) 1 N − |ℓ|

[x(n + |ℓ|) − ˆ µx] [x(n) − ˆ µx]

γu(ℓ) = ˆ γu(−ℓ)

distribution because the process is assumed WSS and ergodic

(N → ∞)

SLIDE 10

Biased versus Unbiased Estimators ˆ γb(ℓ) = N − |ℓ| N ˆ γu(ℓ) ∝

[x(n + |ℓ|) − ˆ µx] [x(n) − ˆ µx]

γb(ℓ) is biased, – The bias is small at small lags – For large lags, the bias is towards 0: ˆ γ(ℓ) → 0 as ℓ → ∞ – This is also a property of the true autocorrelation

(the tail) Biased var{ˆ γb(ℓ)} = O(1/N) Unbiased var{ˆ γu(ℓ)} = O(1/(N − |ℓ|))

Biased versus Unbiased Estimators ˆ γb(ℓ) = N − |ℓ| N ˆ γu(ℓ) ∝

[x(n + |ℓ|) − ˆ µx] [x(n) − ˆ µx]

functions

lag ℓ

Biased is Better? ˆ γb(ℓ) = N − |ℓ| N ˆ γu(ℓ) ∝

[x(n + |ℓ|) − ˆ µx] [x(n) − ˆ µx]

– At small lags, there is little difference between the two estimators – At large lags, the larger bias of the biased model is favorably traded for reduced variance

not been proven rigorously

γ(ℓ) = ˆ γb(ℓ)

Biased versus Unbiased Estimators ˆ γb(ℓ) = N − |ℓ| N ˆ γu(ℓ) ∝

[x(n + |ℓ|) − ˆ µx] [x(n) − ˆ µx]

γb(ℓ) is that it is positive semi-definite (i.e., nonnegative definite)

– We know the true autocovariance has this property – Autoregressive models built with the positive-definite estimates

– Most estimators of power spectral density R(ejω) are nonnegative if they are based on a positive-definite estimate of γ(ℓ)

γu(ℓ) may be positive definite for a particular sequence, but it is not guaranteed in general

SLIDE 11

Estimated Autocorrelation Variance Continued If Gaussian random process, var{ˆ rb(ℓ)} = 1 N

r2(m) + r(m + ℓ)r(m − ℓ)

1/(N − |ℓ|) instead of 1/N

most applications

about our estimators: – Desired properties of the sampling distribution depend on unknown properties of the random process

Estimated Autocorrelation Covariance ˆ γb(ℓ) = N − |ℓ| N ˆ γu(ℓ) ∝

[x(n + |ℓ|) − ˆ µx] [x(n) − ˆ µx]

– Stationary up to order four E [x(n)x(n + k)x(n + ℓ)x(n + m)] = f(k, ℓ, m) – Mean is zero µx = 0, so does not need to be estimated

Estimated ACF The natural estimate of the ACF is ˆ ρb(ℓ) ˆ γb(ℓ) ˆ γ(0) ˆ ρu(ℓ) ˆ γu(ℓ) ˆ γ(0)

– |ˆ ρb(ℓ)| ≤ 1 for all ℓ – Not true in general for ˆ ρu(ℓ)

complicated and based on unknown properties

Estimated Autocorrelation Variance ˆ γb(ℓ) = ˆ rb(ℓ) = 1 N

x(n + |ℓ|)x(n) ˆ γu(ℓ) = ˆ ru(ℓ) = 1 N − |ℓ|

x(n + |ℓ|)x(n)

E [ˆ rb(ℓ)] = N − |ℓ| N r(ℓ) E [ˆ ru(ℓ)] = r(ℓ)

r(ℓ) is complicated and not usable in practice – Depends on fourth joint cumulant of {x(n), x(n + k), x(n + ℓ), x(n + m)} – Depends on true unknown autocorrelation

SLIDE 12

Confidence Intervals

ρb(ℓ) is approximately normal

intervals of an IID sequence

ρb(ℓ)}

Estimated ACF Variance Again, if x(n) is a Guassian process then var{ˆ ρb(ℓ)} ≈ 1 N

ρ2(m) + ρ(m + ℓ)ρ(m − ℓ) + 2ρ2(ℓ)ρ2(m) − 4ρ(ℓ)ρ(m)ρ(m − ℓ)

process with independent inputs

ρ(ℓ) will generally have more correlation than the true ρ(ℓ)

well

Partial Autocorrelation Estimation

autocorrelation to estimate the PACF

bounded

and confidence intervals)

the CLT applies and we can use the same confidence intervals as for the ACF

Confidence Intervals Let x(n) be an IID sequence. Then ρ(0) = 1 ρ(ℓ) = |ℓ| > 0 cov{ˆ ρ(ℓ), ˆ ρ(ℓ + m)} ≈ m = 0 var{ˆ ρb(ℓ)} ≈ 1 N |ℓ| > 0 var{ˆ ρu(ℓ)} ≈ N (N − |ℓ|)2 |ℓ| > 0

estimated ACF because the variance of the estimator depends on the true ACF

purely random process

SLIDE 13

Example 5: AR(1) ACF

10 20 30 40 50 60 70 80 90 −1 −0.5 0.5 1 Lag (l) N=100 a1=−0.9 ρ(l)

Example 5: 1st Order Autoregressive Find the autocorrelation function of a 1st order autoregressive process, AR(1): x(n) = −a1x(n − 1) + w(n) where w(n) ∼ WN(0, σ2

unbiased estimates for N = 100. Do so several times for different values of a1.

Example 5: AR(1) ACF

10 20 30 40 50 60 70 80 90 −1 −0.5 0.5 1 Lag (l) N=100 a1=−0.9 ρ(l)

Example 5: AR(1) Signal

10 20 30 40 50 60 70 80 90 100 −1 1 2 3 4 5 6 Sample (n) N=100 a1=−0.9 x(n)

SLIDE 14

Example 5: AR(1) ACF

10 20 30 40 50 60 70 80 90 −1 −0.5 0.5 1 Lag (l) N=100 a1=0.0 ρ(l)

Example 5: AR(1) PACF

10 20 30 40 50 60 70 80 90 −1 −0.5 0.5 1 Lag (l) N=100 a1=−0.9 α(l)

Example 5: AR(1) ACF

10 20 30 40 50 60 70 80 90 −1 −0.5 0.5 1 Lag (l) N=100 a1=0.0 ρ(l)

Example 5: AR(1) Signal

10 20 30 40 50 60 70 80 90 100 −2.5 −2 −1.5 −1 −0.5 0.5 1 1.5 2 Sample (n) N=100 a1=0.0 x(n)

SLIDE 15

Example 5: AR(1) ACF

10 20 30 40 50 60 70 80 90 −1 −0.5 0.5 1 Lag (l) N=100 a1=0.5 ρ(l)

Example 5: AR(1) PACF

10 20 30 40 50 60 70 80 90 −1 −0.5 0.5 1 Lag (l) N=100 a1=0.0 α(l)

Example 5: AR(1) ACF

10 20 30 40 50 60 70 80 90 −1 −0.5 0.5 1 Lag (l) N=100 a1=0.5 ρ(l)

Example 5: AR(1) Signal

10 20 30 40 50 60 70 80 90 100 −2 −1 1 2 Sample (n) N=100 a1=0.5 x(n)

SLIDE 16

Example 5: AR(1) ACF

10 20 30 40 50 60 70 80 90 −1 −0.5 0.5 1 Lag (l) N=100 a1=0.9 ρ(l)

Example 5: AR(1) PACF

10 20 30 40 50 60 70 80 90 −1 −0.5 0.5 1 Lag (l) N=100 a1=0.5 α(l)

Example 5: AR(1) ACF

10 20 30 40 50 60 70 80 90 −1 −0.5 0.5 1 Lag (l) N=100 a1=0.9 ρ(l)

Example 5: AR(1) Signal

10 20 30 40 50 60 70 80 90 100 −5 5 Sample (n) N=100 a1=0.9 x(n)

SLIDE 17

Summary

processes

– MA: Finite ACF – AR: Finite PACF

solve/estimate the model parameters by solving a set of linear equations (Yule-Walker)

to the unbiased estimates – Less variance (always), and lower MSE (sometimes) – Positive definite (PSD is therefore also nonnegative)

Example 5: AR(1) PACF

10 20 30 40 50 60 70 80 90 −1 −0.5 0.5 1 Lag (l) N=100 a1=0.9 α(l)

Example 5: MATLAB Code