LSI system Input v v1 x v1 x v2 x v2 x + + L + v3 x + v3 - - PDF document

Mathematical Tools for Neural and Cognitive Science

Section 3: Linear Shift-invariant Systems

Fall semester, 2017

Linear shift-invariant (LSI) systems

• Linearity (previously discussed):

“linear combination in, linear combination out”

• Shift-invariance (new property):

“shifted vector in, shifted vector out”

• Note: These two properties are independent

(think of some examples...)

1 2

LSI system

v

Input

v1 x v4 x v3 x v2 x

L

Output

v1 x v4 x v3 x v2 x + + + + + +

As before, express input as a sum of “impulses”, weighted by elements of x

LSI system

v

Input

v1 x v4 x v3 x v2 x

L

Output

v1 x v4 x v3 x v2 x + + + + + +

• Shift-invariance => responses to

impulses are shifted copies of each other

• Linearity => response to x is sum of

responses to impulses, weighted by elements of x

3 4

LSI system

v

Input

v1 x v4 x v3 x v2 x

L

Output

v1 x v4 x v3 x v2 x + + + + + +

LSI systems are characterized by their “impulse response”

Convolution

• Sliding dot products
• Matrix description
• Boundaries: zero-padded, reflected, circular
• Examples: impulse, delay, average, difference

In Out

+

r

1

r2

r3

+

r0 r1 r2

y(n) = X

k

r(n − k)x(k) = X

k

r(k)x(n − k)

5 6

In Out

+

r

1

r2

r3

Feedback LSI system

y(n) = X

k

f(n − k)x(k) + X

k

g(n − k)y(k) (In general, we’ll stick to feedforward (FIR) systems)

• Infinite impulse response (IIR)
• Recursive => possibly unstable

2D convolution

[figure c/o Castleman]

• sliding window

7 8

Discrete Sinusoids

10 20 30 −1 1 10 20 30 −1 1

More generally: “amplitude” “phase” (radians) “frequency” (radians/sample) “frequency” (cycles/vectorLength) , ω = 2πk/N cos(ωn)

Shifting Sinusoids

... via a well-known trigonometric identity: cos(a − b) = cos(a) cos(b) + sin(a) sin(b) We’ll also need conversions between polar and rectangular coordinates: A = p x2 + y2, φ = tan−1(y/x) x = A cos(φ), y = A sin(φ)

x

y A

φ

A cos(ωn − φ) = A cos(φ) cos(ωn) + A sin(φ) sin(ωn)

9 10

Shifting Sinusoids

A cos(ωn − φ) = A cos(φ) cos(ωn) + A sin(φ) sin(ωn) A scaled and shifted sinusoidal vector can be written as a weighted sum of two fixed sinusoidal vectors!

10 20 30 −1 1 10 20 30 −1 1

fixed cos/sin vectors:

A sin φ

φ

A cos φ

A

two scale factors:

10 20 30 −1 1

A = 1.6, φ = 2π0/12

A cos(ωn − φ) = A cos(φ) cos(ωn) + A sin(φ) sin(ωn)

10 20 30 −1 1 10 20 30 −1 1

fixed cos/sin vectors:

10 20 30 −1 1

A = 1.6, φ = 2π1/12

A scaled and shifted sinusoidal vector can be written as a weighted sum of two fixed sinusoidal vectors!

Shifting Sinusoids

11 12

10 20 30 −1 1

A = 1.6, φ = 2π2/12 A = 1.6, φ = 2π3/12

10 20 30 −1 1

A = 1.6, φ = 2π4/12

10 20 30 −1 1

A = 1.6, φ = 2π5/12

10 20 30 −1 1

A = 1.6, φ = 2π6/12

10 20 30 −1 1

A cos(ωn − φ) = A cos(φ) cos(ωn) + A sin(φ) sin(ωn)

10 20 30 −1 1 10 20 30 −1 1

fixed cos/sin vectors:

A scaled and shifted sinusoidal vector can be written as a weighted sum of two fixed sinusoidal vectors!

Shifting Sinusoids

x(n) = cos(ωn)

LSI response to sinusoids

(input)

13 14

x(n) = cos(ωn)

(convolution formula)

LSI response to sinusoids

L

x(n) = cos(ωn)

inner product of impulse response with cos/sin, respectively (trig identity)

LSI response to sinusoids

L

15 16

x(n) = cos(ωn)

L

LSI response to sinusoids

x(n) = cos(ωn)

A sin φ

φ

A cos φ

A Rc(ω) Rs(ω)

cr(ω)

sr(ω)

(convert rectangular to polar coordinates)

LSI response to sinusoids

17 18

x(n) = cos(ωn)

L

“Sinusoid in, sinusoid out” (with modified amplitude/phase)

LSI response to sinusoids

(trig identity, in the opposite direction)

phases add amplitudes multiply

L

“Sinusoid in, sinusoid out” (with modified amplitude/phase) More generally, if input has amplitude and phase , Ax φx

LSI response to sinusoids

19 20

• Construct an orthogonal matrix of sin/cos pairs,

covering different numbers of cycles

The Discrete Fourier transform (DFT)

[all details on board...]

• Frequency multiples of radians/sample,

(specifically, )

• When we apply this matrix to an input vector,

think of output as paired coordinates

• Common to plot these pairs as amplitude/phase
• For , only need the cosine

part (thus, cosines, and sines) N/2 − 1 N/2 + 1

k=0 k=1 k=2 k=3

F =

21 22

The Fourier family

we are here

signal domain frequency domain

Note: the “fast Fourier transform” (FFT) is a computationally efficient implementation of the DFT. Computational cost is Nlog(N) operations, compared to the N2 operations that would be needed for matrix multiplication. x(n) = cos(ωn) NOTE: These dot products are just the Fourier transform of the impulse response r(m)!

LSI response to sinusoids

23 24

⇥ x

L

Fourier & LSI

⇥ x

L

Fourier & LSI

note: only 3 (of many) frequency components shown

25 26

⇥ x

L

Fourier & LSI

note: only 3 (of many) frequency components shown

v

Input

v1 x v4 x v3 x v2 x

L

Output

v1 x v4 x v3 x v2 x + + + + + +

LSI systems are characterized by their frequency response, specified by the Fourier Transform of their impulse response ⇥ x

L

Fourier & LSI

27 28

Complex exponentials: “bundling” sine and cosine

eiθ = cos(θ) + i sin(θ) eiωn = cos(ωn) + i sin(ωn)

n n

real part: imaginary part: eiωn

L

Complex exponentials: “bundling” sine and cosine

29 30

eiωn

L

F.T. of impulse response!

Complex exponentials: “bundling” sine and cosine

eiωn

L

F.T. of impulse response!

Complex exponentials: “bundling” sine and cosine

L

Note: implies that complex exponentials are eigenvectors!

31 32

The “convolution theorem”

convolve with convolve with Fourier Transform inverse Fourier Transform pointwise multiply by

The “convolution theorem”

33 34

The “convolution theorem”

convolve with Fourier Transform inverse Fourier Transform pointwise multiply by L FT F

(diagonal matrix)

⇒ F T ~ y = ˜ RF T~ x

Recap

• Linear system
• defined by superposition
• characterized by a matrix
• Linear Shift-Invariant (LSI) system
• defined by superposition and shift-invariance
• characterized by a vector (the impulse response)
• alternatively, characterized by frequency response

(the Fourier Transform of the impulse response), which specifies an amplitude multiplier and a phase shift for each frequency.

35 36

What do we do with Fourier Transforms?

• Represent/analyze periodic signals
• Analyze/design LSI systems. In particular,

how do you identify the nullspace?

Discrete Fourier transform (with complex numbers)

where ωk = 2πk N (inverse)

k

rn = 1 N

N−1

X

k=0

˜ rk eiωkn

[on board: why minus sign? why 1/N?]

37 38

Visualizing the (Discrete) Fourier transform

• Two conventional choices for frequency axis:
• Plot frequencies from k=0 to k=N/2

(in matlab: 1 to N/2-1)

• Plot frequencies from k=-N/2 to N/2-1

(in matab: use fftshift)

• Typically, plot Amplitude (and possibly

Phase, on a separate graph), instead of the cosine/sine (real/imaginary) parts

Example for k=2, N=32 (note indexing and amplitudes):

10 20 30 −1 1 10 20 30 −10 10 10 20 30 −1 1 −10 10 −10 10 k −10 10 −10 10 k 10 20 30 −10 10 (real part) (imag part)

=>

39 40

More examples

• constant
• sinusoid (see next slide)
• impulse
• Gaussian - “lowpass”
• DoG (difference of 2 Gaussians) - “bandpass”
• Gabor (Gaussian windowed sinusoid) - “bandpass”

[on board]

Retinal ganglion cells (1D)

Enroth-Cugell and Robson (1984)

41 42

Sampling causes “aliasing”

“Aliasing” - one frequency masquerades as another [on board] Sampling process is linear, but many-to-one (non-invertible) Given the samples, it is common/natural to assume that they arose from the lowest compatible frequency... Effect of sampling on the Fourier Transform: Sum of shifted copies |X(ω)| |Xs(ω)|

43 44

Real-world aliasing

downsample by 2

“Moiré pattern”

Pre-filtering to avoid spectral overlap (“aliasing”)

L(ω) L(ω)

lowpass filter, cutoff at π/∆

|X(ω)| |Xs(ω)|

45 46

Real-world aliasing

, with pre-filtering downsample by 2

47