AIMS CDT - Signal Processing Michaelmas Term 2020 Xiaowen Dong - - PowerPoint PPT Presentation

AIMS CDT - Signal Processing

Michaelmas Term 2020

Xiaowen Dong

Department of Engineering Science

/70 2

Representation of Signals

/70

What is a representation of a signal?

• Sum of delta functions in time or space (sampling domain)
• good for display or playback
• not good for analysis (e.g., denoising, compression)

3 T t

/70

What is a representation of a signal?

• In a time-series setting, useful representation could be past samples

4

• Usefulness of the representation depends on the analysis goal
• which may vary but all share the core desire for simplification

Source: http://www.physik.uni-kl.de

• More generally, it involves transformation of the signal into a new domain
• where useful characteristics are revealed
• example: Fourier coefficients reveal rate of change of the signal

/70

Example: Denoising

5

200 400 600 800 1000 1200

• 10
• 5

5 10

Original Signal Denoised Signal

goal: separate signal and noise

/70

Example: Compression

6

goal: compress file size without sacrificing (too much) quality

JPEG 2000 (comp. ratio: 10) JPEG 2000 (comp. ratio: 100)

• riginal

/70

Example: Recognition

7

goal: capture true “causes” of signals given noisy input

true “causes” samples PCA ICA sparse coding KSVD [Tosic11]

/70

Signal representation via dictionaries

• Orthogonal dictionaries

8

= X x

c

signal coefficients dictionary “atoms”

Φ = [φ0 φ1 · · · φN−1]

/70

Signal representation via dictionaries

• Two viewpoints

9

= X X =

synthesis analysis

x

c

x

c

cn = φT

nx

Φ

ΦT

cn = ψT

nx

x = Φc = X

n cnφn

equivalent for complete dictionaries

/70

Signal representation via dictionaries

• Two viewpoints

10

synthesis

x

c

Φ

x = Φc = X

n cnφn

= X

analysis

x

ΦT

= X

a

an = φT

nx

not equivalent for over-complete dictionaries

/70

Signal representation via dictionaries

11

= X

design of dictionaries

• from mathematical modelling of data (transforms/analytic dictionaries)
• from a set of realisations of data (dictionary learning)
• from both sources (parametric training)

= X

/70

Outline

• A historical overview of dictionary design techniques
• signal representation via stochastic models

12

• transforms & analytic dictionaries
• trained dictionaries (dictionary learning)
• Discussion
• applications
• connection with deep learning

/70

1920s-30s: Stochastic models

• Stochastic models
• examples of parametric models
• describe how data were generated
• provide a special representations of signal from a time-series viewpoint
• Typical examples
• autoregressive (AR) models
• moving average (MA) models
• autoregressive moving average (ARMA) models

13

/70

Autocorrelation

• Autocovariance (covariance between the signal and a lagged version of

itself)

14

σxx(T) = 1 N − 1

N

X

t=1

(xt−T − µx)(xt − µx)

signal lagged version by T samples

• Autocorrelation (normalised autocovariance)

rxx(T) = σxx(T) σxx(0)

where σxx(0) = σ2

x

• Both are symmetric or even functions

/70

Autocorrelation

15

20 40 60 80 100 1 2 3 4 5 6 t −100 −50 50 100 −0.5 0.5 1 Lag r

/70

Autoregressive (AR) models

• An AR model predicts the value of a time-series from previous values

16

xt =

p

X

i=1

xt−iai + et

prediction error AR coefficients

et ∼ N(0, σ2

e)

• Matrix form

x = Ma + e

embedding matrix

/70

Autoregressive (AR) models

• The AR model is a special case of the multivariate regression model
• It also provides a special representation of the signal

17

• Use an AR(4) model to analyse the data shown before

ˆ a = [1.46, −1.08, 0.60, −0.186]T

x = Ma + e

ˆ x = Mˆ a

e = x − ˆ x

ˆ a = (MT M)−1MT x

• To compute AR coefficients and predictions

/70

(0.3882 − 0.079)/0.079 = 3.93

Autoregressive (AR) models

18

20 40 60 80 100 −1.5 −1 −0.5 0.5 1 1.5 t 20 40 60 80 100 −1.5 −1 −0.5 0.5 1 1.5 t

σ2

e = 0.079

σ2

x = 0.3882

signal to noise ratio (SNR)

(a) (b)

/70

Autoregressive (AR) models

• Relation to autocorrelation

19

xt = a1xt−1 + a2xt−2 + ... + apxt−p + et xtxt−k = a1xt−1xt−k + a2xt−2xt−k + ... + apxt−pxt−k + etxt−k σxx(k) = a1σxx(k − 1) + a2σxx(k − 2) + ... + apσxx(k − p) + σe,x

multiply by xt−k sum over t and divide by N-1 divide by signal variance

rxx(k) = a1rxx(k − 1) + a2rxx(k − 2) + ... + aprxx(k − p)

/70

Autoregressive (AR) models

• For an AR(4) model

20

r = Ra

Yule-Walker relations

a = R−1r

• More efficient way to estimate AR coefficients:
• autocorrelation matrix is symmetric

and Toeplitz

• efficient computation via a

recursive estimation technique (Levinson-Durbin)

/70

Multivariate processes

• Cross-covariance (covariance between one signal and the lagged version of

another)

21

σxy(T) = 1 N − 1

N

X

t=1

(xt−T − µx)(yt − µy)

• Cross-correlation

rxy(T) = σxy(T) p σxx(0)σyy(0)

• This is not symmetric and has non-unity at lag 0 (unlike

autocorrelation)

/70

Cross-correlation

22

/70

Multivariate AR (MAR) models

23

xt =

p

X

i=1

xt−iai + et

• Extending to multiple time steps

(stacking into embedding matrix )

• Univariate
• Multivariate

X = ˜ MA + E

ˆ A = ( ˜ MT ˜ M)−1 ˜ MT X

for time t:

A

xt

˜ xt

et

˜ xt

˜ M

/70

Multivariate AR (MAR) models

24

solid: original multivariate time-series dashed: predictions

/70

Outline

• A historical overview of dictionary design techniques
• signal representation via stochastic models

25

• transforms & analytic dictionaries
• trained dictionaries (dictionary learning)
• Discussion
• applications
• connection with deep learning

/70

1960s-2010s: Transforms & Dictionaries

26

Fourier basis diagonalises convolution operator g(t)

y(t) x(t) = ejωt

y(t) = Z ∞

−∞

ejω(t−τ)g(τ)dτ = ejωt G(ω)

• recall the LTI system

Y (ω) = X(ω)G(ω)

X(ω)

G(ω)

0.5 1 1.5 2 2.5 3 3.5 4

• 1
• 0.5

0.5 1

Fourier basis functions (real part)

1960s discrete Fourier transform

/70

Transform/analytic dictionary design

27

• discrete Fourier transform (DFT) provides an orthogonal dictionary:

1960s

φn(k) = ej 2π

N nk

discrete Fourier transform

• Fourier basis describes a signal in terms of its global frequency content and hence

is good at representing uniformly smooth signals

W = ej 2π

N

with

• fast Fourier transform (FFT) reduces complexity from to

O(N 2)

O(NlogN)

/70

Transform/analytic dictionary design

28

• DFT produces complex coefficients (“wasteful” for real signals)
• DFT assumes periodic extension (discontinuity at boundary)

Fourier transform cosine transform

x(t) = x(−t)

X(ω) = Z ∞

−∞

x(t)e−jωtdt = Z ∞

−∞

x(t)[cos(ωt) − jsin(ωt)]dt

X(ω) = Z ∞

−∞

x(t)cos(ωt)dt

1960s discrete Fourier transform

• a real and even signal leads to a real cosine transform

/70

Transform/analytic dictionary design

29

• the discrete version is called the discrete cosine transform (DCT)

1974 discrete cosine transform 1960s discrete Fourier transform

• DCT-II is behind the JEPG image compression standard

n

0 1 2 3

• 2 -1
• 3

DCT-I

n

0 1 2 3

• 2 -1
• 3
• 4

DCT-II

• several variants of symmetric extension, which all lead to smoother boundary
• DCT-II provides a real dictionary:

φn(k) = cos[ π N (n + 1 2)k]

/70

Transform/analytic dictionary design

30

• projection onto a fixed subset of DFT or DCT atoms leads to compaction

Σ = ΦΛΦT

known/empirical covariance eigenvectors (dictionary atoms as k largest eigenvectors)

• Karhunen-Loève transform (KLT) or principal component analysis (PCA) fits a low-

dimensional subspace to data

• more representation efficiency (by construction) but computationally expensive

1974 discrete cosine transform 1970-80s Karhunen- Loève transform 1960s discrete Fourier transform

• but data themselves can also be a source of compaction

/70

Transform/analytic dictionary design

31 1974 discrete cosine transform Karhunen- Loève transform 1960s discrete Fourier transform

• DCT atoms (12x12) vs. KLT atoms (trained using 12x12 image patches)

1970-80s

DCT KLT

[Rubinstein10]

/70

Transform/analytic dictionary design

32

• simplicity motivates sparsity

x ≈ X

K ckφk

subset of atoms that is different for each x

• sparsity requires localisation: atoms with concentrated supports
• allow more flexible representations based on local characteristics
• limit effects of irregularities (main source of large coefficients)

1974 discrete cosine transform Karhunen- Loève transform 1960s discrete Fourier transform

• sparsity requires shift from linear to nonlinear approximation

1970-80s

/70

Time-frequency representation

33

µt = 1 ||x||2 Z ∞

−∞

t|x(t)|2dt

∆t = ⇣ 1 ||x||2 Z ∞

−∞

(t − µt)2|x(t)|2dt ⌘ 1

2

time localisation

∆f = ⇣ 1 2π||x||2 Z ∞

−∞

(ω − µf)2|X(ω)|2dω ⌘ 1

2

µf = 1 2π||x||2 Z ∞

−∞

ω|X(ω)|2dω

frequency localisation

[Vetterli] pp. 624

time-frequency plane Heisenberg box (time-frequency tile)

/70

Time-frequency representation

• Consider three basic operations

34

[Vetterli] pp. 625

shift in time

y(t) = x(t − t0)

Y (ω) = e−jωt0X(ω)

FT shift in frequency

y(t) = ejω0tx(t)

Y (ω) = X(ω − ω0)

FT scaling in time

y(t) = √ax(at)

Y (ω) = 1 √aX(ω a )

FT

/70

Time-frequency representation

35

Heisenberg’s uncertainty principle

[Vetterli] pp. 624

Let x ∈ L2(R), then ∆t∆f ≥ 1 2

examples FT FT

/70

Time-frequency representation

• Consider three structured sets of functions

36

[Vetterli] pp. 639

shift in time

ϕ(t − mt0)

shift in frequency

R(ejkω0tϕ(t))

scaling in time

ϕ(a−lt)

/70

Time-frequency representation

37

time shift and scaling

[Vetterli] pp. 641

ϕl,m(t) = ϕ(a−lt − mt0)

l, m ∈ Z

(m, l)

[Vetterli] pp. 640

time shift and modulation ϕk,m(t) = ejkω0tϕ(t − mt0)

k, m ∈ Z

(m, k)

ϕ(a−l(t − malt0))

/70

Transform/analytic dictionary design

38

• delta functions are not localised in frequency

1974 discrete cosine transform Karhunen- Loève transform 1960s discrete Fourier transform 1970-80s

time-domain

ω

t

frequency-domain

ω

t

• Fourier basis functions (complex exponentials) are not localised in time

/70

Transform/analytic dictionary design

39

• consider a set of shifted and modulated versions of a low-pass function

1974 discrete cosine transform Karhunen- Loève transform discrete Fourier transform 1970-80s 1960s

ω

t

ϕk,m(t) = ejkω0tϕ(t − mt0)

k, m ∈ Z

/70

Transform/analytic dictionary design

40 1974 discrete cosine transform Karhunen- Loève transform

• example: consider a box function

and ,

t0 = 1

ω0 = 2π

ϕk,m(t) = ejk2πtϕ(t − m),

1960s discrete Fourier transform 1970-80s

[Vetterli] pp. 646

/70

Transform/analytic dictionary design

41

• we can define the following transform

1974 discrete cosine transform Karhunen- Loève transform 1960s discrete Fourier transform 1970-80s

Xk,m = Z ∞

−∞

x(t)ϕ∗

k,m(t)dt =

Z ∞

−∞

x(t)ϕ(t − mt0)e−jkω0tdt

• more generally

X(ω, τ) = Z ∞

−∞

x(t)ϕ∗

ω,τ(t)dt =

Z ∞

−∞

x(t)ϕ(t − τ)e−jωtdt

• applying time-localised window to the signal before taking Fourier transform:

windowed Fourier transform or short-time Fourier transform (STFT)

• a Gaussian window leads to localisation in frequency as well (Gabor transform)

short-time Fourier transform 1970-80s

• STFT maps a 1-D function into a 2-D function (i.e., it is highly over-complete)

/70

Transform/analytic dictionary design

42 short-time Fourier transform 1974 discrete cosine transform Karhunen- Loève transform 1960s discrete Fourier transform 1970-80s 1970-80s

φk,m(n) = ej 2π

N nkϕ(n − mN)

• discrete STFT provides a non-orthogonal and over-complete dictionary

DCT STFT

[Rubinstein10]

/70

Transform/analytic dictionary design

43

• STFT atoms have fixed time-frequency resolution
• ften times a multi-resolution representation is needed to capture various scales in

natural signals

short-time Fourier transform 1974 discrete cosine transform Karhunen- Loève transform 1960s discrete Fourier transform 1970-80s 1970-80s

/70

Transform/analytic dictionary design

44

• consider a set of shifted and scaled versions of a band-pass function

l, m ∈ Z

ϕl,m(t) = ϕ(a−lt − mt0) = ϕ(t − malt0 al )

short-time Fourier transform 1974 discrete cosine transform Karhunen- Loève transform 1960s discrete Fourier transform 1970-80s 1970-80s

good time resolution good frequency resolution

(m, l)

/70

Transform/analytic dictionary design

45

t0 = 1

a = 2

short-time Fourier transform 1974 discrete cosine transform Karhunen- Loève transform 1960s discrete Fourier transform 1970-80s 1970-80s

• example: consider a square wave

function and ,

[Vetterli] pp. 649

ϕl,m(t) = ϕ(t − 2lm 2l ),

/70

Transform/analytic dictionary design

46

• consider a more general function and define the following transform

ψs,τ(t) = 1 √sψ(t − τ s )

X(s, τ) = Z ∞

−∞

x(t)ψ∗

s,τ(t)dt =

Z ∞

−∞

x(t) 1 √sψ∗(t − τ s )dt

• the prototype function
• has a compact support (small or “-let”)
• is band-pass with zero mean (“wave”):

ψ(t)

• this is called the continuous wavelet transform (CWT)

wavelet transform short-time Fourier transform 1974 discrete cosine transform Karhunen- Loève transform 1980s 1960s discrete Fourier transform 1970-80s 1970-80s

Z ∞

−∞

ψ(t)dt = 0

• CWT maps a 1-D function into a 2-D function (i.e., it is highly over-complete)

/70

Transform/analytic dictionary design

47 wavelet transform short-time Fourier transform discrete Fourier transform 1974 discrete cosine transform Karhunen- Loève transform 1980s 1960s 1970-80s 1970-80s

• examples of prototype function (mother wavelet)
• 4
• 3
• 2
• 1

1 2 3 4

• 1.5
• 1
• 0.5

0.5 1 1.5

Haar wavelet - wavelet

• 4
• 3
• 2
• 1

1 2 3 4

• 1.5
• 1
• 0.5

0.5 1 1.5

• rder 1 Gaussian wavelet
• 4
• 3
• 2
• 1

1 2 3 4

• 1.5
• 1
• 0.5

0.5 1 1.5

Mexican hat wavelet

• 4
• 3
• 2
• 1

1 2 3 4

• 1.5
• 1
• 0.5

0.5 1 1.5

Morlet wavelet

Haar [1909] Derivative of Gaussian Morlet [1984] Marr (Mexican hat)

/70

Transform/analytic dictionary design

48 wavelet transform short-time Fourier transform discrete Fourier transform 1974 discrete cosine transform Karhunen- Loève transform 1980s 1960s 1970-80s 1970-80s

• CWT can be discretised to define a discrete-time wavelet transform (DTWT), but

both CWT and DTWT are non-orthogonal and over-complete

• however, unlike STFT, we can design an orthogonal wavelet transform through a

multi-resolution analysis

ψl,m(t) = 1 √ 2l ψ(t − 2lm 2l )

• functions at given scale form an
• rthogonal basis of a space at scale
• all functions across different scales

are also orthogonal l

l

design principle

/70

Transform/analytic dictionary design

49 wavelet transform short-time Fourier transform discrete Fourier transform 1974 discrete cosine transform Karhunen- Loève transform 1980s 1960s 1970-80s 1970-80s

• 2-D DWT

50 100 150 200 250 50 100 150 200 250

Approximation Coefficients

20 40 60 80 100 120 20 40 60 80 100 120

Horizontal Detail Coefficients

20 40 60 80 100 120 20 40 60 80 100 120

Vertical Detail Coefficients

20 40 60 80 100 120 20 40 60 80 100 120

Diagonal Detail Coefficients

20 40 60 80 100 120 20 40 60 80 100 120

• DWT is behind the JEPG 2000 image compression standard
• the discrete version is the discrete wavelet transform (DWT) which provides an
• rthogonal dictionary

/70

Transform/analytic dictionary design

50 wavelet transform short-time Fourier transform discrete Fourier transform 1974 discrete cosine transform Karhunen- Loève transform 1980s 1960s 1970-80s 1970-80s

DCT KLT STFT DWT

• comparison of the dictionaries we looked at so far

/70

Transform/analytic dictionary design

51 wavelet transform short-time Fourier transform discrete Fourier transform 1974 discrete cosine transform Karhunen- Loève transform 1980s 1960s 1970-80s 1970-80s

• summary

modelling data by a simpler class of mathematical functions smooth functions (Fourier, DCT) piecewise-smooth functions (wavelets) desired properties localisation (STFT, wavelets) multi-resolution (wavelets) adaptivity (KLT, wavelet packets) fast implementation is usually available limited expressiveness

/70

Outline

• A historical overview of dictionary design techniques
• signal representation via stochastic models

52

• transforms & analytic dictionaries
• trained dictionaries (dictionary learning)
• Discussion
• applications
• connection with deep learning

/70

A paradigm shift in dictionary design

53

= X

x

c

Φ

= X

• rthogonal atoms

complete dictionary dense coefficients non-orthogonal atoms

• ver-complete dictionary

sparse coefficients all signals use all atoms different signals use different atoms mathematical modelling adaptation to data realisations

wavelet transform short-time Fourier transform discrete Fourier transform 1974 discrete cosine transform Karhunen- Loève transform 1980s 1960s 1970-80s 1970-80s

/70

Illustrative example

• Modelling assumption: Each data point is a combination of only a few

(sparse) fundamental elements, i.e., dictionary atoms

54

Agarwal et al. 2015

Coefficients Dictionary Signals

/70

Sparse representations

55

= X

x

c

Φ

• express a signal as a linear combination
• f a small number of atoms

min

c ||c||0 subject to x = Φc + η and ||⌘||2 2 ≤ ✏

• the problem is NP-hard :(
• two types of approximation algorithms

matching pursuit (MP) [Mallat93] least absolute shrinkage and selection

• perator (Lasso) [Tibshirani96]

wavelet transform short-time Fourier transform discrete Fourier transform 1974 discrete cosine transform Karhunen- Loève transform 1980s 1960s 1970-80s 1970-80s

/70

Sparse representations

56

= X

x

c

Φ

• MP

choose a subset of atoms from

• ne atom at a time to maximally

(greedily) reduce approximation error

Φ

min

c ||x − Φc||2 2 + λ||c||1

• Lasso

solve a convex relaxation by replacing the 0-norm with 1-norm on c

wavelet transform short-time Fourier transform discrete Fourier transform 1974 discrete cosine transform Karhunen- Loève transform 1980s 1960s 1970-80s 1970-80s

/70

Sparse representations

57

= X

x

c

Φ

• given a dictionary, MP and Lasso can

find a sparse approximation of the data

• the sparsity depends on not only data

but also the dictionary

• finding optimised dictionaries is the goal
• f dictionary learning

wavelet transform short-time Fourier transform discrete Fourier transform 1974 discrete cosine transform Karhunen- Loève transform 1980s 1960s 1970-80s 1970-80s

/70

Dictionary learning: Probabilistic approach

58

• probabilistic approach: maximum likelihood (ML)

Φ∗ = arg max

Φ [log P(x|Φ)]

= arg max

Φ [log

Z

c

P(x|c, Φ)P(c)dc]

= arg min

Φ,c ||x − Φc||2 2 + λ||c||1

Φ∗ = arg min

Φ,c −log[P(x|c, Φ)P(c)]

Olshausen and Field 1996

wavelet transform short-time Fourier transform discrete Fourier transform 1974 discrete cosine transform Karhunen- Loève transform 1980s 1960s 1970-80s 1970-80s 1996 sparse coding

• assumption 1: Laplace distribution of coefficients
• assumption 2: Gaussian distribution of error

ci

η

/70

Dictionary learning: Probabilistic approach

59

• the problem is solved by iterating between two steps

min

Φ,c ||x − Φc||2 2 + λ||c||1

sparse approximation: given , solve for c via Lasso

Φ

• works at patch level for efficiency
• does not necessarily find global optimum
• trained atoms are remarkably similar to mammalian simple-cell receptive fields

Olshausen and Field 1996

wavelet transform short-time Fourier transform discrete Fourier transform 1974 discrete cosine transform Karhunen- Loève transform 1980s 1960s 1970-80s 1970-80s 1996 sparse coding

dictionary update: given c, update via gradient descent

Φ

/70

Dictionary learned with sparse coding

60

Olshausen et al. 2009

/70

Dictionary learning

61

• summary

learning representations directly from data realisations desired properties

• ver-completeness

sparse representations efficiency in training may be combined with analytical dictionary design trained dictionary with structures (e.g., parametric dictionary learning)

wavelet transform short-time Fourier transform discrete Fourier transform 1974 discrete cosine transform Karhunen- Loève transform 1980s 1960s 1970-80s 1970-80s 1996 sparse coding 2006 K-SVD 2010 double sparsity supervised dictionary learning 2000s-2010s

/70

Outline

• A historical overview of dictionary design techniques
• signal representation via stochastic models

62

• transforms & analytic dictionaries
• trained dictionaries (dictionary learning)
• Discussion
• applications
• connection with deep learning

/70

Signal denoising

63

200 400 600 800 1000 1200

• 10
• 5

5 10

Original Signal Denoised Signal

denoising using the order 4 symlets wavelets

/70

Image compression

64

JPEG 2000 (comp. ratio: 10) JPEG 2000 (comp. ratio: 100)

• riginal

compression using the Cohen-Daubechies-Feauveau wavelets

/70

Image reconstruction

65

Aharon et al. 2006

/70

Image restoration

66

Mairal et al. 2008

/70

Connection with deep learning

• Dictionary learning vs. Deep learning

67

= X

x

c

Φ

• both extract feature representations from data realisations
• both apply sparsifying operations such as shrinkage or rectified linear units (ReLU)
• the former leads to representations that are not necessarily hierarchical/no

convolution operators

• the former is normally for reconstruction/approximation (similar to

autoencoders) while the latter is mainly for classification

https://en.wikipedia.org/wiki/File:Typical_cnn.png

/70

Dictionary-inspired deep architectures

• Scattering transform [Bruna and Mallat 2013]

68

/70

Dictionary-inspired deep architectures

• Scattering transform [Bruna and Mallat 2013]
• Multi-layer convolutional sparse coding [Papyan et al. 2018]

69

/70

References

70