[PPT] - Machine Learning for Signal Processing Representing Signals: Images PowerPoint Presentation

SLIDE 1

Machine Learning for Signal Processing

Representing Signals: Images and Sounds

Class 4. 10 Sep 2013 Instructor: Bhiksha Raj

10 Sep 2013 11-755/18-797 1

SLIDE 2

Administrivia

Basics of probability: Will not be covered

– Several very nice lectures on the net

Things to know:

– Basic probability, Bayes rule – Probability distributions over discrete variables – Probability density and Cumulative density over continuous variables

Particularly Gaussian densities

– Moments of a distribution – What is independence – Nice to know

What is maximum likelihood estimation
MAP estimation

10 Sep 2013 11-755/18-797 2

SLIDE 3

Representing Data

The first and most important step in

processing signals is representing them appropriately

10 Sep 2013 11-755/18-797 3

SLIDE 4

Representing an Elephant

It was six men of Indostan,

To learning much inclined, Who went to see the elephant, (Though all of them were blind), That each by observation Might satisfy his mind.

The first approached the elephant,

And happening to fall Against his broad and sturdy side, At once began to bawl: "God bless me! But the elephant Is very like a wall!“

The second, feeling of the tusk,

Cried: "Ho! What have we here, So very round and smooth and sharp? To me 'tis very clear, This wonder of an elephant Is very like a spear!“

The third approached the animal,

And happening to take The squirming trunk within his hands, Thus boldly up and spake: "I see," quoth he, "the elephant Is very like a snake!“

The fourth reached out an eager hand,

And felt about the knee. "What most this wondrous beast is like Is might plain," quoth he; "Tis clear enough the elephant Is very like a tree."

The fifth, who chanced to touch the ear,

Said: "E'en the blindest man Can tell what this resembles most: Deny the fact who can, This marvel of an elephant Is very like a fan.“

The sixth no sooner had begun

About the beast to grope, Than seizing on the swinging tail That fell within his scope, "I see," quoth he, "the elephant Is very like a rope.“

And so these men of Indostan

Disputed loud and long, Each in his own opinion Exceeding stiff and strong. Though each was partly right, All were in the wrong.

10 Sep 2013 11-755/18-797 4

SLIDE 5

Representation

Describe these images

– Such that a listener can visualize what you are describing

More images

10 Sep 2013 11-755/18-797 5

SLIDE 6

Still more images

10 Sep 2013 11-755/18-797 6

How do you describe them?

SLIDE 7

Representation

Pixel-based descriptions are uninformative
Feature-based descriptions are infeasible in

the general case

10 Sep 2013 11-755/18-797 7

SLIDE 8

Sounds

Sounds are just sequences of numbers
When plotted, they just look like blobs

– Which leads to “natural sounds are blobs”

Or more precisely, “sounds are sequences of numbers that, when plotted,

look like blobs”

– Which wont get us anywhere

10 Sep 2013 11-755/18-797 8

SLIDE 9

Representation

Representation is description
But in compact form
Must describe the salient characteristics of the data

– E.g. a pixel-wise description of the two images here will be completely different

Must allow identification, comparison, storage,

reconstruction..

10 Sep 2013 11-755/18-797 9

A A

SLIDE 10

Representing images

The most common element in the image: background

– Or rather large regions of relatively featureless shading – Uniform sequences of numbers

10 Sep 2013 11-755/18-797 10

SLIDE 11

Representing images using a “plain” image

Most of the figure is a more-or-less uniform shade

– Dumb approximation – a image is a block of uniform shade

Will be mostly right!
How to compute the “best” description? Projection

– Represent the images as vectors and compute the projection of the image on the “basis”

10 Sep 2013 11-755/18-797 11

Image =

              N pixel pixel pixel . 2 1               1 . 1 1

B =

age B B B B BW PROJECTION age B pinv W age BW

T T

Im . ) ( Im ) ( Im

1 

   

SLIDE 12

Adding more bases

Lets improve the approximation
Images have some fast varying regions

– Dramatic changes – Add a second picture that has very fast changes

A checkerboard where every other pixel is black and the rest are white

10 Sep 2013 11-755/18-797 12

                   1 1 1 1 1 1 1 1 1 1 B

] [ Im

2 1 2 1 2 2 1 1

B B B w w W B w B w age          

B1 B2 B2 B1

Image . ) ( Image ) ( Image

1 T T

B B B B BW PROJECTION B pinv W BW



   

SLIDE 13

Adding still more bases

Regions that change with different speeds

10 Sep 2013 11-755/18-797 13

age B pinv W age BW Im ) ( Im  

] [ . . ... Im

3 2 1 3 2 1 3 3 2 2 1 1

B B B B w w w W B w B w B w age                    

B1 B2 B3 B4 B5 B6 Getting closer at 625 bases!

SLIDE 14

Representation using checkerboards

A “standard” representation

– Checker boards are the same regardless of the picture you’re trying to describe

As opposed to using “nose shape” to describe faces and “leaf colour” to

describe trees.

Any image can be specified as (for example)

0.8checkerboard(0) + 0.2checkerboard(1) + 0.3*checkerboard(2) ..

The definition is sufficient to reconstruct the image to some

degree

– Not perfectly though

10 Sep 2013 11-755/18-797 14

SLIDE 15

What about sounds?

Square wave equivalents of checker boards

10 Sep 2013 11-755/18-797 15

SLIDE 16

Projecting sounds

10 Sep 2013 11-755/18-797 16

Signal B pinv B BW PROJECTION Signal B pinv W Signal BW )). ( . ( ) (    

] [

3 2 1 3 2 1 3 3 2 2 1 1

B B B B w w w W B w B w B w Signal               

         

B1 B2 B3

                       

3 2 1

w w w

=

SLIDE 17

General Philosophy of Representation

Identify a set of standard structures

– E.g. checkerboards – We will call these “bases”

Express the data as a weighted combination of these bases

– X = w1 B1 + w2 B2 + w3 B3 + …

Chose weights w1, w2, w3.. for the best representation of X

– I.e. the error between X and Si wi Bi is minimized – The error is generally chosen to be ||X – Si wi Bi||2

The weights w1, w2, w3.. fully specify the data

– Since the bases are known beforehand – Knowing the weights is sufficient to reconstruct the data

10 Sep 2013 11-755/18-797 17

SLIDE 18

Bases requirements

Non-redundancy

– Each basis must represent information not already represented by other bases – I.e. bases must be orthogonal

<Bi, Bj> = 0 for i != j

– Mathematical benefit: can compute wi = <Bi,X>

Compactness

– Must be able to represent most of X with fewest bases – Completeness: For D-dimensional data, need no more than D bases

10 Sep 2013 11-755/18-797 18

SLIDE 19

Bases based representation

Place all bases in basis matrix B
For orthogonal bases

10 Sep 2013 11-755/18-797 19

X B Pinv W X BW ) (  

2

|| || ,

i i i

B X B w                        

3 2 1 3 2 1

X X X w w w          

33 32 31 23 22 21 13 12 11

b b b b b b b b b

SLIDE 20

Bases based representation

Challenge: Choice of appropriate bases

10 Sep 2013 11-755/18-797 20

SLIDE 21

Why checkerboards are great bases

We cannot explain one

checkerboard in terms of another

– The two are orthogonal to one another!

This means we can determine

the contributions of individual bases separately

– Joint decomposition with multiple bases gives the same result as separate decomposition with each – This never holds true if one basis can explain another

10 Sep 2013 11-755/18-797 21

                   1 1 1 1 1 1 1 1 1 1 B

B1 B2

] [ Im

2 1 2 1 2 2 1 1

B B B w w W B w B w age          

2

|| || Im ,

i i i

B age B w   

SLIDE 22

Checker boards are not good bases

Sharp edges

– Can never be used to explain rounded curves

10 Sep 2013 11-755/18-797 22

SLIDE 23

Sinusoids ARE good bases

They are orthogonal
They can represent rounded shapes nicely

– Unfortunately, they cannot represent sharp corners

10 Sep 2013 11-755/18-797 23

SLIDE 24

What are the frequencies of the sinusoids

Follow the same format as the

checkerboard:

– DC – The entire length of the signal is

ne period

– The entire length of the signal is two periods.

And so on..
The k-th sinusoid:

– F(n) = sin(2pkn/N)

N is the length of the signal
k is the number of periods in N

samples

10 Sep 2013 11-755/18-797 24

SLIDE 25

How many frequencies in all?

A max of L/2 periods are possible
If we try to go to (L/2 + X) periods, it ends up being identical to having (L/2

– X) periods

– With sign inversion

Example for L = 20

– Red curve = sine with 9 cycles (in a 20 point sequence)

Y(n) = sin(2p9n/20)

– Green curve = sine with 11 cycles in 20 points

Y(n) = -sin(2p11n/20)

– The blue lines show the actual samples obtained

These are the only numbers stored on the computer
This set is the same for both sinusoids

10 Sep 2013 11-755/18-797 25

SLIDE 26

How to compose the signal from sinusoids

The sines form the vectors of the projection matrix

– Pinv() will do the trick as usual

10 Sep 2013 11-755/18-797 26

Signal B B B B BW PROJECTION Signal B pinv W Signal BW

T

. ) ( ) (

1 

   

] [

3 2 1 3 2 1 3 3 2 2 1 1

B B B B w w w W B w B w B w Signal               

         

B1 B2 B3

                       

3 2 1

w w w

=

SLIDE 27

How to compose the signal from sinusoids

The sines form the vectors of the projection matrix

– Pinv() will do the trick as usual

10 Sep 2013 11-755/18-797 27

Signal B pinv W Signal BW ) (  

                               ] 1 [ . ] 1 [ ] [ ] [

3 2 1 3 2 1 3 3 2 2 1 1

L s s s Signal B B B B w w w W B w B w B w Signal

                                                     ] 1 [ . . ] 1 [ ] [ . . /L) ) 1 ).( 2 / ( . sin(2 . . /L) ) 1 .( 1 . sin(2 /L) ) 1 .( . sin(2 . . . . . . . . . . /L) 1 ). 2 / ( . sin(2 . . /L) 1 . 1 . sin(2 /L) 1 . . sin(2 /L) ). 2 / ( . sin(2 . . /L) . 1 . sin(2 /L) . . sin(2

2 / 2 1

L s s s w w w L L L L L L

L

p p p p p p p p p

L/2 columns only

SLIDE 28

Interpretation..

Each sinusoid’s amplitude is adjusted until it gives us the

least squared error

– The amplitude is the weight of the sinusoid

This can be done independently for each sinusoid

10 Sep 2013 11-755/18-797 28

SLIDE 29

Interpretation..

Each sinusoid’s amplitude is adjusted until it gives us the

least squared error

– The amplitude is the weight of the sinusoid

This can be done independently for each sinusoid

10 Sep 2013 11-755/18-797 29

SLIDE 30

Interpretation..

Each sinusoid’s amplitude is adjusted until it gives us the

least squared error

– The amplitude is the weight of the sinusoid

This can be done independently for each sinusoid

10 Sep 2013 11-755/18-797 30

SLIDE 31

Interpretation..

Each sinusoid’s amplitude is adjusted until it gives us the

least squared error

– The amplitude is the weight of the sinusoid

This can be done independently for each sinusoid

10 Sep 2013 11-755/18-797 31

SLIDE 32

Sines by themselves are not enough

Every sine starts at zero

– Can never represent a signal that is non-zero in the first sample!

Every cosine starts at 1

– If the first sample is zero, the signal cannot be represented!

10 Sep 2013 11-755/18-797 32

SLIDE 33

The need for phase

Allow the sinusoids to move!
How much do the sines shift?

10 Sep 2013 11-755/18-797 33

.... ) / 2 sin( ) / 2 sin(

2 2 1 1

      p  p N kn w N kn w signal

Sines are shifted: do not start with value = 0

SLIDE 34

Determining phase

Least squares fitting: move the sinusoid left / right, and

at each shift, try all amplitudes

– Find the combination of amplitude and phase that results in the lowest squared error

We can still do this separately for each sinusoid

– The sinusoids are still orthogonal to one another

10 Sep 2013 11-755/18-797 34

SLIDE 35

Determining phase

Least squares fitting: move the sinusoid left / right, and

at each shift, try all amplitudes

– Find the combination of amplitude and phase that results in the lowest squared error

We can still do this separately for each sinusoid

– The sinusoids are still orthogonal to one another

10 Sep 2013 11-755/18-797 35

SLIDE 36

Determining phase

Least squares fitting: move the sinusoid left / right, and

at each shift, try all amplitudes

– Find the combination of amplitude and phase that results in the lowest squared error

We can still do this separately for each sinusoid

– The sinusoids are still orthogonal to one another

10 Sep 2013 11-755/18-797 36

SLIDE 37

Determining phase

Least squares fitting: move the sinusoid left / right, and

at each shift, try all amplitudes

– Find the combination of amplitude and phase that results in the lowest squared error

We can still do this separately for each sinusoid

– The sinusoids are still orthogonal to one another

10 Sep 2013 11-755/18-797 37

SLIDE 38

The problem with phase

This can no longer be expressed as a simple linear algebraic

equation

– The “basis matrix” depends on the unknown phase

I.e. there’s a component of the basis itself that must be estimated!
Linear algebraic notation can only be used if the bases are

fully known

– We can only (pseudo) invert a known matrix

10 Sep 2013 11-755/18-797 38

                                                              ] 1 [ . . ] 1 [ ] [ . . ) /L ) 1 ).( 2 / ( . sin(2 . . ) /L ) 1 .( 1 . sin(2 ) /L ) 1 .( . sin(2 . . . . . . . . . . ) /L 1 ). 2 / ( . sin(2 . . ) /L 1 . 1 . sin(2 ) /L 1 . . sin(2 ) /L ). 2 / ( . sin(2 . . ) /L . 1 . sin(2 ) /L . . sin(2

2 / 2 1 L/2 1 L/2 1 L/2 1

L s s s w w w L L L L L L

L

 p  p  p  p  p  p  p  p  p

L/2 columns only

SLIDE 39

The cosine is the real part of a complex exponential

– The sine is the imaginary part

A phase term for the sinusoid becomes a multiplicative

term for the complex exponential!!

10 Sep 2013 11-755/18-797 39

) * sin( ] [ n freq n b 

1 ) * sin( ) * cos( ) * * exp( ] [      j n freq j n freq n freq j n b

) * sin( ) * cos( ) exp( ) * * exp( ) * * exp(           n freq j n freq n freq j n freq j

Complex Exponential to the rescue

SLIDE 40

10 Sep 2013 11-755/18-797 40

A x

Explaining with Complex Exponentials

  

B x C x

SLIDE 41

Like sinusoids, a complex exponential of one frequency

can never explain one of another

– They are orthogonal

They represent smooth transitions
Bonus: They are complex

– Can even model complex data!

They can also model real data

– exp(j x ) + exp(-j x) is real

cos(x) + j sin(x) + cos(x) – j sin(x) = 2cos(x)
More importantly

– is real

The complex exponentials with frequencies equally spaced from

L/2 are complex conjugates

10 Sep 2013 11-755/18-797 41

               L n x L j L n x L j ) 2 / ( 2 exp ) 2 / ( 2 exp p p

Complex exponentials are well behaved

SLIDE 42

is real

– The complex exponentials with frequencies equally spaced from L/2 are complex conjugates

“Frequency = k”  k periods in L samples

– Is also real – If the two exponentials are multiplied by numbers that are conjugates of one another the result is real

10 Sep 2013 11-755/18-797 42

               L n x L j L n x L j ) 2 / ( 2 exp ) 2 / ( 2 exp p p                L n x L j a conjugate L n x L j a ) 2 / ( 2 exp ) ( ) 2 / ( 2 exp p p

Complex exponentials are well behaved

SLIDE 43

Complex Exponential bases

Explain the data using L complex exponential bases
The weights given to the (L/2 + k)th basis and the (L/2 – k)th basis should

be complex conjugates, to make the result real

– Because we are dealing with real data

Fortunately, a least squares fit will give us identical weights to both bases

automatically; there is no need to impose the constraint externally

10 Sep 2013 11-755/18-797 43

b0 b1 bL/2

                     

   1 1 2 / 2 / 1 2 /

. .

L L L L

w w w w w

             

=

Complex conjugates

) (

2 / 2 / k L k L

w conjugate w

  

SLIDE 44

Complex Exponential Bases: Algebraic Formulation

Note that SL/2+x = conjugate(SL/2-x) for real s

10 Sep 2013 11-755/18-797 44

                                                       



] 1 [ . . ] 1 [ ] [ . . /L) ) 1 ).( 1 ( . exp(j2 . /L) ) 1 ).( 2 / ( . exp(j2 . /L) ) 1 .( . exp(j2 . . . . . . . . . . /L) 1 ). 1 ( . exp(j2 . . /L) 1 ). 2 / ( . exp(j2 . /L) 1 . . exp(j2 /L) ). 1 ( . exp(j2 . . /L) ). 2 / ( . exp(j2 . /L) . . exp(j2

1 2 /

L s s s S S S L L L L L L L L L

L L

p p p p p p p p p

SLIDE 45

Shorthand Notation

Note that SL/2+x = conjugate(SL/2-x)

10 Sep 2013 11-755/18-797 45

 

                                                      

      

] 1 [ . . ] 1 [ ] [ . . . . . . . . . . . . . . . . . . . . ) / 2 sin( ) / 2 cos( 1 ) / 2 exp( 1

1 2 / 1 , 1 1 , 2 / 1 , 1 , 1 1 , 2 / 1 , , 1 , 2 / , ,

L s s s S S S W W W W W W W W W L kn j L kn L L kn j L W

L L L L L L L L L L L L L L L L L L L L n k L

p p p

SLIDE 46

A quick detour

Real Orthonormal matrix:

– XXT = X XT = I

But only if all entries are real

– The inverse of X is its own transpose

Definition: Hermitian

– XH = Complex conjugate of XT

Conjugate of a number a + ib = a – ib
Conjugate of exp(ix) = exp(-ix)
Complex Orthonormal matrix

– XXH = XH X = I – The inverse of a complex orthonormal matrix is its own Hermitian

10 Sep 2013 11-755/18-797 46

SLIDE 47

W-1 = WH

10 Sep 2013 11-755/18-797 47

                

      1 , 1 1 , 2 / 1 , 1 , 1 1 , 2 / 1 , , 1 , 2 / ,

. . . . . . . . . . . . . . . . . .

L L L L L L L L L L L L L L L L L L

W W W W W W W W W W

) / 2 exp( 1

,

L kn j L W

n k L

p  ) / 2 exp( 1

,

L kn j L W

n k L

p  



                

              ) 1 ( ), 1 ( 2 / ), 1 ( ), 1 ( 1 , 1 2 / , 1 , , 1 1 , 2 / , ,

. . . . . . . . . . . . . . . . . .

L L L L L L L L L L L L L L L L L L H

W W W W W W W W W W

 The complex exponential basis is orthogonal

 Its inverse is its own Hermitian  W-1 = WH

SLIDE 48

Doing it in matrix form

– Because W-1 = WH

10 Sep 2013 11-755/18-797 48

                                                 

              

] 1 [ . . ] 1 [ ] [ . . . . . . . . . . . . . . . . . . . .

) 1 ( ), 1 ( 2 / ), 1 ( ), 1 ( 1 , 1 2 / , 1 , , 1 1 , 2 / , , 1 2 /

L s s s W W W W W W W W W S S S

L L L L L L L L L L L L L L L L L L L L

                                                 

      

] 1 [ . . ] 1 [ ] [ . . . . . . . . . . . . . . . . . . . .

1 2 / 1 , 1 1 , 2 / 1 , 1 , 1 1 , 2 / 1 , , 1 , 2 / ,

L s s s S S S W W W W W W W W W

L L L L L L L L L L L L L L L L L L L L

SLIDE 49

The Discrete Fourier Transform

The matrix to the right is called the “Fourier

Matrix”

The weights (S0, S1. . Etc.) are called the Fourier

transform

10 Sep 2013 11-755/18-797 49

                                                   

              

] 1 [ . . ] 1 [ ] [ . . . . . . . . . . . . . . . . . . . .

) 1 ( ), 1 ( 2 / ), 1 ( ), 1 ( 1 , 1 2 / , 1 , , 1 1 , 2 / , , 1 2 /

L s s s W W W W W W W W W S S S

L L L L L L L L L L L L L L L L L L L L

SLIDE 50

The matrix to the left is the inverse Fourier matrix
Multiplying the Fourier transform by this matrix gives us

the signal right back from its Fourier transform

10 Sep 2013 11-755/18-797 50

                                                   

      

] 1 [ . . ] 1 [ ] [ . . . . . . . . . . . . . . . . . . . .

1 2 / 1 , 1 1 , 2 / 1 , 1 , 1 1 , 2 / 1 , , 1 , 2 / ,

L s s s S S S W W W W W W W W W

L L L L L L L L L L L L L L L L L L L L

The Inverse Discrete Fourier Transform

SLIDE 51

The Fourier Matrix

Left panel: The real part of the Fourier matrix

– For a 32-point signal

Right panel: The imaginary part of the Fourier matrix

10 Sep 2013 11-755/18-797 51

SLIDE 52

The FAST Fourier Transform

The outcome of the transformation with the Fourier matrix is the

DISCRETE FOURIER TRANSFORM (DFT)

The FAST Fourier transform is an algorithm that takes advantage of

the symmetry of the matrix to perform the matrix multiplication really fast

The FFT computes the DFT

– Is much faster if the length of the signal can be expressed as 2N

10 Sep 2013 11-755/18-797 52

SLIDE 53

Images

The complex exponential is two dimensional

– Has a separate X frequency and Y frequency

Would be true even for checker boards!

– The 2-D complex exponential must be unravelled to form one component of the Fourier matrix

For a KxL image, we’d have K*L bases in the matrix

10 Sep 2013 11-755/18-797 53

SLIDE 54

Typical Image Bases

Only real components of bases shown

10 Sep 2013 11-755/18-797 54

SLIDE 55

DFT: Properties

The DFT coefficients are complex

– Have both a magnitude and a phase

Simple linear algebra tells us that

– DFT(A + B) = DFT(A) + DFT(B) – The DFT of the sum of two signals is the DFT of their sum

A horribly common approximation in sound processing

– Magnitude(DFT(A+B)) = Magnitude(DFT(A)) + Magnitude(DFT(B)) – Utterly wrong – Absurdly useful

10 Sep 2013 11-755/18-797 55

) exp( | |

k k k

S j S S   

SLIDE 56

Symmetric signals

If a signal is (conjugate) symmetric around L/2, the Fourier coefficients are real!

– A(L/2-k) * exp(-j *f*(L/2-k)) + A(L/2+k) * exp(-j*f*(L/2+k)) is always real if

A(L/2-k) = conjugate(A(L/2+k))

– We can pair up samples around the center all the way; the final summation term is always real

Overall symmetry properties

– If the signal is real, the FT is (conjugate) symmetric – If the signal is (conjugate) symmetric, the FT is real – If the signal is real and symmetric, the FT is real and symmetric

10 Sep 2013 11-755/18-797 56

* * * * * * * * * * * * * * * * * * * * * * * * *

Contributions from points equidistant from L/2 combine to cancel out imaginary terms

SLIDE 57

The Discrete Cosine Transform

Compose a symmetric signal or image

– Images would be symmetric in two dimensions

Compute the Fourier transform

– Since the FT is symmetric, sufficient to store only half the coefficients (quarter for an image)

Or as many coefficients as were originally in the signal / image

10 Sep 2013 11-755/18-797 57

SLIDE 58

DCT

Not necessary to compute a 2xL sized FFT

– Enough to compute an L-sized cosine transform – Taking advantage of the symmetry of the problem

This is the Discrete Cosine Transform

10 Sep 2013 11-755/18-797 58

                                                          



] 1 [ . . ] 1 [ ] [ . . /2L) ) 1 ).( 5 . ( . cos(2 . . /2L) ) 1 .( 0.5) 1 .( cos(2 /2L) ) 1 ).( 5 . ( . cos(2 . . . . . . . . . . /2L) 1 ). 5 . ( . cos(2 . . /2L) 1 . 0.5) 1 .( cos(2 /2L) 1 ). 5 . ( . cos(2 /2L) ). 5 . ( . cos(2 . . /2L) . 0.5) 1 .( cos(2 /2L) ). 5 . ( cos(2

1 1

L s s s w w w L L L L L L

L

p p p p p p p p p

L columns

SLIDE 59

Representing images

Most common coding is the DCT
JPEG: Each 8x8 element of the picture is converted using a

DCT

The DCT coefficients are quantized and stored

– Degree of quantization = degree of compression

Also used to represent textures etc for pattern recognition

and other forms of analysis

10 Sep 2013 11-755/18-797 59

DCT Multiply by DCT matrix

SLIDE 60

Some tricks to computing Fourier transforms

Direct computation of the Fourier transform

can result in poor representations

Boundary effects can cause error

– Solution : Windowing

The size of the signal can introduce

inefficiency

– Solution: Zero padding

10 Sep 2013 11-755/18-797 60

SLIDE 61

What does the DFT represent

The IDFT can be written formulaically as above
There is no restriction on computing the formula for n < 0
r n > L-1

– Its just a formula – But computing these terms behind 0 or beyond L-1 tells us what the signal composed by the DFT looks like outside our narrow window

10 Sep 2013 11-755/18-797 61

                                                       



] 1 [ . . ] 1 [ ] [ . . /L) ) 1 ).( 1 ( . exp(j2 . /L) ) 1 ).( 2 / ( . exp(j2 . /L) ) 1 .( . exp(j2 . . . . . . . . . . /L) 1 ). 1 ( . exp(j2 . . /L) 1 ). 2 / ( . exp(j2 . /L) 1 . . exp(j2 /L) ). 1 ( . exp(j2 . . /L) ). 2 / ( . exp(j2 . /L) . . exp(j2

1 2 /

L s s s S S S L L L L L L L L L

L L

p p p p p p p p p



 



1

) / 2 exp( ] [

L k k

L kn j S n s p

SLIDE 62

What does the DFT represent

If you extend the DFT-based representation

beyond 0 (on the left) or L (on the right) it repeats the signal!

So what does the DFT really mean

10 Sep 2013 11-755/18-797 62

s[n] DFT

[S0 S1 .. S31]



 



1

) / 2 exp( ] [

L k k

L kn j S n s p

31 63

32

SLIDE 63

What does the DFT represent

The DFT represents the properties of the

infinitely long repeating signal that you can generate with it

– Of which the observed signal is ONE period

This gives rise to some odd effects

10 Sep 2013 11-755/18-797 63

SLIDE 64

The discrete Fourier transform of the above signal actually computes the

properties of the periodic signal shown below

– Which extends from –infinity to +infinity – The period of this signal is 32 samples in this example

10 Sep 2013 11-755/18-797 64

The discrete Fourier transform

SLIDE 65

10 Sep 2013 11-755/18-797 65



The DFT of one period of the sinusoid shown in the figure computes the spectrum of the entire sinusoid from –infinity to +infinity



The DFT of a real sinusoid has only one non zero frequency



The second peak in the figure also represents the same frequency as an effect of aliasing

Windowing

SLIDE 66

10 Sep 2013 11-755/18-797 66



The DFT of one period of the sinusoid shown in the figure computes the spectrum of the entire sinusoid from –infinity to +infinity



The DFT of a real sinusoid has only one non zero frequency



The second peak in the figure also represents the same frequency as an effect of aliasing

Windowing

SLIDE 67

10 Sep 2013 11-755/18-797 67



The DFT of one period of the sinusoid shown in the figure computes the spectrum of the entire sinusoid from –infinity to +infinity



The DFT of a real sinusoid has only one non zero frequency



The second peak in the figure is the “reflection” around L/2 (for real signals) Magnitude spectrum

Windowing

SLIDE 68

10 Sep 2013 11-755/18-797 68



The DFT of any sequence computes the spectrum for an infinite repetition of that sequence



The DFT of a partial segment of a sinusoid computes the spectrum of an infinite repetition of that segment, and not of the entire sinusoid



This will not give us the DFT of the sinusoid itself!

Windowing

SLIDE 69

10 Sep 2013 11-755/18-797 69



The DFT of any sequence computes the spectrum for an infinite repetition of that sequence



The DFT of a partial segment of a sinusoid computes the spectrum of an infinite repetition of that segment, and not of the entire sinusoid



This will not give us the DFT of the sinusoid itself!

Windowing

SLIDE 70

10 Sep 2013 11-755/18-797 70

Magnitude spectrum



The DFT of any sequence computes the spectrum for an infinite repetition of that sequence



The DFT of a partial segment of a sinusoid computes the spectrum of an infinite repetition of that segment, and not of the entire sinusoid



This will not give us the DFT of the sinusoid itself!

Windowing

SLIDE 71

10 Sep 2013 11-755/18-797 71

Magnitude spectrum of segment Magnitude spectrum of complete sine wave

Windowing

SLIDE 72

10 Sep 2013 11-755/18-797 72

 The difference occurs due to two reasons:  The transform cannot know what the signal actually looks like

utside the observed window

 The implicit repetition of the observed signal introduces large

discontinuities at the points of repetition

 This distorts even our measurement of what happens at the

boundaries of what has been reliably observed

Windowing

SLIDE 73

10 Sep 2013 11-755/18-797 73

 The difference occurs due to two reasons:  The transform cannot know what the signal actually looks like

utside the observed window

 The implicit repetition of the observed signal introduces large

discontinuities at the points of repetition

 These are not part of the underlying signal 

We only want to characterize the underlying signal



The discontinuity is an irrelevant detail

Windowing

SLIDE 74

10 Sep 2013 11-755/18-797 74

 While we can never know what the signal looks like outside the

window, we can try to minimize the discontinuities at the boundaries

 We do this by multiplying the signal with a window function

 We call this procedure windowing  We refer to the resulting signal as a “windowed” signal

 Windowing attempts to do the following:

 Keep the windowed signal similar to the original in the central

regions

 Reduce or eliminate the discontinuities in the implicit periodic signal

Windowing

SLIDE 75

10 Sep 2013 11-755/18-797 75

 While we can never know what the signal looks like outside the

window, we can try to minimize the discontinuities at the boundaries

 We do this by multiplying the signal with a window function

 We call this procedure windowing  We refer to the resulting signal as a “windowed” signal

 Windowing attempts to do the following:

 Keep the windowed signal similar to the original in the central

regions

 Reduce or eliminate the discontinuities in the implicit periodic signal

Windowing

SLIDE 76

10 Sep 2013 11-755/18-797 76

 While we can never know what the signal looks like outside the

window, we can try to minimize the discontinuities at the boundaries

 We do this by multiplying the signal with a window function

 We call this procedure windowing  We refer to the resulting signal as a “windowed” signal

 Windowing attempts to do the following:

 Keep the windowed signal similar to the original in the central

regions

 Reduce or eliminate the discontinuities in the implicit periodic signal

Windowing

SLIDE 77

10 Sep 2013 11-755/18-797 77

Magnitude spectrum

Windowing

SLIDE 78

10 Sep 2013 11-755/18-797 78

Magnitude spectrum of windowed signal Magnitude spectrum of complete sine wave Magnitude spectrum of original segment

Windowing

SLIDE 79

10 Sep 2013 11-755/18-797 79

 Cosine windows:

 Window length is M  Index begins at 0

 Hamming: w[n] = 0.54 – 0.46 cos(2pn/M)  Hanning: w[n] = 0.5 – 0.5 cos(2pn/M)  Blackman: 0.42 – 0.5 cos(2pn/M) + 0.08 cos(4pn/M)

Window functions

SLIDE 80

10 Sep 2013 11-755/18-797 80

 Geometric windows:

 Rectangular (boxcar):  Triangular (Bartlett):  Trapezoid:

Window functions

SLIDE 81

Zero Padding

We can pad zeros to the end of a signal to make it a desired length

– Useful if the FFT (or any other algorithm we use) requires signals of a specified length – E.g. Radix 2 FFTs require signals of length 2n i.e., some power of 2. We must zero pad the signal to increase its length to the appropriate number

The consequence of zero padding is to change the periodic signal

whose Fourier spectrum is being computed by the DFT

10 Sep 2013 11-755/18-797 81

SLIDE 82

We can pad zeros to the end of a signal to make it a desired length

– Useful if the FFT (or any other algorithm we use) requires signals of a specified length – E.g. Radix 2 FFTs require signals of length 2n i.e., some power of 2. We must zero pad the signal to increase its length to the appropriate number

The consequence of zero padding is to change the periodic signal

whose Fourier spectrum is being computed by the DFT

10 Sep 2013 11-755/18-797 82

Zero Padding

SLIDE 83

The DFT of the zero padded signal is essentially the same as the DFT
f the unpadded signal, with additional spectral samples inserted in

between

– It does not contain any additional information over the original DFT – It also does not contain less information

10 Sep 2013 11-755/18-797 83

Magnitude spectrum

Zero Padding

SLIDE 84

10 Sep 2013 11-755/18-797 84

Magnitude spectra

SLIDE 85

The DFT of the zero padded signal is essentially the same as the DFT
f the unpadded signal, with additional spectral samples inserted in

between

– It does not contain any additional information over the original DFT – It also does not contain less information

10 Sep 2013 11-755/18-797 85

Zero Padding

Windowed signal

SLIDE 86

10 Sep 2013 11-755/18-797 86

Magnitude spectra

SLIDE 87

Zero padding a speech signal

10 Sep 2013 11-755/18-797 87

8000Hz 8000Hz time frequency frequency

128 samples from a speech signal sampled at 16000 Hz The first 65 points of a 128 point DFT. Plot shows log of the magnitude spectrum The first 513 points of a 1024 point DFT. Plot shows log of the magnitude spectrum

SLIDE 88

The Fourier Transform and Perception: Sound

The Fourier transforms

represents the signal analogously to a bank of tuning forks

Our ear has a bank of

tuning forks

The output of the Fourier

transform is perceptually very meaningful

10 Sep 2013 11-755/18-797 88

+

FT Inverse FT

SLIDE 89

The Fourier Transform and Perception: Sound

Processing Sound:
Analyze the sound using a

bank of tuning forks

Sample the transduced
utput of the turning forks

at periodic intervals

10 Sep 2013 11-755/18-797 89

+

FT Inverse FT

SLIDE 90

Sound parameterization

The signal is processed in segments of 25-64 ms

– Because the properties of audio signals change quickly – They are “stationary” only very briefly

10 Sep 2013 11-755/18-797 90

SLIDE 91

The signal is processed in segments of 25-64 ms

– Because the properties of audio signals change quickly – They are “stationary” only very briefly

Adjacent segments overlap by 15-48 ms

10 Sep 2013 11-755/18-797 91

Sound parameterization

SLIDE 92

The signal is processed in segments of 25-64 ms

– Because the properties of audio signals change quickly – They are “stationary” only very briefly

Adjacent segments overlap by 15-48 ms

10 Sep 2013 11-755/18-797 92

Sound parameterization

SLIDE 93

The signal is processed in segments of 25-64 ms

– Because the properties of audio signals change quickly – They are “stationary” only very briefly

Adjacent segments overlap by 15-48 ms

10 Sep 2013 11-755/18-797 93

Sound parameterization

SLIDE 94

The signal is processed in segments of 25-64 ms

– Because the properties of audio signals change quickly – They are “stationary” only very briefly

Adjacent segments overlap by 15-48 ms

10 Sep 2013 11-755/18-797 94

Sound parameterization

SLIDE 95

The signal is processed in segments of 25-64 ms

– Because the properties of audio signals change quickly – They are “stationary” only very briefly

Adjacent segments overlap by 15-48 ms

10 Sep 2013 11-755/18-797 95

Sound parameterization

SLIDE 96

The signal is processed in segments of 25-64 ms

– Because the properties of audio signals change quickly – They are “stationary” only very briefly

Adjacent segments overlap by 15-48 ms

10 Sep 2013 11-755/18-797 96

Sound parameterization

SLIDE 97

10 Sep 2013 11-755/18-797 97

Each segment is typically 25-64 milliseconds wide

Audio signals typically do not change significantly within this short time interval

Segments shift every 10- 16 milliseconds

Sound parameterization

SLIDE 98

10 Sep 2013 11-755/18-797 98

Each segment is windowed and a DFT is computed from it Windowing Frequency (Hz) Complex spectrum

Sound parameterization

SLIDE 99

10 Sep 2013 11-755/18-797 99

Each segment is windowed and a DFT is computed from it Windowing