In the name of Allah the compassionate, the merciful Digital Video - - PowerPoint PPT Presentation

In the name of Allah

the compassionate, the merciful

Digital Video Systems

• S. Kasaei
• S. Kasaei

Room: CE 307 Department of Computer Engineering Sharif University of Technology E-Mail: skasaei@sharif.edu Webpage: http://sharif.edu/~skasaei

• Lab. Website: http://ipl.ce.sharif.edu

Acknowledgment

Most of the slides used in this course have been provided by: Prof. Yao Wang (Polytechnic University, Brooklyn) based on the book: Video Processing & Communications written by: Yao Wang, Jom Ostermann, & Ya-Oin Zhang Prentice Hall, 1st edition, 2001, ISBN: 0130175471. [SUT Code: TK 5105 .2 .W36 2001].

Chapter 2

Fourier Analysis of Video Signals & Frequency Response of the HVS

Kasaei 6

Outline

Fourier transform over multidimensional

space:

Continuous-space FT (CSFT) Discrete-space FT (DSFT)

Frequency domain characterization of video

signals:

Spatial frequency Temporal frequency Temporal frequency caused by motion

Kasaei 7

Outline

Frequency response of the HVS:

Spatial frequency response Temporal frequency response & flicker Spatio-temporal response Smooth pursuit eye movement

Video sampling (a brief discussion)

Kasaei 8

Continuous-Space Signals

K-D space continuous signals: Convolution: Example function:

Dirac delta function:

k K

R x x x ∈ = ] ,..., , [ ), (

2 1

x x ψ y y y x x x d h h

k

R

∫

− = ) ( ) ( ) ( * ) ( ψ ψ

real or complex

R: set of real numbers X: K-D continuous variable

Kasaei 9

Continuous-Space Fourier Transform (CSFT)

Forward transform: Inverse transform: Convolution theorem:

x x f x f d j

k

R T c

∫

− = Ψ ) 2 exp( ) ( ) ( π ψ

f x f f x d j

k

R T c

∫Ψ

= ) 2 exp( ) ( ) ( π ψ ) ( * ) ( ) ( ) ( ) ( ) ( ) ( * ) ( f f x x f f x x

c c c c

H h H h Ψ ⇔ Ψ ⇔ ψ ψ

Kasaei 10

Continuous-Space Fourier Transform (CSFT)

More on inverse transform:

The inverse CSFT shows that any signal can be

expressed as a linear combination of complex exponential function with different frequencies.

The CSFT at a particular frequency represents the

contribution of the corresponding complex exponential basis function.

The transform determines the correlation between

the input signal & its projection on some defined basis function.

Orthogonal basis functions preserve the signal

energy in the transform domain.

f x f f x d j

k

R T c

∫Ψ

= ) 2 exp( ) ( ) ( π ψ

Kasaei 11

Continuous-Space Systems

General system over K-D continuous space: Linear & (Space) Shift-Invariant (LSI)

System:

LSI system can be completely described by

its impulse response:

k

R T ∈ = x x x )), ( ( ) ( ψ φ

( )

) ( ) ( ) ( ) ( * ) ( ) ( ) ( ) ( f f f x x x x x

c c c

H h T h Ψ = Φ ⇔ = = ψ φ δ ) ( )) ( ( )) ( ) ( ( ) ( ) (

2 2 1 1 2 2 1 1

x x x x x x x x + = + + = + φ ψ ψ α ψ α φ α φ α T T

Kasaei 12

Discrete-Space Signals

K-D space discrete signals: Convolution: Example function:

Kronecker delta function:

K K

Z n n n ∈ = ] ,..., , [ ), (

2 1

n n ψ

∑

∈

− =

K

Z

h h

m

m m n n n ) ( ) ( ) ( * ) ( ψ ψ

Kasaei 13

Discrete-Space Fourier Transform (DSFT)

Forward transform: Inverse transform: Convolution theorem:

{ }

) 2 / 1 , 2 / 1 ( , : period l Fundamenta 1

• f

period with dimension each in periodic is ) ( ) 2 exp( ) ( ) ( − ∈ = Ψ − = Ψ

∑

∈ k K d R T d

f I j

K

f f n f n f

n

π ψ

∫

∈

Ψ =

K

I T d

d j

f

f n f f n ) 2 exp( ) ( ) ( π ψ

) ( * ) ( ) ( ) ( ) ( ) ( ) ( * ) ( f f n n f f n n

d d d d

H h H h Ψ ⇔ Ψ ⇔ ψ ψ

freq:

unit hypercube repeats @ integer points

Kasaei 14

Frequency Domain Characterization

• f Video Signals

Spatial frequency Temporal frequency Temporal frequency caused by motion

Kasaei 15

Spatial Frequency

Spatial frequency measures how fast the

image intensity changes in the image plane.

Spatial frequency can be completely

characterized by the variation frequencies in two orthogonal directions (e.g., horizontal & vertical):

fx: cycles/horizontal unit distance. fy : cycles/vertical unit distance.

It can also be specified by magnitude & angle

• f change:

) / arctan( ,

2 2 x y y x m

f f f f f = + = θ

Kasaei 16

Illustration of Spatial Frequency

Kasaei 17

Angular Frequency

ee) cycle/degr ( f 180 f f (degree) 180 n) h/2d(radia 2 (radian) ) 2 / arctan( 2

s s

h d d h d h π θ π θ

θ

= = = ≈ =

Problem with previous defined spatial frequency:

Perceived speed of change depends on the

viewing distance.

Kasaei 18

Angular Frequency

For the same picture, the angular frequency

increases as the viewing distance increases.

For a fixed viewing distance, a larger screen

size leads to lower angular frequency.

The same picture appears to change more

rapidly when viewed farther away, & it changes more slowly if viewed from larger screen.

It depends on both the spatial frequency in

the signal & the viewing conditions.

Kasaei 19

Temporal Frequency

Temporal frequency measures temporal

variation (cycles/s).

In a video, the temporal frequency is spatial

position dependent, as every point may change differently.

Temporal frequency is caused by camera or

• bject motion.

It depends not only on the motion, but also

• n the spatial frequency of the object.

Kasaei 20

Temporal Frequency caused by Linear Motion

Kasaei 21

) ( : frequency temporal and spatial, motion, between Relation ) ( ) , ( ) , , ( ) , ( ) , , (

y y x x t y y x x t y x t y x y x

f v f v f f v f v f f f f f f t v y t v x t y x + − = + + Ψ = Ψ ⇔ − − = δ ψ ψ

The temporal frequency of the image of a moving object depends on motion as well as the spatial frequency of the object (projection of v on f). Example: A plane with vertical bar pattern, moving vertically, causes no temporal change; but moving horizontally, it causes fastest temporal change.

Relation between Motion, Spatial, & Temporal Frequency

is at time pattern image the ), , ( is at pattern image the Assume ). , ( speed with moving

• bject

an Consider t y x t v v

y x

ψ =

under CIA

X’ y’ 2-D CSFT nonzero

• n plane

convolution

Kasaei 22

Illustration of the Relation

= ⇒ = =

t y x

f f f

Kasaei 23

Frequency Response of HVS

Temporal frequency response & flicker Spatial frequency response Spatio-temporal response Smooth pursuit eye movement

Kasaei 24

Frequency Responses of HVS

Most of the video systems are ultimately

targeted for human viewers.

It is important to understand how the

human perceives a video signal.

Sensitivity of the HVS to a visual pattern

depends on the spatial & temporal frequency content of the pattern.

Kasaei 25

Frequency Responses of HVS

The visual sensitivity is highest at some

intermediate spatial & temporal frequencies.

It then falls off & diminishes at some cut-off

frequencies.

Spatial or temporal changes above these

frequencies are invisible to the human eye.

They form the basis for determining the

frame & line rates in video capture & display systems.

Kasaei 26

Temporal Frequency Responses

The temporal frequency response of the

HVS refers to the visual sensitivity to a temporally varying pattern at different frequencies.

The temporal response of an observer

depends on viewing distance, display brightness, ambient lightning, & …

The temporal response of the HVS is similar

to a BPF.

Kasaei 27

Temporal Frequency Responses

The peak increases with the mean

brightness of the image.

One reason that the eye reduces sensitivity

at higher temporal frequencies is because the eye can retain the sensitivity of an image for a short time interval (even when the actual image has been removed).

This phenomenon is known as the

persistence of vision.

Kasaei 28

Flicker Perception

It causes temporal blurring, if a pattern

changes in a rate faster than the refresh rate

• f the HVS.

It allows the display of a video signal as a

consecutive sequence of frames.

As long as the frame interval is shorter than

the visual persistence period, the eye perceives a continuously varying image.

Otherwise, the eye will observe frame

flicker.

Kasaei 29

Flicker Perception

The lowest frame rate at which the eye does

not perceive flicker is known as the critical flicker frequency.

The brighter the display, the higher the

critical flicker frequency.

The motion picture industry uses 24

frames/sec.

Kasaei 30

Flicker Perception

The TV industry uses 50/60 fields/sec. The computer display uses 72 frames/sec. A troland is the unit used to describe the

intensity of light entered the retina.

Kasaei 31

Temporal Response

200 100 50 20 10 5 2 1 2 5 10 20 50 Frequency (Hz) Contrast sensitivity

0.06 trolands 850 trolands 9300 trolands 7.1 trolands 77 trolands 0.65 trolands

Figure 2.5 The temporal frequency response of the HVS obtained by a visual

• experiment. Different curves represent the responses obtained with different mean

brightness levels, B, measured in trolands. The horizontal axis represents the flicker frequency f , measured in Hz. Reprinted from D. H. Kelly, Visual responses to time-dependent stimuli. I. Amplitude sensitivity measurements, J. Opt. Soc. Am. (1961) 51:422–29, by permission of the Optical Society of America.

Critical flicker frequency: The lowest frame rate at which the eye does not perceive flicker. It provides a guideline for determining the frame rate, when designing a video system. Critical flicker frequency depends on the mean brightness of the display:

60 Hz is typically sufficient

for watching TV.

Watching a movie needs

lower frame rate than TV.

Kasaei 32

MTF of the Visual System

Modulation transfer function (MTF) of human visual system (HVS). (a) Contrast versus spatial frequency sinusoidal grating. (b) A typical MTF plot. cpd

Kasaei 33

Spatial Frequency Response

The spatial frequency response of the HVS

refers to the visual sensitivity to a stationary spatial pattern with different spatial frequencies.

The visual sensitivity is isotropic with

respect to the direction of spatial variation.

The spatial frequency response of the HVS is

also similar to a BPF, with a peak response at about 2-5 cpd & diminishing at about 30 cpd.

Kasaei 34

Spatial Frequency Response

The eye often jumps from one fixation position

to another very rapidly. This is known as saccadic eye movement.

It enhances the contrast sensitivity, but reduces

the peak of the frequency response.

The viewer’s sensitivity is about 10 fold higher

with normal eye movement than without.

The peak response occurs at about 2 cpd with

normal eye movement, but is shifted to about 4 cpd with complete removal of eye movement.

Kasaei 35

Spatial Frequency Response

The spatial frequency

response of the HVS,

• btained by a visual

experiment.

• Filled circles: under

normal, unstabilized conditions.

• Open squares: with
• ptical gain setting for

stabilization.

• Open circles: with
• ptical gain changed

about 5%.

Kasaei 36

Spatiotemporal Frequency Response

At higher temporal frequencies, both the

peak & cut-off frequencies in the spatial frequency response shift downwards.

A similar trend happens with the temporal

frequency response.

When an image patter removes very fast, the

eye will not be able to differentiate the very high spatial frequencies.

The eye is more sensitive to temporal

variations caused be motion than by flickering.

Kasaei 37

Spatiotemporal Response

The reciprocal relation between spatial & temporal sensitivity was used in TV system design: Interlaced scan provides a tradeoff between spatial & temporal resolution. (a) SF:

• Open circles: 1 Hz.
• Filled circles: 6 Hz.
• Open triangles: 16 Hz.
• Filled triangles: 22 Hz.

(b) TF:

• Open circles: 0.5 cpd.
• Filled circles: 4 cpd.
• Open triangles: 16 cpd.
• Filled triangles: 22 cpd.

Kasaei 38

Smooth Pursuit Eye Movement (SPEM)

Smooth pursuit: the eye tracks moving objects. Net effect: reduces the velocity of moving objects on the

retinal plane, so that the eye can perceive much higher raw temporal frequencies than indicated by the temporal frequency response (80 Hz).

y y x x t y y x x t t y x y y x x t y x

v v v v f f v f v f f v v f v f v f v v = = = + + = + − = ~ , ~ if ~ ) ~ ~ ( ~ : ) ~ , ~ ( at moving is eye when the retina at the frequency temporal Observed ) ( : ) , ( at moving is

• bject

n the motion whe

• bject

by caused frequency Temporal Tracking the object motion, reduces the observed temporal frequency at the retina.

Kasaei 39

Smooth Pursuit Eye Movement (SPEM)

Spatiotemporal response of HVS under SPEM: (a) Without SPEM. (b) With eye velocity of 2 deg/s. (c) With eye velocity of 10 deg/s. SPEM extends the nonzero region of the visual response to a large TF range (1000 Hz).

Kasaei 40

Homework 2

Reading assignment:

Chapter 2

Written assignment:

• Prob. 2.1,2.2,2.5,2.6, & 2.7

The End