In the name of Allah In the name of Allah the compassionate, the - - PowerPoint PPT Presentation

In the name of Allah In the name of Allah

the compassionate, the merciful the compassionate, the merciful

Digital Video Systems Digital Video Systems

• S. Kasaei
• S. Kasaei

Room: CE 307 Room: CE 307 Department of Computer Engineering Department of Computer Engineering Sharif University of Technology Sharif University of Technology E E-

• Mail:

Mail: skasaei@sharif.edu skasaei@sharif.edu Webpage: Webpage: http:// http://sharif.edu sharif.edu/~skasaei /~skasaei

• Lab. Website:
• Lab. Website: http://

http://mehr.sharif.edu mehr.sharif.edu/~ipl /~ipl

Acknowledgment Acknowledgment

Most of the slides used in this course have been provided by: Most of the slides used in this course have been provided by: Prof.

• Prof. Yao

Yao Wang Wang (Polytechnic University, Brooklyn) (Polytechnic University, Brooklyn) based on based on the the book: book: Video Processing & Communications Video Processing & Communications written by: written by: Yao Yao Wang, Wang, Jom Jom Ostermann Ostermann, & , & Ya Ya-

• Oin

Oin Zhang Zhang Prentice Hall, 1 Prentice Hall, 1st

st edition, 2001, ISBN: 0130175471.

edition, 2001, ISBN: 0130175471. [SUT Code: TK 5105 .2 .W36 2001]. [SUT Code: TK 5105 .2 .W36 2001].

Chapter 1 Chapter 1

Foundation of Video Coding

Part II: Scalar & Vector Quantization

Kasaei Kasaei 6 6

Outline Outline

• Overview of source coding systems

Overview of source coding systems

• Scalar Quantization

Scalar Quantization

• Vector Quantization

Vector Quantization

• Rate

Rate-

• distortion characterization of lossy

distortion characterization of lossy coding coding

• Operational rate

Operational rate-

• distortion function

distortion function

• Rate

Rate-

• distortion bound (lossy coding bound)

distortion bound (lossy coding bound)

Kasaei Kasaei 7 7

Components of a Coding Components of a Coding System System

Kasaei Kasaei 8 8

Lossy Coding Lossy Coding

• For discrete sources:

For discrete sources:

• Lossless coding: bitrate >= entropy rate.

Lossless coding: bitrate >= entropy rate.

• One can further quantize source samples to reach a lower rate.

One can further quantize source samples to reach a lower rate.

• For continuous sources:

For continuous sources:

• Lossless coding will require an

Lossless coding will require an infinite infinite bitrate! bitrate!

• One must quantize source samples to reach a finite bitrate.

One must quantize source samples to reach a finite bitrate.

• Lossy coding rate is bounded by the mutual information between

Lossy coding rate is bounded by the mutual information between the original source & the quantized source that satisfy a the original source & the quantized source that satisfy a distortion criterion. distortion criterion.

• Quantization methods:

Quantization methods:

• Scalar quantization.

Scalar quantization.

• Vector quantization.

Vector quantization.

Kasaei Kasaei 9 9

Scalar Quantization Scalar Quantization

• General description

General description

• Uniform quantization

Uniform quantization

• MMSE

MMSE quantizer quantizer

• Lloyd algorithm

Lloyd algorithm

Kasaei Kasaei 10 10

Function Representation Function Representation

l l

B f g f Q ∈ = if , ) (

Reconstruction Value Boundary Value (Decision Value)

Kasaei Kasaei 11 11

Line Partition Representation Line Partition Representation

l l l l l l l

B f g f Q g b b B b L ∈ = =

−

if , ) ( : mapping Quantizer : s tion value Reconstruc ) , [ : regions Partition : alues Boundary v : levels

• n

Quantizati

1

reconstruction value f symbol boundary value

Kasaei Kasaei 12 12

Distortion Measure Distortion Measure

General measure:

Mean Square Error (MSE):

2 1

) ( ) , ( g f g f d − =

mean dist. Incurred in each region

Kasaei Kasaei 13 13

Uniform Quantization Uniform Quantization

Uniform source:

Each additional bit provides 6dB gain! quantization stepsize:

2 R q B / L B

−

= =

Kasaei Kasaei 14 14

Minimum MSE (MMSE) Minimum MSE (MMSE) Quantizer Quantizer

MSE minimize to , Determine

l l g

b

: yields 0, Setting

2 2

= ∂ = ∂

l q l q

g b σ σ

Nearest-Neighbor Condition Centroid Condition

• r

centroid of Bl

Kasaei Kasaei 15 15

High High-

• Rate Approximation of SQ

Rate Approximation of SQ

• For a source with an

For a source with an arbitrary arbitrary pdf pdf, when the , when the bitrate bitrate (or (or quantization level) is quantization level) is very high very high so that the so that the pdf pdf within within each partition region can be approximated as each partition region can be approximated as flat flat we we have: have:

1 : source Gaussian for Bound VLC) (w/o 71 . 2 : source Gaussian i.i.d 1 : source Uniform

2 2 2

= = = ε ε ε

pdf of a normalized unit variance source

Lloyd Algorithm Lloyd Algorithm

• Iterative algorithms for

Iterative algorithms for determining determining MMSE MMSE quantizer quantizer parameters parameters ( (bl bl, , gl gl). ).

• Can be based on a

Can be based on a pdf pdf

• r a
• r a training data

training data. .

• Iterates

Iterates between between centroid centroid condition & condition & nearest nearest

• n

eighbor n eighbor condition. condition.

Update D0=D1

Kasaei Kasaei 17 17

Vector Quantization Vector Quantization

• General description

General description

• Nearest

Nearest-

• neighbor

neighbor quantizer quantizer

• MMSE

MMSE quantizer quantizer

• Generalized Lloyd algorithm

Generalized Lloyd algorithm

Kasaei Kasaei 18 18

Vector Quantization: Vector Quantization: General Description General Description

• Motivation: quantize a

Motivation: quantize a group group of samples (a

• f samples (a

vector) together, to exploit the vector) together, to exploit the correlation correlation between these samples. between these samples.

• Each sample vector is replaced by one of

Each sample vector is replaced by one of representative vectors representative vectors (or patterns) that (or patterns) that often

• ften
• ccurs
• ccurs in the signal.

in the signal.

• The task is to find the

The task is to find the L L most most popular patters popular patters. .

Kasaei Kasaei 19 19

Vector Quantization: Vector Quantization: General Description General Description

• Applications:

Applications:

• Color quantization

Color quantization: Quantizes all colors : Quantizes all colors appearing in an image to appearing in an image to L L colors to display colors to display

• n a monitor that can only display
• n a monitor that can only display L

L distinct distinct colors colors at a time at a time Adaptive palette. Adaptive palette.

• Image quantization

Image quantization: Quantizes every : Quantizes every NxN NxN block into one of the block into one of the L L typical patterns typical patterns (obtained through training). More efficient with (obtained through training). More efficient with a larger block size (but the block size is a larger block size (but the block size is limited by the complexity). limited by the complexity).

Kasaei Kasaei 20 20

VQ as Space Partition VQ as Space Partition

Every point in a region (Bl) is replaced by (quantized to) the point indicated by the circle (gl), the codeword.

L N R L l C B Q B L R

l l l l l N 2

log 1 : rate Bit } ,..., 2 , 1 , { : Codebook if , ) ( : mapping Quantizer : (codeword) r tion vecto Reconstruc : regions Partition : levels

• n

Quantizati : vector Original = = = ∈ = ∈ g f g f g f codeword

Kasaei Kasaei 21 21

Distortion Measure Distortion Measure

General measure:

MSE:

Kasaei Kasaei 22 22

Nearest Nearest-

• Neighbor (NN)

Neighbor (NN) Quantizer Quantizer

Challenge: How to determine the codebook?

Kasaei Kasaei 23 23

Complexity of NN VQ Complexity of NN VQ

• Complexity analysis:

Complexity analysis:

• Must compare the input vector with all of the

Must compare the input vector with all of the codewords codewords. .

• Each comparison takes

Each comparison takes N N operations.

• perations.
• Needs

Needs L=2^{NR} L=2^{NR} comparisons. comparisons.

• Total operation =

Total operation = N 2^{NR}. N 2^{NR}.

• Total storage space =

Total storage space = N 2^{NR}. N 2^{NR}.

• Both computation & storage requirement increases

Both computation & storage requirement increases exponentially exponentially with with N N! !

Kasaei Kasaei 24 24

Complexity of NN VQ Complexity of NN VQ

• Example:

Example:

• N=4x4

N=4x4 pixels, pixels, R=1 R=1 bpp bpp: : 16x2^16=2^20=1 16x2^16=2^20=1 Million operation/vector. Million operation/vector.

• Apply to

Apply to video frames video frames, , 720x480 720x480 pels pels/frame, /frame, 30 30 fps: fps: 2^20*(720x480/16)*30= 2^20*(720x480/16)*30=6.8 E+11 6.8 E+11 operations/s!

• perations/s!
• When applied to image, block size is typically limited to <=

When applied to image, block size is typically limited to <= 4x4. 4x4.

• Fast algorithms:

Fast algorithms:

• Structured codebook

Structured codebook so that one can conduct binary tree search. so that one can conduct binary tree search.

• Product VQ

Product VQ: can search sub : can search sub-

• vectors separately.

vectors separately.

• Lattice VQ

Lattice VQ: an analogue of the uniform : an analogue of the uniform quantizer quantizer in the VQ case, in the VQ case, in which all partition regions have the same shape & size (shift in which all partition regions have the same shape & size (shifted ed versions of a versions of a Voronoi Voronoi cell cell). ).

Kasaei Kasaei 25 25

LVQ LVQ

Kasaei Kasaei 26 26

LVQ LVQ

Kasaei Kasaei 27 27

MMSE Vector MMSE Vector Quantizer Quantizer

• Necessary conditions for MMSE:

Necessary conditions for MMSE:

• Nearest

Nearest-

• neighbor condition:

neighbor condition:

• Generalized

Generalized centroid centroid condition: condition:

• MSE as distortion:

MSE as distortion:

Kasaei Kasaei 28 28

Caveats Caveats

• Both quantizers satisfy the NN & centroid condition, but the

quantizer on the right is better! NN & centroid conditions are necessary but NOT sufficient for MSE optimality!

Kasaei Kasaei 29 29

Generalized Lloyd Generalized Lloyd Algorithm Algorithm (LBG Algorithm) (LBG Algorithm)

• Starts with

Starts with initial initial codewords codewords. .

• Iterates

Iterates between between finding best finding best partition partition (using NN (using NN condition), & condition), & updating updating codewords codewords (using (using centroid centroid condition). condition).

Update D0=D1

Example: Example:

For the same date, the codewords are aligned.

Kasaei Kasaei 31 31

Rate Rate-

• Distortion Characterization

Distortion Characterization

• f Lossy Coding
• f Lossy Coding
• Operational rate

Operational rate

• d

istortion function of a d istortion function of a quantizer quantizer: :

• Relates rate & distortion:

Relates rate & distortion: R(D). R(D).

• A vector

A vector quantizer quantizer reaches a different point on its reaches a different point on its R(D) R(D) curve by curve by using a different number of using a different number of codewords codewords. .

• Can also use distortion

Can also use distortion-

• rate function

rate function D(R). D(R).

• Rate

Rate

• distortion bound for a source:

distortion bound for a source:

• Minimum rate

Minimum rate R R needed to describe the source with distortion needed to describe the source with distortion <= <=D. D.

• RD optimal

RD optimal quantizer quantizer: :

• Minimize

Minimize D D for given for given R R (or vice versa). (or vice versa).

Typical D(R) curve

R

D (bound) ) (R D quantizer) given (a ) (R D

Kasaei Kasaei 32 32

Lossy Coding Bound Lossy Coding Bound

(Shannon Lossy Coding Theorem) (Shannon Lossy Coding Theorem)

• Q

QD,N

D,N: all coding schemes (or mappings

: all coding schemes (or mappings q( q(g g| |f f) )) that satisfy distortion criterion ) that satisfy distortion criterion d dN

N(f,

(f,g g)<=D. )<=D.

• I

IN

N(

(F F, ,G G): ): mutual information between mutual information between F F & & G G, information provided by , information provided by G G about about F. F.

• RG(D): RD bound for Gaussian source with the same variance.
• h(F): differential entropy of source F.

Kasaei Kasaei 33 33

RD Bound for Gaussian Source RD Bound for Gaussian Source

• i.i.d. 1

i.i.d. 1

• D

Gaussian: D Gaussian:

• i.i.d. N

i.i.d. N

• D

Gaussian with independent components: D Gaussian with independent components:

• N

N

• D

Gaussian with covariance matrix D Gaussian with covariance matrix C C: :

• Gaussian source with power spectrum (FT of correlation

Gaussian source with power spectrum (FT of correlation function): function):

Kasaei Kasaei 34 34

Summary Summary

• Coding system:

Coding system:

• riginal data
• riginal data

model parameters model parameters quantization quantization

• binary encoding.

binary encoding.

• Quantization:

Quantization:

• Scalar quantization:

Scalar quantization:

• Uniform

Uniform quantizer quantizer. .

• MMSE

MMSE quantizer quantizer (Nearest (Nearest-

• neighbor &

neighbor & centroid centroid condition). condition).

• Vector quantization:

Vector quantization:

• Nearest

Nearest-

• neighbor

neighbor quantizer quantizer. .

• MMSE

MMSE quantizer quantizer. .

• Generalized Lloyd algorithm.

Generalized Lloyd algorithm.

• Uniform

Uniform quantizer quantizer: :

• Can be realized by a lattice

Can be realized by a lattice quantizer quantizer. .

Kasaei Kasaei 35 35

Summary Summary

• Rate

Rate-

• distortion characterization of lossy coding:

distortion characterization of lossy coding:

• Bound on lossy coding.

Bound on lossy coding.

• Operational RD function of practical

Operational RD function of practical quantizers quantizers. .

Kasaei Kasaei 36 36

Homework 7 Homework 7

• Reading assignment:

Reading assignment:

• Sec. 8.5
• Sec. 8.5-
• 8.7, 8.3.2,8.3.3

8.7, 8.3.2,8.3.3

• Written assignment:

Written assignment:

• Prob. 8.8,8.11,8.14
• Prob. 8.8,8.11,8.14
• Computer assignment:

Computer assignment:

• Option 1: Write a program to perform vector quantization on a

Option 1: Write a program to perform vector quantization on a gray scale image using 4x4 pixels as a vector. You should gray scale image using 4x4 pixels as a vector. You should design your codebook using all the blocks in the image as design your codebook using all the blocks in the image as training data, using the generalized Lloyd algorithm. Then training data, using the generalized Lloyd algorithm. Then quantize the image using your codebook. You can choose the quantize the image using your codebook. You can choose the codebook size, say, L=128 or 256. If your program can work with codebook size, say, L=128 or 256. If your program can work with any specified codebook size L, then you can observe the quality any specified codebook size L, then you can observe the quality

• f quantized images with different L.
• f quantized images with different L.

Kasaei Kasaei 37 37

Homework 7 Homework 7

• Option 2: Write a program to perform color quantization on a

Option 2: Write a program to perform color quantization on a color RGB image. Your vector dimension is now 3, containing color RGB image. Your vector dimension is now 3, containing R,G,B values. The training data are the colors of all the pixels R,G,B values. The training data are the colors of all the pixels. . You should design a color palette (i.e. codebook) of size L, usi You should design a color palette (i.e. codebook) of size L, using ng generalized Lloyd algorithm, and then replace the color of each generalized Lloyd algorithm, and then replace the color of each pixel by one of the color in the palette. You can choose a fixed pixel by one of the color in the palette. You can choose a fixed L L

• r let L be a user
• r let L be a user-
• selectable variable. In the later case, observe

selectable variable. In the later case, observe the quality of quantized images with different L. the quality of quantized images with different L.

The End The End