In the name of Allah In the name of Allah the compassionate, the - - PowerPoint PPT Presentation

In the name of Allah In the name of Allah

the compassionate, the merciful the compassionate, the merciful

Digital Video Systems Digital Video Systems

• S. Kasaei
• S. Kasaei

Room: CE 307 Room: CE 307 Department of Computer Engineering Department of Computer Engineering Sharif University of Technology Sharif University of Technology E E-

• Mail:

Mail: skasaei@sharif.edu skasaei@sharif.edu Webpage: Webpage: http://sharif.edu/~skasaei http://sharif.edu/~skasaei

• Lab. Website:
• Lab. Website: http://ipl.ce.sharif.edu

http://ipl.ce.sharif.edu

Acknowledgment Acknowledgment

Most of the slides used in this course have been provided by: Most of the slides used in this course have been provided by: Prof.

• Prof. Yao Wang

Yao Wang (Polytechnic University, Brooklyn) (Polytechnic University, Brooklyn) based on based on the the book: book: Video Processing & Communications Video Processing & Communications written by: written by: Yao Wang, Jom Ostermann, & Ya Yao Wang, Jom Ostermann, & Ya-

• Oin Zhang

Oin Zhang Prentice Hall, 1 Prentice Hall, 1st

st edition, 2001, ISBN: 0130175471.

edition, 2001, ISBN: 0130175471. [SUT Code: TK 5105 .2 .W36 2001]. [SUT Code: TK 5105 .2 .W36 2001].

Chapter 1 Chapter 1

Foundation of Video Coding

Part II: Scalar & Vector Quantization

Kasaei Kasaei 6 6

Outline Outline

• Overview of source coding systems

Overview of source coding systems

• Scalar Quantization

Scalar Quantization

• Vector Quantization

Vector Quantization

• Rate

Rate-

• distortion characterization of lossy

distortion characterization of lossy coding coding

• Operational rate

Operational rate-

• distortion function

distortion function

• Rate

Rate-

• distortion bound (lossy coding bound)

distortion bound (lossy coding bound)

Kasaei Kasaei 7 7

Components of a Coding Components of a Coding System System

Kasaei Kasaei 8 8

Lossy Coding Lossy Coding

• For discrete sources:

For discrete sources:

• Lossless coding: bitrate >= entropy rate.

Lossless coding: bitrate >= entropy rate.

• One can further quantize source samples to reach a lower rate.

One can further quantize source samples to reach a lower rate.

• For continuous sources:

For continuous sources:

• Lossless coding will require an

Lossless coding will require an infinite infinite bitrate! bitrate!

• One must quantize source samples to reach a finite bitrate.

One must quantize source samples to reach a finite bitrate.

• Lossy coding rate is bounded by the mutual information between

Lossy coding rate is bounded by the mutual information between the original source & the quantized source that satisfy a the original source & the quantized source that satisfy a distortion criterion. distortion criterion.

• Quantization methods:

Quantization methods:

• Scalar quantization.

Scalar quantization.

• Vector quantization.

Vector quantization.

Kasaei Kasaei 9 9

Scalar Quantization Scalar Quantization

• General description

General description

• Uniform quantization

Uniform quantization

• MMSE quantizer

MMSE quantizer

• Lloyd algorithm

Lloyd algorithm

Kasaei Kasaei 10 10

Function Representation Function Representation

l l

B f g f Q ∈ = if , ) (

Reconstruction Value Boundary Value (Decision Value)

Kasaei Kasaei 11 11

Line Partition Representation Line Partition Representation

l l l l l l l

B f g f Q g b b B b L ∈ = =

−

if , ) ( : mapping Quantizer : s tion value Reconstruc ) , [ : regions Partition : alues Boundary v : levels

• n

Quantizati

1

Reconstruction Value f Symbol Boundary Value

Kasaei Kasaei 12 12

Distortion Measure Distortion Measure

General measure:

Mean Square Error (MSE):

2 1

) ( ) , ( g f g f d − =

mean dist. incurred in each region

Kasaei Kasaei 13 13

Uniform Quantization Uniform Quantization

Uniform source:

Each additional bit provides 6dB gain! quantization stepsize:

2 R q B / L B

−

= =

Kasaei Kasaei 14 14

Minimum MSE (MMSE) Minimum MSE (MMSE) Quantizer Quantizer

: MSE minimize to & Determine

l l

g b

: yields & Setting

2 2

= ∂ = ∂

l q l q

g b σ σ

Nearest-Neighbor Condition Centroid Condition

• r

centroid of Bl centroid of Bl

Kasaei Kasaei 15 15

High High-

• Rate Approximation of SQ

Rate Approximation of SQ

• For a source with an

For a source with an arbitrary pdf arbitrary pdf, when the , when the bitrate bitrate (or (or quantization level) is quantization level) is very high very high so that the pdf within so that the pdf within each partition region can be approximated as each partition region can be approximated as flat flat, , we we have: have:

1 : source Gaussian for Bound VLC) (w/o 71 . 2 : source Gaussian i.i.d 1 : source Uniform

2 2 2

= = = ε ε ε

pdf of a normalized Unit-variance source

Lloyd Algorithm Lloyd Algorithm

• Iterative algorithms for

Iterative algorithms for determining determining MMSE MMSE quantizer parameters quantizer parameters ( (bl, gl bl, gl). ).

• Can be based on a

Can be based on a pdf pdf

• r a
• r a training data

training data. .

• Iterates

Iterates between between centroid condition & centroid condition & nearest nearest-

• neighbor

neighbor condition. condition.

Update D0=D1

Kasaei Kasaei 17 17

Vector Quantization Vector Quantization

• General description

General description

• Nearest

Nearest-

• neighbor quantizer

neighbor quantizer

• MMSE quantizer

MMSE quantizer

• Generalized Lloyd algorithm

Generalized Lloyd algorithm

Kasaei Kasaei 18 18

Vector Quantization: Vector Quantization: General Description General Description

• Motivation: quantize a

Motivation: quantize a group group of samples (a

• f samples (a

vector) together, to exploit the vector) together, to exploit the correlation correlation between these samples. between these samples.

• Each sample vector is replaced by one of

Each sample vector is replaced by one of representative vectors representative vectors (or patterns) that (or patterns) that

• ften occurs
• ften occurs in the signal.

in the signal.

• The task is to find the

The task is to find the L L most most popular popular patters patters. .

Kasaei Kasaei 19 19

Vector Quantization: Vector Quantization: General Description General Description

Kasaei Kasaei 20 20

Vector Quantization: Vector Quantization: General Description General Description

• Codebook Generation:

Codebook Generation:

• Best results are obtained when the

Best results are obtained when the codebook codebook is is generated from the generated from the content content itself (local codebooks). itself (local codebooks).

• Computationally intensive task.

Computationally intensive task.

• Creates

Creates overhead

• verhead (codebook has to be transmitted to

(codebook has to be transmitted to the receiver as overhead). the receiver as overhead).

• Global Codebooks:

Global Codebooks:

• Linde

Linde-

• Buzo

Buzo-

• Gray

Gray (LBG) clustering algorithm: (LBG) clustering algorithm:

• Training content from the same class of content is used.

Training content from the same class of content is used.

• The larger the codebook the higher the bit rate, the higher

The larger the codebook the higher the bit rate, the higher the quality of the content. the quality of the content.

Kasaei Kasaei 21 21

Vector Quantization: Vector Quantization: General Description General Description

• Applications:

Applications:

• Color quantization

Color quantization: Quantizes all colors : Quantizes all colors appearing in an image to appearing in an image to L L colors to display colors to display

• n a monitor that can only display
• n a monitor that can only display L

L distinct distinct colors colors at a time at a time Adaptive palette. Adaptive palette.

• Image quantization

Image quantization: Quantizes every : Quantizes every NxN NxN block into one of the block into one of the L L typical patterns typical patterns (obtained through training). (obtained through training). More efficient with a larger block size (but the More efficient with a larger block size (but the block size is limited by the complexity). block size is limited by the complexity).

Kasaei Kasaei 22 22

VQ as Space Partition VQ as Space Partition

Every point in a region (Bl) is replaced by (quantized to) the point indicated by the circle (gl), the codeword.

L N R L l C B Q B L R

l l l l l N 2

log 1 : rate Bit } ,..., 2 , 1 , { : Codebook if , ) ( : mapping Quantizer : (codeword) r tion vecto Reconstruc : regions Partition : levels

• n

Quantizati : vector Original = = = ∈ = ∈ g f g f g f codeword

Kasaei Kasaei 23 23

Distortion Measure Distortion Measure

General measure:

MSE:

Kasaei Kasaei 24 24

Nearest Nearest-

• Neighbor (NN)

Neighbor (NN) Quantizer Quantizer

Challenge: How to determine the codebook?

Kasaei Kasaei 25 25

Complexity of NN VQ Complexity of NN VQ

• Complexity analysis:

Complexity analysis:

• Must compare the input vector with all of the codewords.

Must compare the input vector with all of the codewords.

• Each comparison takes

Each comparison takes N N operations.

• perations.
• Needs

Needs L=2^{NR} L=2^{NR} comparisons. comparisons.

• Total operation =

Total operation = N 2^{NR}. N 2^{NR}.

• Total storage space =

Total storage space = N 2^{NR}. N 2^{NR}.

• Both computation & storage requirement increases

Both computation & storage requirement increases exponentially exponentially with with N N! !

Kasaei Kasaei 26 26

Complexity of NN VQ Complexity of NN VQ

• Example:

Example:

• N=4x4

N=4x4 pixels, pixels, R=1 R=1 bpp: bpp: 16x2^16=2^20=1 16x2^16=2^20=1 Million operation/vector. Million operation/vector.

• Apply to

Apply to video frames video frames, , 720x480 720x480 pels/frame, pels/frame, 30 30 fps: fps: 2^20*(720x480/16)*30= 2^20*(720x480/16)*30=6.8 E+11 6.8 E+11 operations/s!

• perations/s!
• When applied to image, block size is typically limited to <=

When applied to image, block size is typically limited to <= 4x4. 4x4.

• Fast algorithms:

Fast algorithms:

• Structured codebook

Structured codebook so that one can conduct binary tree search. so that one can conduct binary tree search.

• Product VQ

Product VQ: can search sub : can search sub-

• vectors separately.

vectors separately.

• Lattice VQ

Lattice VQ: an analogue of the uniform quantizer in the VQ case, : an analogue of the uniform quantizer in the VQ case, in which all partition regions have the same shape & size (shift in which all partition regions have the same shape & size (shifted ed versions of a versions of a Voronoi cell Voronoi cell). ).

Kasaei Kasaei 27 27

Lattice Vector Quantization Lattice Vector Quantization

Kasaei Kasaei 28 28

Lattice Vector Quantization Lattice Vector Quantization

Kasaei Kasaei 29 29

MMSE Vector Quantizer MMSE Vector Quantizer

• Necessary conditions for MMSE:

Necessary conditions for MMSE:

• Nearest

Nearest-

• neighbor condition:

neighbor condition:

• Generalized centroid condition:

Generalized centroid condition:

• MSE as distortion:

MSE as distortion:

Kasaei Kasaei 30 30

Caveats Caveats

• Both quantizers satisfy the NN & centroid condition, but the

quantizer on the right is better! NN & centroid conditions are necessary but NOT sufficient for MSE optimality!

Kasaei Kasaei 31 31

Generalized Lloyd Generalized Lloyd Algorithm Algorithm (LBG Algorithm) (LBG Algorithm)

• Starts with

Starts with initial initial codewords codewords. .

• Iterates

Iterates between between finding best finding best partition partition (using NN (using NN condition), & condition), & updating updating codewords codewords (using (using centroid condition). centroid condition).

Update D0=D1

X: data : representation

Example: Example:

For the same data, the codewords are aligned.

Kasaei Kasaei 33 33

Rate Rate-

• Distortion Characterization

Distortion Characterization

• f Lossy Coding
• f Lossy Coding
• Operational rate

Operational rate-

• distortion function of a quantizer:

distortion function of a quantizer:

• Relates rate & distortion:

Relates rate & distortion: R(D). R(D).

• A vector quantizer reaches a different point on its

A vector quantizer reaches a different point on its R(D) R(D) curve by curve by using a different number of codewords. using a different number of codewords.

• Can also use distortion

Can also use distortion-

• rate function

rate function D(R). D(R).

• Rate

Rate-

• distortion bound for a source:

distortion bound for a source:

• Minimum rate

Minimum rate R R needed to describe the source with distortion needed to describe the source with distortion <= <=D. D.

• RD optimal quantizer:

RD optimal quantizer:

• Minimize

Minimize D D for a given for a given R R (or vice versa). (or vice versa).

Typical D(R) curve

R

D (bound) ) (R D quantizer) given (a ) (R D

Kasaei Kasaei 34 34

Lossy Coding Bound Lossy Coding Bound

(Shannon Lossy Coding Theorem) (Shannon Lossy Coding Theorem)

• Q

QD,N

D,N: all coding schemes (or mappings

: all coding schemes (or mappings q( q(g g| |f f) )) that satisfy distortion criterion ) that satisfy distortion criterion d dN

N(f,

(f,g g)<=D. )<=D.

• I

IN

N(

(F F, ,G G): ): mutual information between mutual information between F F & & G G (information provided by (information provided by G G about about F F) ). .

• RG(D): RD bound for Gaussian source with the same variance.
• h(F): differential entropy of source F.

Kasaei Kasaei 35 35

RD Bound for Gaussian Source RD Bound for Gaussian Source

• i.i.d. 1

i.i.d. 1-

• D Gaussian:

D Gaussian:

• i.i.d. N

i.i.d. N-

• D Gaussian with independent components:

D Gaussian with independent components:

• N

N-

• D Gaussian with covariance matrix

D Gaussian with covariance matrix C C: :

• Gaussian source with power spectrum (FT of correlation

Gaussian source with power spectrum (FT of correlation function): function):

Kasaei Kasaei 36 36

Summary Summary

• Coding system:

Coding system:

• riginal data
• riginal data

model parameters model parameters quantization quantization

• binary encoding.

binary encoding.

• Quantization:

Quantization:

• Scalar quantization:

Scalar quantization:

• Uniform quantizer.

Uniform quantizer.

• MMSE quantizer (Nearest

MMSE quantizer (Nearest-

• neighbor & centroid condition).

neighbor & centroid condition).

• Vector quantization:

Vector quantization:

• Nearest

Nearest-

• neighbor quantizer.

neighbor quantizer.

• MMSE quantizer.

MMSE quantizer.

• Generalized Lloyd algorithm.

Generalized Lloyd algorithm.

• Uniform quantizer:

Uniform quantizer:

• Can be realized by a lattice quantizer.

Can be realized by a lattice quantizer.

Kasaei Kasaei 37 37

Summary Summary

• Rate

Rate-

• distortion characterization of lossy coding:

distortion characterization of lossy coding:

• Bound on lossy coding.

Bound on lossy coding.

• Operational RD function of practical quantizers.

Operational RD function of practical quantizers.

Kasaei Kasaei 38 38

Homework 7 Homework 7

• Reading assignment:

Reading assignment:

• Sec. 8.5
• Sec. 8.5-
• 8.7, 8.3.2,8.3.3

8.7, 8.3.2,8.3.3

• Written assignment:

Written assignment:

• Prob. 8.8,8.11,8.14
• Prob. 8.8,8.11,8.14
• Computer assignment:

Computer assignment:

• Option 1: Write a program to perform vector quantization on a

Option 1: Write a program to perform vector quantization on a gray scale image using 4x4 pixels as a vector. You should gray scale image using 4x4 pixels as a vector. You should design your codebook using all the blocks in the image as design your codebook using all the blocks in the image as training data, using the generalized Lloyd algorithm. Then training data, using the generalized Lloyd algorithm. Then quantize the image using your codebook. You can choose the quantize the image using your codebook. You can choose the codebook size, say, L=128 or 256. If your program can work with codebook size, say, L=128 or 256. If your program can work with any specified codebook size L, then you can observe the quality any specified codebook size L, then you can observe the quality

• f quantized images with different L.
• f quantized images with different L.

Kasaei Kasaei 39 39

Homework 7 Homework 7

• Option 2: Write a program to perform color quantization on a

Option 2: Write a program to perform color quantization on a color RGB image. Your vector dimension is now 3, containing color RGB image. Your vector dimension is now 3, containing R,G,B values. The training data are the colors of all the pixels R,G,B values. The training data are the colors of all the pixels. . You should design a color palette (i.e. codebook) of size L, usi You should design a color palette (i.e. codebook) of size L, using ng generalized Lloyd algorithm, and then replace the color of each generalized Lloyd algorithm, and then replace the color of each pixel by one of the color in the palette. You can choose a fixed pixel by one of the color in the palette. You can choose a fixed L L

• r let L be a user
• r let L be a user-
• selectable variable. In the later case, observe

selectable variable. In the later case, observe the quality of quantized images with different L. the quality of quantized images with different L.

The End The End