In the name of Allah In the name of Allah the compassionate, the - - PowerPoint PPT Presentation

In the name of Allah In the name of Allah

the compassionate, the merciful the compassionate, the merciful

Digital Video Systems Digital Video Systems

• S. Kasaei
• S. Kasaei

Room: CE 307 Room: CE 307 Department of Computer Engineering Department of Computer Engineering Sharif University of Technology Sharif University of Technology E E-

• Mail:

Mail: skasaei@sharif.edu skasaei@sharif.edu Webpage: Webpage: http:// http://sharif.edu sharif.edu/~skasaei /~skasaei

• Lab. Website:
• Lab. Website: http://

http://ipl.ce.sharif.edu ipl.ce.sharif.edu

Acknowledgment Acknowledgment

Most of the slides used in this course have been provided by: Pr Most of the slides used in this course have been provided by: Prof.

• f. Yao

Yao Wang Wang (Polytechnic University, Brooklyn) (Polytechnic University, Brooklyn) based on the based on the book: book: Video Processing & Communications Video Processing & Communications written by: written by: Yao Yao Wang, Wang, Jom Jom Ostermann Ostermann, & , & Ya Ya-

• Oin

Oin Zhang Zhang Prentice Hall, 1 Prentice Hall, 1st

st edition, 2001, ISBN: 0130175471.

edition, 2001, ISBN: 0130175471. [SUT Code: TK 5105 .2 .W36 2001]. [SUT Code: TK 5105 .2 .W36 2001].

Chapter 8 Chapter 8

Foundation of Video Coding

Part I: Overview & Binary Encoding

Outline Outline

• Overview of video coding systems

Overview of video coding systems

• Review of probability & information theory

Review of probability & information theory concepts concepts

• Binary encoding:

Binary encoding:

• Information theory bounds

Information theory bounds

• Huffman coding

Huffman coding

• Arithmetic coding

Arithmetic coding

Kasaei 6

Video Coding Video Coding

• Video coding is an important application of video

Video coding is an important application of video processing. processing.

• The goal of video coding is to

The goal of video coding is to reduce the data rate reduce the data rate of a

• f a

video sequence (transmission, storage, & retrieval). video sequence (transmission, storage, & retrieval).

• Channel bandwidth varies depending on application &

Channel bandwidth varies depending on application & transmission media (20 kbps~6 mbps). transmission media (20 kbps~6 mbps).

• Two main classes of algorithms:

Two main classes of algorithms:

• Waveform

Waveform-

• based (

based (pels pels & blocks). & blocks).

• Content

Content-

• dependent (objects

dependent (objects’ ’ motion & behavior). motion & behavior).

• Two main types of algorithms:

Two main types of algorithms:

• Lossless.

Lossless.

• Lossy

Lossy. .

Kasaei 7

Video Coding Video Coding

Kasaei 8

Video Coding Video Coding

• Different types of

Different types of redundancy redundancy to be to be removed/decreased in the compression algorithm removed/decreased in the compression algorithm include: include:

1. 1.

Spatial Spatial redundancy (correlation between neighboring redundancy (correlation between neighboring pixel values), pixel values),

2. 2.

Spectral Spectral redundancy (correlation between different redundancy (correlation between different color planes or spectral bands), color planes or spectral bands),

3. 3.

Temporal Temporal redundancy (correlation between different redundancy (correlation between different frames in a video sequence), frames in a video sequence),

4. 4.

Psychovisual Psychovisual redundancy (HVS), & redundancy (HVS), &

5. 5.

Coding Coding redundancy (entropy encoding). redundancy (entropy encoding).

Kasaei 9

Components of a Coding System Components of a Coding System

Kasaei 10

Model

Video Coding Techniques based Video Coding Techniques based

• n Different Source Models
• n Different Source Models

Kasaei 11

Waveform-based techniques Content-dependent-techniques

Statistical Characterization Statistical Characterization

• f Random Sources
• f Random Sources

Kasaei 12

A continuous random process (F

F).

.

Fn Fn (RV) f fn ( n (n n-

• th

th sample) sample) each realization

Statistical Characterization Statistical Characterization

• f Random Sources
• f Random Sources
• In statistical representation of image, each

In statistical representation of image, each pixel pixel is considered is considered as an as an RV RV. .

• We think of a given

We think of a given image image as a as a sample function sample function of an

• f an

ensemble of images ensemble of images. .

Kasaei 13

Ensemble of images or discrete random field (F

F).

Sample function, realization, or random image.

Fn Fn sample sample Fn Fn (RV) (RV)

Statistical Characterization Statistical Characterization

• f Random Sources
• f Random Sources
• Source: A random sequence (discrete

Source: A random sequence (discrete-

• time random process),

time random process),

• Ex 1: An

Ex 1: An image image that follows a certain statistics: that follows a certain statistics:

• F

Fn

n represents the

represents the possible value possible value ( (random value random value) ) of the

• f the n

n-

• th

th pixel of the image, pixel of the image, n n=( =(m,n m,n). ).

• f

fn

n represents the

represents the actual value actual value taken. taken.

• Ex 2: A

Ex 2: A video video that follows a certain statistics: that follows a certain statistics:

• F

Fn

n represents the

represents the possible value possible value of the

• f the n

n-

• th

th pixel of a video, pixel of a video, n n=( =(k,m,n k,m,n). ).

• f

fn

n represents the actual value taken.

represents the actual value taken.

• Continuous

Continuous-

• source

source: F : Fn

n takes continuous values (analog image).

takes continuous values (analog image).

• Discrete

Discrete-

• source:

source: F Fn

n takes discrete values (digital image).

takes discrete values (digital image).

Kasaei 14

Statistical Characterization Statistical Characterization

• f Random Sources
• f Random Sources
• Stationary source

Stationary source: statistical distribution is : statistical distribution is invariant to time invariant to time (space) shift. (space) shift.

• Probability distribution

Probability distribution: :

• Probability mass function

Probability mass function ( (pmf pmf) for discrete sources & ) for discrete sources & probability density function probability density function ( (pdf pdf) for continuous sources: ) for continuous sources:

• r
• r
• Joint

Joint pmf pmf or

• r pdf

pdf: :

• r
• r
• Conditional

Conditional pmf pmf or

• r pdf

pdf: :

• r

r

Kasaei 15

Statistical Characterization Statistical Characterization

• f Random Sources
• f Random Sources
• Independent & identically distributed

Independent & identically distributed ( (i.i.d i.i.d.) source .) source ( (memoryless memoryless): ): & &

• Markov process

Markov process: :

• More generally, an

More generally, an M M-

• th

th order Markov process is one in which a sample

• rder Markov process is one in which a sample

depends only on its previous depends only on its previous M M samples. samples.

• A Gaussian process is Markov (

A Gaussian process is Markov (Gauss Gauss-

• Markov process

Markov process), if the covariance ), if the covariance between two samples & has the form of: between two samples & has the form of:

Kasaei 16 1 2 1 2

( , ,..., ) ( ) ( )... ( )

N N

p f f f p f p f p f =

1 1 1 1

( | , ,..., ) ( )

M M M M

p f f f f p f

+ − +

=

1 1 1 1

( | , ,..., ) ( | )

M M M M M

p f f f f p f f

+ − +

=

n

F

m

F

2 ( )

( , )

n m n m

C F F δ ρ −

−

=

Information Content Information Content Characterization of Discrete RVs Characterization of Discrete RVs

• Entropy

Entropy of a discrete RV:

• f a discrete RV:
• Entropy is a measure of

Entropy is a measure of uncertainty uncertainty or

• r information

information content. content.

• Very uncertain (uniform distribution)

Very uncertain (uniform distribution) low information content (max low information content (max entropy). entropy).

• Joint entropy

Joint entropy of two discrete RVs:

• f two discrete RVs:
• Uncertainty of two RVs together:

Uncertainty of two RVs together:

Kasaei 17

(bits)

Information Content Information Content Characterization of Discrete RVs Characterization of Discrete RVs

• Conditional entropy

Conditional entropy between two RVs: between two RVs:

• Uncertainty of one RV given the other RV:

Uncertainty of one RV given the other RV:

• Mutual information

Mutual information between two RVs : between two RVs :

• Information provided by

Information provided by G G about about F F (reduction in required

(reduction in required bitrate bitrate to specify to specify F F): ):

Kasaei 18

Information Content Characterization Information Content Characterization

• f Discrete Sources
• f Discrete Sources
• N

N-

• th

th order entropy of a discrete stationary source:

• rder entropy of a discrete stationary source:
• M

M-

• th

th order

• rder conditional

conditional entropy of a discrete stationary source : entropy of a discrete stationary source :

Kasaei 19

N-fold Cartesian product of A A

Information Content Characterization Information Content Characterization

• f Discrete Sources
• f Discrete Sources
• Entropy rate

Entropy rate lossless coding bound. lossless coding bound.

• N

N-

• th

th order mutual information

• rder mutual information

lossy lossy coding bound. coding bound.

• The information provided by a block of

The information provided by a block of N N quantized samples ( quantized samples (G G) about ) about the original the original N N samples ( samples (F F). ).

Kasaei 20

Information Theory for Source Information Theory for Source Coding Coding

• Source coding

Source coding: Process of converting samples : Process of converting samples from a source into a stream of binary bits. from a source into a stream of binary bits.

• Channel coding

Channel coding: Further manipulating the : Further manipulating the source bits to add protection against source bits to add protection against transmission errors. transmission errors.

• Shannon

Shannon (the father of IT) theory established (the father of IT) theory established bounds on the minimal bounds on the minimal bitrate bitrate required to required to realize lossless & realize lossless & lossy lossy coding. coding.

Kasaei 21

Lossless Coding Lossless Coding (Binary Encoding) (Binary Encoding)

• Binary encoding is a necessary step in any coding system.

Binary encoding is a necessary step in any coding system.

• Applied to:

Applied to:

• riginal symbols (
• riginal symbols (e.g.,

e.g., image pixels) in a discrete source, image pixels) in a discrete source,

• converted symbols (

converted symbols (e.g., e.g., quantized transformed coefficients), or quantized transformed coefficients), or

• model parameters.

model parameters.

• Binary encoding process (scalar coding):

Binary encoding process (scalar coding):

Kasaei 22

Binary Encoding Codeword ci (bit length li) Symbol ai Probability Table pi bitrate (average bits/symbol)

Bound for Lossless Coding Bound for Lossless Coding

• Scalar lossless coding

Scalar lossless coding: :

• Assigns one codeword to

Assigns one codeword to each symbol each symbol at a time. at a time.

• Coded sequence must be

Coded sequence must be uniquely decodable uniquely decodable. .

• Problem: could differ from the entropy by up to 1 bit/symbol.

Problem: could differ from the entropy by up to 1 bit/symbol.

• Vector lossless coding

Vector lossless coding: :

• Assigns one codeword for each

Assigns one codeword for each group group of

• f N

N symbols. symbols.

• Larger

Larger N N lower rate (but higher complexity). lower rate (but higher complexity).

Kasaei 23

minimum bitrate minimal bits per vector minimal bits per symbol source entropy rate

Bound for Lossless Coding Bound for Lossless Coding

• Conditional lossless coding

Conditional lossless coding (context (context-

• based coding):

based coding):

• The codeword for the current symbol depends on the pattern (cont

The codeword for the current symbol depends on the pattern (context) ext) formed by the previous formed by the previous M M symbols. symbols.

• A separate codebook is designed for each possible context (based

A separate codebook is designed for each possible context (based on the

• n the

conditional distribution of the output symbol given this context conditional distribution of the output symbol given this context). ).

Kasaei 24

entropy of the M-th order conditional distribution under context m average minimal bitrate source entropy rate

Binary Encoding: Requirement Binary Encoding: Requirement

• A good code should be:

A good code should be:

• Uniquely decodable.

Uniquely decodable.

• Instantaneously decodable

Instantaneously decodable – – prefix code. prefix code.

Kasaei 25

Huffman Coding Huffman Coding

• Idea: more frequent symbols

Idea: more frequent symbols shorter shorter codewords codewords. .

• Algorithm:

Algorithm:

Kasaei 26

• Huffman coding generates prefix code. ☺
• Can be applied to one symbol at a time (scalar coding), or a group of symbols

(vector coding), or one symbol conditioned on previous symbols (conditional coding).

Huffman Coding Example: Huffman Coding Example: Source description Source description

• Four symbols:

Four symbols: “ “a a” ”, , “ “ b b” ”, , “ “ c c” ”, , “ “ d d” ”

• pmf

pmf: :

• 1

1st

st order conditional

• rder conditional pmf

pmf: :

• 2

2nd

nd order (joint)

• rder (joint) pmf

pmf: :

Kasaei 27

] 1154 . , 1703 . , 2143 . , 5000 . [ =

T

p

0938 . 1875 . * 5 . ) " /" " (" ) " (" ) " (" Ex. = = = a b q a p ab p

q(i|j) q(i|j) q(2|1) q(2|1)

Huffman Coding Example: Huffman Coding Example: Scalar Coding Scalar Coding

Kasaei 28

Huffman Coding Example: Huffman Coding Example: Vector Coding Vector Coding

Huffman Coding Example: Huffman Coding Example: Conditional Coding Conditional Coding

Kasaei 30

6922 . 1 , 8829 . 1 , 5016 . 1 7500 . 1 , 9375 . 1 , 5625 . 1

1 , " " , " " , " " , " " , 1 , " " , " " , " " , " " ,

= = = = = = = = = =

C d C c C b C a C C d C c C b C a C

H H H H H R R R R R

Arithmetic Coding Arithmetic Coding

• Basic idea:

Basic idea:

• Represent a sequence of symbols by an interval with length equal

Represent a sequence of symbols by an interval with length equal to its to its probability. probability.

• The interval is specified by its lower boundary (

The interval is specified by its lower boundary (l l), upper boundary ( ), upper boundary (u u), & ), & length length d d (=probability). (=probability).

• The codeword for the sequence is the common bits in binary

The codeword for the sequence is the common bits in binary representations of representations of l l & & u. u.

• The interval is calculated sequentially starting from the first

The interval is calculated sequentially starting from the first symbol: symbol:

• The initial interval is determined by the first symbol.

The initial interval is determined by the first symbol.

• The next interval is a subinterval of the previous one, determin

The next interval is a subinterval of the previous one, determined by the ed by the next symbol. next symbol.

Kasaei 31

Encoding: Decoding:

/2

3/8 - 1/4 = 1/8 1/8 / 2 = 1/16 1/4 + 1/16 = 5/16 Binary Representation:

Huffman vs. Arithmetic Coding Huffman vs. Arithmetic Coding

• Huffman coding:

Huffman coding:

• Converts a fixed number of symbols into a variable length codewo

Converts a fixed number of symbols into a variable length codeword. rd.

• Efficiency:

Efficiency:

• To approach entropy rate, must code a large number of symbols to

To approach entropy rate, must code a large number of symbols together. gether.

• Used in all image and video coding standards.

Used in all image and video coding standards.

• Arithmetic coding:

Arithmetic coding:

• Converts a variable number of symbols into a variable length cod

Converts a variable number of symbols into a variable length codeword. eword.

• Efficiency:

Efficiency:

• Can approach the entropy rate by processing one symbol at a time

Can approach the entropy rate by processing one symbol at a time. .

• Easy to adapt to changes in source statistics.

Easy to adapt to changes in source statistics.

• Integer implementation is available.

Integer implementation is available.

• Used as advanced options in image and video coding standards.

Used as advanced options in image and video coding standards.

Kasaei 33

Summary Summary

• Coding system:

Coding system:

• Original data

Original data model parameters model parameters quantization quantization

• binary encoding.

binary encoding.

• Waveform

Waveform-

• based vs. content

based vs. content-

• dependent coding.

dependent coding.

• Lossless coding:

Lossless coding:

• Bitrate

Bitrate bounded by entropy rate of the source. bounded by entropy rate of the source.

Kasaei 34

Summary Summary

• Huffman coding:

Huffman coding:

• Scalar, vector, conditional coding.

Scalar, vector, conditional coding.

• Can achieve the bound only if a large number of

Can achieve the bound only if a large number of symbols are coded together. symbols are coded together.

• Huffman coding generates prefix code

Huffman coding generates prefix code (instantaneously decodable). (instantaneously decodable).

• Arithmetic coding:

Arithmetic coding:

• Can achieve the bound by processing one symbol at a

Can achieve the bound by processing one symbol at a time. time.

• More complicated than scalar or short vector

More complicated than scalar or short vector Huffman coding. Huffman coding.

Kasaei 35

Homework 6 Homework 6

• Reading assignment:

Reading assignment:

• Sec. 8.1
• Sec. 8.1-
• 8.5 (excluding Sec. 8.3.2)

8.5 (excluding Sec. 8.3.2)

• Written assignment:

Written assignment:

• Prob. 8.1,8.3,8.5,8.6,8.7
• Prob. 8.1,8.3,8.5,8.6,8.7
• (for Prob. 8.7, you do not have to complete the

(for Prob. 8.7, you do not have to complete the encoding for the entire sequence. Do the first few encoding for the entire sequence. Do the first few letters.) letters.)

Kasaei 36

The End The End