In the name of Allah the compassionate, the merciful p , f D D - - PowerPoint PPT Presentation

In the name of Allah

the compassionate, the merciful p , f

D V P DIGITAL VIDEO PROCESSING

• S. Kasaei

Room: CE 307 Department of Computer Engineering Sh if U i it f T h l Sharif University of Technology E-Mail: skasaei@sharif.edu Webpage: http://sharif.edu/~skasaei Webpage: http://sharif.edu/ skasaei

• Lab. Website: http://ipl.ce.sharif.edu

ACKNOWLEDGMENT ACKNOWLEDGMENT

Most of the slides used in this course have been provided by: Prof Yao Wang (Polytechnic provided by: Prof. Yao Wang (Polytechnic University, Brooklyn) based on the book: Video Processing & Communications g by: Yao Wang, Jom Ostermann, & Ya-Oin Zhang Prentice Hall, 1st edition, 2001, ISBN: 0130175471. [SUT Code: TK 5105 2 W36 2001] [SUT Code: TK 5105 .2 .W36 2001].

CHAPTER 14

Error Control in Video Communications

OUTLINE OUTLINE

Necessity & challenge for error control

Kas

y g

Characteristics of typical applications & networks Overview of techniques

saei

Error resilient encoding Error concealment decoding Encoder-decoder-network interactive error control

6

VIDEO COMMUNICATION SYSTEM VIDEO COMMUNICATION SYSTEM

Kasaei

A typical video communication system.

7

CHALLENGE FOR VIDEO COMMUNICATIONS COMMUNICATIONS

Effective video communication includes:

Kas

Reduction of video data rate. Handling errors & losses in communication networks.

Data communications are not usually subject to

saei

Data communications are not usually subject to

strict delay constraints.

Can be handled using network protocols; that use

resubmission to ensure error-free delivery. y

Transmission error categories include:

Random bit errors.

Caused by imperfections of physical channels;

bit inversion/insertion/deletion.

Erasure errors.

C d b k t l t i k t t k l b t

Caused by packet lost in packet networks, long burst errors

in storage media due to physical defects, system failures/link downs, random bit errors in VLC coded streams.

8

CHALLENGE FOR VIDEO COMMUNICATIONS COMMUNICATIONS

Real networks are unreliable

Kas

In wireless networks: random bit errors, long burst

errors, & possibly link downs.

In the Internet: packet loss & variable delay due to

saei

In the Internet: packet loss & variable delay due to network congestion.

For real-time applications:

Excessive delay = loss Excessive delay = loss.

Real networks are heterogeneous in bandwidth

& reliability & reliability.

Video signals are delay-sensitive:

O l i i f

One cannot rely on retransmission for error

control because of the stringent delay requirement!

9

DIFFICULTIES OF CONVENTIONAL SOURCE CODING TECHNIQUE CONVENTIONAL SOURCE CODING TECHNIQUE

Conventional source coding techniques difficulties:

Kas

g q

Optimal performance is obtained only for fixed rates &

perfect channels.

saei

It results in a poor reconstruction quality when parts of

coded data are lost.

10

DIFFICULTIES OF CONVENTIONAL SOURCE CODING TECHNIQUE CONVENTIONAL SOURCE CODING TECHNIQUE

Compressed video data is very sensitive to

Kas

p y transmission errors because of utilizing:

Variable length coding (VLC). Temporal predictive coding. saei

p p g

Spatial predictive coding.

All contribute to error propagation within the All contribute to error propagation within the

same frame as well as the following frames:

One bit error or packet loss can render the following

received data useless received data useless.

11

SPATIOTEMPORAL ERROR PROPAGATION SPATIOTEMPORAL ERROR PROPAGATION

Kasaei

Illustration of spatiotemporal error propagation Illustration of spatiotemporal error propagation.

12

DRIFT PROBLEM (REFERENCE MISMATCH) (REFERENCE MISMATCH)

Motion compensated temporal prediction

Kas

Motion compensated temporal prediction

should be retained to preserve the coding efficiency.

saei

efficiency.

But, loss in a previous frame can cause

mismatch between the reference frame used in the encoder & that in the decoder.

Encoder & decoder will be out of sync.

13

DRIFT PROBLEM (REFERENCE MISMATCH) (REFERENCE MISMATCH)

Kasaei

Di t ti

refere

Satellite dish

Distortion

ref ence

Transmission

ference encoder decoder

14

EFFECT OF TRANSMISSION ERRORS EFFECT OF TRANSMISSION ERRORS

Kasaei

coded, no loss 3% loss 5% loss 10% loss

15

Reconstructed video frames using a H.263 coded sequence, subject to packet losses.

CHANNEL CODING BASICS CHANNEL CODING BASICS

Channel coding: forward error correction

Kas

C g f (FEC):

Adding redundancy bits on compressed source

bit to e able e

• detectio & co

ectio

saei

bits to enable error detection & correction.

Simple example:

Adding a parity check bit at the end of a block of data

stream can detect all single bit errors.

Channel coding rate:

For every k source bits, add l channel bits, to create

n=k+l bits channel coding rate r=k/n.

Well designed code (e.g., Reed-Solomon (RS) code) can

correct t=l/2 error bits in each n-bit block.

16

CHANNEL CODING BASICS CHANNEL CODING BASICS

Classical Shannon information theory states that:

Kas

One can separately design the source & channel coders

to achieve error-free delivery of a compressed bit stream, as long as the source is represented by a rate below the channel capacity

saei

below the channel capacity.

Source coding minimizes the bitrate necessary to satisfy a

distortion criterion (Shannon rate-distortion theory).

Channel coding adds just enough redundancy bits to

d th h l t t th itt d l l reduce the raw channel error rate to the permitted level.

Such ideal error-free delivery requires infinite

delays in implementing FEC delays in implementing FEC.

It is only valid for stationary source & channel. Requires processing of infinitely long blocks of data.

Delay = infinity! Delay = infinity! 17

FEC FOR VIDEO TRANSMISSION FEC FOR VIDEO TRANSMISSION

For wireless networks, FEC is necessary to reduce raw bit

error rates

Kas

error rates.

For the Internet, errors are mainly due to congestion-

caused packet losses.

FEC can be applied across packets to correct/detect packet

losses

saei

losses.

Unequal error protection (UEP): Using stronger channel codes (r=k/n smaller) for more

“important” bit streams (base layer).

Best implemented with rate compatible punctured Best implemented with rate-compatible punctured

convolutional (RCPC) code.

18

TRANSMISSION ERROR RESILIENCE TRANSMISSION ERROR RESILIENCE

Joint source & channel coding is often a

Kas

g more riable scheme.

Which allocates a total amount of redundancy

between the source & channel coding.

saei

g

All error resilient encoding methods work

d thi i under this promise.

This is usually accomplished by carefully This is usually accomplished by carefully

designing both the predictive coding loop & the VLC, to limit the extend of error propagation propagation.

19

ERROR CONCEALMENT ERROR CONCEALMENT

When a part of image is missing due to

i i h d d i h

Kas

transmission errors, the decoder estimates them based on surrounding received samples.

This is accomplished by using the inherent correlation

among spatially & temporally adjacent samples

saei

among spatially & temporally adjacent samples.

It does not employ any additional bitrates. ☺

But adds computational complexity at the decoder But, adds computational complexity at the decoder.

To facilitate error concealment in the decoder, the

codec & the network transmission protocol must p

• cooperate. For example:

Important parts should be assigned as a more stringent

set of QoS parameters. N t k id f db k h l

Network may provide a feedback channel.

The encoder does not use this part for prediction.

20

TYPICAL VIDEO APPLICATIONS & COMMUNICATION NETWORKS & COMMUNICATION NETWORKS

Some basic definitions:

Kas

Latency: maximally allowed average end-to-end delay. Jitter: delay variation.

R l ti d li d d t t f d t

saei

Real-time delivery: compressed data are transferred at

a speed that matches the coded video source rate.

21

QOS REQUIREMENTS

OF TYPICAL VIDEO APPLICATIONS OF TYPICAL VIDEO APPLICATIONS

Video applications consist of:

Kas

pp

Interactive two-way visual communications. One-way video streaming.

O id d l di

saei

One-way video downloading.

No difference from file downloading.

22

INTERACTIVE TWO-WAY VISUAL COMMUNICATIONS COMMUNICATIONS

Examples include:

T l f i Vid l h Vi l l

Kas

Teleconferencing, Video telephony, Virtual classrooms. They have very stringent delay requirements.

saei

The latency & jitter must be kept within a certain

limit.

For intercontinental telephone conversations (&

video conferencing), the ITU-T G.114 standard recommends:

<=150 ms (one way) desired <=150 ms (one way) desired. 150-400 ms can be acceptable. >400 ms not acceptable. Audio & video must be in sync to maintain lip

h i ti synchronization.

Both encoding & decoding must be completed in real-time.

23

INTERACTIVE TWO-WAY VISUAL COMMUNICATIONS COMMUNICATIONS

Only low-to-intermediate video quality is

Kas

O y q y required: ☺

QCIF at 5-10 fps acceptable for video telephony.

CIF f i f f id

saei

CIF at 10-20 fps satisfactory for video

conferencing.

Moderate amount of compression/transmission

p artifacts can be tolerated.

R

id h li it d ti i t

Raw video has limited motion easier to

code & conceal errors. ☺

24

ONE-WAY VIDEO STREAMING ONE WAY VIDEO STREAMING

Examples include:

Kas

p

TV broadcasting, Multicast of a conference/

event, Video streaming over the Internet.

saei

Except for live broadcast/multicast/

unicast, it can precompress the video, but decoding must be done in real-time decoding must be done in real-time.

Initial play-out delay can be up to a few

seconds: seconds:

Receiver uses a large smoothing buffer to store

several seconds of video frames before starting to display the first received frame to display the first received frame.

25

ONE-WAY VIDEO STREAMING ONE WAY VIDEO STREAMING

Bitrate (video quality) can vary widely

Kas

( q y) y y depending on the application.

saei

Recipients of the same video source may

be connected to the network with different access links (e g wireless modem to 100 access links (e.g., wireless modem to 100 mbps fast Ethernet) & the receiving terminal may have varying computing power (palm vs. laptop vs. desktop):

Scalable coding is desired.

26

ONE-WAY VIDEO DOWNLOADING ONE WAY VIDEO DOWNLOADING

Precoded video (& audio) is downloaded to a

Kas

( ) destination.

saei

Receiving terminal starts to play the

material when the entire video has arrived.

It is the least demanding application in

terms of delay requirement terms of delay requirement.

However, excessive delay may cause the

receiver to abort downloading.

27

ONE-WAY VIDEO DOWNLOADING ONE WAY VIDEO DOWNLOADING

Scalable coding & progressive transmission

Kas

g p g are desirable.

saei

Retransmission can be used to handle any

lost or erroneously delivered part of the data data.

No special error control mechanisms are No special error control mechanisms are

necessary beyond those typically used for data transfer.

28

MAJOR TYPES

OF COMMUNICATION NETWORKS OF COMMUNICATION NETWORKS

Public switched telephone network (PSTN):

Kas

p ( )

Modem link, 56 kbps, QCIF. Asymmetric digital subcarrier loop (ADSL):

saei

Asymmetric digital subcarrier loop (ADSL): Streaming MPEG-1 & MPEG-2 movies through

video-on-demand services.

Integrated service digital network (ISDN): Circuit switching (a particular conversation once Circuit switching (a particular conversation once

the connection is established), very reliable, px64 kbps (p=1-24), QCIF @ 5-10 fps for videophone, CIF@ 15-30 fps for video conferencing CIF@ 15-30 fps for video conferencing.

29

MAJOR TYPES

OF COMMUNICATION NETWORKS OF COMMUNICATION NETWORKS

Broadband ISDN (B-ISDN):

Kas

( )

Virtual ATM packet-switching using fixed-size small cells,

quite reliable.

Cell loss rate (CLR) 10-6 -10 -4, >=155 mbps, real-time

saei

( ) , p , applications. Internet:

Is composed of multiple physical networks interconnected

by computers (routers).

Most widely used physical network for supporting local

area network (LAN) is the Ethernet which has a rate of 10-100 mbps.

Interconnected LANs in a local region form a metropolitan

area network (MAN) area network (MAN).

Several interconnected MANs form a wide area network

(WAN).

30

MAJOR TYPES

OF COMMUNICATION NETWORKS OF COMMUNICATION NETWORKS

Internet:

Kas

Transport of packets is governed by the TCP/IP protocol

suite which includes:

Internet protocol (IP); the lower layer protocol on top of

saei

Internet protocol (IP); the lower layer protocol on top of

which UDP & TCP operate.

User datagram protocol (UDP); for unreliable

connectionless packet-delivery service. p y

Transmission control protocol (TCP); for reliable stream

service.

Guarantees delivery by using automatic retransmission request

(ARQ) (ARQ).

31

MAJOR TYPES

OF COMMUNICATION NETWORKS OF COMMUNICATION NETWORKS

Internet:

Kas

The Internet & its associated TCP/IP protocol was

• riginally designed for data communications which are

not delay sensitive.

saei

y

For real-time applications the retransmission

mechanism in TCP is not appropriate.

New protocols have been developed for real-time data

p p transport:

Real-time transport protocol (RTP) & its companion

real-time control protocol (RTCP); which are periodically f db k h l b h k sent across a feedback channel to probe the network status.

Real-time streaming protocol (RTSP).

Et

Etc.

32

MAJOR TYPES

OF COMMUNICATION NETWORKS OF COMMUNICATION NETWORKS

Wireless networks:

Kas

W

Cellular networks: mobile voice, wireless

modems, under 20 kbps, can be connected to a packet based wireless data network (64 384

saei

packet-based wireless data network (64-384 kbps) [vehicular].

Wireless LAN: mobile IP, wireless ATM, IEEE

802 11b 11 b 802 11 56 b [i d ] 802.11b: 11 mbps, 802.11a: 56 mbps [indoor].

Broadband wireless IP networks: access to the

Internet through small rooftop antennas communicating to a base station using microwave radio, multichannel multipoint distribution service (MMDS): 1mbps, local multipoint distribution service (LMDS): 100-600 mbps [indoor].

33

CHARACTERISTICS OF MAJOR VIDEO COMMUNICATIONS APPLICATIONS COMMUNICATIONS APPLICATIONS

Kasaei

34

H.262: MPEG-2, H.222.0: MPEG2 systems, BER: bit-error rate.

ERROR CONTROL TECHNIQUES

IN VIDEO COMMUNICATIONS IN VIDEO COMMUNICATIONS

Error control techniques include:

Kas

Error control techniques include: Transport-level error control. Error-resilient encoding (ERE).

saei

g ( )

Decoder error concealment (DEC). Encoder-decoder-network interactive error

control (EDNIER). control (EDNIER).

Decoder

(Error Concealment)

Encoder

(Error Resilient)

Network

(Transport)

35

TRANSPORT-LEVEL ERROR CONTROL TRANSPORT LEVEL ERROR CONTROL

Transport-level error control techniques:

Kas

Transport level error control techniques:

Applied on coded video streams. Forward error correction (FEC):

• Error detection & correction through FEC; increases

saei

g ; transmission overhead.

Error-resilient packetization & multiplexing:

• Error can be isolated within a small region, unequal-sized

packets. packets.

FEC & error-resilient techniques are low level error

control mechanisms.

36

TRANSPORT-LEVEL ERROR CONTROL TRANSPORT LEVEL ERROR CONTROL

Delay-constrained retransmission:

Kas

y

• Retransmission of lost packets.
• For non-real-time data transmission:

Automatic retransmission request (ARQ).

• For real-time data transmission:

saei

• For real-time data transmission:

Delay-constrained retransmission. Combination of layered coding with prioritized retransmission.

By placing base layer packets in front of the transmission queue

to increase the number of retransmission trials for the base layer. y

Sending multiple copies of a lost packet.

Unequal error protection:

• Picture header & other side information are protected to be

delivered with a much lower error rate delivered with a much lower error rate.

• Retransmission & strong FEC can be applied on the base

layer.

• Using different transport protocols.

37

ERROR-RESILIENT ENCODING ERROR RESILIENT ENCODING

Error resilient encoding (ERE) techniques:

Kas

Error-resilient encoding (ERE) techniques: Source coding methods (robust to transmission errors):

saei Basic idea: intentionally insert redundancy in source

coding to help the decoder to recover bit errors.

Design goal: minimize the redundancy to achieve a

g g y desired level of resilience.

These techniques include:

Error isolation (part of H.263/MPEG-4):

(p )

Inserting resynchronization markers, data partitioning. Robust binary encoding (part of H.263/MPEG-4): Error resilient entropy coding (EREC), reversible VLC

(RVLC).

38

ERROR-RESILIENT ENCODING ERROR RESILIENT ENCODING

Kasaei

RVLC codewords can be parsed in both forward & backward directions (making it possible to recover more data from a corrupted data stream). MPEG-4 syntax is assumed in the figure, but the basic principle holds true for

39

• ther RVLC-coded data as well.

ERROR-RESILIENT ENCODING ERROR RESILIENT ENCODING

Error-resilient prediction (part of H.263/MPEG-4):

Kas

Insertion of intra-blocks or frames, independent segment

prediction (spatiotemporal prediction within same segments).

Joint source & channel coding:

Q ti ti d t di g t th d & FEC

saei

Quantization and entropy coding at the source coder & FEC

and modulation schemes at the channel coder.

40

ERROR-RESILIENT ENCODING ERROR RESILIENT ENCODING

Layered coding with unequal error protection:

Kas

y g q p

The most promising approach for combating channel

errors.

Base layer provides acceptable quality, enhancement saei

layer refines the quality.

Base layer transmitted in error-free channel (strong

FEC + ARQ).

Good for a network with differentiated service

(do NOT exist today over the Internet, may become part of emerging wireless standards).

Problems: Problems:

• Any error in the base layer causes severe

degradation.

• Repetitive ARQ may incur unacceptable delay,

Repetitive ARQ may incur unacceptable delay, strong FEC may be too complex or cause extra delay.

• The enhancement layer is useless by itself.

41

ERROR-RESILIENT ENCODING ERROR RESILIENT ENCODING

Kasaei

Block diagram of a system using layered coding & prioritized transport.

42

ERROR-RESILIENT ENCODING

(GENERIC TWO DESCRIPTION CODER) (GENERIC TWO DESCRIPTION CODER)

Kasaei

Decoder 1

Decoded Signal S1

MDC Encoder Channel 1 Decoder Ch l 2

Source Signal

from S1 (D1). D d d S1 (R1)

Decoder Channel 2

Decoded Signal from S1 & S2 (D0). S2 (R2)

Decoder 2

Decoded Signal from S2 (D2) Balanced MDC: R1=R2, D1=D2. R: bitrate, D: distortion, S: signal. 43 (D2).

MDC Decoder

R: bitrate, D: distortion, S: signal.

ERROR-RESILIENT ENCODING ERROR RESILIENT ENCODING

Multiple-description (bit stream) coding (MDC):

Kas Several correlated descriptions (bit streams) of the

same source signal are generated & transmitted over separate channels.

Any description provides low but acceptable quality. saei Additional descriptions provide incremental

improvements.

No retransmission required low delay ☺ But, less correlation reduced coding efficiency

But, less correlation reduced coding efficiency

Design goal: maximize the robustness to channel

errors at a permissible level of redundancy.

Assumptions:

• Existence of multiple channels between source & destination
• Existence of multiple channels between source & destination.
• Independent error & failure events.
• Probability that all channels fail simultaneously is low.
• Good model for the Internet & wireless networks when data

are properly packetized & interleaved. p p y p

44

REDUNDANCY RATE-DISTORTION REDUNDANCY RATE DISTORTION

Kas

D0 D1

saei

Rate-Distortion (RD) Function

D1

Redundancy Rate-Distortion (RRD) Function

D0

*

D R

*( )

(SDC) D R

0 ( ) (MDC)

ρ D D

1

( ; )

*

ρ Design criteria for MD coders:

R R* R

D R

0 ( ) (SDC)

ρ ρ Design criteria for MD coders:

Minimize D1 for a given ρ, for fixed R* or D0

*.

Minimizing the average distortion given channel loss rates, for

a given total rate a given total rate.

Can easily vary the ρ vs. D1 trade-off to match

network conditions.

45

CHALLENGE IN DESIGNING MD VIDEO CODER CHALLENGE IN DESIGNING MD VIDEO CODER

To achieve a high coding efficiency, the

Kas

g g y, encoder should retain the temporal prediction loop. Sh ld h di i b b d

saei

Should the prediction be based on

reconstruction from both descriptions or individual descriptions? individual descriptions?

Prediction based on two-description

reconstruction:

Higher prediction efficiency. Mismatch problem at decoder.

46

CHALLENGE IN DESIGNING MD VIDEO CODER CHALLENGE IN DESIGNING MD VIDEO CODER

Prediction based on single-description

Kas

g p reconstruction:

Lower prediction efficiency.

N i h bl

saei

No mismatch problem. How to provide a proper trade-off between

prediction efficiency & mismatch: prediction efficiency & mismatch:

Predict based on two-description

reconstruction, but explicitly code the i t h mismatch error.

47

VIDEO REDUNDANCY CODING

IN H.263+ IN H.263+

Coding even frames & odd frames as separate

Kas

Coding even frames & odd frames as separate

threads:

High redundancy (~30%) due to reduced prediction

i b f l di t b t f

saei

gain; because of longer distance between frames.

Hard to vary the redundancy based on channel loss

characteristics.

Even frames Even frames.

48

Odd frames.

MULTIPLE DESCRIPTION MOTION COMPENSATION (MDMC) (WANG & LIN 2001) COMPENSATION (MDMC) (WANG & LIN, 2001)

A description contains even (or odd) frames only,

but each frame is predicted (central predictor) from

Kas

but each frame is predicted (central predictor) from both even & odd past frames.

Code the central prediction error:

saei

Code the central prediction error:

Sufficient if both descriptions are received.

To avoid mismatch a side predictor for even frames To avoid mismatch, a side predictor for even frames

predicts only from the past even frame, & the mismatch signal (difference between central & side prediction) is also coded.

The predictors & the mismatch error quantizer

control the redundancy of the coder, & can be designed based on the channel loss characteristics. designed based on the channel loss characteristics.

49

SPECIAL CASE: TWO-TAP PREDICTOR

Kas

MV1

1

a MV2 ,

2

a

saei

~ ) ( ˆ ) ( : error prediction Central ) 2 ( ~ ) 1 ( ~ ) ( ˆ : predictor Central

2 1

→ − = − + − = (n) e n n (n) e n a n a n ψ ψ ψ ψ ψ MV1 ,

1

a MV2 MV1 ~ ~ : Send ~ ) ( ˆ ) ( ˆ : error Mismatch ) 2 ( ~ ) ( ˆ : predictor Side ) ( ) ( p

1 1 1 1 3 1

→ − − = − = (n) e (n) e (n) e (n) q n n (n) e n a n ( ) ( ) ψ ψ ψ ψ ψ ψ

n-3 n-2 n-1 n

1 , 1 : predictor leaky

• Non

MV2 MV1, , , : Send

3 2 1 1

= = + a a a (n) e (n) e

)) 2 ( ) 1 ( b th (h i d d i ti b th If

MV2 ,

3

a

)) 2 (

• nly

(have received is n descriptio

• ne

If ) ( ) ( ~ ) ( ˆ ) ( )) 2 ( ), 1 ( both (have received ns descriptio both If

1

n n q n (n) e n n n n − + = + = − − ψ ψ ψ ψ ψ ψ

50

) ( ) ( ~ ~ ) ( ˆ ) (

1 1 1 1

n q n (n) e (n) e n n + = + + = ψ ψ ψ

RRD PERFORMANCE

OF VRC & MDMC OF VRC & MDMC

Kasaei

51

SAMPLE RECONSTRUCTED FRAMES

(10% RANDOM PACKET LOSS) (10% RANDOM PACKET LOSS)

Kasaei

MDMC

52

VRC

DECODER ERROR CONCEALMENT DECODER ERROR CONCEALMENT

Decoder error concealment (DEC):

Kas

Decoder error concealment (DEC):

Due to various constrains (coding delay,

implementation complexity, availability of a good d l) d d id bi ill

saei

source model) a coded video bit stream still possesses a certain degree of statistical redundancy.

Error concealment recovers lost/damaged regions

based on image/video characteristics & human visual system (HVS) properties at the decoder. y ( ) p p

Decoder optimization issue, is not a part of video

coding standard! D d th k t diff i th i

Decoders on the market differ in their error

concealment capabilities.

53

DECODER ERROR CONCEALMENT DECODER ERROR CONCEALMENT

Kasaei

Without concealment. With concealment.

54

DECODER ERROR CONCEALMENT DECODER ERROR CONCEALMENT

Basic idea:

Kas Recover damaged regions by interpolating from

surrounding (in the same frame & in nearby frames) regions.

Basic error concealment techniques include: saei Basic error concealment techniques include:

Recovery of texture information: motion-compensated

temporal interpolation, spatial interpolation, maximally smooth recovery (spatiotemporally smoothly connected pels). pels).

Motion-compensated temporal interpolation:

• Replaces a damaged MB by its corresponding MB in the

reference frame. If the MV is also lost it has to estimate the MV first;

• If the MV is also lost, it has to estimate the MV first;

typically by copying the MV of the MB above.

• Simple & quite effective, if the data was appropriately

partitioned.

Maximally smooth recovery (Wang/Zhu 1993): Maximally smooth recovery (Wang/Zhu, 1993):

• Estimates the missing DCT coefficients in a block so

that a combination of spatial & temporal smoothness measure is maximized.

55

DECODER ERROR CONCEALMENT DECODER ERROR CONCEALMENT

Recovery of coding modes & motion vectors:

Kas Mean, median, set to zero.

Syntax-based repair:

Directly detects & repairs errors in the bit stream (due saei

to few valid combinations of codewords).

56

ENCODER-DECODER-NETWORK ERROR CONTROL ERROR CONTROL

Encoder-decoder-network interactive error control

Kas

Encoder decoder network interactive error control

(EDNIER):

Sender & receiver cooperate in the process of error

control using a backward channel from the decoder to

saei

g the encoder.

It can be realized at either the source coding or the

transport-level.

At the source coder, coding parameters can be adapted

based on the feedback information from the decoder.

At the transport-level, the feedback information can be

employed to change the percentage of the total employed to change the percentage of the total bandwidth used for FEC or retransmission.

Feedback message is not a part of the video syntax but

is transmitted in a different layer of the protocol stack (in which control information is exchanged) (in which control information is exchanged).

Feedback channel not necessarily involves extra

coding delay.

57

ENCODER-DECODER-NETWORK ERROR CONTROL ERROR CONTROL

B i d d d t k i t ti t l

Kas

Basic encoder-decoder-network interactive error control

techniques include:

Coding-parameter adaptation based on

h l di i i di ll i &

saei

channel conditions: periodically estimate & update the QoS parameters (bandwidth, delay, packet loss rates), appropriately y, p ), pp p y determine the coding parameters (intra- block rate, frequency of synchronization markers range of prediction) markers, range of prediction).

Changes intra-rate based on average loss rates.

58

ENCODER-DECODER-NETWORK ERROR CONTROL ERROR CONTROL

Reference picture selection based on feedback

Kas

Reference picture selection based on feedback

information (part of H.263/MPEG-4): decides to code the next P-frame relative not to the most recent frame:

Following a damaged frame (feedback info from

i ) d g d i f

saei

receiver), uses undamaged previous frame as reference frame for temporal prediction.

Error tracking based on feedback information:

determines which MBs are affected following a lost MB (feedback info), avoids using those MBs as reference pixels.

Requires a feedback channel, not necessarily involving

extra coding delay. g y

Retransmission without waiting. Multiple path transport with multiple bit stream coding.

59

ENCODER-DECODER-NETWORK ERROR CONTROL ERROR CONTROL

Kasaei

Illustration of error propagation when error tracking is used & damaged blocks are intra-coded upon the receipt of the negative acknowledgement (NACK) information .

60

VIDEO TRANSPORT USING PATH DIVERSITY VIDEO TRANSPORT USING PATH DIVERSITY

Kas

Wireless ad hoc networks

saei

Wireless ad hoc networks, cellular/WLAN with MIMO antennas, Internet with different relay servers.

L a p t o p c o m p u t e r L a p t o p c o m p u t e r

A

MD/Layered Encoder

L a p t o p c o m p u t e r L a p t o p c o m p u t e r L a p t o p c o m p u t e r

A B

MD/Layered Decoder

61

L a p t o p c o m p u t e r

WHY USING MULTIPLE PATHS WHY USING MULTIPLE PATHS

Enables sending high bandwidth signal

Kas

g g g (video) that exceeds the capacity of a single path (e.g., low bandwidth wireless links).

saei

Can withstand individual path failures in

wireless networks or excessive delay due to ti ti l th i th congestion on a particular path in the Internet.

Enables traffic dispersion & load balancing,

which in turn helps to reduce congestion & consequently packet losses consequently packet losses.

62

WHY USING MULTIPLE PATHS WHY USING MULTIPLE PATHS

If one path is known to be better than the

Kas

If one path is known to be better than the

• ther, or if it is feasible to set up such a

path, use layered coding.

saei

If the paths are symmetric in QoS & no

p y Q reliable path can be set up, use MDC.

63

VIDEO OVER AD HOC NETWORKS

JOINT WORK WITH SHIVENDRA PANWAR SHUNAN LIN SHIWEN MAO JOINT WORK WITH SHIVENDRA PANWAR, SHUNAN LIN, SHIWEN MAO

Wireless ad hoc networks:

Kas

Has no fixed infrastructure, peer to peer. Neighboring nodes are continuously changing.

saei

Multiple paths exit between two end users &

many proposed routing protocols return multiple paths to destination.

A path may become invalid during a connection

due to a link down.

Proposed solution: Integration of multistream

coding with multipath transport.

64

VIDEO OVER AD HOC NETWORKS

JOINT WORK WITH SHIVENDRA PANWAR SHUNAN LIN SHIWEN MAO JOINT WORK WITH SHIVENDRA PANWAR, SHUNAN LIN, SHIWEN MAO

Scope of the Project:

Kas

p j

Set-up/update an active path set (Haas,

Cornell).

QoS monitoring of existing paths (Panwar

saei

QoS monitoring of existing paths (Panwar,

Poly).

MDC/LC based on QoS parameters of active

th (W P l ) paths (Wang, Poly).

Testbed development.

65

PROPOSED SOLUTION: MULTISTREAM CODING + MULTIPATH MULTISTREAM CODING + MULTIPATH TRANSPORT

Kasaei

66

HOW TO GENERATE MULTIPLE SOURCE

STREAMS? STREAMS?

Trade-off between coding efficiency (due to less

Kas

g y ( correlation) & error-resilience.

We examine 3 types of coding & transport

saei

yp g p schemes:

Multiple description coding. Layered coding with selective ARQ.

y g Q

Reference picture selection based on channel feedback. These schemes differ in:

Requirement for a feedback channel.

q

Delay. Buffer requirement. Video quality under different channel loss patterns.

67

LAYERED CODING WITH SELECTIVE ARQ LAYERED CODING WITH SELECTIVE ARQ

Kas

Lost Base Layer

saei

Better Path

Base layer is sent on the better path, lost base layer packets are retransmitted over the enhancement layer path, while dropping the corresponding enhancement layer

68

• packets. Redundancy is due to prediction using base layer reconstruction only.

REFERENCE PICTURE SELECTION REFERENCE PICTURE SELECTION

Kas

E

saei

Even Frames

Referred to

Odd Frames

Not Referred to Referred to Even/odd frames are sent on separate paths. Predicts damaged frames based on NACK on each path & uses undamaged frames as reference pictures

69

frames as reference pictures. Compatible with the RPS option in H.263+ standard. ☺

SIMULATION RESULTS

BASED ON MARKOV LINK MODELS BASED ON MARKOV LINK MODELS

Path model: Each path consists of 3 links, chosen

Kas

p , randomly from an available link pool.

Each link is modeled by a 3-state Markov chain (down,

poor, good).

saei

The path is updated frequently.

We run a large set of tests to compare the 3

schemes under different packet loss patterns. schemes under different packet loss patterns.

We investigate the influence of the following

channel characteristics:

Mea lo ate

Mean loss rate. Mean burst length. Symmetric vs. asymmetric paths.

70

When no feedback or retransmission is allowed, MDC

is significantly better than LC.

Otherwise, LC+ARQ & RPS are better:

RPS is best at low loss rates.

LC ARQ i b tt t hi h l t ( t th f

LC+ARQ is better at higher loss rates (at the expense of

extra delay).

Simulation Results based on Markov Link Models

Performance of all 3 schemes improves gradually when

Kas

Performance of all 3 schemes improves gradually when

the burst length increases:

When the loss rate is the same, longer bursts means fewer

f ff t d (b t i b tt th d

saei

frames are affected (bursty error is better than random error for compressed video!)

Longer burst lengths increase the diversity gain from using

2 th 2 paths.

This trend reverses when the burst length exceeds more

than 1 frame time.

When the average loss rate is the same, the 3 schemes

perform similar in the symmetric vs. asymmetric cases.

72

OPNET SIMULATION RESULTS OPNET SIMULATION RESULTS

Kas

• 16 nodes, 600 m by 600 m, 10 m/s, transmission range=250 m, multipath

ti d i ti (DSR)

saei

routing dynamic source routing (DSR):

73

Mobility can actually help!

A WIRELESS VIDEO STREAMING TESTBED A WIRELESS VIDEO STREAMING TESTBED

Kasaei

74

The current testbed consists of 4 mobile nodes: 1 source, 1 destination, 2 relays.

MORE ON THE TESTBED MORE ON THE TESTBED

Testbed Setup:

Kas

Testbed Setup:

Each mobile node is a Thinkpad notebook computer

equipped with IEEE 802.11b cards working in the ad hoc mode.

saei

The source node sends a pre-encoded compressed bit

stream (MDMC or LC) to the destination node via 2 separate relays. Th d ti ti d bl th i d th k t

The destination node assembles the received the packets

from 2 paths, decodes, displays the video in real-time (10 fps).

With the LC scheme the destination node detects & With the LC scheme, the destination node detects &

requests retransmission of lost BL packets, the source node resend appropriate BL packets & drops corresponding EL packets.

75

MORE ON THE TESTBED MORE ON THE TESTBED

Results:

Kas

Results:

Tested when the nodes are placed in a Polytechnic

building.

With slow moving (walking) of the nodes & limited

saei

g ( g) interference from other traffic, we can get good video quality within both 2 s & 300 ms playout delay.

Results are fairly consistent with the simulations results

i th M k d l i ll ith th MDMC using the Markov model, especially with the MDMC scheme.

76

SUMMARY OF THE VIDEO OVER AD HOC PROJECT AD HOC PROJECT

Multiple path transport is feasible &

Kas

p p p effective for increasing both throughput & reliability in ad hoc networks.

saei

The right source coding strategy depends

• n application requirement & path

pp q p conditions:

MDC is effective when:

O t h diff ti l t t t i th

One cannot have differential treatment in the

network layer.

Delay constraint is very stringent & round trip delay

is long is long.

77

SUMMARY OF THE VIDEO OVER AD HOC PROJECT AD HOC PROJECT

Layered coding is effective when the base layer

Kas

y g y can be transported reliably, through:

A reliable physical channel, if available. Retransmission, if delay is acceptable. saei

, y p

RPS is effective when a feedback channel is

available & channel loss rates are not too high.

But not suitable for video streaming applications But not suitable for video streaming applications

where video is pre-encoded.

Reference: S Mao S Lin YWang & S Panwar ``Video Transport over Ad Reference: S.Mao, S.Lin, Y.Wang, & S.Panwar, Video Transport over Ad

Hoc Networks: Multistream Coding with Multipath Transport,'' IEEE J.

• Select. Areas Commun., Dec. 2003.

78

SUMMARY SUMMARY

Transport-level error control:

Kas

p

Guarantees a basic level of quality. Retransmission is effective within the delay

t i t

saei

constraint.

Unequal error protection has practical

difficulties against separation of source coding & transport layer.

Error resilient encoding: Error-resilient encoding: Trade-off coding efficiency for error resilience. Some techniques are only useful for bit-error

q y dominated channels.

79

SUMMARY SUMMARY

Decoder error concealment:

Kas

Does not involve extra redundancy, motion-

compensated temporal concealment is simple & yet offers visible improvements

saei

& yet offers visible improvements.

Encoder-decoder-network interactive error

control:

Requires feedback info, may not be available. Choice of techniques depends on

underlying application & network. underlying application & network.

80

HOMEWORK 11 HOMEWORK 11

Reading assignment:

Kas

Reading assignment:

• Chap. 14.

Computer assignment:

saei

Co pute ass g e t:

Course project.

81

THE END