Image and Video Coding: Hybrid Video Coding s n 1 [ x , y ] s n [ - - PowerPoint PPT Presentation

▶
image and video coding hybrid video coding
SMART_READER_LITE
LIVE PREVIEW

Image and Video Coding: Hybrid Video Coding s n 1 [ x , y ] s n [ - - PowerPoint PPT Presentation

Jul 12, 2023 529 likes •963 views

Image and Video Coding: Hybrid Video Coding s n 1 [ x , y ] s n [ x , y ] m k = ( m x , m y ) k x y Hybrid Video Coding Last Lectures: Block-Based Image Coding s k [ x , y ] u k [ x , y ] t k [ x , y ] q k [ x , y ] bitstream scalar

slide-1
SLIDE 1

Image and Video Coding: Hybrid Video Coding

s′

n−1[ x, y ]

x y mk = (mx, my)k sn[ x, y ]

slide-2
SLIDE 2

Hybrid Video Coding

Last Lectures: Block-Based Image Coding

2D block transform scalar quantization entropy coding input picture qk[x, y] tk[x, y] bitstream sk[x, y] intra-picture prediction buffer for current picture − ˆ sk[x, y] uk[x, y] scaling &

  • inv. transform

u′

k[x, y]

s′

k[x, y]

1 Transform Coding of Sample Blocks

Transform: Energy compaction / decorrelation Quantization: Approximation of input signal Entropy Coding: Lossless compression of quantization indexes

2 Intra-Picture Prediction between Transform Blocks

Utilization of statistical dependencies between transform blocks

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 2 / 42

slide-3
SLIDE 3

Hybrid Video Coding / Frame Difference Coding

Similarities between Successive Pictures in a Video Sequence t

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 3 / 42

slide-4
SLIDE 4

Hybrid Video Coding / Frame Difference Coding

General Idea of Hybrid Video Coding

“It has been customary in the past to transmit successive complete images of the transmitted picture. [· · · ] In accordance with this invention, this difficulty is avoided by transmitting only the difference between successive images of the object.”

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 4 / 42

slide-5
SLIDE 5

Hybrid Video Coding / Frame Difference Coding

Simple Variant: Frame Difference Coding

  • rec. previous picture s′

prev[x, y]

current original picture s[x, y] prediction error u[x, y]

Two coding modes for each block

( transmit flag per block)

1 Intra prediction:

ˆ s[x, y] = f ipred( s′[x, y] ) (use neighboring reconstructed samples)

2 Inter prediction:

ˆ s[x, y] = s′

prev[x, y]

(copy co-located samples of rec. previous picture)

Transform coding of prediction error

Get prediction error: u[x, y] = s[x, y] − ˆ s[x, y] Transform coding: u[x, y] → u′[x, y] ( transmit quantization indexes { qk } ) Reconstruction: s′[x, y] = ˆ s[x, y] + u′[x, y]

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 5 / 42

slide-6
SLIDE 6

Hybrid Video Coding / Frame Difference Coding

Frame Difference Coding: Inter-Picture Prediction Error

Johnny (original) frame difference (42.6 dB)

video properties static background low object motion few spatial details small inter prediction error

BQTerrace (original) frame difference (27.7 dB)

video properties moving camera moderate object motion lots of spatial details large inter prediction error

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 6 / 42

slide-7
SLIDE 7

Hybrid Video Coding / Frame Difference Coding

Coding Efficiency of Frame Difference Coding

34 35 36 37 38 39 40 41 42 43 44 2 4 6 8 10 Intra FD PSNR (Y) [dB] bit rate [Mbit/s] Johnny (1280×720, 60 Hz) 30 31 32 33 34 35 36 37 38 10 20 30 40 50 60 70 80 Intra FD PSNR (Y) [dB] bit rate [Mbit/s] BQTerrace (1920×1080, 60 Hz)

Large gains for video conferencing content (static background, low motion) Comparably small gains for videos with strong camera or object motion

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 7 / 42

slide-8
SLIDE 8

Hybrid Video Coding / Motion-Compensated Prediction

Motion-Compensated Prediction

x y displacement vector for current block m = (mx, my) current picture displaced object best matching block in reference picture reconstructed reference picture moving object

General Idea for Improving the Inter-Picture Prediction Predict a current block using a displaced block in an already coded picture ˆ s[ x, y ] = s′

ref[ x + mx, y + my ]

Transmit displacement vectors m = (mx, my) as part of the bitstream Encoder has to select suitable displacement vector m = (mx, my)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 8 / 42

slide-9
SLIDE 9

Hybrid Video Coding / Motion-Compensated Prediction

Estimation of Displacement Vector / Motion Vector

Straightforward Motion Search Define search range R =

  • −mmax; mmax
  • ×
  • −mmax; mmax
  • Test all vectors m = (mx, my) in search range R

Calculate cost measure C(m) Compare to current minimum Cmin

Choose vector m that minimizes cost measure C(m) Typical Cost Measures for Motion Search Sum of absolute differences (SAD)

DSAD(mx, my) =

  • x,y
  • s[x, y] − s′

ref[x + mx, y + my]

  • Lagrangian cost of SAD and bit rate for motion vector

J(mx, my) = DSAD(mx, my) + λ · R(mx, my)

  • rec. reference picture s′

ref[ x, y ]

current original picture s[ x, y ]

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 9 / 42

slide-10
SLIDE 10

Hybrid Video Coding / Motion-Compensated Prediction

Illustration of Motion-Compensated Prediction

  • rec. reference picture s′

ref[x, y]

  • riginal current picture s[x, y]

motion vector field m[x, y] referenced blocks in s′

ref[x, y]

prediction ˆ s[x, y] for current picture

  • pred. error u[x, y] = s[x, y] − ˆ

s[x, y]

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 10 / 42

slide-11
SLIDE 11

Hybrid Video Coding / Motion-Compensated Prediction

Motion-Compensated Prediction: Inter-Picture Prediction Error

Johnny (original) frame difference (42.6 dB) MCP error (44.6 dB) BQTerrace (original) frame difference (27.7 dB) MCP error (33.7 dB)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 11 / 42

slide-12
SLIDE 12

Hybrid Video Coding / Motion-Compensated Prediction

Do We Still Need Intra-Picture Coding Modes ?

  • rec. reference picture s′

ref[x, y]

current original picture s[x, y] prediction error u[x, y]

Not all regions can be well predicted by motion-compensated prediction

Uncovered background New regions entering the image area (due to object or camera motion) Motion that cannot be well represented by a block-wise translational motion field

Unmatched prediction can decrease coding efficiency

Prediction error can contain more high-frequency signal energy than original signal Direct coding or intra-picture prediction can lead to better coding efficiency

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 12 / 42

slide-13
SLIDE 13

Hybrid Video Coding / Motion-Compensated Prediction

Coding Efficiency Impact of Intra-Predicted Blocks

32 33 34 35 36 37 38 2 4 6 8 10 all coding modes intra-picture coding modes are disabled in inter pictures PSNR (Y) [dB] bit rate [Mbit/s] Basketball Drive, IPPP 5 10 15 20 25 30 32 33 34 35 36 37 38 bit-rate increase due to disabling of intra-picture coding modes in inter pictures (on average, 11%) bit-rate increase [%] PSNR (Y) [dB] Basketball Drive, IPPP

Intra coding modes are important for areas that cannot be well predicted by MCP (uncovered background, regions with complicated non-translational motion, ...) Enabling of intra coding modes increases coding efficiency

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 13 / 42

slide-14
SLIDE 14

Hybrid Video Coding / Block-based Hybrid Video Coding

Hybrid Video Coding

  • rec. reference picture s′

ref[x, y]

current original picture s[x, y] prediction error u[x, y]

Two coding modes for each block

( transmit coding mode)

1 Intra prediction:

ˆ s[x, y] = f ( s′[x, y], p intra ) ( transmit intra prediction mode p intra )

2 Inter prediction:

ˆ s[x, y] = s′

ref[ x + mx, y + my ]

( transmit motion vector m = (mx, my) )

Transform coding of prediction error

Prediction error: u[x, y] = s[x, y] − ˆ s[x, y] Transform coding: u[x, y] → u′[x, y] ( transmit quantization indexes { qk } ) Reconstruction: s′[x, y] = ˆ s[x, y] + u′[x, y]

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 14 / 42

slide-15
SLIDE 15

Hybrid Video Coding / Block-based Hybrid Video Coding

Block-Based Image Encoder with Intra-Picture Prediction

scaling &

  • inv. transform

transform & quantization intra mode decision intra-picture prediction entropy coding buffer for current picture input picture sk[x, y] uk[x, y] quantization indexes qk[x, y] u′

k[x, y]

s∗

k [x, y]

bitstream intra prediction modes f iprd( s∗

k [x, y] )

ˆ sk[x, y] = f iprd( s∗

k [x, y] )

−

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 15 / 42

slide-16
SLIDE 16

Hybrid Video Coding / Block-based Hybrid Video Coding

Hybrid Video Encoder

scaling &

  • inv. transform

transform & quantization intra mode decision intra-picture prediction entropy coding in-loop filtering decoded picture buffer motion-comp. prediction motion estimation coding mode decision buffer for current picture first input picture input pictures sk[x, y] uk[x, y] quantization indexes qk[x, y] u′

k[x, y]

s∗

k [x, y]

bitstream s∗

k [x, y]

s′

k[x, y]

intra prediction modes intra prediction modes intra prediction modes f iprd( s∗

k [x, y] )

f iprd( s∗

k [x, y] )

ˆ sk[x, y] = f iprd( s∗

k [x, y] )

ˆ sk[x, y] − motion data motion data s′

ref[x + mx, y + my]

coding modes, intra prediction modes, motion data

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 16 / 42

slide-17
SLIDE 17

Hybrid Video Coding / Block-based Hybrid Video Coding

Hybrid Video Decoder

scaling &

  • inv. transform

intra-picture prediction entropy decoding in-loop filtering decoded picture buffer motion-comp. prediction buffer for current picture reconstructed pictures bitstream f iprd( s∗

k [x, y] )

s′

ref[x + mx, y + my]

prediction signal ˆ s[x, y] quantization indexes qk[x, y] u′

k[x, y]

s∗

k [x, y]

s∗

k [x, y]

s′

k[x, y]

intra prediction modes motion data

  • utput

coding modes

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 17 / 42

slide-18
SLIDE 18

Hybrid Video Coding / Block-based Hybrid Video Coding

Coding Efficiency of Hybrid Video Coding

34 36 38 40 42 44 46 1 2 3 4 5 6 7 8 Intra FD simple MCP improved MCP PSNR (Y) [dB] bit rate [Mbit/s] Johnny (1280×720, 60 Hz) 30 32 34 36 38 40 10 20 30 40 50 60 70 80 Intra FD simple MCP improved MCP PSNR (Y) [dB] bit rate [Mbit/s] BQTerrace (1920×1080, 60 Hz)

Simple MCP: Straightforward implementation of motion-compensated prediction Improved MCP: Various improvements (will be discussed later) Motion-compensated prediction yields significant improvement in coding efficiency

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 18 / 42

slide-19
SLIDE 19

Hybrid Video Coding / Block-based Hybrid Video Coding

Aspects that Impact the Effectiveness of Motion-Compensated Prediction

Accuracy of Motion Description Accuracy of motion vectors and choice of interpolation filters Higher-order motion models Selection of Regions with Constant Motion Rectangular blocks or more flexible regions Adaptive block size selection Coding of Motion Parameters Prediction of motion parameters Inference of motion parameters Advanced Concepts for Motion-Compensated Prediction Multiple reference pictures and multi-hypotheses prediction Coding order of pictures In-loop filters

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 19 / 42

slide-20
SLIDE 20

Theoretical Analysis of Motion-Compensated Prediction

Theoretical Analysis: Model for Hybrid Video Coding

r-d optimal intra-picture encoder input pictures bitstream motion- compensated prediction decoded picture buffer s′

ref[x, y]

− s[x, y] u[x, y] intra-picture decoder u′[x, y] ˆ s[x, y] = s′

ref[x + mx, y + my]

s′[x, y]

Assume rate-distortion optimal intra-picture coding for prediction signal Compare theoretical rate-distortion function for original and prediction error pictures Use simple models (not completely realistic)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 20 / 42

slide-21
SLIDE 21

Theoretical Analysis of Motion-Compensated Prediction / Models

Theoretical Analysis: Gaussian Random Signals

Gaussian Random Signals Original and residual pictures are realizations of stationary Gaussian random processes Keeps problem mathematically tractable Rate-Distortion Function for Gaussian Processes Parametric formulation with θ > 0 D(θ) = 1 4π2 π

−π

π

−π

min

  • θ, ΦSS(ωx, ωy)
  • dωx dωy

R(θ) = 1 4π2 π

−π

π

−π

max

  • 0, 1

2 log2 ΦSS(ωx, ωy) θ

  • dωx dωy

with ΦSS(ωx, ωy) being the given power spectral density (Fourier transform of autocorr. function) ΦSS(ωx, ωy) = F{ RSS(ζ, η) } with RSS(ζ, η) = E

  • S[x, y] S[x − ζ, y − η]
  • Heiko Schwarz (Freie Universität Berlin)

— Image and Video Coding: Hybrid Video Coding 21 / 42

slide-22
SLIDE 22

Theoretical Analysis of Motion-Compensated Prediction / Models

Theoretical Analysis: Model for Original Signal

Model for Original Images Original images s[ x, y ] have an isotropic autocorrelation function RSS(ζ, η) = E

  • S[x, y] S[x − ζ, y − η]
  • = σ2

S · ̺

√

ζ2+η2

Assume correlation factor of ̺ = 0.9 (typical value for natural pictures) Model for Inter-Picture Dependencies Using the correct displacement vector d = (dx, dy), the current picture sn[x, y] deviates from the previous picture sn−1[ x, y ] by a noise term n[x, y] sn[ x, y ] = sn−1( x + dx, y + dy) + n[ x, y ] where sn−1(x, y) represents the ideally interpolated image sn−1[ x, y ] The noise term n[ x, y ] (assume white noise) includes all deviation not caused by object motion Characterized by signal-to-noise ratio Θ = 10 log10

  • σ2

S/σ2 N

  • Heiko Schwarz (Freie Universität Berlin)

— Image and Video Coding: Hybrid Video Coding 22 / 42

slide-23
SLIDE 23

Theoretical Analysis of Motion-Compensated Prediction / Models

Theoretical Analysis: Model for Motion-Compensated Prediction

Motion Vector Accuracy Use finite precision motion vectors m = (mx, my) where mx and my are given in units of 2−β Integer-sample (β = 0), half-sample (β = 1), quarter-sample (β = 2), ... Displacement errors ∆x = dx − mx and ∆y = dy − my are uniformly distributed in interval [ −0.5 · 2−β... 0.5 · 2−β ] Motion-Compensated Prediction with Sub-Sample Accuracy Sub-sample interpolation with continuous interpolation kernel g(x, y) ˆ sn[ x, y ] =

  • g ∗ s′

n−1

  • ( x + mx, y + my)

Consider two types of interpolation kernels g(x, y) Ideal interpolator: All-pass filter with transfer function G(ωx, ωy) = 1 Optimal interpolator: Signal-adaptive Wiener filter (minimizes ΦUU(ωx, ωy))

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 23 / 42

slide-24
SLIDE 24

Theoretical Analysis of Motion-Compensated Prediction / Results

Theoretical Results: Power Spectral Density

dashed lines : ideal interpolator; solid lines : Wiener filter

  • 30
  • 20
  • 10

10 20 30 0.5 1 1.5 2 2.5 3 Φ(ω) / σS2 [dB] spatial frequency |ωx| (ωy=0) ρ = 0.9, Θ = 5 dB signal spectrum ΦSS ΦUU (integer-sample accuracy) ΦUU (quarter-sample accuracy)

  • 30
  • 20
  • 10

10 20 30 0.5 1 1.5 2 2.5 3 Φ(ω) / σS2 [dB] spatial frequency |ωx| (ωy=0) ρ = 0.9, Θ = 20 dB signal spectrum ΦSS ΦUU (integer-sample accuracy) ΦUU (quarter-sample accuracy)

Derivation of Results (for details, see [ Girod 1987/1993 ] or [ Schwarz/Wiegand 2016 ]) Derive power spectral density ΦUU(ωx, ωy) for prediction error pictures using models Calculate rate-distortion curves using Gaussian assumption

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 24 / 42

slide-25
SLIDE 25

Theoretical Analysis of Motion-Compensated Prediction / Results

Theoretical Results: Rate-Distortion Functions

dashed lines : ideal interpolator; solid lines : Wiener filter

5 10 15 20 25 30 35 40 0.5 1 1.5 2 2.5 3 SNR: 10 log10( σS2 / D ) [dB] bit rate [bits/sample] ρ = 0.9, Θ = 5 dB intra-picture coding MCP (integer-sample accuracy) MCP (quarter-sample accuracy) 5 10 15 20 25 30 35 40 0.5 1 1.5 2 2.5 3 SNR: 10 log10( σS2 / D ) [dB] bit rate [bits/sample] ρ = 0.9, Θ = 20 dB intra-picture coding MCP (integer-sample accuracy) MCP (quarter-sample accuracy)

Small SNR: No coding gain without suitable interpolation filter, accuracy has small effect Large SNR: Higher precision increases coding efficiency, interpolation has smaller effect

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 25 / 42

slide-26
SLIDE 26

Theoretical Analysis of Motion-Compensated Prediction / Results

Theoretical Results: Summary

  • 4
  • 2

2 4 6 8 10 1/1 1/2 1/4 1/8 1/16 SNR gain rel. to intra coding motion vector accuracy in samples ρ = 0.9 Θ = 20 dB Θ = 15 dB Θ = 10 dB Θ = 5 dB

  • 25
  • 20
  • 15
  • 10
  • 5

5 0.5 1 1.5 2 2.5 3 20 log10 |Gopt(ω)| [dB] spatial frequency |ωx| (ωy=0) ρ = 0.9, β = 2 Θ = 20 dB Θ = 15 dB Θ = 10 dB Θ = 5 dB

Need stronger low-pass filter for small SNR values Θ Wiener filter (optimal interpolator) approaches ideal interpolator for large SNR values Θ Inter-picture coding gain increases with SNR parameter Θ Coding gain increases with motion vector accuracy, but approaches a limit

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 26 / 42

slide-27
SLIDE 27

Accuracy of Motion-Compensated Prediction / Motion Vector Accuracy

Sub-Sample Accurate Motion Vectors

reference picture sref[ x ]

mx

current picture s[ x ] = sref[ x − mx ]

Real Object Motion Real object motion is not aligned to sample grid of video pictures Prediction signal for sample-accurate motion vectors may significantly deviate from original signal Prediction may be improved by using sub-sample accurate motion vectors

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 27 / 42

slide-28
SLIDE 28

Accuracy of Motion-Compensated Prediction / Motion Vector Accuracy

Motion Compensation with Sub-Sample Precision Motion Vectors

Motion Vector Precision In practice: Motion vector precision of 2−n, n ∈ N

Integer-sample precision (n = 0) Half-sample precision (n = 1) Quarter-sample precision (n = 2)

Transmit integer values (2n · mx) and (2n · my) Motion-Compensated Prediction Split vector m = (mx, my) into integer part mi and phase φ

mi = mix miy

  • =

floor(mx) floor(my)

  • ,

φ = φx φy

  • =

mx − mix my − miy

  • Integer vector mi determines location in reference picture

Phase φ = (φx, φy) determines interpolation filter

integer-sample locations half-sample locations quarter-sample locations

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 28 / 42

slide-29
SLIDE 29

Accuracy of Motion-Compensated Prediction / Motion Vector Accuracy

Simplest Interpolation: Bilinear Interpolation

Bilinear Interpolation Value is given by weighted average of surrounding samples X = (1 − φy)(1 − φx) · A + (1 − φy)φx · B + φy(1 − φx) · C + φyφx · D Separable Interpolation of Sub-Sample Values

1 Horizontal interpolation (for required values of y ∗)

˜ s[ x + mx, y ∗ ] = (1 − φx) · s[ x + mix, y ∗ ] + φx · s[ x + mix + 1, y ∗ ]

2 Vertical interpolation (yielding final values)

s[ x + mx, y + my ] = (1 − φy) · ˜ s[ x + mx, y + miy ] + φx · ˜ s[ x + mx + 1, y + miy + 1 ]

A B C D X φy φx

(x + mix, y + miy) (x + mx, y + my)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 29 / 42

slide-30
SLIDE 30

Accuracy of Motion-Compensated Prediction / Motion Vector Accuracy

Impact of Motion Vector Accuracy (Bilinear Interpolation)

28 30 32 34 36 38 40 42 2 4 6 8 10 12 14 MCP (1/1 sample) MCP (1/2 sample) MCP (1/4 sample) Intra PSNR (Y) [dB] bit rate [Mbit/s] BQSquare (416×240, 60 Hz) IPPP, 16×16 MC blocks 34 36 38 40 42 44 1 2 3 4 5 MCP (1/1 sample) MCP (1/2 sample) MCP (1/4 sample) Intra PSNR (Y) [dB] bit rate [Mbit/s] Johnny (1280×720, 60 Hz) IPPP, 16×16 MC blocks

Large gains for half-sample precision motion vectors Additional small gains for quarter-sample precision motion vectors

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 30 / 42

slide-31
SLIDE 31

Accuracy of Motion-Compensated Prediction / Interpolation Filters

Interpolation: Convolution with Interpolation Kernel

Non-linear interpolation:

∗ =

Linear interpolation:

∗ =

Interpolation as Convolution Convolution of discrete signal s[ k ] with continuous interpolation kernel g(x) ˜ s(x) = (s ∗ g)(x) =

  • k

s[ k ] · g(x − k) with g(x) = 1 : x = 0 : x ∈ Z, x = 0 Suitable interpolation kernels yield smooth interpolated signal

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 31 / 42

slide-32
SLIDE 32

Accuracy of Motion-Compensated Prediction / Interpolation Filters

Interpolation Filters: Sampled Interpolation Kernel

˜ s(xi + φ) = s[ xi − 1 ] · g(φ + 1) + s[ xi ] · g(φ) + s[ xi + 1 ] · g(φ − 1) + s[ xi + 2 ] · g(φ − 2) Interpolation Filters For a certain phase φ, we require only a finite number of values of the interpolation kernel ˜ s(xi + φ) =

  • k

s[ k ] · g(xi − k + φ)

  • nly need

g(k + φ), ∀k ∈ Z Small number of values for typical finite size interpolation kernels The set of values { g(k + φ) } is referred to as interpolation filter for phase φ Longer interpolation filters provide more degrees of freedom for their design

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 32 / 42

slide-33
SLIDE 33

Accuracy of Motion-Compensated Prediction / Interpolation Filters

Sub-Sample Interpolation: Separable Interpolation Filters

Predefined Filters Set of 1D interpolation filters h(φ) for different phases φ Separable Interpolation of Sub-Sample Values

1 Horizontal interpolation

˜ s[ x+mx, y ∗ ] =

kmax(φx)

  • k=kmin(φx)

hφx[ k ] · s[ x+mix +k, y ∗ ]

2 Vertical interpolation

˜ s[ x+mx, y +my ] =

kmax(φy)

  • k=kmin(φy)

hφy [ k ] · ˜ s[ x+mx, y +miy +k ] Choice of filters depends on phase (φx, φy)

(x +mix, y +miy)

φy φx

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 33 / 42

slide-34
SLIDE 34

Accuracy of Motion-Compensated Prediction / Interpolation Filters

Selected Interpolation Filters

filter coefficients h[k] (division by 64) phase

  • 3
  • 2
  • 1

1 2 3 4 1/4 48 16 2-tap 1/2 32 32 3/4 16 48 1/4

  • 4

54 16

  • 2

4-tap 1/2

  • 4

36 36

  • 4

3/4

  • 2

16 54

  • 4

1/4 1

  • 5

52 20

  • 5

1 6-tap 1/2 2

  • 10

40 40

  • 10

2 3/4 1

  • 5

20 52

  • 5

1 1/4

  • 1

4

  • 10

58 17

  • 5

1 8/7-tap 1/2

  • 1

4

  • 11

40 40

  • 11

4

  • 1

3/4 1

  • 5

17 58

  • 10

4

  • 1

linear interpolation chroma interpolation filter in H.265 | HEVC luma interpolation filter in H.264 | AVC luma interpolation filter in H.265 | HEVC

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 34 / 42

slide-35
SLIDE 35

Accuracy of Motion-Compensated Prediction / Interpolation Filters

Frequency Responses of Sub-Sample Interpolation Filters

  • 10
  • 8
  • 6
  • 4
  • 2

2 0.5 1 1.5 2 2.5 3 Magnitude 20 log10 |G(ω)| [dB] spatial frequency |ω| half-sample locations 2-tap filter 4-tap filter 6-tap filter 8-tap filter

  • 10
  • 8
  • 6
  • 4
  • 2

2 0.5 1 1.5 2 2.5 3 Magnitude 20 log10 |G(ω)| [dB] spatial frequency |ω| quarter-sample locations 2-tap filter 4-tap filter 6-tap filter 7-tap filter

Longer filters have less low-pass characteristics Filters for quarter-sample locations (1/4, 3/4) have less low-pass characteristics Selected sub-sample position also determines strength of low-pass filtering

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 35 / 42

slide-36
SLIDE 36

Accuracy of Motion-Compensated Prediction / Interpolation Filters

Phase Delay of Sub-Sample Interpolation Filters

0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.5 1 1.5 2 2.5 3 phase delay [samples] spatial frequency |ω| quarter-sample locations 2-tap filter 4-tap filter 6-tap filter 7-tap filter

Note: All half-sample filters have a constant phase delay of 0.5 (due to symmetry) Longer filters also have more constant phase delay

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 36 / 42

slide-37
SLIDE 37

Accuracy of Motion-Compensated Prediction / Interpolation Filters

Coding Efficiency of Interpolation Filters (Half-Sample Accuracy)

10 20 30 40 50 60 70 80 28 30 32 34 36 38 4-tap filter 6-tap filter 8-tap filter bit-rate saving vs 2-tap filter [%] PSNR (Y) [dB] BQSquare (416×240, 60 Hz) 1/2-sample acc. (IPPP, 16×16 MC blocks)

  • 30
  • 20
  • 10

10 20 30 40 50 34 36 38 40 42 44 4-tap filter 6-tap filter 8-tap filter bit-rate saving vs 2-tap filter [%] PSNR (Y) [dB] Johnny (1280×720, 60 Hz) 1/2-sample acc. (IPPP, 16×16 MC blocks)

Coding Experiment (HEVC): IPPP, fixed block size (16×16), half-sample accuracy Longer filters provide better coding efficiency for sequences with high signal-to-noise ratio Best filter depends on sequence characteristics

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 37 / 42

slide-38
SLIDE 38

Accuracy of Motion-Compensated Prediction / Interpolation Filters

Coding Efficiency of Interpolation Filters (Quarter-Sample Accuracy)

10 20 30 40 50 60 70 80 28 30 32 34 36 38 4-tap filters 6-tap filters 8/7-tap filters bit-rate saving vs 2-tap filters [%] PSNR (Y) [dB] BQSquare (416×240, 60 Hz) 1/4-sample acc. (IPPP, 16×16 MC blocks)

  • 30
  • 20
  • 10

10 20 30 40 50 34 36 38 40 42 44 4-tap filters 6-tap filters 8/7-tap filters bit-rate saving vs 2-tap filters [%] PSNR (Y) [dB] Johnny (1280×720, 60 Hz) 1/4-sample acc. (IPPP, 16×16 MC blocks)

Coding Experiment (HEVC): IPPP, fixed block size (16×16), quarter-sample accuracy Longer filters provide better coding efficiency for sequences with high signal-to-noise ratio Best filter depends on sequence characteristics

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 38 / 42

slide-39
SLIDE 39

Accuracy of Motion-Compensated Prediction / Interpolation Filters

Coding Efficiency of Interpolation Filters (Quarter-Sample Accuracy)

10 20 30 40 50 60 70 80 28 30 32 34 36 38 4-tap filters 6-tap filters 8/7-tap filters bit-rate saving vs 2-tap filters [%] PSNR (Y) [dB] BQSquare (416×240, 60 Hz) 1/4-sample acc. (IBBB, all MC block sizes)

  • 30
  • 20
  • 10

10 20 30 40 50 34 36 38 40 42 44 4-tap filters 6-tap filters 8/7-tap filters bit-rate saving vs 2-tap filters [%] PSNR (Y) [dB] Johnny (1280×720, 60 Hz) 1/4-sample acc. (IBBB, all MC block sizes)

Coding Experiment (HEVC): IBBB, variable block sizes, quarter-sample accuracy Longer filters provide best coding efficiency for most sequences Larger differences for sequences with high signal-to-noise ratio

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 39 / 42

slide-40
SLIDE 40

Accuracy of Motion-Compensated Prediction / Motion Vector Accuracy with Improved Interpolation

Impact of Motion Vector Accuracy (HEVC Interpolation Filters)

28 30 32 34 36 38 40 42 2 4 6 8 10 12 14 MCP (1/1 sample) MCP (1/2 sample) MCP (1/4 sample) Intra PSNR (Y) [dB] bit rate [Mbit/s] BQSquare (416×240, 60 Hz) IPPP, 16×16 MC blocks 34 36 38 40 42 44 1 2 3 4 5 MCP (1/1 sample) MCP (1/2 sample) MCP (1/4 sample) Intra PSNR (Y) [dB] bit rate [Mbit/s] Johnny (1280×720, 60 Hz) IPPP, 16×16 MC blocks

Coding Experiment (HEVC): IPPP, fixed block size (16×16), quarter-sample accuracy Coding efficiency significantly increase with motion vector accuracy Very large coding gains relative to integer-accurate motion vectors

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 40 / 42

slide-41
SLIDE 41

Accuracy of Motion-Compensated Prediction / Motion Vector Accuracy with Improved Interpolation

Motion Vector Accuracy: Coding Efficiency for Larger Test Set

entertainment-quality video content bit-rate savings relative to ... Intra MCP (1/1) MCP (1/2) MCP (1/1) 51.0 % MCP (1/2) 59.6 % 18.9 % MCP (1/4) 62.6 % 25.2 % 8.9 % video conferencing content bit-rate savings relative to ... Intra MCP (1/1) MCP (1/2) MCP (1/1) 82.8 % MCP (1/2) 86.6 % 21.2 % MCP (1/4) 88.4 % 30.0 % 11.7 %

Average Bit-Rate Savings for 2 Classes of HD Sequences IPPP coding with 16×16 blocks, intra-picture coding, and ...

Inter-picture coding with integer-sample precision motion vectors Inter-picture coding with half-sample precision motion vectors Inter-picture coding with quarter-sample precision motion vectors

Quarter-sample precision motion vectors are typically a reasonable choice

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 41 / 42

slide-42
SLIDE 42

Summary

Summary of Lecture

Motion-Compensated Prediction Key idea: Utilize dependencies between successive video picture Predict block by displaced block in already reconstructed preceding picture Need to estimate and transmit a motion vector per block Hybrid Video Coding Block-adaptive choice between intra-picture prediction and motion-compensated prediction Transform coding of prediction error signal Most successful video coding design Accuracy of Motion-Compensated Prediction Sub-sample accurate vectors improve coding efficiency (good choice: quarter-sample precision) Choice of interpolation filter has significant impact Longer interpolation filters improve coding efficiency compared to bilinear interpolation

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Hybrid Video Coding 42 / 42