Image and Video Coding: Improved Inter-Picture Prediction Review of - - PowerPoint PPT Presentation

Image and Video Coding: Improved Inter-Picture Prediction

Review of Hybrid Video Coding

Last Lectures: Hybrid Video Coding

Hybrid Video Coding Partitioning of pictures into blocks Block-adaptive selection of prediction method

1 intra-picture prediction 2 motion-compensated prediction

Transform coding of prediction error Effectiveness of Motion-Compensated Prediction Accuracy of motion vectors Choice of interpolation filters Prediction and coding of motion parameters Estimation algorithm for motion parameters ...

• rec. reference picture s′

ref[ x, y ]

current original picture s[ x, y ]

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 2 / 39

Advanced Motion-Compensated Prediction / Multiple Reference Pictures

Multiple Reference Pictures

s′

k−4[x, y]

s′

k−3[x, y]

r =2 s′

k−2[x, y]

r =1 s′

k−1[x, y]

r =0 sk[x, y] current picture decoded picture buffer (DPB)

Motion-Compensated Prediction with Multiple Reference Pictures Store multiple reconstructed pictures in a decoded picture buffer (DPB)

Example: Sliding window buffer (FIFO), which contains the N most recently decoded pictures

For each block: Select reference picture used for prediction Transmit reference picture index r in addition to motion vector

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 3 / 39

Advanced Motion-Compensated Prediction / Multiple Reference Pictures

Encoder Control: Reference Picture Selection

Typical Approach

1 Determine motion vector mr = (mx, my) for each possible reference picture r ∈ [ 0; N − 1 ] 2 Select motion parameters {r, mr} among the pre-determined sets

Criterion for Reference Picture Selection Lagrangian decision similar to motion search Choose motion parameters {r, mr} according to r ∗ = arg min

∀r D(r, mr) + λm · R(r, mr)

with D(r, mr) : Distortion between original block s[x, y] and prediction signal s′

r[x+mx, y +my]

R(r, mr) : Number of bits for reference index r and motion vector mr Usually: Same distortion measure as for sub-sample search (often SAD in Hadamard domain) Complexity Impact: Need N motion searches per block (instead of one)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 4 / 39

Advanced Motion-Compensated Prediction / Multiple Reference Pictures

Coding Efficiency: Multiple Reference Pictures

10 20 30 40 50 60 26 28 30 32 34 36 38 40 2 ref. pics. (avg. 16.6%) 4 ref. pics. (avg. 29.0%) 8 ref. pics. (avg. 35.7%) bit-rate saving vs 1 ref. pic. [%] PSNR (Y) [dB] BQSquare (416×240, 60 Hz) 5 10 15 20 25 34 36 38 40 42 44 2 ref. pics. (avg. 5.3%) 4 ref. pics. (avg. 9.0%) 8 ref. pics. (avg. 10.1%) bit-rate saving vs 1 ref. pic. [%] PSNR (Y) [dB] Johnny (1280×720, 60 Hz)

Coding Experiment (HEVC): IPPP coding Use N = 1, 2, 4, 8 preceeding pictures as reference picture (sliding window mechanism) Coding efficiency typically increases with the number of reference pictures Can have very large impact (depends on source material)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 5 / 39

Advanced Motion-Compensated Prediction / Multi-Hypothesis Prediction

Multi-Hypothesis Prediction

s′

k−4[x, y]

s′

k−3[x, y]

r =2 s′

k−2[x, y]

r =1 s′

k−1[x, y]

r =0 sk[x, y] current picture decoded picture buffer (DPB)

Motion-Compensated Prediction with Multiple Motion Hypotheses Weighted summation of multiple displaced reference blocks s′

rk[ x + mk,x, y + mk,y ]

Simplest and most commonly used variant (equal weigthing of individual prediction signals) ˆ s[ x, y ] = 1 K

K−1

• k=0

s′

rk[ x + mk,x, y + mk,y ]

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 6 / 39

Advanced Motion-Compensated Prediction / Multi-Hypothesis Prediction

Why Multi-Hypothesis Prediction ?

Idealized Assumptions Individual prediction errors uk[x, y] = s[x, y] − s′

rk[x + mr,x, y + mr,y] are uncorrelated

All individual prediction errors have zero mean and the same variance σ2

single

Effect of Multi-Hypothesis Prediction Variance of multi-hypothesis prediction error for idealized assumptions σ2

multi = E

• U2

= E   

• 1

K

K−1

• k=0

Uk

• 2 

  = 1 K 2

K−1

• k=0

E

• U2

k

• =

1 K 2

K−1

• k=0

σ2

single

σ2

multi =

1 K · σ2

single

Prediction error variance is reduced by factor K (compared to conventional prediction) Require less bit rate for achieving same distortion

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 7 / 39

Advanced Motion-Compensated Prediction / Multi-Hypothesis Prediction

Block-Adaptive Multi-Hypothesis Prediction

Coding Efficiency of Multi-Hypothesis Prediction Idealized assumptions: Prediction error variance is reduced by factor K σ2

multi ≈

1 K · σ2

single

In reality, reduction of prediction error variance is smaller (increase is also possible!) Bit rate for motion data (motion vectors and reference indexes) is increased Multi-hypothesis prediction does not improve coding efficiency for all blocks Block-Adaptive Selection of Number of Motion Hypotheses Select number of motion hypotheses for each block (signaled in bitstream) Video coding standards: Only up to two motion hypotheses

1 Uni-predicted blocks:

Conventional motion-compensated prediction (one hypothesis)

2 Bi-predicted blocks:

Averaging of two motion-compensated prediction signals (two hypotheses)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 8 / 39

Advanced Motion-Compensated Prediction / Bi-Prediction in Video Coding Standards

Picture Types in Early Video Coding Standards

I P P P P P P P P P 1 2 3 4 5 6 7 8 9 I P 1 B 2 B 3 P 4 B 5 B 6 P 7 B 8 B 9

IPPP Coding All pictures are coded in acquisition/display order Bi-prediction is not supported IBBP Coding One or more B pictures are inserted between two I/P pictures

coded after P picture succeeding in display order support bi-prediction using last two coded I/P pictures as references never used as reference pictures

P pictures use last I/P picture as reference picture

I Pictures all block are intra-coded P Pictures intra or uni-prediction last I/P as reference stored as reference picture B Pictures intra, uni-, or bi-prediction last two I/P as references not used as reference pic.

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 9 / 39

Advanced Motion-Compensated Prediction / Bi-Prediction in Video Coding Standards

Bi-Prediction in Early Video Coding Standards

I/P (n) B (n+2) B (n+3) I/P (n+1) forward prediction backward prediction bi-prediction

Bi-Prediction in MPEG-2 Video, H.263, and MPEG-4 Visual B pictures support three inter-picture coding modes (selected on block basis)

Forward prediction: Single-hypothesis prediction from preceding I/P picture Backward prediction: Single-hypothesis prediction from succeeding I/P picture Bi-directional prediction: Bi-prediction from preceding and succeeding I/P picture

! B pictures can only be coded as part of an BBP or BBI group

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 10 / 39

Advanced Motion-Compensated Prediction / Bi-Prediction in Video Coding Standards

Picture Types in Modern Video Coding Standards

I B B B P P P P B B 1 2 3 4 5 6 7 8 9 I B 1 P 2 B 3 B 4 B 5 P 6 B 7 B 8 P 9

H.264 | AVC, H.265 | HEVC, and H.266 | VVC Generalized concept of bi-prediction and multiple reference pictures Arbitrary coding order of pictures (only limited by size of decoded picture buffer) Picture types (I, P, B) decoupled from coding order (actually: slice types instead of picture types) Reference Picture Lists P slices: One reference picture list (constructed from pictures in decoded picture buffer) B slices: Two reference picture lists (referred to as list 0 and list 1)

Both reference picture list can include pictures from past and future Each stored picture (in DPB) can be included in either or both lists

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 11 / 39

Advanced Motion-Compensated Prediction / Bi-Prediction in Video Coding Standards

Bi-Prediction in Modern Video Coding Standards

r1 = 1 r0 = 0 r1 = 1 current picture r0 = 2 r1 = 0

Inter-Picture Coding Modes in B Slices List 0 prediction: Uni-prediction with reference picture from list 0 List 1 prediction: Uni-prediction with reference picture from list 1 Bi-prediction: Bi-prediction with a picture from list 0 and a picture from list 1

Both motion hypotheses can use the same picture or different pictures Both motion hypotheses can stem from the same or different directions

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 12 / 39

Advanced Motion-Compensated Prediction / Motion Estimation for Multi-Hypotheses Prediction

Multi-Hypothesis Prediction: Motion Estimation Problem

Sub-Optimality of Independent Motion Estimation Example: SSD Distortion for bi-prediction Dbi

• m0, m1
• =
• x
• s[ x ] − 1

2

• s′

0[ x + m0 ] + s′ 1[ x + m1 ]

• 2

= 1 4

• x
• s[ x ] − s′

0[ x + m0 ]

• +
• s[ x ] − s′

1[ x + m1 ]

• 2

= 1 4 D0

• m0
• + 1

4 D1

• m1
• + 1

2

• x
• s[ x ] − s′

0[ x + m0 ]

• s[ x ] − s′

1[ x + m1 ]

• Minimization of D0 and D1 does not imply minimization of Dbi

Motion Estimation for Multi-Hypothesis Prediction Need to estimate multiple motion vectors for a block Independent estimation of individual motion vectors is sub-optimal Motion estimation in product space is way too complex

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 13 / 39

Advanced Motion-Compensated Prediction / Motion Estimation for Multi-Hypotheses Prediction

Multi-Hypothesis Prediction: Conditional Motion Estimation

Conditional Motion Estimation Consider bi-prediction with given first motion vector m0 Consider general p-norm distortion measure Choose second motion vector m1 so that overall distortion is minimized Dbi

• m1 | m0
• =
• x
• s[ x ] − 1

2

• s′

0[ x + m0 ] + s′ 1[ x + m1 ]

• p

= 1 2p

• x
• 2 · s[ x ] − s′

0[ x + m0 ] − s′ 1[ x + m1 ]

• p

= 1 2p

• x
• s∗[ x ] − s′

1[ x + m1 ]

• p

Conventional motion estimation for m1, but with modified original block s∗ s∗[ x ] = 2 · s[ x ] − s′

0[ x + m0 ]

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 14 / 39

Advanced Motion-Compensated Prediction / Motion Estimation for Multi-Hypotheses Prediction

Multi-Hypothesis Prediction: Iterative Motion Estimation

m0 m1

Iterative Motion Estimation Algorithm for Bi-Prediction

1 Initialization (re-use estimations for list 0 and list 1 estimation)

Independent estimation of motion vector m0 Independent estimation of motion vector m1

2 Iterative refinement (smaller search range)

Conditional refinement of motion vector m0 (for fixed motion vector m1) Conditional refinement of motion vector m1 (for fixed motion vector m0)

Straightforward extension to more than two motion hypotheses

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 15 / 39

Advanced Motion-Compensated Prediction / Coding Efficiency of Bi-Prediction

Coding Efficiency: Block-Adaptive Bi-Prediction

5 10 15 20 25 26 28 30 32 34 36 38 40 1 ref. pic. (avg. 13.2%) 2 ref. pics. (avg. 14.9%) 4 ref. pics. (avg. 14.2%) 8 ref. pics. (avg. 13.0%) bit-rate saving vs uni-pred. [%] PSNR (Y) [dB] BQSquare (416×240, 60 Hz) 5 10 15 20 25 30 34 36 38 40 42 44 1 ref. pic. (avg. 8.3%) 2 ref. pics. (avg. 10.5%) 4 ref. pics. (avg. 11.4%) 8 ref. pics. (avg. 12.1%) bit-rate saving vs uni-pred. [%] PSNR (Y) [dB] Johnny (1280×720, 60 Hz)

Coding Experiment (HEVC): IBBB vs IPPP coding Coding order equal to acquisition/display order Block-adaptive bi-prediction: Comparison with uni-prediction for same number of reference pictures Support of bi-prediction often yields significant coding gain

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 16 / 39

Advanced Motion-Compensated Prediction / Coding Efficiency of Bi-Prediction

Coding Efficiency: Block-Adaptive Bi-Prediction (Averages)

5 10 15 20 1 2 4 8 average bit-rate saving [%] number of reference pictures entertainment-quality video content uni-prediction adaptive bi-prediction 5 10 15 20 1 2 4 8 average bit-rate saving [%] number of reference pictures video conferencing content uni-prediction adaptive bi-prediction

Coding Experiment (HEVC): IBBB and IPPP coding with multiple reference picture Results represent averages for two classes of HD sequences Gains for multiple reference picture and bi-prediction are roughly additive Combination of bi-prediction and multiple reference pictures provides significant gains

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 17 / 39

Advanced Motion-Compensated Prediction / Generalized Multi-Hypothesis Prediction

Generalized Multi-Hypothesis Prediction

General Form of Multi-Hypothesis Prediction Permit unequal weights for individual motion hypotheses ˆ s[ x, y ] =

K−1

• k=0

wk · s′

rk[ x + mk,x, y + mk,y ]

Generalized Bi-Prediction in Video Coding Standards

1 Weighted prediction: Mostly used for improving coding efficiency for fade-in, fade-out, cross-fades a Explicit weights:

Weights are explicitly coded for pictures in reference picture lists

b Implicit weights:

Weights are derived based on temporal distance (as in linear interpolation)

2 Block-adaptive prediction weights

ˆ s[ x, y ] =

• (8 − w) · ˆ

s0[ x, y ] + w · ˆ s1[ x, y ] + 4

• ≫ 3

with weight w chosen from list {4, 5, 3, 10, −2}

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 18 / 39

Coding Structures / Hierarchical Coding Structures

Conventional Coding Structures

I P P P P P P P P P 1 2 3 4 5 6 7 8 9 I P 1 B 2 B 3 P 4 B 5 B 6 P 7 B 8 B 9

IPPP / IBBB Coding Structures All pictures are coded in acquisition/display order Pictures can be coded as P/B pictures (or P/B slices) Usage of multiple reference pictures can be enabled Coding Structures with Conventional B pictures Pictures are coded using B..BP or B..BI groups P pictures can be replaced with B pictures Multiple reference pictures can be enabled Two hierarchy levels: I/P pictures & B pictures

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 19 / 39

Coding Structures / Hierarchical Coding Structures

Hierarchical Coding Structures

I0 B0 1 B1 2 B2 3 B3 4 B3 5 B2 6 B3 7 B3 8 group of pictures (GOP) B0 9 B1 10 B2 11 B3 12 B3 13 B2 14 B3 15 B3 16 group of pictures (GOP) B0 17 B1 18 B2 19 B3 20 B3 21 B2 22 B3 23 B3 24 group of pictures (GOP)

Hierarchical B Pictures Modern video coding standards allow more sophisticated coding structures Can support multiple hierarchy levels (most common: dyadic structures) Video is partitioned into so-called group of pictures (GOPs) Can be used for providing temporal scalability Can be combined with multiple reference pictures (not shown in figure)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 20 / 39

Coding Structures / Hierarchical Coding Structures

QP Cascading for Hierarchical Coding Structures

I0 B0 1 B1 2 B2 3 B3 4 B3 5 B2 6 B3 7 B3 8 group of pictures (GOP) B0 9 B1 10 B2 11 B3 12 B3 13 B2 14 B3 15 B3 16 group of pictures (GOP) B0 17 B1 18 B2 19 B3 20 B3 21 B2 22 B3 23 B3 24 group of pictures (GOP)

Cascading of Quantization Parameter Reconstruction quality of hierarchy levels has different importance Key pictures (level 0) have highest importance (all pictures depend on these pictures) Can be utilized by using different quantization parameters Simple and efficient concept (with QP0 being the QP for key pictures) k > 0 : QPk = QP0 + δ1 + (k − 1) ( suitable first offset: δ1 = 4 )

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 21 / 39

Coding Structures / Coding Efficiency Impact of Choosing Coding Structure

Coding Efficiency: Hierarchical Coding Structures

I/B B B B B B B B B IBBB I/B B0 B0 B0 B1 B1 B0 B1 B1 GOP2 I/B B0 B0 B1 B1 B1 B1 B1 B1 GOP4 (simple) I/B B0 B0 B1 B2 B2 B1 B2 B2 GOP4 (hier.) I/B B0 B1 B2 B3 B3 B2 B3 B3 GOP8 (hier.)

5 10 15 20 25 30 35 40 GOP2 GOP4 (simple) GOP4 (hier.) GOP8 (hier.)

• avg. bit-rate saving vs IBBB [%]

cascaded QP assignment with δ1=4 BasketballDrive BQTerrace Cactus Kimono ParkScene Average

QP assignment (δ1 = 4) k > 0 : QPk = QP0 + 3 + k

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 22 / 39

Coding Structures / Coding Efficiency Impact of Choosing Coding Structure

Coding Efficiency: Impact of QP Cascading

I/B B B B B B B B B IBBB I/B B0 B0 B0 B1 B1 B0 B1 B1 GOP2 I/B B0 B0 B1 B1 B1 B1 B1 B1 GOP4 (simple) I/B B0 B0 B1 B2 B2 B1 B2 B2 GOP4 (hier.) I/B B0 B1 B2 B3 B3 B2 B3 B3 GOP8 (hier.)

• 5

5 10 15 20 25 30 GOP2 GOP4 (simple) GOP4 (hier.) GOP8 (hier.)

• avg. bit-rate saving vs IBBB [%]

average of entertainment-quality video content same QP δ1=1 δ1=2 δ1=3 δ1=4

same QP : QPk = QP0 cascading : QPk = QP0 + δ1 + (k − 1)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 23 / 39

Coding Structures / Coding Efficiency Impact of Choosing Coding Structure

Hierarchical Coding Structures: Effect of QP Cascading

34.0 34.5 35.0 35.5 36.0 36.5 37.0 37.5 38.0 8 16 24 32 40 48 56 64 72 80 88 96 IBBB (same QP, QP0=30): Average = 35.97 dB at 5504 kbit/s GOP8 (same QP, QP0=30): Average = 36.04 dB at 5080 kbit/s GOP8 (cascaded QP, δ1=4, QP0=30): Average = 35.49 dB at 3077 kbit/s GOP8 (cascaded QP, δ1=4, QP0=28): Average = 36.24 dB at 4052 kbit/s PSNR (Y) [dB] picture number (in display order) Cactus (1920×1080, 50 Hz)

Same QP: Roughly same PSNR over all pictures (similar as for IBBB coding) QP cascading: Higher PSNR for key pictures (nonetheless, video appears smooth) PSNR losses for non-key pictures are out-weighted by bit rate savings

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 24 / 39

Coding Structures / Random Access and Low Delay

Random Access Points: Closed GOP

B0 B0 B0 B1 B2 B2 B1 B2 B2 random access I0 B0 B0 B1 B2 B2 B1 B2 B2 5 1 2 3 4 6 7 8

Random Access Required for channel switching (broadcast) or fast forwarding (streaming) Clean random access: Insert intra pictures in regular intervals (e.g., 1-2 seconds) Closed GOP Intra picture at random access point is treated as first picture of a video sequence Decoding can be started at the intra picture Disadvantage: Disturbing temporal blocking artefacts due to interruption of prediction chain

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 25 / 39

Coding Structures / Random Access and Low Delay

Random Access Points: Open GOP

B0 I0 B0 B1 B2 B2 B1 B2 B2 – – – random access B0 B0 B1 B2 B2 B1 B2 B2 5 1 2 3 4 6 7 8

Random Access with Open GOP Structures Intra pictures at random access points are coded as normal key pictures There are pictures that succeed intra pictures in coding order, but precede them in display order Start Decoding at Random Access Point Find next intra picture in bitstream (indicated in header information) and start decoding Discard pictures that precede intra picture in display order Advantage: Attenuation of temporal blocking artefacts (often not visible)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 26 / 39

Coding Structures / Random Access and Low Delay

Low-Delay Hierarchical Coding Structures

I B B B B B B B B I B1 B1 B1 B0 B1 B1 B1 B0

5 10 15 20 25 30 30 32 34 36 38 40 42 FourPeople Johnny KristenAndSara Vidyo1 Vidyo3 Vidyo4 bit-rate saving vs IBBB [%] PSNR (Y) [dB] interactive video content average: 16%

Low-Delay Coding Structures Requirement: Pictures have to be coded in acquisition/display order Can still change reference picture list and use QP cascading Usage of hierarchical structures often improves coding efficiency

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 27 / 39

Loop Filters

In-Loop Filters in Hybrid Video Coding

scaling &

• inv. transform

transform & quantization intra mode decision intra-picture prediction entropy coding in-loop filtering decoded picture buffer motion-comp. prediction motion estimation coding mode decision buffer for current picture input pictures sk[x, y] uk[x, y] quantization indexes qk[x, y] u′

k[x, y]

s∗

k [x, y]

bitstream s∗

k [x, y]

s′

k[x, y]

intra prediction modes f iprd( s∗

k [x, y] )

ˆ sk[x, y] − motion data s′

ref[x + mx, y + my]

coding modes, intra prediction modes, motion data

Filtering of Reconstructed Pictures Reduce visual impact of coding artefacts Pictures are filtered before output and storage into decoded picture buffer Post Filter or In-Loop Filter Both improve subjective quality of pictures In-loop filters also improve prediction signal for following pictures In-loop filters are normative Support in Video Coding Standards Modern video coding standards include

• ne or more in-loop filters (can be disabled)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 28 / 39

Loop Filters

Often Observed Coding Artifacts at Low Bit Rates

Blocking Artifacts Block-wise motion-compensated prediction Block-based transform coding of prediction error Pictures are collections of coded blocks Block structures visible at low bit rates Ringing Artifacts Large transform basis functions Rather long interpolation filters Artifacts close to edges at low bit rates Particular disturbing at moving objects

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 29 / 39

Loop Filters / Deblocking Filter

Deblocking Filter: Basic Principle

block boundary block P block Q reconstructed samples before deblocking modified samples after deblocking p3 p2 p1 p0 q0 q1 q2 q3 p'

1

p' q' q'

1

p′

0 = p0 + ∆0

q′

0 = q0 − ∆0

p′

1 = p1 + ∆p1

q′

1 = q1 − ∆q1

Signal-Adaptive Smoothing of 1D Line Segments across Block Bounadries Correction values ∆0, ∆p1, and ∆q1 depend on

Differences between samples value and QP value (guess: real edge or coding artifact) Boundary filtering strength, which depends on intra/inter, motion data, whether non-zero residual

Stronger filtering for higher QP values (i.e., stronger quantization)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 30 / 39

Loop Filters / Deblocking Filter

Deblocking Filter Example

without deblocking filter with deblocking filter

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 31 / 39

Loop Filters / Deblocking Filter

Deblocking Filter: In-Loop Filter vs Post Filter

2 4 6 8 10 30 32 34 36 38 40 42 bit-rate saving [%] PSNR (Y) [dB] post filter (GOP8 coding structure) BasketballDrive BQTerrace Cactus Kimono ParkScene average: 1.5% 2 4 6 8 10 30 32 34 36 38 40 42 bit-rate saving [%] PSNR (Y) [dB] in-loop filter (GOP8 coding structure) BasketballDrive BQTerrace Cactus Kimono ParkScene average: 4.2%

Coding Experiment (HEVC): Using Hierarchical B pictures (GOP8) Non-normative post filter versus normative in-loop filter In-loop filter also improves prediction signal for following pictures In-loop filter provides large coding gains

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 32 / 39

Loop Filters / Sample Adaptive Offset

Sample Adaptive Offset: (1) Edge Offset Mode

sample value sample position

• riginal

signal reconstructed signal

a c b

class 0

a c b

class 1

a c b

class 2

a c b

class 3

First Operation Mode: Edge Offset Mode Goal: Reduce ringing artifacts around edges Select one of four edge classes for a CTU (specifies main edge direction) Classify each sample into one of five categories Transmit offset for each category (on CTU basis)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 33 / 39

Loop Filters / Sample Adaptive Offset

Edge Offset Mode: Sample Classification and Modification

a c b

category 1

a c b a c b

category 2

a c b a c b

category 3

a c b

category 4

Edge Offset Mode of Sample Adaptive Offset Filter Each sample is classified into one of five categories

depends on relation to neighboring samples

• nly two neighbors in edge mode direction are considered

Transmit non-negative offset for categories 1 to 4 Smooth reconstructed signal in gradient direction Reduction of ringing artifacts

category condition none of the following 1 (c < a) ∧ (c < b) 2 ((c = a) ∧ (c < b)) ∨ ((c < a) ∧ (c = b)) 3 ((c = a) ∧ (c > b)) ∨ ((c > a) ∧ (c = b)) 4 (c > a) ∧ (c > b)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 34 / 39

Loop Filters / Sample Adaptive Offset

Sample Adaptive Offset: (2) Band Offset Mode

sample value sample position band k band k+1 band k+2 band k+3

• riginal

signal reconstructed signal before offset correction PSNR = 42.1 dB sample value sample position Δ0 = 1 Δ1 = 1 Δ2 = -2 Δ3 = -2

• riginal

signal reconstructed signal after offset correction band signals before correction PSNR = 48.4 dB

Second Operation Mode: Band Offset Mode Divide range of sample values into 32 equally sized bands Transmit offset for four consecutive bands (signal first band index) SAO mode (none, edge offset, band offset) and parameters are transmitted at CTU level

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 35 / 39

Loop Filters / Sample Adaptive Offset

Sample Adaptive Offset (SAO): Coding Efficiency

5 10 15 20 25 30 32 34 36 38 40 42 44 46 bit-rate saving [%] PSNR (Y) [dB] IPPP coding structure FourPeople Johnny KristenAndSara Vidyo1 Vidyo3 Vidyo4 average: 13.6%

• 6
• 4
• 2

2 4 6 8 10 12 32 34 36 38 40 42 44 46 bit-rate saving [%] PSNR (Y) [dB] IBBB coding structure FourPeople Johnny KristenAndSara Vidyo1 Vidyo3 Vidyo4 average: 2.3%

Coding Experiment (HEVC): IPPP and IBBB coding Significant coding gains for IPPP coding Smaller coding gains for IBBB coding Support of bi-prediction already reduces ringing artifacts

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 36 / 39

Loop Filters / Adaptive Wiener Filter

Adaptive Wiener Filter

Signal-Adaptive Linear Filter Apply linear filter to block of picture or complete picture s′′[ x, y ] =

n

• k=−n

n

• ℓ=−n

h[ k, ℓ ] · s′[ x + k, y + ℓ ] Convolution of signal with discrete 2D filter h[ k, ℓ ] Encoder can choose filter coefficients that minimize distortion (MSE: linear equation system) Filter coefficients have to be transmitted Application of Adaptive Linear Filter Different variants: Separable or non-separable filter H.264 | AVC and H.265 | HEVC don’t specify a normative filter

but support transmission of filter coefficients for non-normative post filter

H.266 | VVC specifies normative filter, applied as in-loop filter

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 37 / 39

Loop Filters / Adaptive Wiener Filter

Adaptive Loop Filter (ALF) in H.266 | VVC

Picture is partitioned into 4×4 blocks of samples for ALF Each 4×4 luma block is partitioned into one of 25 classes

depending on direction of local gradients depending on magnitude of local gradients

Diamond shaped filters

7×7 shape used for luma 5×5 shape used for chroma

Filter coefficients are coded in special parameter set

25 classes for luma 1 class for chroma (no classification)

VVC include another cross-component ALF

Filtered luma component is added to chroma

Provides large coding gains: About 7% bit-rate savings

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 38 / 39

Summary

Summary of Lecture

Advanced Motion-Compensated Prediction Multiple reference pictures: Block-adaptive selection of reference picture used Multi-hypothesis prediction: Typically block-adaptive choice between uni- and bi-prediction Can be combined: Roughly additive gains Coding Structures Simplest variant: Code pictures in acquisition/display order Conventional B pictures: B pictures inserted between successive I/P pictures State-of-the-art: Hierarchical B pictures (multiple hierarchy levels) In-Loop Filters Deblocking filter Sample adaptive offset filter (de-ringing filter) Signal-adaptive linear filter (Wiener filter) In-loop filters improve coding efficiency relative to post filters

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Improved Inter-Picture Prediction 39 / 39