Image and Video Coding: Introduction bitstream encoder decoder - - PowerPoint PPT Presentation

▶
image and video coding introduction
SMART_READER_LITE
LIVE PREVIEW

Image and Video Coding: Introduction bitstream encoder decoder - - PowerPoint PPT Presentation

Nov 19, 2022 246 likes •839 views

Image and Video Coding: Introduction bitstream encoder decoder Motivation Image and Video Coding video bitstream bitstream video transmission encoder decoder or storage data data Main Goal of Image and Video Coding Efficient

slide-1
SLIDE 1

Image and Video Coding: Introduction

encoder decoder bitstream

slide-2
SLIDE 2

Motivation

Image and Video Coding

encoder transmission

  • r storage

decoder

bitstream bitstream video data video data

Main Goal of Image and Video Coding Efficient transmission or storage of images and videos Reduce the bit rate for a given amount of video data Image and Video Coding: Enabling Technology Enables new application or makes them economically feasible

Distribution and storage of images and video Digital television Video streaming Video conferencing

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 2 / 48

slide-3
SLIDE 3

Motivation

Important Image and Video Coding Standards of ITU-T and ISO/IEC

JPEG (1992)

[ ISO/IEC 10918-1 | ITU-T Rec. T.81 ]

Storage and distribution of digital images MPEG-2 Video (1995)

[ ITU-T Rec. H.262 | ISO/IEC 13818-2 ]

Standard definition (SD): Storage (DVD-Video) and digital television broadcast (DVB-T) H.264 | AVC : Advanced Video Coding (2003)

[ ITU-T Rec. H.264 | ISO/IEC 14496-10 ]

High definition (HD): Storage (Blu-ray) and digital television broadcast (DVB-S) Video streaming H.265 | HEVC : High Efficiency Video Coding (2013)

[ ITU-T Rec. H.265 | ISO/IEC 23008-2 ]

Ultra-high definition (UHD) video storage (UHD Blu-ray), Image storage (BPG, HEIF) Broadcast (HD: DVB-T2, UHD: DVB-S2), UHD video streaming Current project: H.266 | VVC : Versatile Video Coding (to be finalized in July 2020)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 3 / 48

slide-4
SLIDE 4

Motivation

Estimated Global Internet and Video Traffic

[ Cisco: “The Zettabyte Era: Trends and Analysis”, 2017 ]

Exabytes per month

73% 82%

2016 2017 2018 2019 2020 2021 50 100 150 200 250 300 global Internet traffic Internet video traffic Still Need Better Video Coding

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 4 / 48

slide-5
SLIDE 5

Source Data for Image and Video Coding / Gray-Level Images

Single-Component Image

Matrix of integer samples s[x, y] with x = 0, 1, · · · , W − 1 y = 0, 1, · · · , H − 1 Each sample can take values in a given range s[x, y] ∈

  • 0, 2B − 1
  • with

B = bit depth Single-component image is characterized by Image width W Image height H Sample bit depth B Number of bits for raw image data Nbits = W · H · B

x y H W

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 5 / 48

slide-6
SLIDE 6

Source Data for Image and Video Coding / Gray-Level Images

Gray-Level Image Example: Impact of Image Size (Spatial Resolution)

400×300 samples 200×150 samples 100×75 samples 50×38 samples

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 6 / 48

slide-7
SLIDE 7

Source Data for Image and Video Coding / Gray-Level Images

Gray-Level Image Example: Impact of Image Size (Spatial Resolution)

400×300 samples 200×150 samples interpolated to 400×300

  • 100×75 samples

interpolated to 400×300

  • 50×38 samples

interpolated to 400×300

  • Heiko Schwarz (Freie Universität Berlin)

— Image and Video Coding: Introduction 6 / 48

slide-8
SLIDE 8

Source Data for Image and Video Coding / Gray-Level Images

Gray-Level Image Example: Impact of Sample Bit Depth

8 bits per sample 6 bits per sample 4 bits per sample 2 bits per sample

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 7 / 48

slide-9
SLIDE 9

Source Data for Image and Video Coding / Color Images

Color Images

Human eye has three types of color receptors Require three color components Cameras/displays typically use red, green, blue A color image is represented as three matrices

  • f samples (one for each color component)

Number of bits for raw image data Nbits = 3 · W · H · B red, green, blue sample values

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 8 / 48

slide-10
SLIDE 10

Source Data for Image and Video Coding / Color Images

YCC Color Formats

RGB (red,green,blue) format unsuitable for compression: Large amount of statistical dependencies YCC formats: Transform samples at the same spatial position (x, y) RGB → YCC :   Y [x, y] C1[x, y] C2[x, y]   = round     M3×3  ·   R[x, y] G[x, y] B[x, y]   +   2B−1 2B−1     YCC → RGB :   R[x, y] G[x, y] B[x, y]   = round     M−1

3×3

 ·     Y [x, y] C1[x, y] C2[x, y]   −   2B−1 2B−1       (transform matrix M3×3 depends on actual YCC format and RGB color space) Most common format in image and video coding: YCbCr format Y : Luma component (representing brightness) Cb : Scaled difference between blue and luma Cr : Scaled difference between red and luma

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 9 / 48

slide-11
SLIDE 11

Source Data for Image and Video Coding / Color Images

Example: Comparison of RGB and YCbCr format

sample values RGB YCbCr sample values

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 10 / 48

slide-12
SLIDE 12

Source Data for Image and Video Coding / Color Images

Color Sampling Formats

RGB YCbCr 4:4:4 YCbCr 4:2:2 YCbCr 4:2:0 most common color format

half the number

  • f RGB samples
  • Heiko Schwarz (Freie Universität Berlin)

— Image and Video Coding: Introduction 11 / 48

slide-13
SLIDE 13

Source Data for Image and Video Coding / Videos

Video: Sequence of Images

Video Sequence of images Characterized by Image size W ×H Sample bit depth B Color format (typically, YCbCr 4:2:0) Frame rate F (pictures per second)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 12 / 48

slide-14
SLIDE 14

Source Data for Image and Video Coding / Videos

Video Example: Impact of Frame Rate

F = 50 Hz ( 50 pictures per second ) F = 5 Hz ( 5 pictures per second )

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 13 / 48

slide-15
SLIDE 15

Source Data for Image and Video Coding / Raw Data Rate

Raw Video Data Rate

Raw Video Data Rate: Bit rate of raw video data Rraw = (samples per time unit) · (bit depth per sample) = (frame rate F) · (image size W ·H) · (color format factor C) · (bit depth B) Example 1: Ultra High Definition (UHD) Video 60 pictures per second (US, Japan) 3840×2160 luma samples, YCbCr 4:2:0 color format, 10 bits per sample raw data rate: Rraw = 60 Hz · 3840 · 2160 · (3/2) · 10 bits ≈ 7.5 Gbits/s two-hour video: file size ≈ 6.7 TByte (without audio) Example 2: Full HD Lecture Video 60 pictures per second 1920×1080 luma samples (full HD), RGB color format (screen capture), 8 bits per sample raw data rate: Rraw = 60 Hz · 1920 · 1080 · 3 · 8 bits ≈ 2.986 Gbits/s 90 min. lecture: file size = 2.986 Gbits/s · 90 min · 60 s/min / (8 bit/byte) ≈ 2 TByte

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 14 / 48

slide-16
SLIDE 16

The Image and Video Coding Problem / Video Communication

Typical Video Communication Scenario

capture raw input data samples raw input data samples pre- processing video encoder video encoder transmission channel (can be replaced by storage) channel encoder modulator channel demodulator channel decoder video decoder video decoder post- processing raw output data samples raw output data samples

  • utput and

perception encoded bitstream received bitstream no transmission errors bitstream

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 15 / 48

slide-17
SLIDE 17

The Image and Video Coding Problem / Video Communication

Typical Video Communication Scenario

capture raw input data samples raw input data samples pre- processing video encoder video encoder transmission channel (can be replaced by storage) channel encoder modulator channel demodulator channel decoder video decoder video decoder post- processing raw output data samples raw output data samples

  • utput and

perception encoded bitstream received bitstream no transmission errors bitstream

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 15 / 48

slide-18
SLIDE 18

The Image and Video Coding Problem / Video Communication

Typical Video Communication Scenario

capture raw input data samples raw input data samples pre- processing video encoder video encoder transmission channel (can be replaced by storage) channel encoder modulator channel demodulator channel decoder video decoder video decoder post- processing raw output data samples raw output data samples

  • utput and

perception encoded bitstream received bitstream no transmission errors bitstream

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 15 / 48

slide-19
SLIDE 19

The Image and Video Coding Problem / Video Communication

Application Examples

HD movie UHD broadcast Video conference

  • n Blu-ray Disc
  • ver DVB-S2
  • ver the Internet

raw video format 1920×1080 luma samples 3840×2160 luma samples 1280×720 luma samples YCbCr 4:2:0 color format YCbCr 4:2:0 color format YCbCr 4:2:0 color format 8 bits per sample 10 bits per sample 8 bits per sample 24 frames per second 60 frames per second 50 frames per second raw data rate

  • ca. 600 Mbit/s
  • ca. 7.5 Gbit/s
  • ca. 550 Mbit/s

channel bit rate 36 Mbit/s (read speed) 58 Mbit/s (8PSK 2/3) depends on connection video bit rate

  • ca. 20 Mbit/s
  • ca. 15 Mbit/s
  • ca. 1 Mbit/s

required compression

  • ca. 30 : 1
  • ca. 500 : 1
  • ca. 500 : 1

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 16 / 48

slide-20
SLIDE 20

The Image and Video Coding Problem / Video Communication

Types of Image and Video Compression

Lossless Compression Invertible / reversible: Original input data can be completely recovered Examples: PNG, JPEG-LS for images H.265 | HEVC lossless for video Achievable compression ratios typically in range from 2:1 to 3:1 Lossy Compression Not invertible: Only approximation of original input data can be recovered Achieves much higher compression ratios Examples: JPEG, JPEG-2000 for images MPEG-2, H.264 | AVC, H.265 | HEVC for video Dominant form of compression for images and video

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 17 / 48

slide-21
SLIDE 21

The Image and Video Coding Problem / The Coding Problem

The Basic Image and Video Coding Problem

Image and Video Coding Problem Two equivalent formulations: Representing images/videos with the highest fidelity possible within an available bit rate and Representing images/videos using the lowest bit rate possible while maintaining a specified reproduction quality Image/Video Codec Codec: System of encoder and decoder

encoder decoder bitstream

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 18 / 48

slide-22
SLIDE 22

The Image and Video Coding Problem / The Coding Problem

Video Coding in Practice

Characteristics of Video Codecs Bit rate: Throughput of the communication channel Quality: Fidelity of the reconstructed signal Delay: Start-up latency, end-to-end delay Complexity: Computational complexity, memory requirement, memory access requirements Practical Video Coding Problem Given a maximum allowed complexity and a maximum delay, achieve an optimal trade-off between bit rate and reconstruction quality for the transmission problem in the targeted application In this course: Will concentrate on basic video codec Ignore aspects of transmission channel (e.g., transmission errors)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 19 / 48

slide-23
SLIDE 23

The Image and Video Coding Problem / Summary

Intermediate Summary: Goal of Image and Video Coding

Raw Data Format for Images and Videos Gray-Level Images: Matrix of samples (image size, bit depth per sample) Color Images: Three arrays of samples (typically YCbCr 4:2:0) Video: Sequence of images (frame rate) Extremely large raw data rate (for example: 7.5 GBits/s for UHD 60Hz) Image and Video Coding Represent image/video data using much smaller bit rate (suitable for certain application) Require lossy compression (approximation of original input data) Main goal of image and video: Best possible quality for given bit rate, or Smallest possible bit rate for given quality In practice: Take into account delay and complexity

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 20 / 48

slide-24
SLIDE 24

Image Compression Example: JPEG / Overview

Image Compression Example: JPEG Baseline

YCbCr 4:2:0 color format Y Cb Cr Partition color components (Y, Cb, Cr) into blocks of 8 × 8 samples Transform coding of 8 × 8 blocks of samples

2d block transform scalar quantization entropy coding block of samples sequence

  • f bits

2d inverse transform scaling entropy decoding reconstructed block sequence

  • f bits

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 21 / 48

slide-25
SLIDE 25

Image Compression Example: JPEG / Overview

JPEG Principle — Transform Coding of Sample Blocks

input component input component

partitioned into blocks

  • input component

partitioned into blocks

  • 222 199 148

97 205 180 129 82 189 160 117 49 184 150 88 64

  • riginal

samples

2d block transform scalar quantization entropy coding

bitstream

2d block transform

1 Partitioning into Blocks 2 Transform Coding of Blocks

a 2D Transform:

Energy compaction

b Quantization:

Approximate signal (remove invisible details)

c Entropy Coding: Represent data with

as little bits as possible

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 22 / 48

slide-26
SLIDE 26

Image Compression Example: JPEG / Transform

Orthogonal Transform of Sample Vectors

forward transform:

t

=

A

·

s s′

=

A−1

·

t′

inverse transform:

Linear Transform of a Vector of Samples Consider vector s of neighboring samples (e.g., row or column of a block) Forward transform and inverse transform: Matrix-vector multiplications t = A · s A : transform matrix s′ = A−1 · t′ t : vector of transform coefficients Orthogonal Transform Transform matrix A has the property: A−1 = AT (rotation/reflection in signal space) Same mean squared error (MSE) in sample and transform domain

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 23 / 48

slide-27
SLIDE 27

Image Compression Example: JPEG / Transform

Example: 2D Histogram for Natural Gray-Level Images

sn sn+1 N(sn, sn+1) joint histogram

  • f two horizontally

adjacent samples 15 test images (each 768×512)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 24 / 48

slide-28
SLIDE 28

Image Compression Example: JPEG / Transform

Effect of Transform for Correlated Sample Vectors

s0 s1 t0 t1

s0 s1 Orthogonal transform: A = 1 √ 2 1 1 1 −1

  • concentration of

signal energy into few coefficients

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 25 / 48

slide-29
SLIDE 29

Image Compression Example: JPEG / Transform

Transforms in Image and Video Coding s t = A AT and t′ s′ = AT A

Separable Orthogonal Block Transforms Transform of rows and columns of a block Transforms in Image and Video Coding Most often: Discrete Cosine Transform (DCT) or approximation thereof

  • riginal

block horizontal DCT vertical DCT after 2d DCT

Effect of Transform: Compaction of Signal Energy

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 26 / 48

slide-30
SLIDE 30

Image Compression Example: JPEG / Transform

JPEG Principle — Transform Coding of Sample Blocks

input component

partitioned into blocks

  • 222 199 148

97 205 180 129 82 189 160 117 49 184 150 88 64

  • riginal

samples

entropy coding

bitstream

2d block transform 2d block transform

1810 620 159 13 223 −7 −31 8 32 −10 34 −27 −3 18 −34 28 transform coefficients

scalar quantization scalar quantization

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 27 / 48

slide-31
SLIDE 31

Image Compression Example: JPEG / Quantization

Scalar Quantization

∆ −∆ 2∆ −2∆ 3∆ −3∆ 4∆ −4∆ quantization indexes k: 1 −1 2 −2 3 −3 4 −4 ∆ t f (t)

Typical Scalar Quantizer: Uniform Reconstruction Quantizer Reconstruction levels: Uniformly spaced and centered around zero (quantization step size ∆) Simple decoder operation: t′ = k · ∆ (k: quantization index) Encoder: Freedom to adapt decision to source and entropy coding Simplest encoder: k = round (t/∆) Effect of Quantization: Approximation of Original Signal Quantization step size ∆ determines trade-off between quality and bit rate

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 28 / 48

slide-32
SLIDE 32

Image Compression Example: JPEG / Quantization

JPEG Principle — Transform Coding of Sample Blocks

input component

partitioned into blocks

  • 222 199 148

97 205 180 129 82 189 160 117 49 184 150 88 64

  • riginal

samples

2d block transform

1810 620 159 13 223 −7 −31 8 32 −10 34 −27 −3 18 −34 28 transform coefficients bitstream

∆ = 100 scalar quantization scalar quantization

18 6 2 2 quantization indexes

entropy coding entropy coding

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 29 / 48

slide-33
SLIDE 33

Image Compression Example: JPEG / Entropy Coding

Entropy Coding of Quantization Indexes

1 −1 2 −2 3 −3 4 −4 k p(k)

Lossless Coding of Quantization Indexes Simplest approach: Codeword table Consider symbol probabilities Example

indexes k: 18, 6, 2, 0, 2, 0, · · ·, 0 (16 values) simple: 7 + 7 + 2 · 7 + 12 · 6 = 100 bits better: 10 + 6 + 2 · 4 + 12 · 1 = 36 bits

Entropy Coding: Minimize Number of Bits

simple better k codewords codewords 000000 ±1 000001 s 10 0 s ±2 000010 s 10 1 s ±3 000011 s 110 00 s ±4 000100 s 110 01 s ±5 000101 s 110 10 s ±6 000110 s 110 11 s ±7 000111 s 1110 000 s ±8 001000 s 1110 001 s · · · · · · · · · ±14 001110 s 1110 111 s ±15 001111 s 11110 0000 s ±16 010000 s 11110 0001 s ±17 010001 s 11110 0010 s ±18 010010 s 11110 0011 s · · · · · · · · · ±63 111111 s 1111110 000000 s

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 30 / 48

slide-34
SLIDE 34

Image Compression Example: JPEG / Entropy Coding

Entropy Coding: Exploitation of Dependencies

18 6 2 2 18 6 2 2

18, 6, 2, 0, 0, 2, 0, · · · , 0 Typical Blocks of Quantization Indexes Most zeros are at high-frequency locations Simple Entropy Coding Improvement Scan quantization indexes using zig-zag scan Include end-of-block symbol (eob) in code Example

scanned k: 18, 6, 2, 0, 0, 2, 0, · · ·, 0 (16 values) new code: 11 + 7 + 5 + 2 · 1 + 5 + 2 = 32 bits

k code with eob (eob) 10 ±1 110 0 s ±2 110 1 s ±3 1110 00 s ±4 1110 01 s ±5 1110 10 s ±6 1110 11 s ±7 11110 000 s ±8 11110 001 s · · · · · · ±14 11110 111 s ±15 111110 0000 s ±16 111110 0001 s ±17 111110 0010 s ±18 111110 0011 s · · · · · · ±63 11111110 000000 s

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 31 / 48

slide-35
SLIDE 35

Image Compression Example: JPEG / Entropy Coding

JPEG Principle — Transform Coding of Sample Blocks

input component

partitioned into blocks

  • riginal

samples

2d block transform

transform coefficients bitstream

scalar quantization

quantization indexes

∆ = 100

222 199 148 97 205 180 129 82 189 160 117 49 184 150 88 64 1810 620 159 13 223 −7 −31 8 32 −10 34 −27 −3 18 −34 28

entropy coding

18 6 2 2

entropy coding

1111 1000 1101 1101 1011 0100 0110 1010

18 6 2 2

1111 1000 1101 1101 1011 0100 0110 1010

4×4×8 bit = 128 bit 32 bit Reduction to 25% of raw data size Compression factor: 4.0

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 32 / 48

slide-36
SLIDE 36

Image Compression Example: JPEG / Entropy Coding

JPEG Principle — Transform Coding of Sample Blocks

input component

partitioned into blocks

  • riginal

samples

2d block transform

transform coefficients bitstream

scalar quantization

quantization indexes

∆ = 100

222 199 148 97 205 180 129 82 189 160 117 49 184 150 88 64 1810 620 159 13 223 −7 −31 8 32 −10 34 −27 −3 18 −34 28

entropy coding

18 6 2 2

entropy coding

1111 1000 1101 1101 1011 0100 0110 1010

18 6 2 2

1111 1000 1101 1101 1011 0100 0110 1010

4×4×8 bit = 128 bit 32 bit Reduction to 25% of raw data size Compression factor: 4.0

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 32 / 48

slide-37
SLIDE 37

Image Compression Example: JPEG / Decoding

JPEG Principle — Transform Coding of Sample Blocks

input component

partitioned into blocks

  • riginal

samples

2d block transform

transform coefficients

scalar quantization

quantization indexes

∆ = 100 entropy coding

1111 1000 1101 1101 1011 0100 0110 1010

bitstream bitstream bitstream

entropy decoding

18 6 2 2

entropy decoding

quantization indexes

same indexes

scaling

1800 600 200 200

scaling

reconstructed transform coefficients

∆ = 100

approximation lossy part

2d inverse transform

214 195 142 92 202 184 131 81 185 166 113 63 173 155 102 52

2d inverse transform

reconstructed samples

reconstructed component

approximation

8 4 6 5 3 −4 −2 1 4 −6 4 −14 11 −5 −14 12

difference signal / reconstruction error vs. number of bits trade-off is determined by quantization step size ∆

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 33 / 48

slide-38
SLIDE 38

Image Compression Example: JPEG / Decoding

JPEG Principle — Transform Coding of Sample Blocks

input component

partitioned into blocks

  • riginal

samples

2d block transform

transform coefficients

scalar quantization

quantization indexes

∆ = 100 entropy coding

1111 1000 1101 1101 1011 0100 0110 1010

bitstream bitstream bitstream

entropy decoding

18 6 2 2

entropy decoding

quantization indexes

same indexes

scaling

1800 600 200 200

scaling

reconstructed transform coefficients

∆ = 100

approximation lossy part

2d inverse transform

214 195 142 92 202 184 131 81 185 166 113 63 173 155 102 52

2d inverse transform

reconstructed samples

reconstructed component

approximation

8 4 6 5 3 −4 −2 1 4 −6 4 −14 11 −5 −14 12

difference signal / reconstruction error vs. number of bits trade-off is determined by quantization step size ∆

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 33 / 48

slide-39
SLIDE 39

Image Compression Example: JPEG / JPEG Examples

Image Compression: Quality versus Compression Ratio

100 %

Original Image (1024×640 image points, 1966 KB) Lossy Compressed: JPEG (Quality 95)

11.76 % 8.5 : 1

Lossy Compressed: JPEG (Quality 75)

3.08 % 32 : 1

Lossy Compressed: JPEG (Quality 50)

1.89 % 53 : 1

Lossy Compressed: JPEG (Quality 25)

1.08 % 92 : 1

Lossy Compressed: JPEG (Quality 1)

0.24 % 414 : 1

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 34 / 48

slide-40
SLIDE 40

Image Compression Example: JPEG / JPEG Examples

Image Compression: Quality versus Compression Ratio

100 %

Original Image (1024×640 image points, 1966 KB) Lossy Compressed: JPEG (Quality 95)

11.76 % 8.5 : 1

Lossy Compressed: JPEG (Quality 75)

3.08 % 32 : 1

Lossy Compressed: JPEG (Quality 50)

1.89 % 53 : 1

Lossy Compressed: JPEG (Quality 25)

1.08 % 92 : 1

Lossy Compressed: JPEG (Quality 1)

0.24 % 414 : 1

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 34 / 48

slide-41
SLIDE 41

Image Compression Example: JPEG / JPEG Examples

Image Compression: Quality versus Compression Ratio

100 %

Original Image (1024×640 image points, 1966 KB) Lossy Compressed: JPEG (Quality 95)

11.76 % 8.5 : 1

Lossy Compressed: JPEG (Quality 75)

3.08 % 32 : 1

Lossy Compressed: JPEG (Quality 50)

1.89 % 53 : 1

Lossy Compressed: JPEG (Quality 25)

1.08 % 92 : 1

Lossy Compressed: JPEG (Quality 1)

0.24 % 414 : 1

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 34 / 48

slide-42
SLIDE 42

Image Compression Example: JPEG / JPEG Examples

Image Compression: Quality versus Compression Ratio

100 %

Original Image (1024×640 image points, 1966 KB) Lossy Compressed: JPEG (Quality 95)

11.76 % 8.5 : 1

Lossy Compressed: JPEG (Quality 75)

3.08 % 32 : 1

Lossy Compressed: JPEG (Quality 50)

1.89 % 53 : 1

Lossy Compressed: JPEG (Quality 25)

1.08 % 92 : 1

Lossy Compressed: JPEG (Quality 1)

0.24 % 414 : 1

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 34 / 48

slide-43
SLIDE 43

Image Compression Example: JPEG / JPEG Examples

Image Compression: Quality versus Compression Ratio

100 %

Original Image (1024×640 image points, 1966 KB) Lossy Compressed: JPEG (Quality 95)

11.76 % 8.5 : 1

Lossy Compressed: JPEG (Quality 75)

3.08 % 32 : 1

Lossy Compressed: JPEG (Quality 50)

1.89 % 53 : 1

Lossy Compressed: JPEG (Quality 25)

1.08 % 92 : 1

Lossy Compressed: JPEG (Quality 1)

0.24 % 414 : 1

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 34 / 48

slide-44
SLIDE 44

Image Compression Example: JPEG / JPEG Examples

Image Compression: Quality versus Compression Ratio

100 %

Original Image (1024×640 image points, 1966 KB) Lossy Compressed: JPEG (Quality 95)

11.76 % 8.5 : 1

Lossy Compressed: JPEG (Quality 75)

3.08 % 32 : 1

Lossy Compressed: JPEG (Quality 50)

1.89 % 53 : 1

Lossy Compressed: JPEG (Quality 25)

1.08 % 92 : 1

Lossy Compressed: JPEG (Quality 1)

0.24 % 414 : 1

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 34 / 48

slide-45
SLIDE 45

Image Compression Example: JPEG / Summary

Intermediate Summary: Basic Principle of JPEG

JPEG Baseline Coding of color images in YCbCr 4:2:0 format All color components are partitioned into 8×8 blocks of samples Each 8×8 block is coded using transform coding Transform Coding of Sample Blocks

1 Orthogonal Block Transform

Compaction of signal energy (exploit strong dependencies between neighboring samples)

2 Scalar Quantization of Transform Coefficients

Approximation of transform coefficients (divide by quantization step size, rounding)

3 Entropy Coding of Quantization Indexes

Represent quantization indexes (integer numbers) with as little bits as possible

Quantization step size controls trade-off between bit rate and quality

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 35 / 48

slide-46
SLIDE 46

Video Coding Basics / Inter-Picture Similarities

Example Video (832×480 @ 50 Hz)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 36 / 48

slide-47
SLIDE 47

Video Coding Basics / Inter-Picture Similarities

Successive Pictures in Video Sequences

successive picture are typically very similar (exception: scene cuts)

t

Important: Utilize large amount of dependencies between video picture Basic Idea: Predict current picture from already coded previous picture

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 37 / 48

slide-48
SLIDE 48

Video Coding Basics / Motion-Compensated Prediction

Simple Variant: Frame Difference Coding

  • rec. previous picture s′

n−1

current original picture sn prediction error un (difference)

Partition current picture sn into rectangular blocks Block-wise coding of current picture sn

1 Get prediction error:

un[x, y] = sn[x, y] − s′

n−1[x, y]

2 Transform coding:

un[x, y] → u′

n[x, y]

(similar as in JPEG)

3 Transmit in bitstream: Quantization indexes {k} 4 Reconstruction:

s′

n[x, y] = s′ n−1[x, y] + u′ n[x, y]

Problem: Ineffective for moving regions

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 38 / 48

slide-49
SLIDE 49

Video Coding Basics / Motion-Compensated Prediction

Important Improvement: Motion-Compensated Prediction

  • rec. previous picture s′

n−1

current original picture sn prediction error un (MCP)

Estimate motion of blocks in current picture relative to previous picture s′

n−1

Motion is specified by displacement / motion vector (mx, my)

Block-wise coding of current picture sn

1 Get prediction error:

un[x, y] = sn[x, y] − s′

n−1[x + mx, y + my]

2 Transform coding:

un[x, y] → u′

n[x, y]

(similar as in JPEG)

3 Transmit in bitstream: Motion vector (mx, my) and quantization indexes {k} 4 Reconstruction:

s′

n[x, y] = s′ n−1[x + mx, y + my] + u′ n[x, y]

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 39 / 48

slide-50
SLIDE 50

Video Coding Basics / Motion-Compensated Prediction

Efficiency of Motion-Compensated Prediction ≈ 80 %

10 20

Kimono, 1920×1080, 24Hz

JPEG difference coding MCP coding target quality bit rate [Mbit/s] Quality

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 40 / 48

slide-51
SLIDE 51

Video Coding Basics / Video Coding Examples

Video Compression: Quality versus Compression Ratio

H.264 | AVC @ 2.7 Mbit/s (90 : 1) H.264 | AVC @ 400 kbit/s (600 : 1) Original: 240 Mbits/s (832×480, 50Hz, YCbCr 4:2:0, 8 bit)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 41 / 48

slide-52
SLIDE 52

Coding Efficiency / Measuring Bit Rate and Quality

Measuring Coding Efficiency: Average Bit Rate

Core Properties of a Bitstream Average bit rate (in practice: distribution over time matters also) Quality of reconstructed image or video Average Bit rate images: R = number of bits in bitstream for the image

number of luma samples in the image

video: R = number of bits in bitstream for the video sequence

nominal duration of the video sequence

Reconstruction Quality Ideally: Quality as perceived by human being In practice: Often use Peak Signal-to-Noise Ratio (PSNR) (based on square error)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 42 / 48

slide-53
SLIDE 53

Coding Efficiency / Measuring Bit Rate and Quality

Measuring Coding Efficiency: Peak Signal-To-Noise Ratio (PSNR)

Distortion Measure: Mean Squared Error (MSE) MSE = 1 W · H

  • ∀x,y
  • s′[x, y] − s[x, y]
  • 2

s[x, y] :

  • riginal samples

s′[x, y] : reconstructed samples W , H : image width and height Quality Measure: Peak Signal-To-Noise Ratio (PSNR) component: PSNR [dB] = 10 · log10 s2

max

MSE

  • smax = 2B − 1
  • RGB image:

PSNR [dB] = 10 · log10

  • 3 · s2

max

MSER + MSEG + MSEB

  • similar for YCbCr
  • video:

PSNR [dB] = 1 N

N−1

  • k=0

PSNR(k)

  • average over pictures
  • Heiko Schwarz (Freie Universität Berlin)

— Image and Video Coding: Introduction 43 / 48

slide-54
SLIDE 54

Coding Efficiency / Rate-PSNR Plots

Trade-Off between Quality and Compression Ratio

Coding Efficiency Ability to trade-off bit rate and reconstruction quality Want best reconstruction quality for a given bitrate (or vice versa) codec A better codec B bit rate quality (e.g. PSNR)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 44 / 48

slide-55
SLIDE 55

Coding Efficiency / Example: Comparison of JPEG and HEVC

Image Coding Comparion: JPEG versus HEVC 100 : 1

0.5 1 1.5 2 28 30 32 34 36 38

JPEG HEVC

bit rate [bits per pixel] PSNRRGB [dB]

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 45 / 48

slide-56
SLIDE 56

Coding Efficiency / Example: Comparison of JPEG and HEVC

Visual Comparison: JPEG versus HEVC ≈ 100:1 compression

JPEG (RGB-PSNR = 28.2 dB) HEVC (RGB-PSNR = 31.4 dB)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 46 / 48

slide-57
SLIDE 57

Summary

Summary of Lecture

Image and Video Applications Very large raw data rates / file sizes Require compression for transmission and storage Goal of Image and Video Coding Minimize bit rate while preserving certain reconstruction quality, or Maximize reconstruction quality while not exceeding given bit rate budget Basic Techniques in Image and Video Coding Transform coding of sample blocks (transform, quantization, entropy coding) Motion-compensated prediction Coding Efficiency Trade-off between bit rate and reconstruction quality Simple quality measure: Peak Signal-to-Noise Ratio (PSNR)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 47 / 48

slide-58
SLIDE 58

Summary

Outline of Course

Raw Data Formats Human vision, image capture, image display, representation formats Image Coding The JPEG Standard Improvements after JPEG Hybrid Video Coding Motion-compensated coding Improved inter-picture coding concepts State-of-the art video coding standards Exercises Collaborative implementation of our own image codec Step-by-step improvement using concepts discussed in lectures

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Introduction 48 / 48