MPEG- -4 Standards 4 Standards MPEG-4 Standards MPEG Fernando - - PowerPoint PPT Presentation

mpeg 4 standards 4 standards mpeg 4 standards mpeg
SMART_READER_LITE
LIVE PREVIEW

MPEG- -4 Standards 4 Standards MPEG-4 Standards MPEG Fernando - - PowerPoint PPT Presentation

MPEG- -4 Standards 4 Standards MPEG-4 Standards MPEG Fernando Pereira Fernando Pereira Fernando Pereira Klagenfurt, Austria, October 2008 Klagenfurt, Austria, October 2008 Audiovisual Compression: from Basics to Systems, Fernando Pereira


slide-1
SLIDE 1

Audiovisual Compression: from Basics to Systems, Fernando Pereira

MPEG-4 Standards

Fernando Pereira Klagenfurt, Austria, October 2008

MPEG MPEG-

  • 4 Standards

4 Standards

Fernando Pereira Fernando Pereira Klagenfurt, Austria, October 2008

slide-2
SLIDE 2

Audiovisual Compression: from Basics to Systems, Fernando Pereira

The Old Analogue Times: the TV Paradigm The Old Analogue Times: the TV Paradigm The Old Analogue Times: the TV Paradigm

  • Video data modeled as a sequence of pictures with a certain number of

lines

  • One audio channel is added to the video signal
  • Video and audio have an analogue representation
  • User chooses among the available broadcast programmes
slide-3
SLIDE 3

Audiovisual Compression: from Basics to Systems, Fernando Pereira

Evolving Multimedia Context ... Evolving Multimedia Context ...

  • More information is in digital form, ...
  • More information is on-line, ...
  • More information is multimedia, …
  • Multimedia information now covers all bitrates and all

networks

  • Applications & services become ‘multimedia’ …
  • Applications & services become ‘interactive’ …
  • Internet is growing …
slide-4
SLIDE 4

Audiovisual Compression: from Basics to Systems, Fernando Pereira

New Technologies, New Needs … New Technologies, New Needs … New Technologies, New Needs …

  • Having multimedia information available wherever you are, covering

a wide range of access conditions

  • More freedom to interact with what is within the content
  • Reusing the multimedia content, combining elements of content in

new ways

  • Hyperlinking from elements of the content
  • Finding and selecting the information you need
  • Identifying, managing and protecting rights on content
  • Common technology for many types of services, notably broadcasting,

communications, retrieval Demands come from users, producers and providers !

slide-5
SLIDE 5

Audiovisual Compression: from Basics to Systems, Fernando Pereira

We and the World around us … We and the World around us … We and the World around us …

slide-6
SLIDE 6

Audiovisual Compression: from Basics to Systems, Fernando Pereira

Towards the Real World: The Object-based Representation Model T Towards

  • wards t

the he Real Real World World: : The The Object Object-

  • b

based ased Representation Representation Model Model

  • Audiovisual scene represented as a composition of objects
  • Integration of objects from different nature: A&V, natural and

synthetic, text & graphics, animated faces, arbitrary and rectangular video shapes, generic 3D, speech and music, ...

  • Object-based hyperlinking, processing, coding and description
  • Interaction with objects and their descriptions is possible
  • Object-based content may be reused in different contexts
  • Object composition principle is independent of bitrate: from low

bitrates to (virtually) lossless quality …

slide-7
SLIDE 7

Audiovisual Compression: from Basics to Systems, Fernando Pereira

Object-based Content … Object Object-

  • based Content …

based Content …

Sports results: Benfica - Sporting Sports results: Benfica - Sporting Stock information ... Stock information ...

slide-8
SLIDE 8

Audiovisual Compression: from Basics to Systems, Fernando Pereira

Conventional Audiovisual System Conventional Audiovisual System Conventional Audiovisual System

demultiplexer sync & multiplexer enc. enc. dec. dec. compositor

...

slide-9
SLIDE 9

Audiovisual Compression: from Basics to Systems, Fernando Pereira

Object-based Audiovisual System Object Object-

  • based Audiovisual System

based Audiovisual System

demultiplexer sync & multiplexer enc. enc.

... ...

Comp. enc.

Comp. Comp. Info Info

dec. dec.

Comp. dec.

compositor

... ...

slide-10
SLIDE 10

Audiovisual Compression: from Basics to Systems, Fernando Pereira

Object-based Audiovisual System Object Object-

  • based Audiovisual System

based Audiovisual System

demultiplexer sync & multiplexer

AV objects AV objects coded coded

AV objects AV objects uncoded uncoded

enc. enc.

... ...

Comp. enc.

Comp. Comp. Info Info

dec. dec.

Comp. dec.

compositor

... ...

dec.

AV objects AV objects coded coded

slide-11
SLIDE 11

Audiovisual Compression: from Basics to Systems, Fernando Pereira

Object-based Audiovisual System Object Object-

  • based Audiovisual System

based Audiovisual System

demultiplexer sync & multiplexer

AV objects AV objects coded coded

AV objects AV objects uncoded uncoded

enc. enc.

... ...

Comp. enc.

Comp. Comp. Info Info

dec. dec.

Comp. dec.

compositor

... ...

dec.

AV objects AV objects coded coded

interaction interaction

slide-12
SLIDE 12

Audiovisual Compression: from Basics to Systems, Fernando Pereira

MPEG-4: Object-Based Coding Standard MPEG MPEG-

  • 4: Object

4: Object-

  • Based Coding Standard

Based Coding Standard

  • Adopts the object-based model giving

a semantic value to the data structure

  • Integration of natural and synthetic

content, both aural and visual

  • Object-based functionalities, e.g., re-

using and manipulation capabilities

  • Powerful data model for interaction

and personalisation

  • Exploitation of synergies, e.g.,

between Video Coding, Computer Vision and Computer Graphics

slide-13
SLIDE 13

Audiovisual Compression: from Basics to Systems, Fernando Pereira

MPEG-4: Visual Coding Architecture MPEG MPEG-

  • 4: Visual Coding Architecture

4: Visual Coding Architecture

Visual Object Segment.

Visual Object 0 Encoder Visual Object 1 Encoder Visual Object N Encoder Visual Object 2 Encoder Visual Object 0 Decoder Visual Object 1 Decoder Visual Object N Decoder Visual Object 2 Decoder

Compo- sitor

Multiplexer Demultiplexer ... ...

Composition inform. Composition inform.

slide-14
SLIDE 14

Audiovisual Compression: from Basics to Systems, Fernando Pereira

Basic MPEG-4 Video Decoding Basic MPEG Basic MPEG-

  • 4 Video Decoding

4 Video Decoding

Coded shape bitstream Coded texture bitstream

Shape decoding Motion decoding

Coded motion bitstream

Variable length decoding Inverse scan Inverse quantization Inverse DCT

Motion compensation

Previous reconstructed VOP

Demultiplexer

video_object_layer_shape

Texture Decoding Texture Decoding

VOP reconstruction

Inverse AC/DC prediction

Decoded VOP

slide-15
SLIDE 15

Audiovisual Compression: from Basics to Systems, Fernando Pereira

Segmentation: a Limitation or not so Much ? Segmentation: a Limitation or not so Much ? Segmentation: a Limitation or not so Much ?

slide-16
SLIDE 16

Audiovisual Compression: from Basics to Systems, Fernando Pereira

The ‘Weather’ Girl ... The ‘Weather’ Girl ... The ‘Weather’ Girl ...

slide-17
SLIDE 17

Audiovisual Compression: from Basics to Systems, Fernando Pereira

Segmentation: the Problem that Sometimes does not Exist ... Segmentation: the Problem that Sometimes does not Segmentation: the Problem that Sometimes does not Exist ... Exist ...

slide-18
SLIDE 18

Audiovisual Compression: from Basics to Systems, Fernando Pereira

Segmentation: Automatic and Real-Time ? Segmentation: Automatic and Real Segmentation: Automatic and Real-

  • Time ?

Time ?

slide-19
SLIDE 19

Audiovisual Compression: from Basics to Systems, Fernando Pereira

Synthetic Content: Facial Animation and More … Synthetic Content: Facial Animation and More … Synthetic Content: Facial Animation and More …

slide-20
SLIDE 20

Audiovisual Compression: from Basics to Systems, Fernando Pereira

The MPEG-4 Tools (1): The Codecs The MPEG The MPEG-

  • 4 Tools (1)

4 Tools (1): : The The Codecs Codecs

  • Efficiently encode video data from very low bitrates, notably in view
  • f low bitrate channels such as the telephone line or mobile

environments, to very high quality conditions;

  • Efficiently encode music and speech data for a very wide bitrate

range, notably from transparent music to very low bitrate speech;

  • Efficiently encode text and graphics;
  • Efficiently encode time
  • c

h anging 3D generic objects as well as

some more specific 3D objects such as human faces and bodies;

  • Efficiently encode synthetically generated speech and music as

well as 3D audio spaces;

  • Provide error resilience in the encoding layer for the various data types

involved, notably in view of critical channel conditions;

slide-21
SLIDE 21

Audiovisual Compression: from Basics to Systems, Fernando Pereira

The MPEG-4 Tools (2): Systems Tools The MPEG The MPEG-

  • 4 Tools (2)

4 Tools (2): : Systems Tools Systems Tools

  • Independently represent the various objects in the scene, notably

visual objects, allowing to independently access, manipulate and re- use these objects;

  • Compose aural and visual, natural and synthetic, objects in one

audiovisual scene;

  • Describe objects and events in the scene;
  • Provide hyperlinking and interaction capabilities;
  • Provide some means to protect audiovisual content so that only

authorised users can consume it.

slide-22
SLIDE 22

Audiovisual Compression: from Basics to Systems, Fernando Pereira

MPEG-4 Systems: the MP4 File Format MPEG MPEG-

  • 4 Systems: the MP4 File Format

4 Systems: the MP4 File Format

The MP4 file format defines the storage of MPEG-4 audio, scenes and multimedia content using the ISO Base Media File Format. The MP4 file format is life-cycle file format—one in which the files can be used when capturing media, editing it, and combining it; when serving the media as a file download or as a stream; and when exchanging partial or complete presentations.

slide-23
SLIDE 23

Audiovisual Compression: from Basics to Systems, Fernando Pereira

About the MPEG File Formats About the MPEG File Formats About the MPEG File Formats

  • The family of the storage file formats is based in the concept of box-structured
  • files. A box-structured file consists of a series of boxes (sometimes called atoms),

which have a size and a type. The type field is usually four printable characters.

  • Box structured files are used in a number of applications, and it is possible to form

‘multi-purpose’ files which contain the boxes required by more than one

  • specification. Examples include the ISO Base File Format family and the JPEG

2000 file format family, which for the most part is a still-image file format.

  • The ISO Base Media File Format additionally contains structural and media data

information for timed presentations of media data such as audio, video, etc. This structure is intentionally general, so that by structuring files in different ways the same base specification can be used for files for

  • capture and local playback;
  • exchange and download, including incremental download and play;
  • editing, composition, and lay-up;
  • streaming from streaming servers;
  • more specialized uses include the use for the storage of a partial or complete MPEG-4

scene and associated object descriptions like in the MP4 file format.

slide-24
SLIDE 24

Audiovisual Compression: from Basics to Systems, Fernando Pereira

MPEG-4 Application Examples MPEG MPEG-

  • 4 Application Examples

4 Application Examples

  • Streamed video on the Internet/Intranet
  • Advanced real-time (mobile) communications
  • Multimedia broadcasting
  • Video cameras
  • Content-based storage and retrieval
  • Interactive DVD
  • Remote surveillance, monitoring
  • Studio and television post-production
  • Virtual meetings
  • ...
slide-25
SLIDE 25

Audiovisual Compression: from Basics to Systems, Fernando Pereira

The Bloomberg Case … Today ! The Bloomberg Case … Today ! The Bloomberg Case … Today !

  • Coding efficiency
  • Automatic/manual

customization of content

  • Automatic/manual

customization of screen layout based on:

  • global content and objects,

content-based AV events, language, complex user defined criteria, …

slide-26
SLIDE 26

Audiovisual Compression: from Basics to Systems, Fernando Pereira

Using Objects … Using Objects … Using Objects …

slide-27
SLIDE 27

Audiovisual Compression: from Basics to Systems, Fernando Pereira

3D Games for 3G … in Korea … 3D Games for 3G … in Korea … 3D Games for 3G … in Korea …

slide-28
SLIDE 28

Audiovisual Compression: from Basics to Systems, Fernando Pereira

MPEG-4 Objects: Old is Also New ... MPEG MPEG-

  • 4 Objects: Old is Also New ...

4 Objects: Old is Also New ...

slide-29
SLIDE 29

Audiovisual Compression: from Basics to Systems, Fernando Pereira

Video Coding in MPEG-4 Video Coding in MPEG Video Coding in MPEG-

  • 4

4

There are two Parts in the MPEG-4 standard dealing with video coding:

  • Part 2: Visual (1998)

Part 2: Visual (1998) – Specifies several coding tools targeting the efficient and error resilient of video, including arbitrarily shaped video; it also includes coding of 3D faces and bodies.

  • Part 10: Advanced Video Coding (AVC) (2003)

Part 10: Advanced Video Coding (AVC) (2003) – Specifies more efficient (about 50%) and more resilient frame based video coding tools; this Part has been jointly developed by ISO/IEC MPEG and ITU-T through the Joint Video Team (JVT) and it is often known as H.264/AVC. Each of these 2 Parts specifies several profiles with different video coding functionalities and compression efficiency versus complexity trade-

  • ffs. Part 10 only addresses rectangular frames !
slide-30
SLIDE 30

Audiovisual Compression: from Basics to Systems, Fernando Pereira

MPEG-4 Visual (Part 2) Profiles in the Market MPEG MPEG-

  • 4 Visual (Part 2) Profiles in the Market

4 Visual (Part 2) Profiles in the Market

Simple and Advanced Simple are the most used MPEG

  • 4

Visual profiles ! Simple and Advanced Simple are the most used MPEG

  • 4

Visual profiles !

  • The Simple profile is rather similar to the

H.263 standard with the addition of some error resilience tools. There are many products in the market using this profile, notably video cameras.

  • The Advanced Simple profile, more efficient,

uses also global and ¼ pel motion compensation and allows to code interlaced video.

slide-31
SLIDE 31

Audiovisual Compression: from Basics to Systems, Fernando Pereira

Conclusions Conclusions Conclusions

  • The MPEG-4 standard adopted a large number of new concepts,

notably an object-based coding architecture.

  • The MPEG-4 standard specifies a large number of visual and

audio codecs, notably for natural and synthetic content.

  • The MPEG-4 standard specifies the composition of the objects in

the scene.

  • Although the concept of object is already largely used in content

production, it is still not much used in the content available to the users, either stored or transmitted.

  • Frame based video coding in MPEG-4 has been ‘improved’ after

Part 2, Visual leading to the emerging H.264/AVC (Advanced Video Coding) standard.

slide-32
SLIDE 32

Audiovisual Compression: from Basics to Systems, Fernando Pereira

Bibliography Bibliography Bibliography

  • The MPEG-4 Book, Fernando Pereira, Touradj Ebrahimi, Prentice

Hall, 2002

  • H.264 and MPEG-4 Video Compression, Iain Richardson, John

Wiley & Sons, 2003