GPAC: delivery of VR/360 videos using Tiles Ahmed Rida SEKKAT - - PowerPoint PPT Presentation

▶
gpac delivery of vr 360 videos using tiles
SMART_READER_LITE
LIVE PREVIEW

GPAC: delivery of VR/360 videos using Tiles Ahmed Rida SEKKAT - - PowerPoint PPT Presentation

Dec 28, 2023 377 likes •566 views

GPAC: delivery of VR/360 videos using Tiles Ahmed Rida SEKKAT Ahmed JELIJLI Telecom ParisTech FOSDEM17 05/02/2017 Institut Mines-Tlcom What is GPAC? Multimedia Packagers, Streamers and Player Multi-platform: all

slide-1
SLIDE 1

Institut Mines-Télécom

GPAC: delivery of VR/360 videos using Tiles

Ahmed Rida SEKKAT Ahmed JELIJLI Telecom ParisTech FOSDEM17 05/02/2017

slide-2
SLIDE 2

Institut Mines-Télécom

What is GPAC?

 Multimedia Packagers, Streamers and Player

  • Multi-platform: all

desktops, mobile (iOS, Android), embedded linux

  • Features

─ Any codec, any protocol ─ Graphics & Interactivity

 Advocating Multimedia Standards

  • MPEG, W3C, IETF
  • Reference and Utility

Software for various MPEG Standards

 Open Source Software

  • Dual LGPL v2.1 /

Commercial Licensing

  • 600000+ lines of C code
  • Hosted on GitHub
  • Per month: 17000+

visits, 4000+ dl

  • gpac.io
  • gpac-licensing.com

 Academic Dissemination

  • 100+ academic

references

  • 300+ publications using

GPAC

  • IBC ‘14, ‘16, NAB ‘13,

‘15, ’16

 Collaborations

  • 8 EU funded projects
  • 18 French funded

projects

  • Several industrial-funded

projects

 Teaching

  • Labs session
  • Students projects
  • Corporate Training

2

slide-3
SLIDE 3

Institut Mines-Télécom

Key GPAC Tools

 Multimedia Packagers (MP4Box)

  • MPEG-2 TS (Live Multicast, or

DASH/HLS)

  • ISOBMF Packager & Analyzer
  • DASH Segmenter/Live

simulator/Encoder

  • Support HEVC(+layered HEVC),

AVC, HE-AAC…  Multimedia Player (MP4Client)

  • MPEG-2 TS, RTP, ISOBMF,

DASH, HLS

  • Multi-path delivery of layered

coded data

  • SVG/BIFS/VRML + JavaScript
  • VR / 3D / Auto-stereoscopic
  • utput

3

slide-4
SLIDE 4

Institut Mines-Télécom

GPAC NEWS

  • New public test infrastructure.
  • Range extension support for AVC and HEVC.
  • Improved TTML support.
  • Support of VR/360 videos, including using Tiles => This Talk.
  • Hardware decoding
  • for OSX and iOS (VideoToolBox ).
  • for Android (MediaCodec).
  • Coming soon:
  • Support for more PIFF and Smooth Streaming file format (branch).
  • Hardware accelerated encryption (branch).
  • Other projects: check https://github.com/gpac

4

slide-5
SLIDE 5

Institut Mines-Télécom 5

Streaming of VR/360 content

 Bandwidth is expensive

  • 360 videos require at least 4K x 2K.
  • Some claim: 12K x 6K for achieving 4k field of view

resolution.

4K (FOV 120°) 4K (FOV 120°) 4K (FOV 120°) 6K (180°)

slide-6
SLIDE 6

Institut Mines-Télécom 6

 Necessity to reduce bandwidth

  • Video Compression

─ of a 2D video after some projection ─ possibly after some shuffling & packing

  • Adaptive delivery

─ Deliver parts of the video based on viewpoint/viewport ─ Lower quality outside the viewpoint/viewport ─ Necessary to react quickly to motion (motion-to-photon latency)

Streaming of VR/360 content

slide-7
SLIDE 7

Institut Mines-Télécom 7

360 Projection & Packing Examples

 Projected videos will probably be packed and compressed based

  • n rectangular regions (Tiling)
  • MPEG Omnidirectional Media

Application Format (OMAF)

Equirectangular Projection (ERP) ERP with specific packing Cube-map with packing

Text of ISO/IEC CD 23000-20 Omnidirectional Media Application Format, MPEG N16636, Jan. 2017

slide-8
SLIDE 8

Institut Mines-Télécom 8

 Stream tiles with different qualities

Tiling and Adaptive Streaming Principles

Technologies under Consideration for DASH MPEG N16659

  • Jan. 2017
slide-9
SLIDE 9

Institut Mines-Télécom 9

 Or stream some tiles or even only one tile

Tiling and Adaptive Streaming Principles

slide-10
SLIDE 10

Institut Mines-Télécom 10

Tiling and Adaptive Streaming MPEG-DASH SRD

 Description of 2D relationships between videos

  • In the source content (not a composition description)
  • X,Y,W,H (possibly in arbitrary units)

 Codec agnostic

  • No assumption of tiling coding tools
  • Can be used with multiple independent videos

 Can already be used for projected videos

  • MPEG discussions about extending it to 3D relationships
  • O. Niamut, E. Thomas, D. Mico, C. Concolato, F. Denoual and S. Y. Lim,

MPEG DASH SRD - Spatial Relationship Description, ACM MMSys, Klagenfurt, Austria, May 2016

slide-11
SLIDE 11

Institut Mines-Télécom 11

MPEG-DASH SRD and HEVC tiling for VR/360 videos

 Tile-based adaptation using independent videos

slide-12
SLIDE 12

Institut Mines-Télécom 12

MPEG-DASH SRD and HEVC tiling for VR/360 videos

 HEVC Motion-constrained Tile-based adaptation

slide-13
SLIDE 13

Institut Mines-Télécom 13

Tiling and Adaptive Streaming Using HEVC

 HEVC Tiles

  • Motion-Constrained
  • Single Decoder

 Extensions to ISOBMFF

  • Independent access to tile

data

─ HEVC tile tracks “hvt1”

  • Reconstruction of the entire

video

─ Track references “sabt”/”tbas” ─ Reconstruction Rules

 MPEG-DASH

  • SRD
  • Use of @dependencyId

Motion- Constrained HEVC Encoder

Single Compliant

HEVC/ISOBMFF encapsulation DASH Segments and MPD Generator

mp4 tile segments

Input Sequence Tiling

Tile tracks MPD (SRD) HTTP requests/responses

Tile selection & single stream decoding

+ tile base track HEVC bitstream

https://gpac.wp.imt.fr/2017/02/01/hevc-tile-based-adaptation-guide/

slide-14
SLIDE 14

Institut Mines-Télécom 14

Tiling and Adaptive Streaming Streaming Strategies

 Which tiles to favor (region of interest)?  What is the acceptable quality degradation btw. tiles?

  • J. Le Feuvre and C. Concolato,

Tiled-based Adaptive Streaming using MPEG-DASH, ACM MMSys, Klagenfurt, Austria, May 2016

slide-15
SLIDE 15

Institut Mines-Télécom 15

Tiling and Adaptive Streaming 360° video

Demo using GPAC https://gpac.io/2016/05/25/srd/

slide-16
SLIDE 16

Institut Mines-Télécom

MediaCodec statistics

16

Demo using GPAC

slide-17
SLIDE 17

Institut Mines-Télécom 17

Questions ?