GPAC: delivery of VR/360 videos using Tiles Ahmed Rida SEKKAT - PowerPoint PPT Presentation

GPAC: delivery of VR/360 videos using Tiles Ahmed Rida SEKKAT Ahmed JELIJLI Telecom ParisTech FOSDEM17 05/02/2017 Institut Mines-Télécom

What is GPAC?  Multimedia Packagers, Streamers and Player •  Multi-platform: all Academic Dissemination desktops, mobile (iOS, • 100+ academic Android), embedded references linux • 300+ publications using • Features GPAC ─ Any codec, any protocol • IBC ‘14, ‘16, NAB ‘13, ─ Graphics & Interactivity ‘15, ’16  Advocating Multimedia  Collaborations Standards • 8 EU funded projects • MPEG, W3C, IETF • 18 French funded • Reference and Utility projects Software for various • Several industrial-funded MPEG Standards projects  Open Source Software  Teaching • Dual LGPL v2.1 / • Labs session Commercial Licensing • Students projects • 600000+ lines of C code • Corporate Training • Hosted on GitHub • Per month: 17000+ visits, 4000+ dl • gpac.io • gpac-licensing.com 2 Institut Mines-Télécom

Key GPAC Tools  Multimedia Packagers (MP4Box) • MPEG-2 TS (Live Multicast, or DASH/HLS) • ISOBMF Packager & Analyzer • DASH Segmenter/Live simulator/Encoder • Support HEVC(+layered HEVC), AVC, HE- AAC…  Multimedia Player (MP4Client) • MPEG-2 TS, RTP, ISOBMF, DASH, HLS • Multi-path delivery of layered coded data • SVG/BIFS/VRML + JavaScript • VR / 3D / Auto-stereoscopic output 3 Institut Mines-Télécom

GPAC NEWS • New public test infrastructure. • Range extension support for AVC and HEVC. • Improved TTML support. • Support of VR/360 videos, including using Tiles => This Talk. • Hardware decoding • for OSX and iOS (VideoToolBox ). • for Android (MediaCodec). • Coming soon: • Support for more PIFF and Smooth Streaming file format (branch). • Hardware accelerated encryption (branch). • Other projects: check https://github.com/gpac 4 Institut Mines-Télécom

Streaming of VR/360 content  Bandwidth is expensive • 360 videos require at least 4K x 2K. • Some claim: 12K x 6K for achieving 4k field of view resolution. 6K (180°) 4K (FOV 120°) 4K (FOV 120°) 4K (FOV 120°) 5 Institut Mines-Télécom

Streaming of VR/360 content  Necessity to reduce bandwidth • Video Compression ─ of a 2D video after some projection ─ possibly after some shuffling & packing • Adaptive delivery ─ Deliver parts of the video based on viewpoint/viewport ─ Lower quality outside the viewpoint/viewport ─ Necessary to react quickly to motion (motion-to-photon latency) 6 Institut Mines-Télécom

360 Projection & Packing Examples Equirectangular Projection (ERP) ERP with specific packing  Projected videos will probably be packed and compressed based on rectangular regions (Tiling) • MPEG Omnidirectional Media Application Format (OMAF) Cube-map with packing Text of ISO/IEC CD 23000-20 Omnidirectional Media Application Format , MPEG N16636, Jan. 2017 7 Institut Mines-Télécom

Tiling and Adaptive Streaming Principles  Stream tiles with different qualities Technologies under Consideration for DASH MPEG N16659 Jan. 2017 8 Institut Mines-Télécom

Tiling and Adaptive Streaming Principles  Or stream some tiles or even only one tile 9 Institut Mines-Télécom

Tiling and Adaptive Streaming MPEG-DASH SRD  Description of 2D relationships between videos • In the source content (not a composition description) • X,Y,W,H (possibly in arbitrary units)  Codec agnostic • No assumption of tiling coding tools • Can be used with multiple independent videos  Can already be used for projected videos • MPEG discussions about extending it to 3D relationships O. Niamut, E. Thomas, D. Mico, C. Concolato, F. Denoual and S. Y. Lim, MPEG DASH SRD - Spatial Relationship Description , ACM MMSys, Klagenfurt, Austria, May 2016 10 Institut Mines-Télécom

MPEG-DASH SRD and HEVC tiling for VR/360 videos  Tile-based adaptation using independent videos 11 Institut Mines-Télécom

MPEG-DASH SRD and HEVC tiling for VR/360 videos  HEVC Motion-constrained Tile-based adaptation 12 Institut Mines-Télécom

Tiling and Adaptive Streaming Using HEVC Input Sequence Tiling Motion-  HEVC Tiles Constrained • HEVC Encoder Motion-Constrained • Single Decoder Single HEVC Tile tracks Compliant bitstream  Extensions to ISOBMFF DASH • Independent access to tile Segments HEVC/ISOBMFF data and MPD encapsulation ─ HEVC tile tracks “hvt1” Generator • Reconstruction of the entire Tile selection + tile base track video & single ─ Track references stream “ sabt ”/” tbas ” decoding ─ Reconstruction Rules MPD (SRD) HTTP requests/responses  MPEG-DASH • SRD • Use of @dependencyId mp4 tile segments https://gpac.wp.imt.fr/2017/02/01/hevc-tile-based-adaptation-guide/ 13 Institut Mines-Télécom

Tiling and Adaptive Streaming Streaming Strategies  Which tiles to favor (region of interest)?  What is the acceptable quality degradation btw. tiles? J. Le Feuvre and C. Concolato, Tiled-based Adaptive Streaming using MPEG-DASH , ACM MMSys, Klagenfurt, Austria, May 2016 14 Institut Mines-Télécom

Tiling and Adaptive Streaming 360° video Demo using GPAC https://gpac.io/2016/05/25/srd/ 15 Institut Mines-Télécom

MediaCodec statistics Demo using GPAC 16 Institut Mines-Télécom

Questions ? 17 Institut Mines-Télécom