Object-based Layered Depth Images for improved virtual view - PowerPoint PPT Presentation

Object-based Layered Depth Images for improved virtual view synthesis in rate-constrained context Vincent Jantet 1 , Christine Guillemot 2 , Luce Morin 3 1 ENS Cachan, Antenne de Bretagne, Campus de Ker Lann, 35170 Bruz – France 2 INRIA Rennes, Bretagne Atlantique, Campus de Beaulieu, 35042 Rennes – France 3 IETR - INSA Rennes, 20 avenue des Buttes de Coësmes, 35043 Rennes – France ICIP 2011 Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 1 / 21

Context of multi-view videos Functionalities: 3DTV: Depth feeling by stereo-vision simulation. FVV: Live viewpoint selection. Require a virtual view synthesis method. Fig: 3D rendering Acquisition Compression Rendering Representation Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 2 / 21

Table on contents Introduction 1 Object-based classification 2 Rendering results 3 Compression results 4 Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 3 / 21

Outline Introduction 1 Object-based classification 2 Rendering results 3 Compression results 4 Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 4 / 21

Depth Image-Based Rendering (DIBR) Warping algorithm Input : View and associated depth map Depth i View i Output : New viewpoint (texture & depth) Disocclusions Obstructed scene information from reference viewpoint Warping They appear along depth discontinuities Solution: Add additional informations ( LDI ) Fig: Disocclusion Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 5 / 21

Layered Depth Image (LDI) [SGHS98] A set of layers, containing depth pixels from a single viewpoint From a reference viewpoint 1 st layer ⇒ Reference view Visible texture and depth 2 nd layer ⇒ Residual data Hidden texture and depth Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 6 / 21

Layered Depth Image (LDI) [SGHS98] A set of layers, containing depth pixels from a single viewpoint From a reference viewpoint 1 st layer ⇒ Reference view Visible texture and depth 2 nd layer ⇒ Residual data Hidden texture and depth 1 st layer Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 6 / 21

Layered Depth Image (LDI) [SGHS98] A set of layers, containing depth pixels from a single viewpoint From a reference viewpoint 1 st layer ⇒ Reference view Visible texture and depth 2 nd layer ⇒ Residual data Hidden texture and depth 1 st layer 2 nd layer Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 6 / 21

LDI from real multi-view plus depth sequence [JMG09] Compressed depth map 1 st layer 2 nd layer Limitations - Redundant boundaries in both layers - Moving elements in both layers Synthesized virtual view - Layers contain large depth discontinuities (Discontinuities are hard to compress) Fig: Rendering impact of depth map compression Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 7 / 21

Outline Introduction 1 Object-based classification 2 Rendering results 3 Compression results 4 Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 8 / 21

Object-based LDI representation This representation organizes pixels into layers to enhance depth continuity 1 st layer 2 nd layer Foreground Background Fig: Classical LDI depth layers Fig: Object-based LDI depth layers Method based on a growing-region algorithm Region R initialized with pixels where Z FG and Z BG are already defined. For each pixel q outside R : - Extrapolate Z FG and Z BG . - Classify q . Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 9 / 21

Object-based LDI representation This representation organizes pixels into layers to enhance depth continuity 1 st layer 2 nd layer Foreground Background Fig: Classical LDI depth layers Fig: Object-based LDI depth layers Method based on a growing-region algorithm Region R initialized with pixels where Z FG and Z BG are already defined. For each pixel q outside R : - Extrapolate Z FG and Z BG . - Classify q . Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 9 / 21

Classification: Initializing Foreground Background Unclassified Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 10 / 21

Classification: Results Foreground Background Unclassified Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 11 / 21

Background inpainting [CPT03] Advantages - Remove unnecessary boundaries - Inpainting processed once, before encoding stage - No need of inpainting method at rendering stage Fig: Background inpainting Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 12 / 21

Outline Introduction 1 Object-based classification 2 Rendering results 3 Compression results 4 Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 13 / 21

Rendering results Classical LDI rendering Virtual view inpainting O-LDI rendering Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 14 / 21

Fast mesh-based rendering Fig: Object-based LDI Continuous layers can be rendered as Fig: Meshes rendering meshes. Foreground mesh is partially transparent. Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 15 / 21

Outline Introduction 1 Object-based classification 2 Rendering results 3 Compression results 4 Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 16 / 21

LDI and MVD compression schemes MVD compression (MVC) LDI compression (MVC) Input views LDI 4 Input LDI V 1 V 3 V 5 V 7 Compression MVC MVC LDI ′ Compressed LDI 4 V ′ V ′ V ′ V ′ 1 3 5 7 Compressed views Rendering DIBR Rendering VSRS V 6 Final view V ′′ Final view 6 Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 17 / 21

Rate-distortion curve 90 SSIM (%) 88 86 Object-based LDI Classical LDI —– MPEG (MVC/VSRS) 84 0 5 10 15 20 Bitrate (Mbit/s) Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 18 / 21 Fig: Rate distortion curves

Conclusions on Object-Based LDI Advantages - Remove unnecessary boundaries ⇒ Improve compression - Static background along time - Compatible with fast mesh-based rendering - Depth continuity improves rendering quality Limit - No backward compatibility with 2D decoding scheme Questions? Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 19 / 21

[CPT03] A. Criminisi, P. Pérez, and K. Toyama. Object removal by exemplar-based inpainting. In Computer Vision and Pattern Recognition (CVPR), IEEE Computer Society Conference on , volume 2, pages 721–728, June 2003. [JMG09] Vincent Jantet, Luce Morin, and Christine Guillemot. Incremental-ldi for multi-view coding. In 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video , pages 1–4, Potsdam, Germany, May 2009. [SGHS98] Jonathan Shade, Steven Gortler, Li-wei He, and Richard Szeliski. Layered depth images. In SIGGRAPH ’98: Proceedings of the 25th annual conference on Computer graphics and interactive techniques , pages 231–242, New York, NY, USA, July 1998. ACM. Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 20 / 21