Object-based Layered Depth Images for improved virtual view - - PowerPoint PPT Presentation

▶

Mar 08, 2023 210 likes •445 views

Object-based Layered Depth Images for improved virtual view synthesis in rate-constrained context Vincent Jantet 1 , Christine Guillemot 2 , Luce Morin 3 1 ENS Cachan, Antenne de Bretagne, Campus de Ker Lann, 35170 Bruz France 2 INRIA Rennes,

SLIDE 1

Object-based Layered Depth Images for improved virtual view synthesis in rate-constrained context

Vincent Jantet 1, Christine Guillemot 2, Luce Morin 3

1 ENS Cachan, Antenne de Bretagne, Campus de Ker Lann, 35170 Bruz – France 2 INRIA Rennes, Bretagne Atlantique, Campus de Beaulieu, 35042 Rennes – France 3 IETR - INSA Rennes, 20 avenue des Buttes de Coësmes, 35043 Rennes – France

ICIP 2011

Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 1 / 21

SLIDE 2

Context of multi-view videos

Functionalities:

3DTV: Depth feeling by stereo-vision simulation. FVV: Live viewpoint selection. Require a virtual view synthesis method.

Fig: 3D rendering

Acquisition Compression Rendering Representation

Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 2 / 21

SLIDE 3

Table on contents

1

Introduction

2

Object-based classification

3

Rendering results

4

Compression results

Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 3 / 21

SLIDE 4

Outline

1

Introduction

2

Object-based classification

3

Rendering results

4

Compression results

Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 4 / 21

SLIDE 5

Depth Image-Based Rendering (DIBR)

Warping

View i Depth i

Warping algorithm

Input: View and associated depth map Output: New viewpoint (texture & depth)

Disocclusions

Obstructed scene information from reference viewpoint They appear along depth discontinuities Solution: Add additional informations (LDI)

Fig: Disocclusion

Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 5 / 21

SLIDE 6

Layered Depth Image (LDI) [SGHS98]

A set of layers, containing depth pixels from a single viewpoint

From a reference viewpoint

1st layer ⇒ Reference view Visible texture and depth 2nd layer ⇒ Residual data Hidden texture and depth

Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 6 / 21

SLIDE 7

Layered Depth Image (LDI) [SGHS98]

A set of layers, containing depth pixels from a single viewpoint

From a reference viewpoint

1st layer ⇒ Reference view Visible texture and depth 2nd layer ⇒ Residual data Hidden texture and depth 1st layer

Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 6 / 21

SLIDE 8

Layered Depth Image (LDI) [SGHS98]

A set of layers, containing depth pixels from a single viewpoint

From a reference viewpoint

1st layer ⇒ Reference view Visible texture and depth 2nd layer ⇒ Residual data Hidden texture and depth 1st layer 2nd layer

Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 6 / 21

SLIDE 9

LDI from real multi-view plus depth sequence [JMG09]

1st layer 2nd layer

Limitations

Redundant boundaries in both layers
Moving elements in both layers
Layers contain large depth discontinuities

(Discontinuities are hard to compress) Compressed depth map Synthesized virtual view

Fig: Rendering impact of depth map compression

Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 7 / 21

SLIDE 10

Outline

1

Introduction

2

Object-based classification

3

Rendering results

4

Compression results

Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 8 / 21

SLIDE 11

Object-based LDI representation

This representation organizes pixels into layers to enhance depth continuity

1st layer 2nd layer

Fig: Classical LDI depth layers

Foreground Background

Fig: Object-based LDI depth layers

Method based on a growing-region algorithm

Region R initialized with pixels where ZFG and ZBG are already defined. For each pixel q outside R:

Extrapolate ZFG and ZBG.
Classify q.

Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 9 / 21

SLIDE 12

Object-based LDI representation

This representation organizes pixels into layers to enhance depth continuity

1st layer 2nd layer

Fig: Classical LDI depth layers

Foreground Background

Fig: Object-based LDI depth layers

Method based on a growing-region algorithm

Region R initialized with pixels where ZFG and ZBG are already defined. For each pixel q outside R:

Extrapolate ZFG and ZBG.
Classify q.

Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 9 / 21

SLIDE 13

Classification: Initializing

Foreground Unclassified Background

Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 10 / 21

SLIDE 14

Classification: Results

Foreground Unclassified Background

Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 11 / 21

SLIDE 15

Background inpainting [CPT03]

Advantages

Remove unnecessary boundaries
Inpainting processed once, before encoding stage
No need of inpainting method at rendering stage

Fig: Background inpainting

Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 12 / 21

SLIDE 16

Outline

1

Introduction

2

Object-based classification

3

Rendering results

4

Compression results

Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 13 / 21

SLIDE 17

Rendering results

Classical LDI rendering Virtual view inpainting O-LDI rendering

Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 14 / 21

SLIDE 18

Fast mesh-based rendering

Fig: Object-based LDI

Continuous layers can be rendered as meshes. Foreground mesh is partially transparent.

Fig: Meshes rendering

Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 15 / 21

SLIDE 19

Outline

1

Introduction

2

Object-based classification

3

Rendering results

4

Compression results

Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 16 / 21

SLIDE 20

LDI and MVD compression schemes

MVD compression (MVC)

V1 V3 V5 V7

Input views

V ′

1

V ′

3

V ′

5

V ′

7 Compressed views

MVC VSRS Rendering V ′′

6

Final view

LDI compression (MVC)

LDI4 MVC LDI′

4

DIBR V6 Input LDI Compression Compressed LDI Rendering Final view

Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 17 / 21

SLIDE 21

Rate-distortion curve

5 10 15 20 84 86 88 90 Bitrate (Mbit/s) SSIM (%) Object-based LDI Classical LDI —– MPEG (MVC/VSRS)

Fig: Rate distortion curves

Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 18 / 21

SLIDE 22

Conclusions on Object-Based LDI

Advantages

Remove unnecessary boundaries ⇒ Improve compression
Static background along time
Compatible with fast mesh-based rendering
Depth continuity improves rendering quality

Limit

No backward compatibility with 2D decoding scheme

Questions?

Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 19 / 21

SLIDE 23

[CPT03]

A. Criminisi, P. Pérez, and K. Toyama.

Object removal by exemplar-based inpainting. In Computer Vision and Pattern Recognition (CVPR), IEEE Computer Society Conference on, volume 2, pages 721–728, June 2003. [JMG09] Vincent Jantet, Luce Morin, and Christine Guillemot. Incremental-ldi for multi-view coding. In 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video, pages 1–4, Potsdam, Germany, May 2009. [SGHS98] Jonathan Shade, Steven Gortler, Li-wei He, and Richard Szeliski. Layered depth images. In SIGGRAPH ’98: Proceedings of the 25th annual conference

n Computer graphics and interactive techniques, pages

231–242, New York, NY, USA, July 1998. ACM.

Vincent Jantet (ENS-Cachan – France) Object-based LDI ICIP 2011 20 / 21