Incremental-LDI for Multi-View Coding Generation & - - PowerPoint PPT Presentation

Incremental-LDI for Multi-View Coding

Generation & Representation Vincent JANTET

IRISA Rennes - TEMICS Team FRANCE Supervisors: Christine Guillemot - IRISA Luce Morin - IETR - INSA Gaël Sourimant - IRISA

April 1, 2010

Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 1 / 19

Context

Multi-view videos

Desired functionalities: 3DTV: Depth feeling by stereovision simulation. FVV: (for Free Viewpoint Video) Live viewpoint selection.

Problems

Acquisition: Synchronization, calibration. . . Compression: Compact representation of the huge amount of data. Rendering: Photo-realistic virtual view generation.

Figure: Multi-view acquisition Figure: 3D rendering

Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 2 / 19

Context

Multi-view videos

Desired functionalities: 3DTV: Depth feeling by stereovision simulation. FVV: (for Free Viewpoint Video) Live viewpoint selection.

Problems

Acquisition: Synchronization, calibration. . . Compression: Compact representation of the huge amount of data. Rendering: Photo-realistic virtual view generation.

Figure: Multi-view acquisition Figure: 3D rendering

Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 2 / 19

Table on contents

1

Introduction

2

Incremental-LDI construction scheme

3

Ghosting

4

Results

Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 3 / 19

Outline

1

Introduction

2

Incremental-LDI construction scheme

3

Ghosting

4

Results

Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 4 / 19

View generation (Warping algorithm)

Warping

View i Depth i Figure: Warping algorithm

Image-based rendering

Input: View and associated Depth map Output: New viewpoint (texture & depth).

Problems

Sampling: Visual artifacts ⇒ Inpainting Disocclusion: Unknown texture. ⇒ Extra information (LDI).

Figure: Disocclusion

Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 5 / 19

LDI (Layered Depth Image) - [SGHS98, YLKH07]

A Layered Depth Image is a set of many layers constituted by depth pixels. Contains texture of occluded area. 1st layer 2nd layer 3rd layer 4th layer · · ·

Figure: First layers of an LDI frame.

Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 6 / 19

Classical LDI construction [CSY07]

Every input views are warped

• nto a reference viewpoint,

and then merged together.

Merging policy

Eliminate pixels described twice. . . . Reference viewpoint Merging .

LDI View i View j

Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 7 / 19

Advantages and limits

Advantages

Disocclusion: Filled by real texture. Camera freedom: Virtual camera can move inside a large area. Compactness: Eliminate some correlated pixels and reduce data size.

Limits

Compression: Many layers, partially empty with scattered pixels distribution. Visual artifacts: Ghosting, Bluring, . . .

Figure: Scattered pixels distribution. Figure: Ghosting artifacts.

Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 8 / 19

Outline

1

Introduction

2

Incremental-LDI construction scheme

3

Ghosting

4

Results

Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 9 / 19

Incremental-LDI construction scheme (I-LDI)

Current I-LDI is warped iteratively on every other viewpoint. Residual information is computed by exclusion difference with the real view. This information is warped back into the I-LDI. .

I-LDI

Warp Ref to i

Viewpoint i

Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 10 / 19

Incremental-LDI construction scheme (I-LDI)

Current I-LDI is warped iteratively on every other viewpoint. Residual information is computed by exclusion difference with the real view. This information is warped back into the I-LDI. Exclusion difference .

I-LDI

Warp i to Ref Warp Ref to i

View i Viewpoint i

Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 10 / 19

Incremental-LDI construction scheme (I-LDI)

Current I-LDI is warped iteratively on every other viewpoint. Residual information is computed by exclusion difference with the real view. This information is warped back into the I-LDI. Exclusion difference .

I-LDI

Warp i to Ref Insert Warp Ref to i

View i Viewpoint i

Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 10 / 19

Incremental-LDI construction scheme (I-LDI)

Current I-LDI is warped iteratively on every other viewpoint. Residual information is computed by exclusion difference with the real view. This information is warped back into the I-LDI. Exclusion difference .

I-LDI

Warp j to Ref Insert Warp Ref to j

View j Viewpoint j

Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 10 / 19

Outline

1

Introduction

2

Incremental-LDI construction scheme

3

Ghosting

4

Results

Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 11 / 19

Ghosting Artifacts

Ghosting is due to pixels with blended color between background and foreground. Detect depth discontinuity [Canny]. Classify background and foreground pixels near each boundaries. Ignore background blended pixels from data.

First layer of an I-LDI frame.

Depth Map Canny edges Boundaries Ghost removal

Figure: Ghosting artifacts removal

Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 12 / 19

Outline

1

Introduction

2

Incremental-LDI construction scheme

3

Ghosting

4

Results

Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 13 / 19

First layers of an LDI and an I-LDI Comparison.

LDI frame: many pixels, scattered pixels distribution, ...

(a) 1st layer (b) 2nd layer (c) 3rd layer (d) 4th layer

...

(e)

I-LDI frame: less layer and less pixels, compact distribution, ...

(f) 1st layer (g) 2nd layer (h) 3rd layer (i) 4th layer

...

(j)

Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 14 / 19

LDI & I-LDI comparison.

(a) Layers completion rate. (b) Pixels ratio taken from different views.

Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 15 / 19

Figure: LDI rendering.

PSNR Original LDI Original 30.23 I-LDI 30.22 46.26 SSIM Original LDI Original 79.62% I-LDI 79.68% 99.52%

Figure: PSNR & SSIM Figure: I-LDI rendering.

(a) LDI. (b) I-LDI.

Figure: Differences.

Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 16 / 19

Rendering result.

Figure: Rendering result of an I-LDI.

Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 17 / 19

Rendering result.

Figure: Rendering result of an I-LDI.

Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 17 / 19

Rendering result.

Figure: Rendering result of an I-LDI.

Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 17 / 19

Conclusion

Advantages

80% less pixels for the same quality. Less correlation between layers. Compact pixels distribution.

Limits

Sampling artifacts. Some textures will never be inserted into the I-LDI.

Future works.

Look for an efficient I-LDI compression algorithm. Questions?

Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 18 / 19

References I

• X. Cheng, L. Sun, and S. Yang.

Generation of layered depth images from multi-view video. Image Processing, 2007. ICIP 2007. IEEE International Conference on, 5:V –225–V –228, 16 2007-Oct. 19 2007.

• J. Shade, S. Gortler, L. He, and R. Szeliski.

Abstract layered depth images. 1998. S.-U. Yoon, E.-K. Lee, S.-Y. Kim, and Y.-S. Ho. A framework for representation and processing of multi-view video using the concept of layered depth image. Journal of VLSI Signal Processing Systems for Signal Image and Video Technology, 46:87–102, 2007.

Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 19 / 19