Depth Camera Based System for Auto-Stereoscopic Displays Fran cois - - PowerPoint PPT Presentation

▶
depth camera based system for auto stereoscopic displays
SMART_READER_LITE
LIVE PREVIEW

Depth Camera Based System for Auto-Stereoscopic Displays Fran cois - - PowerPoint PPT Presentation

Jan 08, 2024 128 likes •497 views

Depth Camera Based System for Auto-Stereoscopic Displays Fran cois de Sorbier Yuko Uematsu Hideo Saito Keio University - Japan IEEE MMSP 2010 October 6 th 2010 Outline 1 Introduction 2 Depth Camera 3 GPU Rendering 4 Conclusion Fran

slide-1
SLIDE 1

Depth Camera Based System for Auto-Stereoscopic Displays

Fran¸ cois de Sorbier Yuko Uematsu Hideo Saito

Keio University - Japan

IEEE MMSP 2010 October 6th 2010

slide-2
SLIDE 2

Outline

1 Introduction 2 Depth Camera 3 GPU Rendering 4 Conclusion

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 2/21

slide-3
SLIDE 3

Auto-Stereoscopic Display

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 3/21

slide-4
SLIDE 4

Auto-Stereoscopic Display

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 3/21

slide-5
SLIDE 5

Auto-Stereoscopic Display

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 3/21

slide-6
SLIDE 6

Input Format of Auto-Stereoscopic Displays

Input format depends on the:

  • number of required input views
  • interlacing
  • sub-pixel arrangement
  • on-board technology
  • Video plus depth

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 4/21

slide-7
SLIDE 7

On-line Video Streams Input

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 5/21

slide-8
SLIDE 8

On-line Video Streams Input

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 5/21

slide-9
SLIDE 9

On-line Video Streams Input

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 5/21

slide-10
SLIDE 10

Goal

  • Reduce the number of cameras
  • Generate several virtual viewpoints
  • Real-time process

Depth Camera Based System for Auto-Stereoscopic Displays

  • Use a depth camera
  • Mesh based approach
  • GPU to speed-up the process

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 6/21

slide-11
SLIDE 11

Depth Camera

What is a Depth Camera?

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 7/21

slide-12
SLIDE 12

Depth Camera

What is a Depth Camera?

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 7/21

slide-13
SLIDE 13

Depth Camera

What is a Depth Camera?

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 7/21

slide-14
SLIDE 14

Depth Camera

What is a Depth Camera?

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 7/21

slide-15
SLIDE 15

Depth Camera

What is a Depth Camera?

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 7/21

slide-16
SLIDE 16

Depth Camera

What is a Depth Camera?

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 7/21

slide-17
SLIDE 17

Depth Camera

What is a Depth Camera?

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 7/21

slide-18
SLIDE 18

Depth Camera

Limitations

  • Low resolution (176×144)
  • Color information is missing ⇒ Add a color camera
  • Viewpoints are different

⇒ A calibration stage is required

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 8/21

slide-19
SLIDE 19

Depth Camera

Calibration and Rendering

  • Color camera’s pose estimation
  • 2D/3D correspondences between the depth map and the color image
  • Mesh-based approach
  • Rendering viewpoint corresponds to color camera viewpoint

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 9/21

slide-20
SLIDE 20

Depth Camera

Calibration and Rendering

  • Color camera’s pose estimation
  • 2D/3D correspondences between the depth map and the color image
  • Mesh-based approach
  • Rendering viewpoint corresponds to color camera viewpoint

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 9/21

slide-21
SLIDE 21

Depth Camera

Create content for Auto-Stereoscopic Displays

  • Video plus depth format: easy
  • Multiple views: similar to computer graphics stereoscopy.
  • Projective texture to resolve color occlusions

Mtexture = 0.5 0 0 0

0 0.5 0 0 0 1 0 0.5 0.5 0 1

  • × Mprojection × Mmodelview

(1)

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 10/21

slide-22
SLIDE 22

Depth Camera

Create content for Auto-Stereoscopic Displays

  • Video plus depth format: easy
  • Multiple views: similar to computer graphics stereoscopy.
  • Projective texture to resolve color occlusions

Mtexture = 0.5 0 0 0

0 0.5 0 0 0 1 0 0.5 0.5 0 1

  • × Mprojection × Mmodelview

(1)

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 10/21

slide-23
SLIDE 23

Depth Camera

Create content for Auto-Stereoscopic Displays

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 11/21

slide-24
SLIDE 24

GPU Rendering

Rendering limitations

  • Increasing the number of rendering passes decreases the frame-rate
  • Redundant computation among the different views
  • Compute properties of vertices only once using the GPU

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 12/21

slide-25
SLIDE 25

GPU Rendering

GPU for geometry duplication

  • 3 programmable units on GPU
  • Vertex Shader: Vertices properties ⇒ common part
  • Geometry shader: Geometry assembly ⇒ duplication
  • Pixel shader: No need

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 13/21

slide-26
SLIDE 26

GPU Rendering

Limitations of OpenGL

  • 8 output textures maximum ⇒ only 8 views
  • Results share a single depth buffer ⇒ Artifacts
  • Solution: A single shared texture

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 14/21

slide-27
SLIDE 27

GPU Rendering

Duplication stage

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 15/21

slide-28
SLIDE 28

GPU Rendering

Duplication stage

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 15/21

slide-29
SLIDE 29

GPU Rendering

Duplication stage

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 15/21

slide-30
SLIDE 30

GPU Rendering

Clipping stage

  • Some triangles can overlap other view’s areas
  • Apply openGL user’s clipping
  • Distance values to the borders of the area
  • If one value is negative, clip
  • Else keep

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 16/21

slide-31
SLIDE 31

GPU Rendering

Context of Results

  • bi-Xeon 2.5Ghz with nVIDIA GeForce GTX285
  • Number of triangles: 50000 (176 × 144 × 2)
  • Resolution of each view : 1024×768
  • Number of views : 5 (Tridility Screen)

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 17/21

slide-32
SLIDE 32

GPU Rendering

Results

  • The Frame-rate depends on the number of views and triangles
  • 1 view: 51 frames per second
  • 5 views: 28 frames per second (instead of 15)

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 18/21

slide-33
SLIDE 33

Conclusion

Summary

  • Capture system
  • Depth camera
  • Color camera
  • Real-Time mesh based approach
  • Multi-view rendering applied on GPU

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 19/21

slide-34
SLIDE 34

Conclusion

Future works

  • Reduce problem of occlusions
  • Several layers
  • Reduce noise of depth map
  • More cameras (depth/color)
  • Consistency over time

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 20/21

slide-35
SLIDE 35

Conclusion

Questions

Thank you for your attention

Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 21/21