3D RECONSTRUCTION Reconstruction method Reconstruction from images - PowerPoint PPT Presentation

3D RECONSTRUCTION

Reconstruction method Reconstruction from images Reconstruction from video

Using Kinect Raw Depth Image Infrared laser projector Monochrome CMOS sensor

Demo Kinect Raw data

Real-time Reconstruction

Pipeline Measurement Noise Surface Compute Update Raw Depth Reduction Pose Prediction Surface Vertex Reconstruction Image Bilateral Estimation ICP and Normal TSDF Ray-cast Filtering Map r k R k T gk V k, N k S k V k, N k Input: 20 frames * 640 * 480 * 12 = 614.8 MB/s

Bilateral Filtering Demo

Pipeline Measurement Noise Surface Compute Update Raw Depth Reduction Pose Prediction Surface Vertex Reconstruction Image Bilateral Estimation ICP and Normal TSDF Ray-cast Filtering Map r k R k T gk V k, N k S k V k, N k

ICP 3D shape alignment SVD

ICP 3D shape alignment Demo

Pipeline Measurement Noise Surface Compute Update Raw Depth Reduction Pose Prediction Surface Vertex Reconstruction Image Bilateral Estimation ICP and Normal TSDF Ray-cast Filtering Map r k R k T gk V k, N k S k V k, N k

TSDF Signed Distance Function The value in the cube corresponds to the signed distance to the closest zero crossing( surface).

TSDF Signed Distance Function Truncated Signed Distance Function

TSDF Signed Distance Function Truncated Signed Distance Function Integrate the cubes from different position.

TSDF -1 Depth Map from Kinect

TSDF -1 -0.2 Depth Map from Kinect

TSDF 0.05 -1 -0.2 Depth Map from Kinect

TSDF 0.05 -1 0.2 -0.2 Depth Map from Kinect

TSDF 1 0.05 -1 0.2 -0.2 Depth Map from Kinect

TSDF 1 1 0.05 -1 0.2 -0.2 Depth Map from Kinect

TSDF 1 1 0.05 -1 0.5 -0.2 -1 -0.8 -0.1 0.3 1 1 We have depth maps from different camera positions, -1 -1 -0.5 0.05 1 1 how can we integrate them together ? -1 -1 -0.5 0.1 1 1 Integration? or update? Weighted? or add up? -0.8 -1 -0.05 0.3 1 1 What makes integration -1 -0.5 -0.03 0.5 1 1 possible ?

TSDF 1 1 0.05 -1 0.5 -0.2 -1 -0.8 -0.1 0.3 1 1 We have depth maps from different camera positions, -1 -1 -0.5 0.05 1 1 how can we integrate them together ? -1 -1 -0.5 0.1 1 1 Integration? or update? Weighted? or add up? -0.8 -1 -0.05 0.3 1 1 What makes integration -1 -0.5 -0.03 0.5 1 1 possible ?

TSDF 1 1 0.05 -1 0.5 -0.2 -1 -0.8 -0.1 0.3 1 1 We have depth maps from different camera positions, -1 -1 -0.5 0.05 1 1 how can we integrate them together ? -1 -1 -0.5 0.1 1 1 Integration? or update? Weighted? or add up? -0.8 -1 -0.05 0.3 1 1 What makes integration -1 -0.5 -0.03 0.5 1 1 possible ?

TSDF 1 1 0.05 -1 0.5 -0.2 -1 -0.8 -0.1 0.3 1 1 To get the surface behind the surface. The camera is -1 -1 -0.5 0.05 1 1 moving! -1 -1 -0.5 0.1 1 1 Only part of distance data is needed, so we can -0.8 -1 truncate the distance. -0.05 0.3 1 1 -1 -0.5 -0.03 0.5 1 1

TSDF 1 time update ! 1 1 0.05 -0.3 0.5 -0.2 -1 -0.8 -0.1 0.3 1 1 To get the surface behind the surface. The camera is -1 -1 -0.5 0.05 1 1 moving! -1 -1 -0.5 0.1 1 1 Only part of distance data is needed to represent the -0.8 -1 object, so we can truncate -0.05 0.3 1 1 the distance. -1 -0.5 -0.03 0.5 1 1

TSDF 2 times update ! 1 1 0.05 0 0.5 -0.2 -1 -0.8 -0.1 0.3 1 1 To get the surface behind the surface. The camera is -1 -1 -0.5 0.05 1 1 moving! -1 -1 -0.5 0.1 1 1 Only part of distance data is needed to represent the -0.8 -1 object, so we can truncate -0.05 0.3 1 1 the distance. -1 -0.5 -0.03 0.5 1 1

TSDF 3 times update ! 1 1 0.05 0.3 0.5 -0.2 -1 -0.8 -0.1 0.3 1 1 To get the surface behind the surface. The camera is -1 -1 -0.5 0.05 1 1 moving! -1 -1 -0.5 0.1 1 1 Only part of distance data is needed to represent the -0.8 -1 object, so we can truncate -0.05 0.3 1 1 the distance. -1 -0.5 -0.03 0.5 1 1

Pipeline Measurement Noise Surface Compute Update Raw Depth Reduction Pose Prediction Surface Vertex Reconstruction Image Bilateral Estimation ICP and Normal TSDF Ray-cast Filtering Map r k R k T gk V k, N k S k V k, N k

RAY CASTING Cast only, no chasing. Transfer the TSDF cube in to some thing the computer can understand, Vertex fusion. Take a photo using X-ray.

RAY CASTING 1 1 0.05 -1 0.5 -0.2 -1 -0.8 -0.1 0.3 1 1 -1 -1 -0.5 0.05 1 1 -1 -1 -0.5 0.1 1 1 Detect the sign change. -0.8 -1 -0.05 0.3 1 1 Two scales search -1 -0.5 -0.03 0.5 1 1 Linear regression

RAY CASTING 1 1 0.05 -1 0.5 -0.2 -1 -0.8 -0.1 0.3 1 1 -1 -1 -0.5 0.05 1 1 1 -1 -1 -0.5 0.1 1 Detect the sign change. -0.8 -1 -0.05 0.3 1 1 Two scales search -1 -0.5 -0.03 0.5 1 1 Linear regression Normal Vectors

Real-time Reconstruction Demo

Reference [1] KinectFusion: Real-Time Dense Surface Mapping and Tracking. Microsoft Research [2] B. Curless and M. Levoy. A volumetric method for building complex models from range images. [3] M. Harris, S. Sengupta, and J. D. Owens. Parallel prefix sum (scan) with CUDA. In H. Nguyen, editor, GPU Gems 3, chapter 39, pages 851 – 876. Addison Wesley, August 2007. 3.5 [4] C. Tomasi and R. Manduchi. Bilateral filtering for gray and color images. In Proceedings of the ICCV, 1998. [5] C. Rasch and T. Satzger. Remarks on the O(N) implementation of the fast marching method. [6] Y. Chen and G. Medioni. Object modeling by registration of multiple range images. Image and Vision Computing (IVC), 10(3):145 – 155,1992 [7] Kok-Lim Low Linear Least-Squares Optimization for Point-to-Plane ICP Surface Registration