360-Degree 3D Ground Surface Reconstruction Using a Single Rotating - - PowerPoint PPT Presentation

IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd.

360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera

Kouma Motooka†, Shigeki Sugimoto†, Masatoshi Okutomi† Takeshi Shima‡

†Tokyo Institute of Technology ‡ Hitachi Ltd.

IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd.

• Backgrounds

– 360-degree surrounding ground surfaces.

• Required for heavy machineries in construction sites.
• Related works

Introduction

2

Catadioptric stereo camera [18] Laser scanner

[18] M. Schonbein, A. Geiger. “Omnidirectional 3D reconstruction in augmented Manhattan worlds.” IROS2014

・ Hard to calibrate ・ Expensive

IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd.

Objective

3

3D ground surface

Camera positions

Camera rotation

To robustly and precisely reconstruct a 360-degree 3D ground surface from the images captured by a single rotating camera.

Single rotating camera: easy to calibrate, low cost. Robust and precise Reconstruction: via effective combination of feature-based and pixel-based methods.

IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd.

Overview

4

Surface refinement by minimizing variances of pixels

Refined surface

Robust initial surface estimation by fitting to 3D points of SfM

Surface fitted to 3D Points Regular mesh Ground coordinate system

IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd.

Robust initial surface estimation

5

Surface fitted to 3D Points Regular mesh Ground coordinate system

Robust initial surface estimation by fitting to 3D points of SfM

IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd.

Overview

6

Surface refinement by minimizing variances of pixels

Refined surface

Robust initial surface estimation by fitting to 3D points of SfM

Surface fitted to 3D Points Regular mesh Ground coordinate system

IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd.

Surface refinement

7

𝒚𝑡

𝐷𝑄𝐸 𝒜 =

𝒚𝑡

′

1 𝑜𝑡

𝑜𝑡

𝐽𝑜𝑡 𝒒𝑜𝑡 𝒚𝑡 𝒚′𝑡, 𝒜 − 𝐽𝑜𝑡(𝒜)

2

The variance of the pixel values 𝒒𝑜𝑡 𝒚′𝑡

𝑦-𝑧 plane

Cost function: 𝐷𝑄 𝒜 = 𝐷𝑄𝐸 𝒜 + 𝛽𝑄𝑇𝐷𝑄𝑇(𝒜)

Data term Smoothness term The heights of each vertex

Data term :

𝑦 𝑨 𝒚′𝑡 : Sample point on 𝑦-𝑧 plane 𝒚𝑡 : Point on surface 𝒒𝑜𝑡 : Point on image 𝐽𝑜𝑡(𝒜) : Average of the pixel values 𝐽𝑜𝑡: Image which observes 𝒚𝑡 Ground coordinate system 𝑧 𝑜𝑡 : Number of the image 𝐽𝑜𝑡

IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd. 8

Outlier removal

Outlier removal

Detect outlier pixels of the surface point 𝒚𝑡 by taking the median value MED(𝐽𝑜𝑡)

Minimize the cost function 𝐷𝑄𝑇(𝒜) using only inlier pixels 𝐽𝑜𝑡 𝒒𝑜𝑡(𝒚𝑡(𝒜)) − MED(𝐽𝑜𝑡) > 𝜐 (𝑢ℎ𝑠𝑓𝑡ℎ𝑝𝑚𝑒) 𝐽𝑜𝑡 is outlier pixel

Moving shadow Same 3D point on the surface Outlier pixel

IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd.

Implementation Details

9

Hierarchical meshing

(In 4 or 5 pyramid levels)

Pre-defined sample points

• Defined on the x-y

plane using pixel- interval

• Dense in near, sparse

in far.

IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd. 10

Experimental results: Synthetic images

Setting

𝒜 𝒚 𝒛

2.0m 45° 1m

Input image (640 × 480 pixels)

Ground coordinate system

Rotating camera ・ The camera rotated 10 degrees per frame capturing 36 images Ground truth surface

16m 16m

IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd.

Estimated surfaces

11

Initial surface (SfM + surface fitting) 16m 16m

Ground truth surface

[5] Y. Furukawa and J. Ponce. Accurate, “Accurate, Dense, and Robust Multi-View Stereopsis” PAMI2010

SfM + PMVS [5] + surface fitting Final (proposed method)

IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd.

Initial surface (SfM + surface fitting) SfM + PMVS [5] + surface fitting Final (proposed method) 16m 16m

Ground truth surface

Error maps

12

Z error 10cm

IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd.

RMSE in distances

11

2 4 6 8 10 12

2～4m 4～6m 6～8m 8～10m 10m～

RMSE [cm] Distance from the rotation axis

SfM SfM+PMVS Proposed Initial PMVS Final (Proposed)

IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd.

Real scene ①

Input image (640 × 480 pixels) ・ The camera rotated 10 degrees per frame capturing 36 images

Scene

Rotation

1.2m 2.0m

Camera

Setting

12

IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd.

Real scene ①: Estimated surface

Result of 3D ground surface generation （Grid size : 12.5cm） Camera position

12m 12m

Textured surface Scene

IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd.

Real scene ②

16

Input image (640 × 480 pixels)

IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd.

Real scene ②: Comparison

17

Without outlier removal （surface）

With outlier removal （surface） With outlier removal （surface + texture）

IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd.

Conclusion

• 360-Degree 3D Ground Surface Reconstruction

Using a Single Rotating Camera

– Propose the method for robustly and precisely reconstructing a 3D ground surface – Improve the stability by outlier pixels removal

• Future work

– Acceleration by optimizing implementation

18