3D Reconstruction Wei-Chih Tu ( ) National Taiwan University Fall - PowerPoint PPT Presentation

Computer Vision: from Recognition to Geometry Lecture 15 3D Reconstruction Wei-Chih Tu ( 塗偉志 ) National Taiwan University Fall 2018

Outline • Structure from Motion • Use slides from SFMedu • http://3dvision.princeton.edu/courses/SFMedu/ • Multiple View Stereo (supp.) • Large Scale Reconstruction • Depth from Accidental Motion 2

Stereo Matching • For pixel 𝑦 0 in one image, where is the corresponding point 𝑦 1 in another image? • Stereo: two or more input views • Based on the epipolar geometry, corresponding points lie on the epipolar lines • A matching problem 3

Multiple View Stereo State-of-the-art: PMVS: http://grail.cs.washington.edu/software/pmvs/ Accurate, Dense, and Robust Multi-View Stereopsis, Y Furukawa and J Ponce, 2007. Benchmark: http://vision.middlebury.edu/mview/ A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms. SM Seitz, B Curless, J Diebel, D Scharstein, R Szeliski. 2006. Baseline: Multi-view stereo revisited. M Goesele, B Curless, SM Seitz. 2006.

Multiple View Stereo PMVS, 2007 Furukawa and Ponce. Accurate, dense, and robust multi-view stereopsis. In CVPR 2007. 5

Multiple View Stereo • Feature points  sparse set of matches 6

Large Scale Reconstruction • Building Rome in a Day [ICCV 2009] • https://grail.cs.washington.edu/rome/ 7

Large Scale Reconstruction • Building Rome on a Cloudless Day [ECCV 2010] • https://www.youtube.com/watch?v=4cEQZreQ2zQ 8

Large Scale Reconstruction • Reconstructing the World* in Six Days [CVPR 2015] • As captured by the Yahoo 100 million image dataset • http://www.cs.unc.edu/~jheinly/reconstructing_the_world.html • https://youtu.be/bRYqyoqUJuM 9

Large Scale Reconstruction • Structure from Motion Revisited [CVPR 2016] • COLMAP https://demuc.de/colmap/ Sparse model of central Rome using 21K photos produced by COLMAP’s SfM pipeline 10

Depth from Small Motion Clips • Accidental motions are inevitable when we use handheld cameras • iPhone Live Photo: 1.5 sec clips before and after you press the shutter button • https://iphonephotographyschool.com/live-photos/ Example from Ha et al. [CVPR 2016] 11

Related Works • Yu and Gallup [CVPR 2014] • Initiate the problem • Im et al. [ICIP 2015] • Add a geometry constraint to improve depth estimation • Im et al. [ICCV 2015] • Handle the rolling shutter artifacts • Ha et al. [CVPR 2016] • Solve camera calibration and bundle adjustment at the same time 12

Yu and Gallup’s Work [CVPR 2014] • Bundle adjustment • They brought two techniques • Small angel approximation to simplify the problem • Use inverse depth for numerical stability 13

Algorithm Pipeline • Feature extraction • Harris corner detection • Tracking across all frames • KLT tracker • Bundle adjustment • Compute (inverse) depths for feature points • Dense reconstruction • Sparse-to-dense depth propagation 14

Example Results of Bundle Adjustment Front view Side view Top view Input video Yu and Gallup (with pre-calibrated parameters) Final depth map slide from Hyowon Ha 15