3D Vision Andreas Geiger and Torsten Sattler Spring 2017 3D Vision - PowerPoint PPT Presentation

3D Vision Andreas Geiger and Torsten Sattler Spring 2017

3D Vision • Understanding geometric relations • between images and the 3D world • between images • Obtaining 3D information describing our 3D world • from images • from dedicated sensors

3D Vision • Extremely important in robotics and AR / VR • Visual navigation • Sensing / mapping the environment • Obstacle detection, … • Many further application areas • A few examples …

Google Tango

Google Tango

Image-Based Localization

Geo-Tagging Holiday Photos (Li et al. ECCV 2012)

Augmented Reality (Middelberg et al. ECCV 2014)

Large-Scale Structure-from-Motion Video credit: Johannes Schönberger

Virtual Tourism

3D Urban Modeling UNC/UKY UrbanScape project

3D Urban Modeling

Mobile Phone 3D Scanner

Mobile Phone 3D Scanner

Self-Driving Cars

Self-Driving Cars

Self-Driving Cars

Micro Aerial Vehicles

Microsoft HoloLens Mixed Reality

Virtual Reality

Raw Kinect Output: Color + Depth http://grouplab.cpsc.ucalgary.ca/cookbook/index.php/Technologies/Kinect

Human-Machine Interface

3D Video with Kinect

Autonomous Micro-Helicopter Navigation Use Kinect to map out obstacles and avoid collisions

Dynamic Reconstruction

Performance Capture

Performance Capture (Oswald et al. ECCV 14)

Performance Capture

Motion Capture

Interactive 3D Modeling (Sinha et al. Siggraph Asia 08) collaboration with Microsoft Research (and licensed to

Scanning Industrial Sites as-build 3D model of off-shore oil platform

Scanning Cultural Heritage

Cultural Heritage tanford ’ s Digital Michelangelo S Digital archive Art historic studies

Archaeology accuracy ~1/500 from DV video (i.e. 140kb jpegs 576x720)

• Crime scene recording and analysis Forensics

Forensics

Sports

Surgery

3D Vision Course Team Andreas Geiger Torsten Sattler Federico Camposeco CNB G105 CNB 104 CAB G 86.3 andreas.geiger@inf.ethz.ch torsten.sattler@inf.ethz.ch federico.camposeco@inf.ethz.ch Johannes Schönberger Peidong Liu Pablo Speciale CAB G 85.1 CAB G 84.2 CAB G 86.3 jsch@inf.ethz.ch peidong.liu@inf.ethz.ch pablo.speciale@inf.ethz.ch

Course Objectives • To understand the concepts that relate images to the 3D world and images to other images • Explore the state of the art in 3D vision • Implement a 3D vision system/algorithm

Learning Approach • Introductory lectures: • Cover basic 3D vision concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations ( you ) (seminal papers and state-of-the-art, related to your projects) • 3D vision project: • Choose topic, define scope (by week 4) • Implement algorithm/system • Presentation/demo and paper report Grade distribution • Paper presentation & discussions: 25% • 3D vision project & report: 75%

Materials Slides and more http://www.cvg.ethz.ch/teaching/3dvision/ Also check out on-line “shape-from-video” tutorial: http://www.cs.unc.edu/~ marc/tutorial.pdf http://www.cs.unc.edu/~ marc/tutorial/ Textbooks: • Hartley & Zisserman, Multiple View Geometry • Szeliski, Computer Vision: Algorithms and Applications

Schedule Introduction Feb 20 Feb 27 Geometry, Camera Model, Calibration Mar 6 Features, Tracking / Matching Project Proposals by Students Mar 13 Mar 20 Structure from Motion (SfM) + papers Mar 27 Dense Correspondence (stereo / optical flow) + papers Apr 3 Bundle Adjustment & SLAM + papers Student Midterm Presentations Apr 10 Arp17 Easter break Apr 24 Multi-View Stereo & Volumetric Modeling + papers Labour Day May 1 May 8 3D Modeling with Depth Sensors + papers May 15 3D Scene Understanding + papers May 22 4D Video & Dynamic Scenes + papers Student Project Demo Day = Final Presentations May 29

• Quick overview of what is coming… Fast Forward

Camera Models and Geometry Pinhole camera or Geometric transformations in 2D and 3D

Camera Calibration • Know 2D/3D correspondences, compute projection matrix also radial distortion (non-linear)

Feature Tracking and Matching Harris corners, KLT features, S IFT features key concepts: invariance of extraction, descriptors to viewpoint, exposure and illumination changes

3D from Images L 2 m 1 C 1 M? M L 1 Triangulation - calibration m 2 l 2 - correspondences C 2

Epipolar Geometry Fundamental matrix Essential matrix Also how to robustly compute from images

Structure from Motion Initialize Motion Initialize Structure (P 1 ,P 2 compatibel with F) (minimize reprojection error) Extend motion Extend structure (compute pose through matches (Initialize new structure, seen in 2 or more previous views) refine existing structure)

Visual SLAM • Visual Simultaneous Navigation and Mapping (Clipp et al. ICCV’09)

Stereo and Rectification Warp images to simplify epipolar geometry Compute correspondences for all pixels

Multi-View Stereo

Joint 3D Reconstruction and Class Segmentation (Haene et al CVPR13) reconstruction only (isotropic smoothness prior) joint reconstruction and segmentation  Building (ground, building, vegetation, stuff)  Ground  Vegetation  Clutter

Structured Light • Projector = camera • Use specific patterns to obtain correspondences

Papers and Discussion • Will cover recent state of the art • Each student team will present a paper (5min per team member), followed by discussion • “Adversary” to lead the discussion • Papers will be related to projects/topics • Will distribute papers later (depending on chosen projects)

Projects and reports • Project on 3D Vision-related topic • Implement algorithm / system • Evaluate it • Write a report about it • 3 Presentations / Demos: • Project Proposal Presentation (week 4) • Midterm Presentation (week 8) • Project Demos (week 15) • Ideally: Groups of 3 students

Course project example: Build your own 3D scanner! Example: Bouguet ICCV’98

Benchmarking Relative Pose Solvers Implement two pose estimation methods and find which is the Goal: best algorithm for relative motion plus a vertical direction. Description: Many algorithms for upright relative motion (camera to camera motion with a known direction) exist, the most important ones are [1], [2] and [3] (code online). However, these papers do a bad job of comparing with each other and currently no survey that compares them fairly exist. In this project you will implement [1,2,3] and compare their time performance and accuracy using simulated data (provided by the supervisor). You will also compare them against the state-of-the art in relative motion without known vertical direction [4] (code online). [1] Kalantari et al. „A new solution to the relative orientation problem using only 3 points and the vertical direction." J.of Math. Imaging and Vision 2011 [2] Naroditsky et al. "Two efficient solutions for visual odometry using directional correspondence.“ PAMI 2012 [3] Sweeney et al.. „Efficient Computation of Absolute Pose for Gravity-Aware Augmented Reality“. ISMAR 2015. [4] Hartley & Li. “An efficient hidden variable approach to minimal-case camera motion estimation." PAMI 2012. Requirements / Tools: Supervisor: Required: MATLAB Federico Camposeco fede@inf.ethz.ch 3D Vision, Spring Semester 2017

Simulating Structure from Motion Build a synthetic SfM generator in MATLAB. Goal: Description: Many applications, such as localization, rely on using a Structure from Motion (SfM) dataset as an input. The characteristics of the SfM used affect how the algorithms build to work on them behave. Currently, only real-data SfM reconstructions are a valid way of validating such applications, however such data is limited and expensive to reproduce. In this project, you will implement a MATLAB simulator that can build a faithful SfM reconstruction. Requirements / Tools: Supervisor: Required: MATLAB Federico Camposeco fede@inf.ethz.ch 3D Vision, Spring Semester 2017