3D Vision Marc Pollefeys and Viktor Larsson Spring 2020 3D Vision - - PowerPoint PPT Presentation

▶
3d vision
SMART_READER_LITE
LIVE PREVIEW

3D Vision Marc Pollefeys and Viktor Larsson Spring 2020 3D Vision - - PowerPoint PPT Presentation

Jan 18, 2024 230 likes •840 views

3D Vision Marc Pollefeys and Viktor Larsson Spring 2020 3D Vision Understanding geometric relations between images and the 3D world between images Obtaining 3D information describing our 3D world from images from

slide-1
SLIDE 1

3D Vision

Marc Pollefeys and Viktor Larsson

Spring 2020

slide-2
SLIDE 2

3D Vision

  • Understanding geometric relations
  • between images and the 3D world
  • between images
  • Obtaining 3D information describing
  • ur 3D world
  • from images
  • from dedicated sensors
slide-3
SLIDE 3

3D Vision

  • Extremely important in robotics and

AR / VR

  • Visual navigation
  • Sensing / mapping the environment
  • Obstacle detection, …
  • Many further application areas
  • A few examples …
slide-4
SLIDE 4

Google Tango

(officially discontinued, lives on as ARCore)

slide-5
SLIDE 5

Google Tango

slide-6
SLIDE 6

Image-Based Localization

slide-7
SLIDE 7

Geo-Tagging Holiday Photos

(Li et al. ECCV 2012)

slide-8
SLIDE 8

Augmented Reality

(Middelberg et al. ECCV 2014)

slide-9
SLIDE 9

Video credit: Johannes Schönberger

Large-Scale Structure-from-Motion

slide-10
SLIDE 10

Virtual Tourism

slide-11
SLIDE 11

UNC/UKY UrbanScape project

3D Urban Modeling

slide-12
SLIDE 12

3D Urban Modeling

slide-13
SLIDE 13
slide-14
SLIDE 14

Mobile Phone 3D Scanner

slide-15
SLIDE 15

Mobile Phone 3D Scanner

slide-16
SLIDE 16

Self-Driving Cars

slide-17
SLIDE 17

Self-Driving Cars

slide-18
SLIDE 18

Micro Aerial Vehicles

slide-19
SLIDE 19

Mixed Reality

Microsoft HoloLens 2

slide-20
SLIDE 20

Virtual Reality

slide-21
SLIDE 21

Raw Kinect Output: Color + Depth

http://grouplab.cpsc.ucalgary.ca/cookbook/index.php/Technologies/Kinect

slide-22
SLIDE 22

Human-Machine Interface

slide-23
SLIDE 23

Autonomous Micro-Helicopter Navigation

Use Kinect to map out obstacles and avoid collisions

slide-24
SLIDE 24

Dynamic Reconstruction

slide-25
SLIDE 25

Performance Capture

slide-26
SLIDE 26

Performance Capture

(Oswald et al. ECCV 14)

slide-27
SLIDE 27

Motion Capture

slide-28
SLIDE 28

Interactive 3D Modeling

(Sinha et al. Siggraph Asia 08) collaboration with Microsoft Research (and licensed to MS)

slide-29
SLIDE 29

Scanning Industrial Sites

as-build 3D model of off-shore oil platform

slide-30
SLIDE 30

Scanning Cultural Heritage

slide-31
SLIDE 31

Cultural Heritage

Stanford’s Digital Michelangelo

Digital archive Art historic studies

slide-32
SLIDE 32

accuracy ~1/500 from DV video (i.e. 140kb jpegs 576x720)

Archaeology

slide-33
SLIDE 33

Forensics

  • Crime scene recording and analysis
slide-34
SLIDE 34

Forensics

slide-35
SLIDE 35

Sports

slide-36
SLIDE 36

Surgery

slide-37
SLIDE 37

Viktor Larsson CNB G 102.2

viktor.larsson@inf.ethz.ch

Marc Pollefeys CNB G 105

marc.pollefeys@inf.ethz.ch

Peidong Liu CAB G 84.2

peidong.liu@inf.ethz.ch

3D Vision Course Team

Marcel Geppert CAB G 84.2

marcel.geppert@inf.ethz.ch

Sandro Lombardi CAB G 89

sandro.lobardi@inf.ethz.ch

Zuoyue Li CAB G 85.2

li.zuoyue@inf.ethz.ch

Mihai Dusmanu CAB G 101

mihai-alexandru. dusmanu@inf.ethz.ch

Taein Kwon CAB G 85.2

taein.kwon@inf.ethz.ch

slide-38
SLIDE 38
  • To understand the concepts that relate

images to the 3D world and images to

  • ther images
  • Explore the state of the art in 3D vision
  • Implement a 3D vision system/algorithm

Course Objectives

slide-39
SLIDE 39

Learning Approach

  • Introductory lectures:
  • Cover basic 3D vision concepts and approaches.
  • Further lectures:
  • Short introduction to topic
  • Paper presentations (you)

(seminal papers and state of the art, related to your projects)

  • 3D vision project:
  • Choose topic, define scope (by week 4)
  • Implement algorithm/system
  • Presentation/demo and paper report

Grade distribution

  • Paper presentation & discussions: 25%
  • 3D vision project & report: 75%
slide-40
SLIDE 40

Slides and more

http://www.cvg.ethz.ch/teaching/3dvision/

Also check out on-line “shape-from-video” tutorial:

http://www.cs.unc.edu/~marc/tutorial.pdf http://www.cs.unc.edu/~marc/tutorial/

Textbooks:

  • Hartley & Zisserman, Multiple View Geometry
  • Szeliski, Computer Vision: Algorithms and Applications

Materials

slide-41
SLIDE 41

Feb 17 Introduction Feb 24 Geometry, Camera Model, Calibration Mar 2 Features, Tracking / Matching Mar 9 Project Proposals by Students Mar 16 Structure from Motion (SfM) + papers Mar 23 Dense Correspondence (stereo / optical flow) + papers Mar 30 Bundle Adjustment & SLAM + papers Apr 6 Student Midterm Presentations Apr 13 Easter break Apr 20 Multi-View Stereo & Volumetric Modeling + papers Apr 27 3D Modeling with Depth Sensors + papers May 4 3D Scene Understanding + papers May 11 4D Video & Dynamic Scenes + papers May 18 papers May 25 Student Project Demo Day = Final Presentations

Schedule

slide-42
SLIDE 42

Fast Forward

  • Quick overview of what is coming…
slide-43
SLIDE 43

Pinhole camera Geometric transformations in 2D and 3D

  • r

Camera Models and Geometry

slide-44
SLIDE 44
  • Know 2D/3D correspondences,

compute projection matrix also radial distortion (non-linear)

Camera Calibration

slide-45
SLIDE 45

Harris corners, KLT features, SIFT features

key concepts: invariance of extraction, descriptors to viewpoint, exposure and illumination changes

Feature Tracking and Matching

slide-46
SLIDE 46

l2 C1 m1 M? L1 m2 L2 M C2 Triangulation

  • calibration
  • correspondences

3D from Images

slide-47
SLIDE 47

Fundamental matrix Essential matrix

Also how to robustly compute from images

Epipolar Geometry

slide-48
SLIDE 48

Initialize Motion (P1,P2 compatibel with F) Initialize Structure (minimize reprojection error) Extend motion (compute pose through matches seen in 2 or more previous views) Extend structure (Initialize new structure, refine existing structure)

Structure from Motion

slide-49
SLIDE 49
  • Visual Simultaneous Navigation and Mapping

Visual SLAM

(Clipp et al. ICCV’09)

slide-50
SLIDE 50

Stereo and Rectification

Warp images to simplify epipolar geometry Compute correspondences for all pixels

slide-51
SLIDE 51

Multi-View Stereo

slide-52
SLIDE 52

Joint 3D Reconstruction and Class Segmentation

(Haene et al CVPR13)

joint reconstruction and segmentation

(ground, building, vegetation, stuff)

reconstruction only

(isotropic smoothness prior)

 Building  Ground  Vegetation  Clutter

slide-53
SLIDE 53

Structured Light

  • Projector = camera
  • Use specific patterns to obtain

correspondences

slide-54
SLIDE 54

Papers and Discussion

  • Will cover recent state of the art
  • Each student team will present a paper (5min

per team member), followed by discussion

  • “Adversary” to lead the discussion
  • Papers will be related to projects/topics
  • Will distribute papers later

(depending on chosen projects)

slide-55
SLIDE 55

Projects and reports

  • Project on 3D Vision-related topic
  • Implement algorithm / system
  • Evaluate it
  • Write a report about it
  • 3 Presentations / Demos:
  • Project Proposal Presentation (week 4)
  • Midterm Presentation (week 8)
  • Project Demos (week 15)
  • Ideally: Groups of 3-4 students
slide-56
SLIDE 56

Course project example: Build your own 3D scanner!

Example: Bouguet ICCV’98

slide-57
SLIDE 57

Project Topics

slide-58
SLIDE 58

3D Vision, Spring Semester 2018

Goal:

Requirements / Tools: Supervisor:

Description:

Your Own Project Learn about the techniques presented in the lecture Choose your own topic! Available hardware: Google Tango Tablets Microsoft HoloLens GoPro Cameras Intel RealSense Sensor We find one for you

Required: Related to 3D Vision / topics of the lecture

slide-59
SLIDE 59

Your Next Steps

  • Find a group (ideally: groups of 3-4)
  • Find a project (one of ours or your own)
  • Topic subscription via doodle in a few days:
  • For questions contact us via the lecture Moodle

(preferred) or contact Sandro per email

  • First come first serve!
  • Do not contact supervisors directly!
  • After topic assignment: talk with your supervisor
  • Write a project proposal
  • Don’t worry: You’ll get reminders!
slide-60
SLIDE 60

Feb 17 Introduction Feb 24 Geometry, Camera Model, Calibration Mar 2 Features, Tracking / Matching Mar 9 Project Proposals by Students Mar 16 Structure from Motion (SfM) + papers Mar 23 Dense Correspondence (stereo / optical flow) + papers Mar 30 Bundle Adjustment & SLAM + papers Apr 6 Student Midterm Presentations Apr 13 Easter break Apr 20 Multi-View Stereo & Volumetric Modeling + papers Apr 27 3D Modeling with Depth Sensors + papers May 4 3D Scene Understanding + papers May 11 4D Video & Dynamic Scenes + papers May 18 papers May 25 Student Project Demo Day = Final Presentations

Schedule