Variational Smoothing on Delaunay Graphs W. Nicholas Greene Robust - - PowerPoint PPT Presentation

▶
variational smoothing on delaunay graphs
SMART_READER_LITE
LIVE PREVIEW

Variational Smoothing on Delaunay Graphs W. Nicholas Greene Robust - - PowerPoint PPT Presentation

Mar 29, 2024 49 likes •860 views

FLaME: Fast Lightweight Mesh Estimation using Variational Smoothing on Delaunay Graphs W. Nicholas Greene Robust Robotics Group, MIT CSAIL LPM Workshop IROS 2017 September 28, 2017 with Nicholas Roy 1 We want Autonomous Vision-Based

slide-1
SLIDE 1

1

FLaME: Fast Lightweight Mesh Estimation using Variational Smoothing on Delaunay Graphs

  • W. Nicholas Greene

with Nicholas Roy Robust Robotics Group, MIT CSAIL LPM Workshop IROS 2017 September 28, 2017

slide-2
SLIDE 2

2

We want Autonomous Vision-Based Navigation

https://commons.wikimedia.org/wiki/File%3AQuadcopter_Drone.png https://commons.wikimedia.org/wiki/File%3ACanon_90-300mm_camera_lens.jpg

slide-3
SLIDE 3

3

But Dense Monocular SLAM is Expensive

Density Efficiency

Mono-Slam (Davison, ICCV 2007) PTAM (Klein and Murray, ISMAR 2007) SVO (Forster et al., ICRA 2014)

Sparse Methods

https://commons.wikimedia.org/wiki/File%3ANvidia_Titan_XP.jpg https://shop-media.intel.com/api/v2/helperservice/getimage?url=http://images.icecat.biz/img/gallery/23221218_49.jpg&height=550&width=550 https://www.qualcomm.com/sites/ember/files/styles/optimize/public/component-item/flexible-block/thumb/chip_3.png?itok=XncLtDdQ

slide-4
SLIDE 4

4

But Dense Monocular SLAM is Expensive

Density Efficiency

Mono-Slam (Davison, ICCV 2007) PTAM (Klein and Murray, ISMAR 2007) SVO (Forster et al., ICRA 2014)

Sparse Methods

Engel et al. ICCV 2013 LSD-SLAM (Engel et al., ECCV 2014) Mur-Artal and Tardos (RSS 2015) Pillai et al. ICRA 2016

Semi-Dense Methods

https://commons.wikimedia.org/wiki/File%3ANvidia_Titan_XP.jpg https://shop-media.intel.com/api/v2/helperservice/getimage?url=http://images.icecat.biz/img/gallery/23221218_49.jpg&height=550&width=550 https://www.qualcomm.com/sites/ember/files/styles/optimize/public/component-item/flexible-block/thumb/chip_3.png?itok=XncLtDdQ

slide-5
SLIDE 5

5

But Dense Monocular SLAM is Expensive

Density Efficiency

Mono-Slam (Davison, ICCV 2007) PTAM (Klein and Murray, ISMAR 2007) SVO (Forster et al., ICRA 2014)

Sparse Methods

Engel et al. ICCV 2013 LSD-SLAM (Engel et al., ECCV 2014) Mur-Artal and Tardos (RSS 2015) Pillai et al. ICRA 2016

Semi-Dense Methods

DTAM (Newcombe et al., ICCV 2011) Graber et al. (ICCV 2011) MonoFusion (Pradeep et al., ISMAR 2013) REMODE (Pizzoli et al., ICRA 2014)

Dense Methods

https://commons.wikimedia.org/wiki/File%3ANvidia_Titan_XP.jpg https://shop-media.intel.com/api/v2/helperservice/getimage?url=http://images.icecat.biz/img/gallery/23221218_49.jpg&height=550&width=550 https://www.qualcomm.com/sites/ember/files/styles/optimize/public/component-item/flexible-block/thumb/chip_3.png?itok=XncLtDdQ

slide-6
SLIDE 6

6

But Dense Monocular SLAM is Expensive

https://commons.wikimedia.org/wiki/File%3ANvidia_Titan_XP.jpg https://commons.wikimedia.org/wiki/File%3AQuadcopter_Drone.png https://commons.wikimedia.org/wiki/File%3ACanon_90-300mm_camera_lens.jpg

slide-7
SLIDE 7

7

But Dense Monocular SLAM is Expensive

Density Efficiency

Mono-Slam (Davison, ICCV 2007) PTAM (Klein and Murray, ISMAR 2007) SVO (Forster et al., ICRA 2014)

Sparse Methods

Engel et al. ICCV 2013 LSD-SLAM (Engel et al., ECCV 2014) Mur-Artal and Tardos (RSS 2015) Pillai et al. ICRA 2016

Semi-Dense Methods

DTAM (Newcombe et al., ICCV 2011) Graber et al. (ICCV 2011) MonoFusion (Pradeep et al., ISMAR 2013) REMODE (Pizzoli et al., ICRA 2014)

Dense Methods

?

Can we more efficiently estimate dense geometry?

https://commons.wikimedia.org/wiki/File%3ANvidia_Titan_XP.jpg https://shop-media.intel.com/api/v2/helperservice/getimage?url=http://images.icecat.biz/img/gallery/23221218_49.jpg&height=550&width=550 https://www.qualcomm.com/sites/ember/files/styles/optimize/public/component-item/flexible-block/thumb/chip_3.png?itok=XncLtDdQ

slide-8
SLIDE 8

8

What Would a Dense Method Do?

Image

slide-9
SLIDE 9

9

Estimate Depth for Every Pixel

Depthmap

slide-10
SLIDE 10

10

Dense Methods Oversample Geometry

One depth estimate per pixel 100k - 1M pixels per image Depthmap

slide-11
SLIDE 11

11

Dense Methods Make Regularization Hard(er)

One depth estimate per pixel 100k - 1M pixels per image Depthmap

slide-12
SLIDE 12

12

Meshes Encode Geometry with Fewer Depths

Depthmap

slide-13
SLIDE 13

13

Meshes Encode Geometry with Fewer Depths

Depthmap

slide-14
SLIDE 14

14

Meshes Encode Geometry with Fewer Depths

Depthmap

slide-15
SLIDE 15

15

Meshes Encode Geometry with Fewer Depths

One depth estimate per vertex 100 - 10k vertices per mesh Depthmap

slide-16
SLIDE 16

16

Meshes Make Regularization Easy(er)

One depth estimate per vertex 100 - 10k vertices per mesh Depthmap

slide-17
SLIDE 17

17

FLaME: Fast Lightweight Mesh Estimation

Fast (< 5 ms/frame) Lightweight (< 1 Intel i7 CPU core) Runs Onboard MAV

Greene and Roy, ICCV2017

slide-18
SLIDE 18

18

FLaME: Fast Lightweight Mesh Estimation

Image and pose Depthmap interpolation Delaunay Triangulation Variational Regularizer (NLTGV2-L1) Inverse Depth Estimation Feature Selection Output

slide-19
SLIDE 19

19

FLaME: Fast Lightweight Mesh Estimation

Image and pose Depthmap interpolation Delaunay Triangulation Variational Regularizer (NLTGV2-L1) Inverse Depth Estimation Feature Selection Output

slide-20
SLIDE 20

20

Select Easily Trackable Features

  • At each frame we sample trackable pixels or features over the image
  • These features will serve as potential vertices to add to the mesh
slide-21
SLIDE 21

21

Select Easily Trackable Features

  • At each frame we sample trackable pixels or features over the image
  • These features will serve as potential vertices to add to the mesh
slide-22
SLIDE 22

22

Select Easily Trackable Features

  • At each frame we sample trackable pixels over the image domain
  • These features will serve as potential vertices to add to the mesh
slide-23
SLIDE 23

23

Select Easily Trackable Features

  • At each frame we sample trackable pixels over the image domain
  • These features will serve as potential vertices to add to the mesh
slide-24
SLIDE 24

24

Select Easily Trackable Features

  • At each frame we sample trackable pixels over the image domain
  • These features will serve as potential vertices to add to the mesh
slide-25
SLIDE 25

25

Select Easily Trackable Features

  • At each frame we sample trackable pixels over the image domain
  • These features will serve as potential vertices to add to the mesh

Select pixel in each grid cell with maximum score:

slide-26
SLIDE 26

26

FLaME: Fast Lightweight Mesh Estimation

Image and pose Depthmap interpolation Delaunay Triangulation Variational Regularizer (NLTGV2-L1) Inverse Depth Estimation Feature Selection Output

slide-27
SLIDE 27

27

Estimate Inverse Depths for Features

  • We track features across new images using direct epipolar stereo

comparisons

slide-28
SLIDE 28

28

Estimate Inverse Depths for Features

  • We track features across new images using direct epipolar stereo

comparisons

slide-29
SLIDE 29

29

Estimate Inverse Depths for Features

  • We track features across new images using direct epipolar stereo

comparisons

slide-30
SLIDE 30

30

Estimate Inverse Depths for Features

  • We track features across new images using direct epipolar stereo

comparisons

slide-31
SLIDE 31

31

Estimate Inverse Depths for Features

  • We track features across new images using direct epipolar stereo

comparisons

slide-32
SLIDE 32

32

Estimate Inverse Depths for Features

  • We track features across new images using direct epipolar stereo

comparisons

  • Each successful match generates an inverse depth measurement
slide-33
SLIDE 33

33

Estimate Inverse Depths for Features

  • We track features across new images using direct epipolar stereo

comparisons

  • Each successful match generates an inverse depth measurement
  • Measurements are fused over time using Bayesian update
slide-34
SLIDE 34

34

FLaME: Fast Lightweight Mesh Estimation

Image and pose Depthmap interpolation Delaunay Triangulation Variational Regularizer (NLTGV2-L1) Inverse Depth Estimation Feature Selection Output

slide-35
SLIDE 35

35

Triangulate Features into Mesh

  • Once a features inverse depth variance drops below threshold, we insert it

into the mesh using Delaunay triangulations

slide-36
SLIDE 36

36

Triangulate Features into Mesh

  • Once a features inverse depth variance drops below threshold, we insert it

into the mesh using Delaunay triangulations

slide-37
SLIDE 37

37

Triangulate Features into Mesh

  • Once a features inverse depth variance drops below threshold, we insert it

into the mesh using Delaunay triangulations

slide-38
SLIDE 38

38

Triangulate Features into Mesh

  • Once a features inverse depth variance drops below threshold, we insert it

into the mesh using Delaunay triangulations

  • We can project 2D mesh into 3D using the inverse depth for each vertex
slide-39
SLIDE 39

39

Triangulate Features into Mesh

  • Once a features inverse depth variance drops below threshold, we insert it

into the mesh using Delaunay triangulations

  • We can project 2D mesh into 3D using the inverse depth for each vertex
slide-40
SLIDE 40

40

FLaME: Fast Lightweight Mesh Estimation

Image and pose Depthmap interpolation Delaunay Triangulation Variational Regularizer (NLTGV2-L1) Inverse Depth Estimation Feature Selection Output

slide-41
SLIDE 41

41

Smooth Mesh Using Variational Regularization

  • Mesh inverse depths are noisy and prone to outliers
  • Need to spatially regularize/smooth inverse depths

Raw

slide-42
SLIDE 42

42

Smooth Mesh Using Variational Regularization

  • Mesh inverse depths are noisy and prone to outliers
  • Need to spatially regularize/smooth inverse depths
  • Will exploit graph structure to make optimization fast and incremental

Raw Smoothed

slide-43
SLIDE 43

43

Define Variational Cost over Inverse Depthmap

Raw idepthmap

slide-44
SLIDE 44

44

Define Variational Cost over Inverse Depthmap

Raw idepthmap Smoothed idepthmap

slide-45
SLIDE 45

45

Define Variational Cost over Inverse Depthmap

NLTGV2 stands for Non-Local Total Generalized Variation (Second Order) Ranftl et al. 2014, Pinies et al. 2015

Raw idepthmap Smoothed idepthmap

slide-46
SLIDE 46

46

Approximate Cost over Mesh

  • Reinterpret mesh as graph
slide-47
SLIDE 47

47

Approximate Cost over Mesh

  • Reinterpret mesh as graph
slide-48
SLIDE 48

48

Approximate Cost over Mesh

slide-49
SLIDE 49

49

Approximate Cost over Mesh

Raw idepthmap

slide-50
SLIDE 50

50

Approximate Cost over Mesh

slide-51
SLIDE 51

51

Approximate Cost over Mesh

Raw mesh

slide-52
SLIDE 52

52

Approximate Cost over Mesh

Raw mesh Smoothed mesh

slide-53
SLIDE 53

53

Approximate Cost over Mesh

Smoothed mesh Raw mesh

slide-54
SLIDE 54

54

Approximate Cost over Mesh

Smoothed mesh Raw mesh Math…

slide-55
SLIDE 55

55

Approximate Cost over Mesh

Smoothed mesh Raw mesh Math…

slide-56
SLIDE 56

56

Optimize using Primal-Dual on Graph

slide-57
SLIDE 57

57

Optimize using Primal-Dual on Graph

Math…

slide-58
SLIDE 58

58

Optimize using Primal-Dual on Graph

Math…

slide-59
SLIDE 59

59

Optimize using Primal-Dual on Graph

Math…

slide-60
SLIDE 60

60

Optimize using Primal-Dual on Graph

Math…

Edges update using vertices

slide-61
SLIDE 61

61

Optimize using Primal-Dual on Graph

Math…

Vertices update using edges

slide-62
SLIDE 62

62

Optimize using Primal-Dual on Graph

Math…

Optimization steps are fast even without GPU (<< frame rate)

slide-63
SLIDE 63

63

Optimize using Primal-Dual on Graph

Math…

Optimization steps are fast even without GPU (<< frame rate) Optimization convergence is fast (~ frame rate)

slide-64
SLIDE 64

64

Optimize using Primal-Dual on Graph

Math…

Optimization steps are fast even without GPU (<< frame rate) Optimization convergence is fast (~ frame rate) Mesh can be augmented without restarting optimization

slide-65
SLIDE 65

65

FLaME: Fast Lightweight Mesh Estimation

Image and pose Depthmap interpolation Delaunay Triangulation Variational Regularizer (NLTGV2-L1) Inverse Depth Estimation Feature Selection Output

slide-66
SLIDE 66

66

Benchmark Results

Compared FLaME to two existing CPU-only approaches:

  • LSD-SLAM
  • MLM
slide-67
SLIDE 67

67

Benchmark Results

Compared FLaME to two existing CPU-only approaches:

  • LSD-SLAM
  • MLM

Evaluated on two benchmark datasets (ground truth poses):

  • TUM RGBD SLAM (640x480 @ 30 Hz)
  • EuRoC MAV (752x480 @ 20 Hz)
slide-68
SLIDE 68

68

Benchmark Results

Compared FLaME to two existing CPU-only approaches:

  • LSD-SLAM
  • MLM

Evaluated on two benchmark datasets (ground truth poses):

  • TUM RGBD SLAM (640x480 @ 30 Hz)
  • EuRoC MAV (752x480 @ 20 Hz)

Desktop Intel i7 CPU only

slide-69
SLIDE 69

69

Benchmark Results

Lower Error

slide-70
SLIDE 70

70

Benchmark Results

More Dense

slide-71
SLIDE 71

71

Benchmark Results

Less Load

slide-72
SLIDE 72

72

Benchmark Results

Faster

slide-73
SLIDE 73

73

Benchmark Results

Faster

slide-74
SLIDE 74

74

Benchmark Results

slide-75
SLIDE 75

75

Flight Experiments

DJI F450 frame Intel NUC flight computer 10 ms/frame runtime 1.5 core load Indoor Flight at 2.5 m/s Outdoor Flight at 3.5 m/s

slide-76
SLIDE 76

76

Flight Experiments

slide-77
SLIDE 77

77

FLaME Contributions

  • Proposed a lightweight method for dense online monocular depth estimation

Paper Code*

* Coming soon!

  • W. Nicholas Greene (wng@csail.mit.edu)
slide-78
SLIDE 78

78

FLaME Contributions

  • Proposed a lightweight method for dense online monocular depth estimation
  • Formulated the reconstruction problem as a non-local variational
  • ptimization over a Delaunay graph, which allows for a fast, efficient

approach to depth estimation

Paper Code*

* Coming soon!

  • W. Nicholas Greene (wng@csail.mit.edu)
slide-79
SLIDE 79

79

FLaME Contributions

  • Proposed a lightweight method for dense online monocular depth estimation
  • Formulated the reconstruction problem as a non-local variational
  • ptimization over a Delaunay graph, which allows for a fast, efficient

approach to depth estimation

  • Demonstrated improved depth accuracy and density on benchmark

datasets using less than 1 Intel i7 CPU core

Paper Code*

* Coming soon!

  • W. Nicholas Greene (wng@csail.mit.edu)
slide-80
SLIDE 80

80

FLaME Contributions

  • Proposed a lightweight method for dense online monocular depth estimation
  • Formulated the reconstruction problem as a non-local variational
  • ptimization over a Delaunay graph, which allows for a fast, efficient

approach to depth estimation

  • Demonstrated improved depth accuracy and density on benchmark

datasets using less than 1 Intel i7 CPU core

  • Demonstrated real-world applicability with indoor and outdoor flight

experiments running FLaME onboard, in-the-loop

Paper Code*

* Coming soon!

  • W. Nicholas Greene (wng@csail.mit.edu)
slide-81
SLIDE 81

81

Acknowledgments

This material is based upon work supported by DARPA under Contract No. HR0011-15-C-0110 and by the NSF Graduate Research Fellowship under Grant

  • No. 1122374. Any opinion, findings, and conclusions or recommendations

expressed in this material are those of the authors(s) and do not necessarily reflect the views of the National Science Foundation. We also thank John Ware, John Carter, and the rest of the MIT-Draper FLA Team for continued development and testing support.

Paper Code*

* Coming soon!

  • W. Nicholas Greene (wng@csail.mit.edu)