Nonparametric Directional Perception Julian Straub Collaborators: - - PowerPoint PPT Presentation

▶
nonparametric directional perception julian straub
SMART_READER_LITE
LIVE PREVIEW

Nonparametric Directional Perception Julian Straub Collaborators: - - PowerPoint PPT Presentation

May 28, 2023 176 likes •970 views

Nonparametric Directional Perception Julian Straub Collaborators: Oren Freifeld, Jason Chang, Guy Rosman, Trevor Campbell, Randi Cabezas, Nishchal Bhandari, Jonathan P. How, John J. Leonard, John W. Fisher III. This talk is entirely based on my

slide-1
SLIDE 1

Nonparametric Directional Perception Julian Straub

Collaborators: Oren Freifeld, Jason Chang, Guy Rosman, Trevor Campbell, Randi Cabezas, Nishchal Bhandari, Jonathan P. How, John J. Leonard, John W. Fisher III.

This talk is entirely based on my PhD thesis at MIT.

October 5, 2017

slide-2
SLIDE 2

Perception is Key

slide-3
SLIDE 3

Perception

Localization Mapping Scene Understanding

slide-4
SLIDE 4

Perception

Localization Mapping Scene Understanding

Simultaneous Localization and Mapping (SLAM)

slide-5
SLIDE 5

Perception

Localization Mapping Scene Understanding

Simultaneous Localization and Mapping (SLAM)

Semantic SLAM

slide-6
SLIDE 6

Perception

Localization Mapping Scene Understanding

Simultaneous Localization and Mapping (SLAM)

Semantic SLAM Directional Scene Understanding

slide-7
SLIDE 7

Outlinep

Background 1.

slide-8
SLIDE 8

Outlinep

Background 1. Directional Scene Understanding Manhattan Nonparametric Nonparametric Manhattan Manhattan 2.

slide-9
SLIDE 9

Outlinep

Background 1. Directional Scene Understanding Manhattan Nonparametric Nonparametric Manhattan Manhattan 2. Nonparametric Directional SLAM 3.

slide-10
SLIDE 10

Outlinep

Background 1. Directional Scene Understanding Manhattan Nonparametric Nonparametric Manhattan Manhattan 2. Nonparametric Directional SLAM 3. Nonparametric Directional Perception

capture and use regularities of man- made environments revealed in their surface normal distribution

slide-11
SLIDE 11

Outlinep

Background Directional Scene Understanding Nonparametric Directional SLAM Nonparametric Directional Perception

capture and use regularities of man- made environments revealed in their surface normal distribution

Manhattan Nonparametric Nonparametric Manhattan Manhattan 1. 2. 3.

slide-12
SLIDE 12

RGB and Depth Image

slide-13
SLIDE 13

Point Cloud and Surface Normals RGB and Depth Image

slide-14
SLIDE 14

Point Cloud and Surface Normals RGB and Depth Image True Surface

slide-15
SLIDE 15

Point Cloud and Surface Normals RGB and Depth Image True Surface Point Cloud Sensing

slide-16
SLIDE 16

Point Cloud and Surface Normals RGB and Depth Image True Surface Point Cloud Sensing Surface Normals Normal Extraction

slide-17
SLIDE 17

Point Cloud and Surface Normals Surface Normal Space: Sphere S2 RGB and Depth Image True Surface Point Cloud Sensing Surface Normals Normal Extraction

slide-18
SLIDE 18

Point Cloud and Surface Normals Surface Normal Space: Sphere S2 RGB and Depth Image True Surface Point Cloud Sensing Surface Normals Normal Extraction

slide-19
SLIDE 19

Scene Structure and Distribution of Normals

Small Scale Large Scale

slide-20
SLIDE 20

Scene Structure and Distribution of Normals

Small Scale Large Scale

slide-21
SLIDE 21

Scene Structure and Distribution of Normals

Small Scale Large Scale Surface normal clusters capture environment regularities.

slide-22
SLIDE 22

Directional Clustering and Segmentation

Scene Surface Normals

slide-23
SLIDE 23

Directional Clustering and Segmentation

Scene Surface Normals Directional Clustering Bayesian directional mixture models for directional clustering.

slide-24
SLIDE 24

Directional Clustering and Segmentation

Scene Surface Normals Directional Clustering Directional Segmentation Bayesian directional mixture models for directional clustering.

slide-25
SLIDE 25

Outline

Background Directional Scene Understanding Nonparametric Directional SLAM Nonparametric Directional Perception

capture and use regularities of man- made environments revealed in their surface normal distribution

Manhattan Nonparametric Nonparametric Manhattan Manhattan 1. 2. 3.

slide-26
SLIDE 26

Scene Representations

Real World ≈ MW

Manhattan World (MW)

[Coughlan 1999]

slide-27
SLIDE 27

Scene Representations

Real World ≈ MW

Manhattan World (MW)

[Coughlan 1999]

AW

Atlanta World (AW)

[Schindler 2004]

MMF

Mixture of Manhattan Frames (MMF)

[Straub 2014]

slide-28
SLIDE 28

Scene Representations

Real World ≈ MW

Manhattan World (MW)

[Coughlan 1999]

AW

Atlanta World (AW)

[Schindler 2004]

MMF

Mixture of Manhattan Frames (MMF)

[Straub 2014]

Manhattan Constrained Directional Models

slide-29
SLIDE 29

Manhattan World R3

slide-30
SLIDE 30

Manhattan World R3

  • Orth. Vanishing Points

R2

Projection

[Caprile 1990, Coughlan 1999, Bose 2003, Lee 2009, Neverova 2013, Liu 2015, . . . ]

slide-31
SLIDE 31

Manhattan World R3

  • Orth. Vanishing Points

R2

Projection

[Caprile 1990, Coughlan 1999, Bose 2003, Lee 2009, Neverova 2013, Liu 2015, . . . ]

sparse line observations ⇒ fragile

slide-32
SLIDE 32

Manhattan World R3 S2 Manhattan Frame

Surface Normal Extraction

  • Orth. Vanishing Points

R2

Projection

[Caprile 1990, Coughlan 1999, Bose 2003, Lee 2009, Neverova 2013, Liu 2015, . . . ]

sparse line observations ⇒ fragile

slide-33
SLIDE 33

Manhattan World R3 S2 Manhattan Frame

Surface Normal Extraction

  • Orth. Vanishing Points

R2

Projection

[Caprile 1990, Coughlan 1999, Bose 2003, Lee 2009, Neverova 2013, Liu 2015, . . . ]

sparse line observations ⇒ fragile

slide-34
SLIDE 34

Manhattan World R3 S2 Manhattan Frame

Surface Normal Extraction

  • Orth. Vanishing Points

R2

Projection

[Caprile 1990, Coughlan 1999, Bose 2003, Lee 2009, Neverova 2013, Liu 2015, . . . ]

sparse line observations ⇒ fragile

[Straub 2014, Straub 2015, Ghanem 2015, Joo 2016, Straub 2017]

dense surface normal observations ⇒ accurate, robust

slide-35
SLIDE 35
slide-36
SLIDE 36
slide-37
SLIDE 37
slide-38
SLIDE 38

Mixture of Manhattan Frames [CVPR 2014, TPAMI 2017]

slide-39
SLIDE 39

Mixture of Manhattan Frames [CVPR 2014, TPAMI 2017]

slide-40
SLIDE 40

Mixture of Manhattan Frames [CVPR 2014, TPAMI 2017]

MF 1 MF 2 MF 3

slide-41
SLIDE 41

Manhattan Frame: Mixture over Axes Distributions

MF 1 MF 2 MF 3

slide-42
SLIDE 42

Manhattan Frame: Mixture over Axes Distributions

MF 1 MF 2 MF 3 MF Axes Assignments

slide-43
SLIDE 43

Manhattan Frame: Mixture over Axes Distributions

MF 1 MF 2 MF 3 MF Axes Assignments Sampling-based algorithm allows inference of number of MFs.

slide-44
SLIDE 44
slide-45
SLIDE 45
slide-46
SLIDE 46

Scene Representations

Real World ≈ MW

Manhattan World (MW)

[Coughlan 1999]

AW

Atlanta World (AW)

[Schindler 2004]

MMF

Mixture of Manhattan Frames (MMF)

[Straub 2014]

SCW

Stata Center World (SCW)

[Straub 2015]

Nonparametric Unconstrained Directional Model

slide-47
SLIDE 47

Scene Representations

Real World ≈ Planes

Planes

MW

Manhattan World (MW)

[Coughlan 1999]

AW

Atlanta World (AW)

[Schindler 2004]

MMF

Mixture of Manhattan Frames (MMF)

[Straub 2014]

SCW

Stata Center World (SCW)

[Straub 2015]

Nonparametric Unconstrained Directional Model

slide-48
SLIDE 48

Stata Center World

Stata Center World R3

slide-49
SLIDE 49

Stata Center World

Stata Center World R3 Vanishing Points R2

Projection

slide-50
SLIDE 50

Stata Center World

Stata Center World R3 Vanishing Points R2

Projection

[Collins 1990, Antone 2000, Tardif 2009, Barinova 2010, Xu 2013, Lezama 2014, Kroeger 2015, . . . ]

sparse observations, no MW constraints ⇒ even more fragile

slide-51
SLIDE 51

Stata Center World

Stata Center World R3 S2

  • Dir. Clusters

Surface Normal Extraction

Vanishing Points R2

Projection

[Collins 1990, Antone 2000, Tardif 2009, Barinova 2010, Xu 2013, Lezama 2014, Kroeger 2015, . . . ]

sparse observations, no MW constraints ⇒ even more fragile

slide-52
SLIDE 52

Stata Center World

Stata Center World R3 S2

  • Dir. Clusters

Surface Normal Extraction

Vanishing Points R2

Projection

[Collins 1990, Antone 2000, Tardif 2009, Barinova 2010, Xu 2013, Lezama 2014, Kroeger 2015, . . . ]

sparse observations, no MW constraints ⇒ even more fragile

slide-53
SLIDE 53

Stata Center World

Stata Center World R3 S2

  • Dir. Clusters

Surface Normal Extraction

Vanishing Points R2

Projection

[Collins 1990, Antone 2000, Tardif 2009, Barinova 2010, Xu 2013, Lezama 2014, Kroeger 2015, . . . ]

sparse observations, no MW constraints ⇒ even more fragile

nonparametric surface normal clustering [Triebel 2005, Straub 2015, Straub 2015, Zhou 2016]

dense observations ⇒ accurate, robust

slide-54
SLIDE 54

Outlinep

Background Directional Scene Understanding Nonparametric Directional SLAM Nonparametric Directional Perception

capture and use regularities of man- made environments revealed in their surface normal distribution

Manhattan Nonparametric Nonparametric Manhattan Manhattan 1. 2. 3.

slide-55
SLIDE 55

Nonparametric Directional SLAM

Localization Mapping Directional Scene Understanding Nonparametric Directional SLAM

slide-56
SLIDE 56

Related Geometry-based Semantic SLAM Systems

Planes

[Castle 2007, Taguchi 2013, Salas-Moreno 2014, Kaess 2015, Ma 2016, Hsiao 2017]

slide-57
SLIDE 57

Related Geometry-based Semantic SLAM Systems

Planes

[Castle 2007, Taguchi 2013, Salas-Moreno 2014, Kaess 2015, Ma 2016, Hsiao 2017]

Manhattan World (MW)

[Peasley 2012, Furukawa 2009, Le 2017]

slide-58
SLIDE 58

Related Geometry-based Semantic SLAM Systems

Planes

[Castle 2007, Taguchi 2013, Salas-Moreno 2014, Kaess 2015, Ma 2016, Hsiao 2017]

Manhattan World (MW)

[Peasley 2012, Furukawa 2009, Le 2017]

Vanishing Points (VPs)

[Bosse 2003]

slide-59
SLIDE 59

Related Geometry-based Semantic SLAM Systems

Planes

[Castle 2007, Taguchi 2013, Salas-Moreno 2014, Kaess 2015, Ma 2016, Hsiao 2017]

Manhattan World (MW)

[Peasley 2012, Furukawa 2009, Le 2017]

Vanishing Points (VPs)

[Bosse 2003]

Stata Center World (SCW)

[Straub 2017]

slide-60
SLIDE 60

Related Geometry-based Semantic SLAM Systems

Planes

[Castle 2007, Taguchi 2013, Salas-Moreno 2014, Kaess 2015, Ma 2016, Hsiao 2017]

Manhattan World (MW)

[Peasley 2012, Furukawa 2009, Le 2017]

Vanishing Points (VPs)

[Bosse 2003]

Stata Center World (SCW)

[Straub 2017]

Directional Scene Understanding

slide-61
SLIDE 61

Related Geometry-based Semantic SLAM Systems

Planes

[Castle 2007, Taguchi 2013, Salas-Moreno 2014, Kaess 2015, Ma 2016, Hsiao 2017]

Manhattan World (MW)

[Peasley 2012, Furukawa 2009, Le 2017]

Vanishing Points (VPs)

[Bosse 2003]

Stata Center World (SCW)

[Straub 2017]

Directional Scene Understanding Planes MW VPs SCW scene-wide constraints ≈ ✓ ✓ ✓ dense ✓ ✓ ✗ ✓ flexible ✓ ✗ ✓ ✓

slide-62
SLIDE 62

Directional SLAM Overview

Map represented as surfels with location pi and normal ni. pi ni Surfel i

slide-63
SLIDE 63

Directional SLAM Overview

Stata Center World Directional Segmentation pi ni Surfel i

slide-64
SLIDE 64

Directional SLAM Overview

pi ni Surfel i Can directional segmentation be used to regularize reconstruction?

slide-65
SLIDE 65

Direction-aware Mapping

ni N pi N Surfels

slide-66
SLIDE 66

Direction-aware Mapping

α πk zi ni Dirichlet Process von-Mises-Fisher mixture θk µk G0 K K ∞ ∞ N pi N Surfels and Nonparamet- ric Directional Segmentation Normals ni and locations pi independent!

slide-67
SLIDE 67

Direction-aware Mapping

α πk zi ni θk µk G0 K K ∞ ∞ zj pj pi zj pj Nearest neighbors j of surfel i.

slide-68
SLIDE 68

Direction-aware Mapping

α πk zi ni θk µk G0 K K ∞ ∞ zj pj pi zj pj Two cases: in-plane and out-of-plane neighbors.

slide-69
SLIDE 69

Direction-aware Mapping

α πk zi ni θk µk G0 K K ∞ ∞ zj pj pi zj pj Encourage local planarity within directional segment.

slide-70
SLIDE 70

Direction-aware Mapping

α πk zi ni θk µk G0 K K ∞ ∞ zj pj pi zj pj Connects scene-wide directional seg- mentation with local 3D structure.

slide-71
SLIDE 71

Directional SLAM System

RGB-D Camera Observations Background Realtimeg

slide-72
SLIDE 72

Directional SLAM System

Mapping {pi, ni} RGB-D Camera Observations Background Realtimeg

slide-73
SLIDE 73

Directional SLAM System

Mapping {pi, ni} Stata Center World Segmentation {zi} Gibbs-sampling RGB-D Camera Observations Background Realtimeg

slide-74
SLIDE 74

Directional SLAM System

Mapping {pi, ni} Stata Center World Segmentation {zi} Gibbs-sampling RGB-D Camera Observations Gibbs-sampling: full DP- vMF-MM inference; com- putation of expectations. Background Realtimeg

slide-75
SLIDE 75

Directional SLAM System

Mapping {pi, ni} Stata Center World Segmentation {zi} Gibbs-sampling RGB-D Camera Observations Localization {T} Maximum Likelihood (ICP) Gibbs-sampling: full DP- vMF-MM inference; com- putation of expectations. Background Realtimeg

slide-76
SLIDE 76
slide-77
SLIDE 77

Conclusion

Directional Scene Understanding Nonparametric Directional SLAM Nonparametric Directional Perception

capture and use regularities of man- made environments revealed in their surface normal distribution

Manhattan Nonparametric Nonparametric Manhattan Manhattan 1. 2.