An Introduction to Eric Rosen, Kaiyu Zheng Semantic Mapping in - - PowerPoint PPT Presentation

▶
an introduction to
SMART_READER_LITE
LIVE PREVIEW

An Introduction to Eric Rosen, Kaiyu Zheng Semantic Mapping in - - PowerPoint PPT Presentation

Nov 15, 2023 230 likes •1k views

An Introduction to Eric Rosen, Kaiyu Zheng Semantic Mapping in Robotics 3/22/2019 Outline Timeline of Semantic mapping Why semantic mapping (in robotics)? How? Problem definition Literature review What? (Our

slide-1
SLIDE 1

An Introduction to Semantic Mapping in Robotics

Eric Rosen, Kaiyu Zheng 3/22/2019

slide-2
SLIDE 2

Outline

  • Timeline of “Semantic mapping”
  • Why semantic mapping (in robotics)?
  • How?
  • Problem definition
  • Literature review
  • What? (Our research)
  • TopoNets, GraphSPNs
  • Action-oriented semantic mapping
slide-3
SLIDE 3
  • Originated in linguistics - 1960s
  • Correspondence of hierarchies in languages

For example: phonogram “/həˈlō/” → word “hello”

  • In literacy (vocabulary instruction)
  • Structure of knowledge in graphic form

Semantic Mapping [Johnson et.al., 1986]

Timeline

“Semantic Mapping”

On the Uniqueness of Semantic Mapping [Householder, 1962]

slide-4
SLIDE 4
slide-5
SLIDE 5
  • Appearance in CS (NLP) - 1989
  • Neural network (!)
  • Semantic relations between symbolic data.

Timeline

“Semantic Mapping”

“geometrically or topologically organized maps”

Self-Organizing Semantic Maps [Ritter and Kohonen, 1989] (training data)

slide-6
SLIDE 6

These concepts are central to semantic mapping in robotics!

Self-Organizing Semantic Maps [Ritter and Kohonen, 1989]

slide-7
SLIDE 7
  • Semantic hierarchy of spatial representations
  • [sensorimotor → control] → topology → geometry
  • Definition of terms “metric map”, “topological map”,

“hybrid map” (no semantic map )

  • Hybrid metric-topological-semantic map

Timeline

“Semantic Mapping” (robotics)

A robot exploration and mapping strategy based on a semantic hierarchy

  • f spatial representations [Kuipers and Byun, 1991]

Some Notes on the Use of Hybrid Maps for Mobile Robots [Buschka and Saffotti, 2004] Multi-Hierarchical Semantic Maps for Mobile Robotics [Galindo, Buschka et.al. , 2005]

slide-8
SLIDE 8

Multi-Hierarchical Semantic Maps for Mobile Robotics [Galindo, Buschka et.al. , 2005]

slide-9
SLIDE 9
  • Semantic hierarchy of spatial representations
  • [sensorimotor → control] → topology → geometry
  • Definition of terms “metric map”, “topological map”,

“hybrid map” (no semantic map )

  • Hybrid metric-topological-semantic map
  • 3D laser-based SLAM with scene interpretation

Timeline

“Semantic Mapping” (robotics)

A robot exploration and mapping strategy based on a semantic hierarchy

  • f spatial representations [Kuipers and Byun, 1981]

Some Notes on the Use of Hybrid Maps for Mobile Robots [Buschka and Saffotti, 2004] Multi-Hierarchical Semantic Maps for Mobile Robotics [Galindo, Buschka et.al. , 2005] Towards semantic maps for mobile robots [Nüchter, Hertzberg et.al. , 2008]

slide-10
SLIDE 10

Recent successor (also 3D SLAM)

Meaningful Maps With Object-Oriented Semantic Mapping [Sünderhauf, et.al. , 2016] Towards semantic maps for mobile robots [Nüchter, Hertzberg et.al. , 2008]

slide-11
SLIDE 11
  • Semantic hierarchy of spatial representations
  • [sensorimotor → control] → topology → geometry
  • Definition of terms “metric map”, “topological map”,

“hybrid map” (no semantic map )

  • Hybrid metric-topological-semantic map
  • 3D laser-based SLAM with scene interpretation
  • Semantic Mapping with Mobile Robots

Timeline

“Semantic Mapping” (robotics)

A robot exploration and mapping strategy based on a semantic hierarchy

  • f spatial representations [Kuipers and Byun, 1981]

Some Notes on the Use of Hybrid Maps for Mobile Robots [Buschka and Saffotti, 2004] Multi-Hierarchical Semantic Maps for Mobile Robotics [Galindo, Buschka et.al. , 2005] Towards semantic maps for mobile robots [Nüchter, Hertzberg et.al. , 2008] PhD thesis [Pronobis, 2011]

[Influencer for us]

slide-12
SLIDE 12

Why semantic mapping?

Robots Are Getting Better

Hardware↑ Sensing↑ Control↑ Robots are getting better at single tasks

Mobility Manipulation

slide-13
SLIDE 13

Moving Forward

Multiple tasks Automated planning and scheduling

Why semantic mapping?

slide-14
SLIDE 14

Why semantic mapping?

Moving Forward

Multiple tasks Automated planning and scheduling World states description Semantic maps

State representation

slide-15
SLIDE 15

How? Problem definition

Semantic Mapping

PhD thesis [Pronobis, 2011]

Input Sensory observations and odometry Prior knowledge of semantic information Output Semantic maps (what is this?)

Capable to facilitate planning

Shared understanding in literature

slide-16
SLIDE 16

Semantic Maps

Problem definition

  • Historically informal

Self-Organizing Semantic Maps [Ritter and Kohonen, 1989] Towards semantic maps for mobile robots [Nüchter, Hertzberg et.al. , 2008] Multi-Hierarchical Semantic Maps for Mobile Robotics [Galindo, Buschka et.al. , 2005] and PhD thesis [Pronobis, 2011] did not provide a formal definition either.

slide-17
SLIDE 17

Semantic Maps

Problem definition

  • Formal, but ignored (and hard to find)

Semantic Maps for Robotics [Lang and Paulus, 2014]

Vague enough Somewhat complicated

slide-18
SLIDE 18

Semantic Maps

Problem definition

  • The attempt in [Zheng et.al.’19 In submission]
  • Did not include “metric map”
  • No semantics in edges
  • Suitable for their research problem

𝑁 = (𝑈, 𝒀, 𝒁) 𝑁: semantic map 𝑈 = (𝑾, 𝑭): topological graph 𝒀 = {𝒀𝑗: 𝑗 ∈ 𝑾}: local observations 𝒁 = {𝒁𝑗: 𝑗 ∈ 𝑾}: semantic attributes Each node 𝑗 is a place (defined later)

slide-19
SLIDE 19
  • Semantic mapping is conceptually clear
  • Formally define semantic mapping?
  • Semantic maps may vary
  • Yet, one should at least include:
  • Spatial information (contained in map)
  • Maps: Metric, topological
  • Anchoring of spatial concepts
  • Place classification (simplification)

(We are talking about mobile robots)

Problem definition

Summary

slide-20
SLIDE 20
  • Metric map
  • Created by SLAM
  • Captures geometry of the world
  • Topological map
  • A graph (V,E)
  • Each node in V is a place
  • Each edge in E indicates navigability
  • Captures structure of the world

Literature review

Maps

Metric map

[Sünderhauf et.al., ICRA’16] [Zheng., senior thesis’17]

Topological maps

slide-21
SLIDE 21

[Pronobis, et. al. ICAPS Workshop’17]

slide-22
SLIDE 22
  • Classification from local sensory information
  • Through detecting objects in the environment

(less common)

  • Through robot sensory observations
  • RGB-D (Visual scene classification)
  • Laser-range
  • Multi-modal (RGB+Laser+Odometry)

Literature review

Local Place Classification

[Li et.al., ECCV Workshops’10] [Viswanathan et.al., CRV’10] [Pronobis et.al., IROS’17] [Pronobis et.al., IJRR’10] [Zhu et.al., CVPR’16] [Friedman et.al., IJCAI’07]

slide-23
SLIDE 23

Literature review

Mainstream: Place classification

  • n a map

Structured prediction

  • Relations between places
  • Probabilistic inference
  • Boost classification results
slide-24
SLIDE 24

Literature review

Place Classification on a Map

[Goeddel et.al., IROS’16]

Map → Planning for robots (Mobility)

Metric Maps

[Friedman et.al., IJCAI’07] [Sünderhauf et.al., ICRA’16]

slide-25
SLIDE 25

Literature review

Place Classification on a Map

[Pronobis et.al., ICRA’12]

Map → Planning for robots (Mobility)

Topological Maps

[Friedman et.al., IJCAI’07] [Zheng et.al.’19 In submission]

slide-26
SLIDE 26

Literature review

Structured Prediction

Voronoi Random Field (VRF) in Factor graph (i.e. MRF) in

[Pronobis et.al., ICRA’12] [Friedman et.al., IJCAI’07]

Graph-Structured Sum-Product Networks (GraphSPNs [Zheng et.al. AAAI’18] )

slide-27
SLIDE 27
  • TopoNets, GraphSPNs
  • Action-oriented semantic mapping

What?

Our Research

slide-28
SLIDE 28
  • TopoNets, GraphSPNs
  • Action-oriented semantic mapping

What?

Our Research

slide-29
SLIDE 29

TopoNets

slide-30
SLIDE 30

Video link: https://www.youtube.com/watch?v=JrXeRsnJin0

slide-31
SLIDE 31

End-to-end Unified Deep Model

Take-away I

Sensory information Local place semantics Global topology Semantics in global context Unified Model

Figure adapted from [Pronobis, et. al. ICAPS Workshop’17]

slide-32
SLIDE 32
  • TopoNets are Sum-Product Networks
  • Viewed in 2 ways:
  • Deep architecture
  • Graphical model
  • Structure semantics:
  • Hierarchical mixture of parts

Take-away II

Tractable Exact Inference

[Poon&Domingos, UAI’11]

Latent Variable Input Variables

slide-33
SLIDE 33

Sum-Product Networks

[Poon & Domingos, UAI’11, Friesen & Domingos, ICML’16]

Naïve Bayes Mixture Model

  • 3 components
  • 2 binary variables

X1 X2 X2 X1

0.3 0.2 0.5

0.2 0.8 0.3 0.7 0.5 0.5 0.6 0.4

Sum (Mixture Model) Weights (Priors) Product (Compositions of Parts) Low-level Features Input Variables

P(X1, X2)

slide-34
SLIDE 34
  • Learn conditional or joint distributions
  • Tractable partition function, exact inference

Sum-Product Networks

[Poon & Domingos, UAI’11, Friesen & Domingos, ICML’16]

slide-35
SLIDE 35

Template-based Method

Take-away III

  • Learn a set of template networks
  • Templates can decompose graphs
  • At inference time, form a single network
  • Adapts structure to topology of the environment

The end-to-end model is called TopoNets. The SPN-based structured prediction method is called GraphSPN

slide-36
SLIDE 36

TopoNets: Template-based Method

Learning

slide-37
SLIDE 37

TopoNets: Template-based Method

Inference

slide-38
SLIDE 38

Dataset

  • Collected by mobile robot
  • 40 semantic maps on 4 floors
  • Built from laser-range and odometry data
  • Two place category setups
  • Cross-validation:
  • Trained on graphs from 3 floors
  • Tested on graphs from remaining floor

Experiments

slide-39
SLIDE 39

Setup

𝑁𝑢𝑓𝑡𝑢 = (𝑈, 𝒀, 𝒁) 𝑁𝑢𝑓𝑡𝑢: test semantic map 𝑈: topological graph 𝒀: local observations 𝒁: semantic attributes 𝒁 = 𝒁𝑞𝑚𝑏𝑑𝑓 ∪ 𝒁𝑞𝑚𝑏𝑑𝑓ℎ𝑝𝑚𝑒𝑓𝑠 Experiments

slide-40
SLIDE 40

Tasks

Task 1: Semantic place classification ෝ 𝒛 = argmax𝑧∈𝒁𝑞𝑚𝑏𝑑𝑓 𝑄(𝒁𝑞𝑚𝑏𝑑𝑓|𝒀) Task 2: Inferring placeholders (unexplored) ෝ 𝒛𝑞𝑚𝑏𝑑𝑓ℎ𝑝𝑚𝑒𝑓𝑠 = argmax𝑧∈𝒁𝑞𝑚𝑏𝑑𝑓ℎ𝑝𝑚𝑒𝑓𝑠 𝑄(𝒁|𝒀) Task 3: Novelty detection Experiments

Note: 𝒁 = 𝒁𝑞𝑚𝑏𝑑𝑓 ∪ 𝒁𝑞𝑚𝑏𝑑𝑓ℎ𝑝𝑚𝑒𝑓𝑠

𝑂: Number of classes 𝒒𝑗: place classifier likelihoods for node 𝑗 ෝ 𝒒𝑗: modified likelihoods for node 𝑗 𝑞𝑗1 𝑞𝑗𝑘 𝑞𝑗𝑙 𝑞𝑗𝑂 In each trial, select two random classes 𝑘, 𝑙. For every node 𝑗, 𝑞𝑗1 𝑞𝑗𝑙 𝑞𝑗𝑘 𝑞𝑗𝑂

Swap

𝒒𝑗 ෝ 𝒒𝑗

slide-41
SLIDE 41
slide-42
SLIDE 42

Quantitative Results I

Experiments

slide-43
SLIDE 43

Quantitative Results II

Experiments

slide-44
SLIDE 44

Novelty Detection Results

Experiments

slide-45
SLIDE 45

Semantic Place Classification

Explanation: GraphSPN synthetic experiment

Noise Level →

Accuracy (%) →

Freiburg Saarbrücken Stockholm

GraphSPN MRF (order 2) MRF (order 3) [Zheng et.al. AAAI’18] MRFs: Sensitive to noise → Sensitive to training maps

slide-46
SLIDE 46

Performance

Experiments

Each local observation: 1176 pixels each pixel 3 possible values ➔1176 x 3 = 3258 indicators Worst case run time (empirical) 10 class case, evaluate P(X,Y) TopoNets: 105 nodes in the topological map: 0.36s 155 nodes in the topological map: 0.49s Local + MRF: worse case MRF doesn’t converge > 45s

slide-47
SLIDE 47

Summary

  • Take-away I : End-to-end Unified Deep Spatial Model
  • Take-away II:Tractable Exact Inference
  • Take-away III: Template-based method

TopoNets

slide-48
SLIDE 48
  • TopoNets, GraphSPNs
  • Action-oriented semantic mapping

What?

Our Research

slide-49
SLIDE 49

Bridging the Semantic Gap for Robots: Action-Oriented Semantic Maps via Mixed Reality

Eric Rosen, Nishanth Kumar, Daniel Ullman, David Whitney, George Konidaris, Stefanie Tellex Brown University

49

slide-50
SLIDE 50

64 ROBOT TASK PLANNING AND EXPLANATION IN OPEN AND UNCERTAIN WORLDS

  • M. Hanheide, M. Göbelbecker, G. Horn, A. Pronobis, K. Sjöö, A. Aydemir, P. Jensfelt, C. Gretton, R. Dearden,
  • M. Janicek, H. Zender, G.-J. Kruijff, N. Hawes, J. Wyatt

In: Artificial Intelligence, 247, 2017.

slide-51
SLIDE 51

65

slide-52
SLIDE 52

66

slide-53
SLIDE 53

Action-Oriented Semantic Maps (AOSMs)

AOSM = <Mo, Mt, A, I>

  • Mo: metric map
  • Mt: topological map
  • A: set of actions
  • a: <Mo, Mt> -> Mt’
  • I: initiation set classifier
  • I: <Mo, Mt, A> -> 2^|A|

68

slide-54
SLIDE 54

Action-Oriented Semantic Maps (AOSMs)

69

slide-55
SLIDE 55

70

slide-56
SLIDE 56

Interaction Model

71

slide-57
SLIDE 57

Interaction Model

72

slide-58
SLIDE 58

73

slide-59
SLIDE 59

74

slide-60
SLIDE 60
slide-61
SLIDE 61
slide-62
SLIDE 62
slide-63
SLIDE 63
slide-64
SLIDE 64
slide-65
SLIDE 65
slide-66
SLIDE 66

82

slide-67
SLIDE 67

Rapid Iterative Testing And Evaluation (RITE)

  • Reset Interaction Step
  • Sensitivity changes
  • Text displaying current interaction step

86

slide-68
SLIDE 68

87

slide-69
SLIDE 69

Contributions

  • Action-Oriented Semantic Maps (AOSMs): Semantic Maps that model

consequences of actions

88

slide-70
SLIDE 70

Contributions

  • Action-Oriented Semantic Maps (AOSMs): Semantic Maps that model

consequences of actions

  • MR interface for humans to aid robots generate and correct AOSMs

89

slide-71
SLIDE 71

Contributions

  • Action-Oriented Semantic Maps (AOSMs): Semantic Maps that model

consequences of actions

  • MR interface for humans to aid robots generate and correct AOSMs
  • Rapid Iterative Testing and Evaluation of MR interface with 3 novice

users

90

slide-72
SLIDE 72

Contributions

  • Action-Oriented Semantic Maps (AOSMs): Semantic Maps that model

consequences of actions

  • MR interface for humans to aid robots generate and correct AOSMs
  • Rapid Iterative Testing and Evaluation of MR interface with 3 novice users
  • Demonstration of AOSMs via MR to perform 3 complex tasks:
  • Pick and Place
  • Flipping light switches
  • Turning off sinks

91

slide-73
SLIDE 73

Conclusion

  • Semantic maps are world descriptions.

Multiple tasks Automated planning and scheduling World states description Semantic maps

State representation

slide-74
SLIDE 74

Conclusion

  • Semantic mapping currently simplifies to

structured place classification.

  • Proposed:
  • TopoNets, end-to-end unified deep model for

semantic mapping over topological graphs

  • Place classification, placeholder inference, novelty

detection

  • Action…
slide-75
SLIDE 75
  • Viswanathan, Pooja, et al. "Automated place classification using object detection." 2010

Canadian Conference on Computer and Robot Vision. IEEE, 2010.

  • Zhu, Hongyuan, Jean-Baptiste Weibel, and Shijian Lu. "Discriminative multi-modal feature

fusion for rgbd indoor scene recognition." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016.

  • Sünderhauf, Niko, et al. "Place categorization and semantic mapping on a mobile

robot." 2016 IEEE international conference on robotics and automation (ICRA). IEEE, 2016.

  • Pronobis, Andrzej, and Patric Jensfelt. "Large-scale semantic mapping and reasoning with

heterogeneous modalities." 2012 IEEE International Conference on Robotics and Automation. IEEE, 2012.

  • Pronobis, Andrzej, and Rajesh PN Rao. "Learning deep generative spatial models for mobile

robots." 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2017.

  • Friedman, Stephen, Hanna Pasula, and Dieter Fox. "Voronoi Random Fields: Extracting

Topological Structure of Indoor Environments via Place Labeling." IJCAI. Vol. 7. 2007.

  • Zheng, Kaiyu, Andrzej Pronobis, and Rajesh PN Rao. "Learning graph-structured sum-

product networks for probabilistic semantic maps." Thirty-Second AAAI Conference on Artificial Intelligence. 2018.

References (for citations without titles)

slide-76
SLIDE 76
  • Zheng, Kaiyu. "Learning Large-Scale Topological Maps Using Sum-Product Networks." arXiv

preprint arXiv:1706.03416(2017).

  • Zheng, Kaiyu, and Andrzej Pronobis. "From Pixels to Buildings: End-to-end Probabilistic

Deep Networks for Large-scale Semantic Mapping." arXiv preprint arXiv:1812.11866(2018).

  • Li, Li-Jia, et al. "Objects as attributes for scene classification." European Conference on

Computer Vision. Springer, Berlin, Heidelberg, 2010.

  • Goeddel, Robert, and Edwin Olson. "Learning semantic place labels from occupancy grids

using CNNs." 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2016.

  • Pronobis, Andrzej, Francesco Riccio, and Rajesh PN Rao. "Deep spatial affordance hierarchy:

Spatial knowledge representation for planning in large-scale environments." ICAPS 2017 Workshop on Planning and Robotics. 2017.

References (for citations without titles)

slide-77
SLIDE 77

Thank you!