An Introduction to Eric Rosen, Kaiyu Zheng Semantic Mapping in Robotics 3/22/2019
Outline • Timeline of “Semantic mapping” • Why semantic mapping (in robotics)? • How? • Problem definition • Literature review • What? (Our research) • TopoNets, GraphSPNs • Action-oriented semantic mapping
Timeline “Semantic Mapping” • Originated in linguistics - 1960s • Correspondence of hierarchies in languages For example: phonogram “/ h əˈ l ō /” → word “hello” On the Uniqueness of Semantic Mapping [Householder, 1962] • In literacy (vocabulary instruction) • Structure of knowledge in graphic form Semantic Mapping [Johnson et.al., 1986]
Timeline “Semantic Mapping” • Appearance in CS (NLP) - 1989 • Neural network (!) • Semantic relations between symbolic data. Self-Organizing Semantic Maps [Ritter and Kohonen, 1989] (training data) “ geometrically or topologically organized maps”
These concepts are central to semantic mapping in robotics! Self-Organizing Semantic Maps [Ritter and Kohonen, 1989]
Timeline “Semantic Mapping” (robotics) • Semantic hierarchy of spatial representations • [sensorimotor → control] → topology → geometry A robot exploration and mapping strategy based on a semantic hierarchy of spatial representations [Kuipers and Byun, 1991] • Definition of terms “metric map”, “topological map”, “hybrid map” (no semantic map ) Some Notes on the Use of Hybrid Maps for Mobile Robots [Buschka and Saffotti, 2004] • Hybrid metric-topological-semantic map Multi-Hierarchical Semantic Maps for Mobile Robotics [Galindo, Buschka et.al. , 2005]
Multi-Hierarchical Semantic Maps for Mobile Robotics [Galindo, Buschka et.al. , 2005]
Timeline “Semantic Mapping” (robotics) • Semantic hierarchy of spatial representations • [sensorimotor → control] → topology → geometry A robot exploration and mapping strategy based on a semantic hierarchy of spatial representations [Kuipers and Byun, 1981] • Definition of terms “metric map”, “topological map”, “hybrid map” (no semantic map ) Some Notes on the Use of Hybrid Maps for Mobile Robots [Buschka and Saffotti, 2004] • Hybrid metric-topological-semantic map Multi-Hierarchical Semantic Maps for Mobile Robotics [Galindo, Buschka et.al. , 2005] • 3D laser-based SLAM with scene interpretation Towards semantic maps for mobile robots [Nüchter, Hertzberg et.al. , 2008]
Recent successor (also 3D SLAM) Meaningful Maps With Object-Oriented Semantic Mapping [Sünderhauf, et.al. , 2016] Towards semantic maps for mobile robots [Nüchter, Hertzberg et.al. , 2008]
Timeline “Semantic Mapping” (robotics) • Semantic hierarchy of spatial representations • [sensorimotor → control] → topology → geometry A robot exploration and mapping strategy based on a semantic hierarchy of spatial representations [Kuipers and Byun, 1981] • Definition of terms “metric map”, “topological map”, “hybrid map” (no semantic map ) Some Notes on the Use of Hybrid Maps for Mobile Robots [Buschka and Saffotti, 2004] • Hybrid metric-topological-semantic map Multi-Hierarchical Semantic Maps for Mobile Robotics [Galindo, Buschka et.al. , 2005] • 3D laser-based SLAM with scene interpretation Towards semantic maps for mobile robots [Nüchter, Hertzberg et.al. , 2008] • Semantic Mapping with Mobile Robots [Influencer for us] PhD thesis [Pronobis, 2011]
Why semantic mapping? Robots Are Getting Better Hardware ↑ Sensing ↑ Control ↑ Robots are getting better at single tasks Mobility Manipulation
Why semantic mapping? Moving Forward Multiple tasks Automated planning and scheduling
Why semantic mapping? Moving Forward Multiple tasks Automated planning and scheduling State representation World states description Semantic maps
How? Problem definition Semantic Mapping Input PhD thesis [Pronobis, 2011] Sensory observations and odometry Prior knowledge of semantic information Output Semantic maps (what is this?) Capable to facilitate planning Shared understanding in literature
Problem definition Semantic Maps • Historically informal Self-Organizing Semantic Maps Towards semantic maps for mobile robots [Ritter and Kohonen, 1989] [Nüchter, Hertzberg et.al. , 2008] Multi-Hierarchical Semantic Maps for Mobile Robotics [Galindo, Buschka et.al. , 2005] and PhD thesis [Pronobis, 2011] did not provide a formal definition either.
Problem definition Semantic Maps • Formal, but ignored (and hard to find) Vague enough Somewhat complicated Semantic Maps for Robotics [Lang and Paulus, 2014]
Problem definition Semantic Maps • The attempt in [Zheng et.al.’19 In submission] 𝑁 = (𝑈, 𝒀, 𝒁) 𝑁 : semantic map 𝑈 = (𝑾, 𝑭) : topological graph 𝒀 = {𝒀 𝑗 : 𝑗 ∈ 𝑾} : local observations 𝒁 = {𝒁 𝑗 : 𝑗 ∈ 𝑾} : semantic attributes Each node 𝑗 is a place (defined later) • Did not include “metric map” • No semantics in edges • Suitable for their research problem
Problem definition Summary • Semantic mapping is conceptually clear • Formally define semantic mapping? • Semantic maps may vary • Yet, one should at least include: • Spatial information (contained in map) • Maps: Metric, topological • Anchoring of spatial concepts • Place classification (simplification) (We are talking about mobile robots)
Literature review Maps Metric map • Metric map • Created by SLAM • Captures geometry of the world • Topological map • A graph ( V,E ) • Each node in V is a place [Sünderhauf et.al., ICRA’16] • Each edge in E indicates navigability Topological maps • Captures structure of the world [Zheng., senior thesis’17]
[Pronobis , et. al. ICAPS Workshop’17]
Literature review Local Place Classification • Classification from local sensory information • Through detecting objects in the environment (less common) [Viswanathan et.al., CRV’10] [Li et.al., ECCV Workshops’10] • Through robot sensory observations • RGB-D (Visual scene classification) [Zhu et.al., CVPR’16] • Laser-range [Pronobis et.al., IROS’17] [Friedman et.al., IJCAI’07] • Multi-modal (RGB+Laser+Odometry) [Pronobis et.al., IJRR’10]
Literature review Mainstream: Place classification on a map Structured prediction • Relations between places • Probabilistic inference • Boost classification results
Literature review Place Classification on a Map Map → Planning for robots (Mobility) Metric Maps [Friedman et.al., IJCAI’07] [Goeddel et.al., IROS’16] [Sünderhauf et.al., ICRA’16]
Literature review Place Classification on a Map Map → Planning for robots (Mobility) Topological Maps [Friedman et.al., IJCAI’07] [Pronobis et.al., ICRA’12] [Zheng et.al.’19 In submission]
Literature review Structured Prediction Voronoi Random Field (VRF) in [Friedman et.al., IJCAI’07] Graph-Structured Sum-Product Networks (GraphSPNs [Zheng et.al. AAAI’18] ) Factor graph (i.e. MRF) in [Pronobis et.al., ICRA’12]
What? Our Research • TopoNets, GraphSPNs • Action-oriented semantic mapping
What? Our Research • TopoNets, GraphSPNs • Action-oriented semantic mapping
TopoNets
Video link: https://www.youtube.com/watch?v=JrXeRsnJin0
Take-away I End-to-end Unified Deep Model Semantics in Unified Model global context Global topology Local place semantics Sensory information Figure adapted from [Pronobis , et. al. ICAPS Workshop’17]
Take-away II Tractable Exact Inference • TopoNets are Sum-Product Networks [Poon&Domingos , UAI’11] • Viewed in 2 ways: • Deep architecture • Graphical model • Structure semantics: • Hierarchical mixture of parts Latent Variable Input Variables
Sum-Product Networks P(X 1 , X 2 ) Sum (Mixture Model) 0.5 Weights (Priors) 0.3 0.2 Product (Compositions of Parts) Low-level Features 0.8 0.7 0.5 0.4 0.2 0.3 0.5 0.6 Naïve Bayes Mixture Model • 3 components X 1 X 1 X 2 X 2 Input Variables • 2 binary variables [Poon & Domingos , UAI’11, Friesen & Domingos , ICML’16]
Sum-Product Networks • Learn conditional or joint distributions • Tractable partition function, exact inference [Poon & Domingos , UAI’11, Friesen & Domingos , ICML’16]
Take-away III Template-based Method • Learn a set of template networks • Templates can decompose graphs • At inference time, form a single network • Adapts structure to topology of the environment The end-to-end model is called TopoNets . The SPN-based structured prediction method is called GraphSPN
TopoNets: Template-based Method Learning
TopoNets: Template-based Method Inference
Experiments Dataset • Collected by mobile robot • 40 semantic maps on 4 floors • Built from laser-range and odometry data • Two place category setups • Cross-validation: • Trained on graphs from 3 floors • Tested on graphs from remaining floor
Experiments Setup 𝑁 𝑢𝑓𝑡𝑢 = (𝑈, 𝒀, 𝒁) 𝑁 𝑢𝑓𝑡𝑢 : test semantic map 𝑈 : topological graph 𝒀 : local observations 𝒁 : semantic attributes 𝒁 = 𝒁 𝑞𝑚𝑏𝑑𝑓 ∪ 𝒁 𝑞𝑚𝑏𝑑𝑓ℎ𝑝𝑚𝑒𝑓𝑠
Recommend
More recommend
