SLIDE 1
Volumetric instance-aware semantic mapping and 3D object discovery - - PowerPoint PPT Presentation
Volumetric instance-aware semantic mapping and 3D object discovery - - PowerPoint PPT Presentation
Volumetric instance-aware semantic mapping and 3D object discovery Margarita Grinvald, Fadri Furrer, Tonci Novkovic, Jen Jen Chung, Cesar Cadena, Roland Siegwart, Juan Nieto IROS, 5 Nov 2019 Instance-aware semantic mapping solves detection
SLIDE 2
SLIDE 3
SLIDE 4
Instance-aware semantic mapping solves detection and recognition at the level of individual objects Semantic mapping classifies scene parts by category but disregards individual object instances Traditional 3D reconstruction fails to provide any high-level interpretation of the scene
SLIDE 5
Instance-aware semantic mapping solves detection and recognition at the level of individual objects Semantic mapping classifies scene parts by category but disregards individual object instances
SLIDE 6
Instance-aware semantic mapping solves detection and recognition at the level of individual objects
SLIDE 7
Object-level mapping in the real-world needs to cope with the complexity of an open-set environment
SLIDE 8
Object-level mapping in the real-world needs to cope with the complexity of an open-set environment
SLIDE 9
Object-level mapping in the real-world needs to cope with the complexity of an open-set environment
SLIDE 10
Volumetric instance-aware semantic mapping and 3D object discovery
Dense object-level mapping with a localized RGB-D camera Object detection in an open-set world by fusing classic and modern computer vision Efficient online framework well-suited for a real-world robotic setup
SLIDE 11
Volumetric instance-aware semantic mapping and 3D object discovery
Dense object-level mapping with a localized RGB-D camera Object detection in an open-set world by fusing classic and modern computer vision Efficient online framework well-suited for a real-world robotic setup
SLIDE 12
SLIDE 13
Depth RGB
SLIDE 14
TSDF grid Fusion Framewise 2D segmentation
A dense volumetric object-level map is built online by incrementally fusing per-frame 2D segmentation
Depth RGB
SLIDE 15
A dense volumetric object-level map is built online by incrementally fusing per-frame 2D segmentation
Framewise 2D segmentation Depth RGB
SLIDE 16
A neural network detects recognized objects in the RGB frame and predicts for each a (loose) segmentation mask
RGB Semantic instance-aware segmentation Mask R-CNN
SLIDE 17
An unsupervised geometric method exhaustively (over)segments the depth frame
Depth Convexity-based segmentation Convexity criterion
SLIDE 18
The semantic masks group sets of convex segments as part of the same object instance
Depth Geometric segmentation RGB Mask R-CNN Semantically refined geometric segmentation Overlap measure
SLIDE 19
The partial per-frame geometry and segmentation observations are incrementally integrated into a volumetric map
Depth Geometric segmentation RGB Mask R-CNN
SLIDE 20
Volumetric instance-aware semantic mapping and 3D object discovery
Dense object-level mapping with a localized RGB-D camera Object detection in an open-set world by fusing classic and modern computer vision Efficient online framework well-suited for a real-world robotic setup
SLIDE 21
Volumetric instance-aware semantic mapping and 3D object discovery
Dense object-level mapping with a localized RGB-D camera Object detection in an open-set world by fusing classic and modern computer vision Efficient online framework well-suited for a real-world robotic setup
SLIDE 22
The framework detects both recognized instances and previously unseen object-like elements
SLIDE 23
SLIDE 24
SLIDE 25
SLIDE 26
SLIDE 27
SLIDE 28
SLIDE 29
SLIDE 30
A sample inventory of objects includes recognized instances as well as previously unseen, discovered elements
“chair”
SLIDE 31
A sample inventory of objects includes recognized instances as well as previously unseen, discovered elements
“chair”
“couch”
SLIDE 32
A sample inventory of objects includes recognized instances as well as previously unseen, discovered elements
“chair” “couch”
“table”
SLIDE 33
A sample inventory of objects includes recognized instances as well as previously unseen, discovered elements
“chair” “couch” “table”
[jacket] [bag] [fan] [fan] [speaker] [box] [case] [heater] [paper roll] [appliance] [pillow] [tissues] [drawer]
SLIDE 34
Volumetric instance-aware semantic mapping and 3D object discovery
Dense object-level mapping with a localized RGB-D camera Object detection in an open-set world by fusing classic and modern computer vision Efficient online framework well-suited for a real-world robotic setup
SLIDE 35
Volumetric instance-aware semantic mapping and 3D object discovery
Dense object-level mapping with a localized RGB-D camera Object detection in an open-set world by fusing classic and modern computer vision Efficient online framework well-suited for a real-world robotic setup
SLIDE 36
The framework has been validated within a real-world setup
SLIDE 37
The object-level map of an office floor is built in an online fashion
SLIDE 38
The final map densely describes individual scene objects without introducing a significant memory overhead
SLIDE 39
Volumetric instance-aware semantic mapping and 3D object discovery
Dense object-level mapping with a localized RGB-D camera Object detection in an open-set world by fusing classic and modern computer vision Efficient online framework well-suited for a real-world robotic setup
SLIDE 40
SLIDE 41
SLIDE 42