Learning of perceptual models of retailproducts using - PowerPoint PPT Presentation

Learning of perceptual models of retailproducts using photo-realistic simulations of supermarkets Ferenc B´ alint-Bencz´ edi IAI, University of Bremen IROS, 5 th October 2018

About our group Backgounrd • building complete cognitive robotic agents that perform everyday manipulation actions • coordinators of the CRC EASE Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5 th October 2018 2

About our group Backgounrd Refills • building complete cognitive • building semantic store maps robotic agents that perform everyday manipulation actions • transferring our systems to a • coordinators of the CRC different domain EASE Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5 th October 2018 3

About our group Backgounrd Refills • building complete cognitive • building semantic store maps robotic agents that perform everyday manipulation actions • transferring our systems to a • coordinators of the CRC different domain EASE One of our main focuses: Virtual realities for robotic applications using game engine technology Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5 th October 2018 4

1. Motivation for using Game Engines 2. Previous work in other domains 3. Transferring to retail environments Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5 th October 2018 5

Motivation for using Game Engines Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5 th October 2018 6

Photo Realism in Game Engines (I) • realistic environments for everyday manipulation (size, realism, detail, machine-understandable, faithful simulation) video source: https://www.youtube.com/watch?v=E3LtFrMAvQ4 Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5 th October 2018 7

Photo Realism in Game Engines (II) Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5 th October 2018 8

Motivation- Why a game engine? → Rendering images is not new. Many existing tools using SoA Ray-tracing libraries • real-time photo-realistic rendering • open-source, open-access • game engines are uniquely suited for scaling to large environments • can have dynamic interaction with the environment • more than a means for generating images Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5 th October 2018 9

Previous work in other domains Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5 th October 2018 10

Modeling our main scenario, creating a robot simulator Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5 th October 2018 11

Variations of Episodic Memories KnowRob Ground Similar Ontology of Truth annotation objects objects Episode O 1 Episodic Generator V. Con fi gurator Object Extract Image & O 2 Memories Object Data alternatives O 3 T2 Con fi guration O 1 EP2 Epispode O 2 EP1 O 1 O 3 T1 O 2 EPn T3 Scene annoations with Learning Models Robotic Agent RoboSherlock GT data obj ((type,'VollMilch), (shape,'Box') (color,'White')) Balint-Benczedi F. and Beetz M.: Variation on a Theme: “It’s a poor sort of memory that only works backwards“ @IROS’18 Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5 th October 2018 12

Variations of Episodic Memories: Results Data Precision Recall Accuracy F-Score Turn table 0.96% 0.94% 0.94% 0.94% Unreal Variations 0.90% 0.83% 0.83% 0.84% Training Data Testing Data Epochs IoU Recall 45k 89.17 % 98.86 % 20k 86.11 % 99.26 % 20 Var 10k 85.39 % 99.13 % 1k 71.30 % 91.32 % 80 Var 45k 80.83 % 99.42 % 20k 80.41 % 97.98 % 4 Robot Episodes 10k 78.16 % 97.69 % 1k 67.88 % 88.76 % 45k 89.68 % 98.82% 20k 88.39 % 99.13 % 20 Var 10k 85.82 % 98.99 % 1k 79.11 % 97.18 % 80 Var + 2 RE 45k 84.47 % 99.37 % 20k 83.85 % 100 % 2 Robot Episodes 10k 81.44 % 100 % 1k 74.6 % 98.1 % Tabelle: IoU and Recall of Yolo trained with different data sets: Variations of the original two scenes (Var), and variations combined with Robot Episodes (RE) Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5 th October 2018 13

Adding more variety Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5 th October 2018 14

Training Mask -R-CNN Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5 th October 2018 15

Results on Synthetic data Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5 th October 2018 16

Preliminary results on real data Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5 th October 2018 17

Transferring to retail environments Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5 th October 2018 18

Digital replications of retail environments • “photo-realistic” rendering • “faithful” physics simulation • implement real environments • implement real activities • machine understandable • pervasive data recording • casted as KnowRob knowledge bases • game engine-enabled Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5 th October 2018 19

Automating the process Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5 th October 2018 20

From VR to the real world Current State: • Generate variations of object placements • Training data generation and recognition • DeCaf: replace soft-max with custom classifier • train on images from unreal and test on robot Future: • Evaluate different recognition approaches • migrate more of the approaches from the kitchen domain • Identifying further learning scenarios Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5 th October 2018 21

Scaling up the environment Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5 th October 2018 22

Thank you for your attention! Questions? Many thanks to: Andrei Haidu, Patrick Mania, Michael Neumann, Franklin Kenghagho, Feroz Siddiky and more More information at: http://www.robcog.org https://www.refills-project.eu http://www.open-ease.org/ Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5 th October 2018 23