Learning of perceptual models of retailproducts using - - PowerPoint PPT Presentation

▶
learning of perceptual models of retailproducts using
SMART_READER_LITE
LIVE PREVIEW

Learning of perceptual models of retailproducts using - - PowerPoint PPT Presentation

Jan 09, 2023 348 likes •592 views

Learning of perceptual models of retailproducts using photo-realistic simulations of supermarkets Ferenc B alint-Bencz edi IAI, University of Bremen IROS, 5 th October 2018 About our group Backgounrd building complete cognitive

slide-1
SLIDE 1

Learning of perceptual models

  • f

retailproducts using photo-realistic simulations

  • f

supermarkets

Ferenc B´ alint-Bencz´ edi IAI, University of Bremen IROS, 5th October 2018

slide-2
SLIDE 2

About our group

Backgounrd

  • building complete cognitive

robotic agents that perform everyday manipulation actions

  • coordinators of the CRC

EASE

Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5th October 2018 2

slide-3
SLIDE 3

About our group

Backgounrd

  • building complete cognitive

robotic agents that perform everyday manipulation actions

  • coordinators of the CRC

EASE

Refills

  • building semantic store maps
  • transferring our systems to a

different domain

Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5th October 2018 3

slide-4
SLIDE 4

About our group

Backgounrd

  • building complete cognitive

robotic agents that perform everyday manipulation actions

  • coordinators of the CRC

EASE

Refills

  • building semantic store maps
  • transferring our systems to a

different domain One of our main focuses: Virtual realities for robotic applications using game engine technology

Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5th October 2018 4

slide-5
SLIDE 5
  • 1. Motivation for using Game Engines
  • 2. Previous work in other domains
  • 3. Transferring to retail environments

Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5th October 2018 5

slide-6
SLIDE 6

Motivation for using Game Engines

Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5th October 2018 6

slide-7
SLIDE 7

Photo Realism in Game Engines (I)

  • realistic environments for everyday manipulation (size, realism, detail,

machine-understandable, faithful simulation)

video source: https://www.youtube.com/watch?v=E3LtFrMAvQ4 Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5th October 2018 7

slide-8
SLIDE 8

Photo Realism in Game Engines (II)

Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5th October 2018 8

slide-9
SLIDE 9

Motivation- Why a game engine?

→ Rendering images is not new. Many existing tools using SoA Ray-tracing libraries

  • real-time photo-realistic rendering
  • open-source, open-access
  • game engines are uniquely suited for scaling to large environments
  • can have dynamic interaction with the environment
  • more than a means for generating images

Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5th October 2018 9

slide-10
SLIDE 10

Previous work in other domains

Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5th October 2018 10

slide-11
SLIDE 11

Modeling our main scenario, creating a robot simulator

Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5th October 2018 11

slide-12
SLIDE 12

Variations of Episodic Memories

Episodic Memories

Extract Image & Object Data

O1 O1 O1 T1 T2

T3

O2 O2 O2 O3 O3

Ground Truth annotation

  • V. Configurator

Object alternatives KnowRob

Ontology of

  • bjects

Similar

  • bjects

Epispode Configuration

Episode Generator

EP1 EP2 EPn

RoboSherlock

  • bj((type,'VollMilch),

(shape,'Box') (color,'White'))

Learning Models Scene annoations with GT data

Robotic Agent

Balint-Benczedi F. and Beetz M.: Variation on a Theme: “It’s a poor sort of memory that only works backwards“ @IROS’18

Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5th October 2018 12

slide-13
SLIDE 13

Variations of Episodic Memories: Results

Data Precision Recall Accuracy F-Score Turn table 0.96% 0.94% 0.94% 0.94% Unreal Variations 0.90% 0.83% 0.83% 0.84% Training Data Testing Data Epochs IoU Recall 80 Var 20 Var 45k 89.17 % 98.86 % 20k 86.11 % 99.26 % 10k 85.39 % 99.13 % 1k 71.30 % 91.32 % 4 Robot Episodes 45k 80.83 % 99.42 % 20k 80.41 % 97.98 % 10k 78.16 % 97.69 % 1k 67.88 % 88.76 % 80 Var + 2 RE 20 Var 45k 89.68 % 98.82% 20k 88.39 % 99.13 % 10k 85.82 % 98.99 % 1k 79.11 % 97.18 % 2 Robot Episodes 45k 84.47 % 99.37 % 20k 83.85 % 100 % 10k 81.44 % 100 % 1k 74.6 % 98.1 %

Tabelle: IoU and Recall of Yolo trained with different data sets: Variations of the original two scenes (Var), and variations combined with Robot Episodes (RE)

Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5th October 2018 13

slide-14
SLIDE 14

Adding more variety

Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5th October 2018 14

slide-15
SLIDE 15

Training Mask -R-CNN

Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5th October 2018 15

slide-16
SLIDE 16

Results on Synthetic data

Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5th October 2018 16

slide-17
SLIDE 17

Preliminary results on real data

Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5th October 2018 17

slide-18
SLIDE 18

Transferring to retail environments

Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5th October 2018 18

slide-19
SLIDE 19

Digital replications of retail environments

  • “photo-realistic” rendering
  • “faithful” physics simulation
  • implement real environments
  • implement real activities
  • machine understandable
  • pervasive data recording
  • casted as KnowRob knowledge bases
  • game engine-enabled

Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5th October 2018 19

slide-20
SLIDE 20

Automating the process

Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5th October 2018 20

slide-21
SLIDE 21

From VR to the real world

Current State:

  • Generate variations of object

placements

  • Training data generation and

recognition

  • DeCaf: replace soft-max with custom

classifier

  • train on images from unreal and test
  • n robot

Future:

  • Evaluate different recognition approaches
  • migrate more of the approaches from the kitchen domain
  • Identifying further learning scenarios

Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5th October 2018 21

slide-22
SLIDE 22

Scaling up the environment

Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5th October 2018 22

slide-23
SLIDE 23

Thank you for your attention! Questions? Many thanks to: Andrei Haidu, Patrick Mania, Michael Neumann, Franklin Kenghagho, Feroz Siddiky and more More information at: http://www.robcog.org https://www.refills-project.eu http://www.open-ease.org/

Motivation for using Game Engines Previous work in other domains Transferring to retail environments Ferenc Bálint-Benczédit IROS, 5th October 2018 23