Luca Iocchi Sapienza University of Rome, Italy Motivation - - PowerPoint PPT Presentation

▶
luca iocchi
SMART_READER_LITE
LIVE PREVIEW

Luca Iocchi Sapienza University of Rome, Italy Motivation - - PowerPoint PPT Presentation

Apr 28, 2023 521 likes •906 views

RoboCup@Home Benchmarking Intelligent Service Robots through Scientific Competitions Luca Iocchi Sapienza University of Rome, Italy Motivation Development of Domestic Service Robots Complex Integrated Systems AI&Robotics Large

slide-1
SLIDE 1

RoboCup@Home

Benchmarking Intelligent Service Robots through Scientific Competitions

Luca Iocchi

Sapienza University of Rome, Italy

slide-2
SLIDE 2

Motivation

  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup
  • Development of Domestic Service Robots
  • Complex Integrated Systems AI&Robotics
  • Large variety of tasks
  • Human-Robot Interaction
slide-3
SLIDE 3

About RoboCup@Home

  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup
  • Starts in 2006
  • 1 international competition per year
  • Many regional competitions
  • Largest competition for domestic and service robots
slide-4
SLIDE 4

Large variety of tasks

  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup

Nimbro@Home, University of Bonn, Germany

slide-5
SLIDE 5

Benchmarking Robotic Abilities

  • Functional benchmarking
  • Planning domains (e.g., IPC)
  • SLAM (e.g., Radish, RAWSEEDS)
  • Person tracking (e.g., PETS)
  • Object recognition (e.g., PASCAL)
  • Navigation (e.g., MoVeMe)
  • Mobile manipulation …
  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup
  • Integrated system benchmarking
  • Benchmarking the entire system => robotic competitions
  • Combining Functional and Integrated system

benchmarking => trend for the future

slide-6
SLIDE 6

Robotic scientific competitions

  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup
  • DARPA / EUROC Challenges
  • RoboCup Soccer, Rescue, @Home, @Work
  • AAAI / ICRA / IROS robot competitions
  • RoboCup Junior, Eurobot

Advantages of Competitions

  • Set up of common test-beds
  • Attractive for many teams

(research groups)

  • Collaboration and knowledge

sharing

  • Evolution over time

Difficulties for DSR

  • Human involved
  • Real environments
  • Integration of several

functionalities

  • Large variety of tasks
slide-7
SLIDE 7

Observations from other Robot Competitions

  • Little HRI involved
  • Limited application
  • rientation
  • No real world environment
  • Very specific rules and

regulations for robots and environment

  • Often requires many

resources (special environment, many robots)

  • Danger of developing

towards local optima

  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup
slide-8
SLIDE 8

Local Optima in Benchmarking

Fixed task and performance improvement over time Local optimum (overfitting)

  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup

Increasing difficulty of tasks and maintaining performance Increase general applicability

Time Performance Given a fixed task Tasks Performance

slide-9
SLIDE 9

RoboCup@Home Scenario and Concepts

  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup
slide-10
SLIDE 10

RoboCup@Home Scenario and Concepts

  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup
  • Autonomous robots
  • Human-Robot Interaction
  • Non-standardised realistic domestic

environment and real public areas

  • Many tests related to desired functionalities
  • Changing tests over the years to keep

performance "constantly good"

  • Statistical evaluation for measuring league

progresses

slide-11
SLIDE 11

RoboCup@Home Scenario and Concepts

  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup

Teams (research groups) are provided with specifications of tasks, environmental setting and performance metrics to evaluate and compare integrated research in domestic and service robots.

slide-12
SLIDE 12

Current focus of RoboCup@Home

Functional abilities:

  • Navigation
  • Mapping
  • Person recognition
  • Person tracking
  • Object recognition
  • Object manipulation
  • Speech recognition
  • Gesture recognition
  • Cognition
  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup
slide-13
SLIDE 13

Current focus of RoboCup@Home

System properties:

  • Ease of use
  • Fast calibration and setup
  • Natural and multi-modal

interaction

  • Attractiveness and

ergonomics of the robot

  • Adaptivity and general

intelligence

  • Robustness
  • General applicability
  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup
slide-14
SLIDE 14

Implementation of RoboCup@Home

General rules

  • 2 stages with different focus
  • Stage 1 for basic tasks
  • Stage 2 for more complex,

integrated tasks

  • High level of uncertainty in the

environment (no standardization)

  • Only natural interaction allowed
  • Very short setup time (usually 1

minute)

  • Partial score system for tests
  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup
slide-15
SLIDE 15

Stage 1

  • Robot Inspection & Poster:

Autonomous registration to the competition, TC inspection, team poster

  • Follow me: Lead the robot quickly on a

path through an external scenario

  • Cocktail Party: Deliver drinks to

people in the apartment

  • Clean up: Clean up a room in the

apartment

  • Emergency Situation: React to an

unknown emergency situation

  • Technical Challenge: Defined by TC

every year

  • Open Challenge: Demonstrate most

important (scientific) achievements (team choice)

  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup
slide-16
SLIDE 16

Stage 2

  • Enduring General Purpose Service Robot: Solve

multiple tasks not known beforehand upon request

  • Restaurant: Mapping and serving drinks and food in a

real unknown restaurant

  • Demo Challenge: Topic-based free demonstration

(e.g., health-care, cooking, …)

  • Finals: Open demonstration with external jury + Exec

evaluation

  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup
slide-17
SLIDE 17

Final

  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup
slide-18
SLIDE 18

Apartment, People and Objects

  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup
slide-19
SLIDE 19

Person names

  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup
slide-20
SLIDE 20

Objects

  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup
slide-21
SLIDE 21

Object categories and default locations

  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup
slide-22
SLIDE 22

Test evolution: 'follow me' example

  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup

2007: proof of concept, special markers on the walker allowed 2008: walker known, but no special markers 2009: walker unknown 2010: outside the arena (in the RoboCup venue) 2011: pre-defined interferences (people passing between walker and robot) 2012-2013: crowded and complex environment (changing floor through an elevator) future: public environment with crowd and unpredictable interferences

slide-23
SLIDE 23

Benchmarking Robot Cognition: General Purpose Service Robot

  • The test is about how much the robot can

understand and reason about the environment and its task

  • No predefined task
  • Task goals are randomly generated at runtime
  • Task goals can include multiple objects/locations,

underspecified objects/locations and wrong information

  • GPSR incorporates the abilities tested in all previous tests.
  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup
slide-24
SLIDE 24

Benchmarking Robot Cognition: General Purpose Service Robot

Task goal is not predefined ! Given a set of known objects, known locations and known persons, execute a randomly generated task from a set

  • f templates.
  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup
slide-25
SLIDE 25

Evaluation of the League

  • Year by year statistical analysis to:
  • Measure overall performance
  • Drive developments
  • Plan for rule changes
  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup
slide-26
SLIDE 26

Score system of RoboCup@Home

  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup

Navigation Mapping Person Recogn. Person Tracking Object Recogn. Object Manipul. Speech/ Gesture Recogn. Cognition Follow Me 49 % 6 % 39 % 6 % Clean Up Cocktail Party Emergency Situation General Purpose Service Robot Restaurant Open Challenge Demo Challenge Final

Defined by the Technical Committee Defined by the teams

slide-27
SLIDE 27

Functionality distribution

Each test includes a set of the functional abilities Distribution of functional abilities over tests evolves over time allowing for proper analysing and planning.

  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup
slide-28
SLIDE 28

Functionality distribution for a test

Example from Follow me 2012 test Distribution of functional abilities over tests evolves over time allowing for proper analysing and planning.

  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup

Navigation Object Recognition Person Recognition Person Tracking Object Manipulation Speech recognition Gesture recognition Mapping Cognition

CP1 300 0.5 0.5 CP2 300 0.3 0.2 0.3 0.1 0.1 CP3 300 0.5 0.5 Complete 100 1 490 60 390 30 30 1000

slide-29
SLIDE 29

Evaluation 2008-2013

  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup

Best/average score of the finalist teams.

slide-30
SLIDE 30

Evaluation 2008-2013

  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup
slide-31
SLIDE 31

Evaluation 2006-2012

  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup

Performance metrics of the RoboCup@Home league over the years Good: we are not going towards a local optimum!!! Performance do not always increase because of changes in the rules (major changes in 2008, 2010, 2012).

slide-32
SLIDE 32

RoboCup@Home Community Resources

  • Web site (information and rules)
  • @Home Wiki (> 50 teams active worldwide) HW/SW/Papers…
  • Mailing lists (active rule discussion)
  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup

www.robocupathome.org

slide-33
SLIDE 33

Scientific Achievements

  • Speech understanding in noisy

environments

  • Speaker localization for following human

guides

  • Detecting and tracking human operators

using laser and RBGD cameras

  • Detecting, learning and recognizing objects
  • Complex two-hands object manipulation

Demonstrated within an integrated system

  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup
slide-34
SLIDE 34
  • More general tests in the real world (also outdoor)
  • Longer operation times
  • Less pre-defined pre-planned behaviors
  • Improved cognitive and social skills

(language skills, social behaviors)

  • Human-robot and Inter-team

robot-robot cooperation

  • Detecting and dealing with failures

and unpredicted situations

Future directions of RoboCup@Home

  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup
slide-35
SLIDE 35
  • RoboCup@Home can drive the development of

effective intelligent robots

  • Benchmarking methodology based on the definition
  • f several variable tests
  • Cognitive abilities can be actually tested in a real

setting

  • Statistical analysis can drive fast achievements of the

league.

  • Research groups can exploit RoboCup@Home to

develop, test, evaluate and disseminate DSR solutions.

Conclusions

  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup
slide-36
SLIDE 36

Tijn van der Zant and Thomas Wisspeintner. RoboCup@Home: Creating and Benchmarking Tomorrows Service Robot Applications. IN: Lima, P. (Ed) Robotic Soccer, Vienna: I-Tech Education and Publishing. pp521-528, 2007. Thomas Wisspeintner, Tijn van der Zant, Luca Iocchi, Stefan Schiffer. RoboCup@Home: Scientific Competition and Benchmarking for Domestic Service Robots. Interaction Studies. 10 (3). 2009. pp. 393--428, 2009. ISSN: 1572-0373. Jörg Stückler, Dirk Holz, and Sven Behnke: RoboCup@Home: Demonstrating Everyday Manipulation Skills in RoboCup@Home. IEEE Robotics & Automation Magazine 19(2):34-42, 2012. Dirk Holz, Luca Iocchi, Tijn van der Zant, "Benchmarking Intelligent Service Robots through Scientific Competitions: The RoboCup@Home Approach", In AAAI Spring Symposium: Designing Intelligent Robots, 2013.

References

  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup
slide-37
SLIDE 37

Thank you for your attention Questions?

www.robocupathome.org

  • L. Iocchi. RoboCup@Home - AAAI'15 Tutorial on RoboCup