how to win the Amazon Robotics Challenge T eam ACRV - - PowerPoint PPT Presentation

▶

Aug 28, 2022 199 likes •463 views

Doug Morrison acrv | arc centre of excellence for robotic vision qut | queensland university of technology RoboticVisionAU how to win the Amazon Robotics Challenge T eam ACRV roboticvision.org #cartman Hardware 1.2m 1.2m 1.5m

SLIDE 1

RoboticVisionAU

Doug Morrison

acrv | arc centre of excellence for robotic vision qut | queensland university of technology

T eam ACRV

how to win the Amazon Robotics Challenge

SLIDE 2

roboticvision.org

SLIDE 3

http://roboticvision.org/

#cartman

Hardware

1.2m 1.2m 1.5m

SLIDE 4

http://roboticvision.org/

Multi-modal End-Effector

#cartman

SLIDE 5

http://roboticvision.org/

In Action

#cartman

SLIDE 6

http://roboticvision.org/

Visual Perception HW

#cartman

SLIDE 7

http://roboticvision.org/

#cartman

Unsupervised Approach

Deep Metric Learning

SLIDE 8

http://roboticvision.org/

Supervised Approach

[Lin et al., CVPR ’17]

#cartman

SLIDE 9

http://roboticvision.org/

Data Collection

#cartman

SLIDE 10

http://roboticvision.org/

Data Collection Unseen

#cartman

SLIDE 11

http://roboticvision.org/

Quick Detour

F0.5 vs F1 vs IOU in Robotic Applications

F0.5 = 83% F1 = 66% IOU = 50% Precision = 100% F0.5 = 100% F1 = 100% IOU = 100% Precision = 100% F0.5 = 63% F1 = 72% IOU = 57% Precision = 57% F0.5 = 83% F1 = 66% IOU = 50% Precision = 100% F0.5 = 100% F1 = 100% IOU = 100% Precision = 100% F0.5 = 63% F1 = 72% IOU = 57% Precision = 57%

F0.5 = 83% F1 = 66% IOU = 50% Precision = 100% F0.5 = 100% F1 = 100% IOU = 100% Precision = 100% F0.5 = 63% F1 = 72% IOU = 57% Precision = 57%

SLIDE 12

http://roboticvision.org/

#cartman

Unsupervised vs Supervised

SLIDE 13

http://roboticvision.org/

#cartman

Training Data Needs

SLIDE 14

http://roboticvision.org/

Perception Results

#cartman

SLIDE 15

http://roboticvision.org/

Perception in Clutter

#cartman

SLIDE 16

http://roboticvision.org/

unknown

vs known

#cartman

SLIDE 17

http://roboticvision.org/

Grasping

#cartman

Incomplete point clouds  (e.g. reflective objects) No valid point cloud  (e.g. clear or black objects) Good quality point clouds (e.g. regular, matt objects)

Surface Normals Point Cloud Centroid RGB Centroid

RGB Image Item Segment Grasp Ranking

no valid points for segment

Grasp Output

No high quality grasps Not enough valid points

SLIDE 18

http://roboticvision.org/

#cartman

Grasp Accuracy

SLIDE 19

http://roboticvision.org/

Finals Scoring

#cartman

SLIDE 20

http://roboticvision.org/

Finals Run

#cartman

SLIDE 21

http://roboticvision.org/

System Robustness

#cartman

Active and Interactive Perception  Multiple viewpoints are used to help locate partially occluded items. If no wanted items are visible, the system will move objects within the storage system based on the likelihood that they are obscuring wanted items. Item Reclassification  Items can be reclassified to correct errors, based on consensus from two sensors (primary/ secondary visual classification and weight). Error Detection and Recovery  A number of sensors are used to detect failed grasps and dropped items. T

ward the end of a task, visual classification is used to double-check that the location of items

matches the internal state.

SLIDE 22

Real-time, Active and Reactive Grasping.

[Closing the Loop for Robotic Grasping, Morrison et al, RSS 2018]

… What Now?

SLIDE 23

The ACRV Picking Benchmark

j.leitner@qut.edu.au

Jürgen ‘Juxi’ Leitner

Juxi #ICRA2017

http://Juxi.net/acrv-picking-benchmark/

vercome limita,ons of current robo,c system comparison

reproducible research on end-to-end TASKS 

SLIDE 24

Tidy Up My Room Challenge

http://Juxi.net/challenge/tidy-up-my-room

SLIDE 25

http://roboticvision.org/

#teamACRV

RoboticVisionAU

Adam Tow Steve Martin Rohan Smith Jordan Erskine Anthony Gillespie Riccardo Grinover Alec Gurman Tom Hunn Darryl Lee Nathan Perkins Gerard Rallos Andrew Razjigaev Juxi Leitner, Ian Reid, Peter Corke

http://facebook.com/T eamACRV

Doug Morrison Matt McTaggert Zheyu Zhuang Norton Kelly-Boxall Sean Wade-McCue Thomas Rowntree Trung Pham Vijay Kumar Ming Cai Saroj Weerasekera Chris Lehnert Anton Milan

SLIDE 26

Brisbane, Australia

We are hiring 

here in Brisbane at QUT!

<j.leitner@qut.edu.au> http://Juxi.net

Thanks!

Come talk to me!

Juxi Leitner

<douglas.morrison@hdr.qut.edu.au>

Doug Morrison