Grasping Team MIT-Princeton @ the Amazon Robotics Challenge 1st - - PowerPoint PPT Presentation
Grasping Team MIT-Princeton @ the Amazon Robotics Challenge 1st place in stowing task Andy Zeng Shuran Song Kuan-Ting Yu Elliott Donlon Francois Hogan Maria Bauza Daolin Ma Orion Taylor Melody Liu Eudald Romo Nima Fazeli Ferran Alet
Andy Zeng Shuran Song Kuan-Ting Yu Prem Qu Nair Isabella Morona Daolin Ma Thomas Funkhouser Alberto Rodriguez Elliott Donlon Francois Hogan Maria Bauza Orion Taylor Eudald Romo Nima Fazeli Nikhil Dafle Rachel Holladay Melody Liu Weber Liu Ferran Alet
1st place in stowing task
Ian Taylor Druck Green
Model-based grasping
Pose estimation ✔ Works well with known objects in structured environments ✘ Can’t handle novel objects in unstructured environments (due to pose estimation) Grasp planning
Model-based grasping
Pose estimation
Model-free grasping
✔ Works well with known objects in structured environments ✘ Can’t handle novel objects in unstructured environments (due to pose estimation) Grasp planning
Visual data Grasp planning
Supersizing Self-Supervision
DexNet 1.0 - 3.0
Grasp Pose Detection
Handles clutter and novel objects
Supersizing Self-Supervision
DexNet 1.0 - 3.0
Grasp Pose Detection
Handles clutter on tabletop scenarios and novel objects selected beforehand
Supersizing Self-Supervision
DexNet 1.0 - 3.0
Grasp Pose Detection
Common limitations: low grasp sample density, small neural network sizes
○ Handles dense clutter in tabletop bin/box scenario Rethink dense clutter:
tossed and stacked on top of each other
○ Handles dense clutter in tabletop bin/box scenario ○ Works for novel objects of all kinds (i.e. any household object should be fair game) 90 - 95% grasping accuracy is not enough Objects without depth data...
○ Handles dense clutter in tabletop bin/box scenario ○ Works for novel objects of all kinds (i.e. any household object should be fair game) ○ Fast and efficient Grasp sampling Grasp ranking Dense pixel-wise predictions Standard: Ours:
○ Handles dense clutter in tabletop bin/box scenario ○ Works for novel objects of all kinds (i.e. any household object should be fair game) ○ Fast and efficient ○ 1st place stowing task at Amazon Robotics Challenge ‘17 (i.e. it works) “The Beast from the East” setup competition footage
Input: multi-view RGB-D images
Input: multi-view RGB-D images Output: dense grasp proposals and affordance scores for 4 primitive grasping behaviors:
suction side suction down grasp down flush grasp
Input RGB-D images fully convolutional ResNet-50
Input RGB-D images suction down suction side
❌ ✓
fully convolutional ResNet-50
Input RGB-D images suction down suction side
❌ ✓
fully convolutional ResNet-50
Input RGB-D images suction down suction side
❌ ✓
fully convolutional ResNet-50 What about grasping?
RGB-D heightmaps Input RGB-D images suction down suction side
❌ ✓
fully convolutional ResNet-50
RGB-D heightmaps Input RGB-D images suction down suction side
❌ ✓
grasp down flush grasp
❌ ✓
fully convolutional ResNet-50
RGB-D heightmaps Input RGB-D images suction down suction side
❌ ✓
grasp down flush grasp
❌ ✓
fully convolutional ResNet-50 predicts horizontal grasp affordances
Suctionable areas Parallel-jaw grasps
Advantages:
Advantages:
Advantages:
Grasp sampling Grasp ranking Standard: Ours:
Advantages:
Limitations:
○ Can trivially extend to more grasp angles
affordances (no real planning)
Model-based grasping
Pose estimation
Model-free grasping
Grasp planning Visual data Grasp planning
Model-based grasping
Pose estimation
Model-free grasping
Grasp planning Visual data Grasp planning
How can we improve model-free by making it more like model-based?
Model-based grasping Model-free grasping
Semantic Scene Completion from a Single Depth Image [Song et al., CVPR ‘17]
○ FCNs to compute dense affordance predictions for multiple grasping behaviors (suction, parallel-jaw) ○ Multiple grasping primitive behaviors → dense clutter in bin/box scenario ○ Multi-view color and depth + diverse training data + robust hardware → handle novel objects of all kinds ○ FCNs for grasping affordance predictions → efficiency and high grasp recall
○ FCNs to compute dense affordance predictions for multiple grasping behaviors (suction, parallel-jaw) ○ Multiple grasping primitive behaviors → dense clutter in bin/box scenario ○ Multi-view color and depth + diverse training data + robust hardware → handle novel objects of all kinds ○ FCNs for grasping affordance predictions → efficiency and high grasp recall
Paper and code are available: arc.cs.princeton.edu
✓ ❌ ❌
match? ℓ2 distance ratio loss
product images
softmax loss for K-Net only match? ℓ2 distance ratio loss
product images
novel feature embedding known
novel feature embedding known input
novel match! feature embedding known input
novel match! feature embedding known input Pre-trained ImageNet features