Form2Fit: Learning Shape Priors for Generalizable Assembly from - - PowerPoint PPT Presentation

▶
form2fit learning shape priors for generalizable assembly
SMART_READER_LITE
LIVE PREVIEW

Form2Fit: Learning Shape Priors for Generalizable Assembly from - - PowerPoint PPT Presentation

Aug 21, 2022 31 likes •1.24k views

Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly Kevin Zakka, Andy Zeng, Johnny Lee, Shuran Song Google Stanford Columbia Research University University Form2Fit: Learning Shape Priors for Generalizable Assembly

slide-1
SLIDE 1

Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly

Kevin Zakka, Andy Zeng, Johnny Lee, Shuran Song

Columbia

University

Stanford

University

Google

Research

slide-2
SLIDE 2

Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly

Kevin Zakka, Andy Zeng, Johnny Lee, Shuran Song

Columbia

University

Stanford

University

Google

Research

includes video narration

slide-3
SLIDE 3

Shape Matching

kit assembly

slide-4
SLIDE 4

Shape Matching

kit assembly everyday interactions

slide-5
SLIDE 5

Shape Matching

kit assembly everyday interactions

slide-6
SLIDE 6

Shape Matching

kit assembly everyday interactions

slide-7
SLIDE 7

Shape Matching

kit assembly everyday interactions

slide-8
SLIDE 8

Towards Flexible Assembly

state-of-the-art robo-kitting solution

Choi et. al.

CAD Model

slide-9
SLIDE 9

Towards Flexible Assembly

state-of-the-art robo-kitting solution

Choi et. al.

CAD Model

slide-10
SLIDE 10

Towards Flexible Assembly

state-of-the-art robo-kitting solution

Choi et. al.

  • require prior knowledge and manual engineering

CAD Model

slide-11
SLIDE 11

Towards Flexible Assembly

state-of-the-art robo-kitting solution

Choi et. al.

  • require prior knowledge and manual engineering

CAD Model

  • cannot quickly adapt to new objects and settings
slide-12
SLIDE 12

Towards Flexible Assembly

state-of-the-art robo-kitting solution can we endow them with generalization abilities?

Choi et. al.

  • require prior knowledge and manual engineering

CAD Model

  • cannot quickly adapt to new objects and settings
slide-13
SLIDE 13

Generalizable Assembly

slide-14
SLIDE 14

Generalizable Assembly

through Shape Matching & Self-Supervision

slide-15
SLIDE 15

Generalizable Assembly

through Shape Matching & Self-Supervision

slide-16
SLIDE 16

Generalizable Assembly

through Shape Matching & Self-Supervision

slide-17
SLIDE 17

Generalizable Assembly

never before seen through Shape Matching & Self-Supervision

slide-18
SLIDE 18

Generalizable Assembly

Form2Fit never before seen through Shape Matching & Self-Supervision

slide-19
SLIDE 19

Generalizable Assembly

Form2Fit 94% novel configurations 86% novel objects & kits never before seen through Shape Matching & Self-Supervision

slide-20
SLIDE 20

Generalizable Assembly

Form2Fit 94% novel configurations 86% novel objects & kits never before seen through Shape Matching & Self-Supervision ~12 hours training

slide-21
SLIDE 21

Key Ideas

Kit Assembly Shape Matching

→

slide-22
SLIDE 22

Key Ideas

Kit Assembly Shape Matching

→

  • learns geometric shape descriptors
slide-23
SLIDE 23

Key Ideas

Kit Assembly Shape Matching

→

  • learns geometric shape descriptors
slide-24
SLIDE 24

Key Ideas

Kit Assembly Shape Matching

→

  • learns geometric shape descriptors
  • generalizes to new shapes
slide-25
SLIDE 25

Key Ideas

Assembly from Disassembly

64x

  • learns geometric shape descriptors
  • generalizes to new shapes

Kit Assembly Shape Matching

→

slide-26
SLIDE 26

Key Ideas

Assembly from Disassembly

  • fully self-supervised

64x

  • learns geometric shape descriptors
  • generalizes to new shapes

Kit Assembly Shape Matching

→

slide-27
SLIDE 27

Key Ideas

Assembly from Disassembly

  • fully self-supervised

64x

  • learns geometric shape descriptors
  • generalizes to new shapes
  • trial and error

Kit Assembly Shape Matching

→

slide-28
SLIDE 28

Key Ideas

Assembly from Disassembly

  • fully self-supervised

64x

  • learns geometric shape descriptors
  • generalizes to new shapes
  • trial and error

Kit Assembly Shape Matching

→

slide-29
SLIDE 29

Key Ideas

Assembly from Disassembly

64x

  • fully self-supervised
  • trial and error
  • learns geometric shape descriptors
  • generalizes to new shapes

Kit Assembly Shape Matching

→

slide-30
SLIDE 30

<< rewind

Key Ideas

Assembly from Disassembly

64x

  • fully self-supervised
  • trial and error
  • learns geometric shape descriptors
  • generalizes to new shapes

Kit Assembly Shape Matching

→

slide-31
SLIDE 31

Method

slide-32
SLIDE 32

Overview of Form2Fit

grayscale-depth heightmaps are generated from 3D pointcloud data

slide-33
SLIDE 33

Kit Heightmap Object Heightmap

Overview of Form2Fit

grayscale-depth heightmaps are generated from 3D pointcloud data

slide-34
SLIDE 34

Overview of Form2Fit

suction network ingests object heightmap and outputs suction heatmap

Kit Heightmap Object Heightmap

slide-35
SLIDE 35

Overview of Form2Fit

suction network ingests object heightmap and outputs suction heatmap

Kit Heightmap Object Heightmap Suction Network

slide-36
SLIDE 36

Overview of Form2Fit

suction network ingests object heightmap and outputs suction heatmap

Kit Heightmap Object Heightmap Suction Network

slide-37
SLIDE 37

Overview of Form2Fit

place network ingests kit heightmap and outputs place heatmap

Kit Heightmap Object Heightmap Suction Network

slide-38
SLIDE 38

Overview of Form2Fit

place network ingests kit heightmap and outputs place heatmap

Kit Heightmap Object Heightmap Suction Network Place Network

slide-39
SLIDE 39

Overview of Form2Fit

place network ingests kit heightmap and outputs place heatmap

Kit Heightmap Object Heightmap Suction Network Place Network

slide-40
SLIDE 40

Overview of Form2Fit

Kit Heightmap Object Heightmap Suction Network Place Network

corresponding pick and place candidates

slide-41
SLIDE 41

Overview of Form2Fit

Kit Heightmap Object Heightmap Suction Network Place Network

corresponding pick and place candidates

slide-42
SLIDE 42

Overview of Form2Fit

Kit Heightmap Object Heightmap Suction Network Place Network

corresponding pick and place candidates

slide-43
SLIDE 43

Overview of Form2Fit

Kit Heightmap Object Heightmap Suction Network Place Network

corresponding pick and place candidates

slide-44
SLIDE 44

Overview of Form2Fit

Kit Heightmap Object Heightmap Suction Network Place Network

corresponding pick and place candidates

slide-45
SLIDE 45

Overview of Form2Fit

Kit Heightmap Object Heightmap Suction Network Place Network

corresponding pick and place candidates

slide-46
SLIDE 46

Overview of Form2Fit

matching network ingests heightmaps and outputs descriptor maps

Kit Heightmap Object Heightmap Suction Network Place Network

slide-47
SLIDE 47

Overview of Form2Fit

Matching Network

matching network ingests heightmaps and outputs descriptor maps

Kit Heightmap Object Heightmap Suction Network Place Network pixel-wise descriptors

slide-48
SLIDE 48

Overview of Form2Fit

Matching Network

closer descriptor distances indicate better object-to-placement correspondences

Kit Heightmap Object Heightmap Suction Network Place Network pixel-wise descriptors

slide-49
SLIDE 49

Overview of Form2Fit

Matching Network

closer descriptor distances indicate better object-to-placement correspondences

Kit Heightmap Object Heightmap Suction Network Place Network pixel-wise descriptors

slide-50
SLIDE 50

Overview of Form2Fit

Matching Network

closer descriptor distances indicate better object-to-placement correspondences

Kit Heightmap Object Heightmap Suction Network Place Network pixel-wise descriptors

slide-51
SLIDE 51

Overview of Form2Fit

Matching Network

closer descriptor distances indicate better object-to-placement correspondences

Kit Heightmap Object Heightmap Suction Network Place Network pixel-wise descriptors

slide-52
SLIDE 52

Overview of Form2Fit

Matching Network

closer descriptor distances indicate better object-to-placement correspondences

Kit Heightmap Object Heightmap Suction Network Place Network pixel-wise descriptors

slide-53
SLIDE 53

Overview of Form2Fit

Matching Network

closer descriptor distances indicate better object-to-placement correspondences

Kit Heightmap Object Heightmap Suction Network Place Network pixel-wise descriptors

× 20 × 20

slide-54
SLIDE 54

Overview of Form2Fit

Matching Network

descriptors are rotation-sensitive

Kit Heightmap Object Heightmap Suction Network Place Network pixel-wise descriptors

× 20 × 20

slide-55
SLIDE 55

Overview of Form2Fit

Matching Network

descriptors are rotation-sensitive

Kit Heightmap Object Heightmap Suction Network Place Network pixel-wise descriptors

× 20 × 20

slide-56
SLIDE 56

Overview of Form2Fit

Matching Network

descriptors are rotation-sensitive

Kit Heightmap Object Heightmap Suction Network Place Network pixel-wise descriptors

× 20 × 20

slide-57
SLIDE 57

Overview of Form2Fit

Matching Network

descriptors are rotation-sensitive

Kit Heightmap Object Heightmap Suction Network Place Network pixel-wise descriptors

× 20 × 20

slide-58
SLIDE 58

Overview of Form2Fit

Matching Network

planner integrates information to produce suction/place poses & end-effector rotation

Kit Heightmap Object Heightmap Suction Network Place Network pixel-wise descriptors

× 20 × 20

slide-59
SLIDE 59

Overview of Form2Fit

Matching Network

planner integrates information to produce suction/place poses & end-effector rotation

Kit Heightmap Object Heightmap Suction Network Place Network pixel-wise descriptors

× 20

θ

p q

Planner

× 20

slide-60
SLIDE 60

Overview of Form2Fit

Matching Network

planner integrates information to produce suction/place poses & end-effector rotation

Kit Heightmap Object Heightmap Suction Network Place Network pixel-wise descriptors

× 20

θ

p q

Planner

× 20

slide-61
SLIDE 61

Overview of Form2Fit

Matching Network

planner integrates information to produce suction/place poses & end-effector rotation

Kit Heightmap Object Heightmap Suction Network Place Network pixel-wise descriptors

× 20

θ

p q

Planner

× 20

slide-62
SLIDE 62

Overview of Form2Fit

Matching Network

planner integrates information to produce suction/place poses & end-effector rotation

Kit Heightmap Object Heightmap Suction Network Place Network pixel-wise descriptors

× 20

θ

p q

Planner

× 20

slide-63
SLIDE 63

Overview of Form2Fit

Matching Network

planner integrates information to produce suction/place poses & end-effector rotation

Kit Heightmap Object Heightmap Suction Network Place Network pixel-wise descriptors

× 20

θ

p q

Planner

× 20

p

Pick Position Place Position

q

slide-64
SLIDE 64

Overview of Form2Fit

Matching Network

planner integrates information to produce suction/place poses & end-effector rotation

Kit Heightmap Object Heightmap Suction Network Place Network pixel-wise descriptors

× 20

θ

p q

Planner

× 20

p

Pick Position Place Position

q

slide-65
SLIDE 65

Data Collection

slide-66
SLIDE 66

Data Collection

12x 12x

500 disassembly sequence (~ 8 to 10 hours) for each kit

12x 12x 12x

slide-67
SLIDE 67

Data Collection

12x 12x

500 disassembly sequence (~ 8 to 10 hours) for each kit

12x 12x 12x

slide-68
SLIDE 68

Data Collection from Disassembly

slide-69
SLIDE 69

Data Collection from Disassembly

suction network predicts a suction candidate

slide-70
SLIDE 70

Data Collection from Disassembly

suction network predicts a suction candidate

slide-71
SLIDE 71

Data Collection from Disassembly

suction network predicts a suction candidate

slide-72
SLIDE 72

Data Collection from Disassembly

slide-73
SLIDE 73

place pose randomly generated (q, θ)

Data Collection from Disassembly

slide-74
SLIDE 74

place pose randomly generated (q, θ)

Data Collection from Disassembly

slide-75
SLIDE 75

place pose randomly generated (q, θ)

θ

Data Collection from Disassembly

slide-76
SLIDE 76

kit is secured to table to prevent accidental displacement from bad suction grasps

Data Collection from Disassembly

slide-77
SLIDE 77

kit is secured to table to prevent accidental displacement from bad suction grasps

Data Collection from Disassembly

slide-78
SLIDE 78

place point ground-truth obtained from suction

Data Collection from Disassembly

slide-79
SLIDE 79

place point ground-truth obtained from suction

Data Collection from Disassembly

slide-80
SLIDE 80

place point ground-truth obtained from suction

Data Collection from Disassembly

slide-81
SLIDE 81

suction point ground-truth obtained from place

Data Collection from Disassembly

slide-82
SLIDE 82

suction point ground-truth obtained from place

Data Collection from Disassembly

slide-83
SLIDE 83

suction point ground-truth obtained from place

Data Collection from Disassembly

slide-84
SLIDE 84

Data Collection from Disassembly

slide-85
SLIDE 85

dense correspondence ground-truth obtained from robot motion

Data Collection from Disassembly

slide-86
SLIDE 86

dense correspondence ground-truth obtained from robot motion

Data Collection from Disassembly

slide-87
SLIDE 87

Results

slide-88
SLIDE 88

Varying Initial Conditions

model trained and tested on each kit

12x 12x 12x 12x 12x

slide-89
SLIDE 89

Varying Initial Conditions

model trained and tested on each kit

12x 12x 12x 12x 12x

slide-90
SLIDE 90

Varying Initial Conditions

model trained and tested on each kit

12x 12x 12x 12x 12x

slide-91
SLIDE 91

Varying Initial Conditions

model trained and tested on each kit

12x 12x 12x 12x 12x

slide-92
SLIDE 92

Varying Initial Conditions

model trained and tested on each kit

12x 12x 12x 12x 12x

slide-93
SLIDE 93

Varying Initial Conditions

model trained and tested on each kit

12x 12x 12x 12x 12x

slide-94
SLIDE 94

Generalization to Novel Settings

slide-95
SLIDE 95

model trained on 2 kits: floss and tape

Generalization to Novel Settings

slide-96
SLIDE 96

model trained on 2 kits: floss and tape

Generalization to Novel Settings

Individual

64x 64x

slide-97
SLIDE 97

model trained on 2 kits: floss and tape

Generalization to Novel Settings

Multiple Individual

64x 64x 64x 64x

slide-98
SLIDE 98

model trained on 2 kits: floss and tape

Generalization to Novel Settings

Multiple Mixture Individual

64x 64x 64x 64x 64x 64x

slide-99
SLIDE 99

Generalization to Novel Objects/Kits

slide-100
SLIDE 100

Generalization to Novel Objects/Kits

64x 64x 64x 64x

slide-101
SLIDE 101

Generalization to Novel Objects/Kits

never before seen animals

4x

slide-102
SLIDE 102

Generalization to Novel Objects/Kits

never before seen animals

4x

slide-103
SLIDE 103

What Has Form2Fit Learned?

slide-104
SLIDE 104

Descriptor Visualization

descriptors encode object orientation

slide-105
SLIDE 105

Descriptor Visualization

descriptors encode object orientation

same orientation

slide-106
SLIDE 106

Descriptor Visualization

descriptors encode object orientation

same orientation different rotation

slide-107
SLIDE 107

descriptors encode spatial correspondence

Descriptor Visualization

slide-108
SLIDE 108

descriptors encode spatial correspondence

Descriptor Visualization

same points share similar descriptors

slide-109
SLIDE 109

descriptors encode object identity

Descriptor Visualization

slide-110
SLIDE 110

descriptors encode object identity

Descriptor Visualization

unique descriptor for different objects

slide-111
SLIDE 111

Limitations & Future Work

slide-112
SLIDE 112

Typical Failure Case

180º rotational flips

12x

slide-113
SLIDE 113

Typical Failure Case

180º rotational flips

12x

slide-114
SLIDE 114

Future Directions

slide-115
SLIDE 115

Future Directions

slide-116
SLIDE 116

Future Directions

  • restricted to planar manipulations
slide-117
SLIDE 117

Future Directions

  • restricted to planar manipulations
slide-118
SLIDE 118

Future Directions

  • restricted to planar manipulations
  • can’t handle fully-transparent objects
slide-119
SLIDE 119

Future Directions

  • restricted to planar manipulations
  • can’t handle fully-transparent objects
  • time-reversal currently restricted to

quasi-static environment

slide-120
SLIDE 120

For details, videos and code, visit: https://form2fit.github.io