Learning to Order Objects using Haptic and Proprioceptive - - PowerPoint PPT Presentation

▶
learning to order objects using haptic and proprioceptive
SMART_READER_LITE
LIVE PREVIEW

Learning to Order Objects using Haptic and Proprioceptive - - PowerPoint PPT Presentation

Dec 23, 2022 151 likes •642 views

Learning to Order Objects using Haptic and Proprioceptive Exploratory Behaviors Jivko Sinapov , Priyanka Khante, Maxwell Svetlik, and Peter Stone Department of Computer Science University of Texas at Austin, Austin TX 78712, USA

slide-1
SLIDE 1

Learning to Order Objects using Haptic and Proprioceptive Exploratory Behaviors

Jivko Sinapov, Priyanka Khante, Maxwell Svetlik, and Peter Stone Department of Computer Science University of Texas at Austin, Austin TX 78712, USA {jsinapov,pkhante,maxwell, pstone}@cs.utexas.edu

slide-2
SLIDE 2

2

Building-Wide Intelligence Project: http://www.cs.utexas.edu/~larg/bwi_web/

slide-3
SLIDE 3

3

Building-Wide Intelligence Project: http://www.cs.utexas.edu/~larg/bwi_web/

slide-4
SLIDE 4

4

Building-Wide Intelligence Project: http://www.cs.utexas.edu/~larg/bwi_web/

slide-5
SLIDE 5

5

Motivation: Grounded Language Learning

Robot, fetch me the green empty bottle

slide-6
SLIDE 6

6

Object Category Recognition in Robotics

Sridharan et al. 2008 Lai et al. 2011 Rusu et al. 2009 Collet et al. 2009

slide-7
SLIDE 7

7

Object Category Learning in Robotics

Thomason, J., Sinapov, J., Svetlik, M., Stone, P., and Mooney, R. (2016). Learning Multi-Modal Grounded Linguistic Semantics by Playing I, Spy Robotics and Vision 3 Session

slide-8
SLIDE 8

8

Now, when and where does this fail...

Consider the word, “weight” - how should it be grounded?

slide-9
SLIDE 9

9

How do humans ground such words?

Sample Montessori toys designed to teach children about the ordinal properties of object weight, height, and size

slide-10
SLIDE 10

10

Object Ordering in Psychology

slide-11
SLIDE 11

11

Object Orderings in Human Environments

slide-12
SLIDE 12

12

Problem Formulation

  • Order Recognition: what property is a given

series of objects ordered by?

“height” “width”

slide-13
SLIDE 13

13

Problem Formulation (2)

  • Order Insertion: given an object series, insert

a new object into the correct position

series test object

slide-14
SLIDE 14

14

Three-Stage Approach

Stage 1: Object Exploration Stage 2: Unsupervised Order Discovery

. . . .

Stage 3: Semantic Grounding

weight width height . . . .

slide-15
SLIDE 15

16

Stage 1: Object Exploration

32 common household and

  • ffice items

The objects vary along three

  • rdinal properties:

1) Weight 2) Width 3) Height

slide-16
SLIDE 16

17

Exploratory Behaviors

slide-17
SLIDE 17

18

Video

slide-18
SLIDE 18

19

Video

slide-19
SLIDE 19

20

Video

slide-20
SLIDE 20

21

Haptic and Proprioceptive Feature Extraction

Time Joint Positions (Prorioception) Joint Efforts (Haptics) . . . . . .

slide-21
SLIDE 21

22

Haptic and Proprioceptive Feature Extraction

Time Joint Positions (Prorioception) Joint Efforts (Haptics) . . . . . .

slide-22
SLIDE 22

23

Haptic and Proprioceptive Feature Extraction

Time Joint Positions (Prorioception) Joint Efforts (Haptics) . . . . . .

slide-23
SLIDE 23

24

Stage 2: Unsupervised Order Discovery

grasp lift hold lower drop

proprioception

Behaviors Sensory Modalities push press

haptics

slide-24
SLIDE 24

26

Unsupervised Order Discovery Example with Synthetic Data

Object order with highest likelihood using the method of [Kemp and Tennenbam, 2008] Input Relational Count Matrix

slide-25
SLIDE 25

27

Example Relational Count Matrix with the Press action and Haptic features

Similarity between objects i and j in the press-haptic sensorimotor context

slide-26
SLIDE 26

28

Resulting Order

(Press behavior and Haptic modality)

The number corresponds to the object's height in millimeters

slide-27
SLIDE 27

29

Stage 2: Unsupervised Order Discovery

grasp lift hold lower drop

proprioception

Behaviors

Sensory Modalities

push press

haptics

slide-28
SLIDE 28

30

Stage 3: Order Grounding Stage

slide-29
SLIDE 29

31

Order Grounding Example: “height”

Positive Examples: Negative Examples:

. . . . . .

slide-30
SLIDE 30

32

Object Order Representation

Training Example:

Object Orders Discovered During Stage 2

. . . .

slide-31
SLIDE 31

33

Object Order Representation

Training Example: . . . .

Object Orders Discovered During Stage 2

slide-32
SLIDE 32

34

Object Order Representation

Training Example: . . . .

Object Orders Discovered During Stage 2

slide-33
SLIDE 33

35

Object Order Representation

Training Example: x1 . . . .

Object Orders Discovered During Stage 2

slide-34
SLIDE 34

36

Object Order Representation

Training Example: x1 . . . .

Object Orders Discovered During Stage 2

slide-35
SLIDE 35

37

Object Order Representation

. . . . Training Example: x1 x2

Object Orders Discovered During Stage 2

slide-36
SLIDE 36

38

Object Order Representation

. . . . Training Example: x1 x2 . . . . xn

Object Orders Discovered During Stage 2

slide-37
SLIDE 37

39

Results: Order Recognition

slide-38
SLIDE 38

40

Sample Learned Decision Trees

Hold Haptics Lower Haptics Lift Haptics Press Proprioception Press Haptics Grasp Proprioception

weight

width height

slide-39
SLIDE 39

41

When does the robot make mistakes?

slide-40
SLIDE 40

42

When does the robot make mistakes?

difficult easy

slide-41
SLIDE 41

43

When does the robot make mistakes?

difficult easy

slide-42
SLIDE 42

44

Object Order Insertion Results

slide-43
SLIDE 43

45

Object Order Insertion Results

slide-44
SLIDE 44

46

Object Order Insertion Results

slide-45
SLIDE 45

47

Object Order Insertion Results

slide-46
SLIDE 46

48

Conclusion

  • A behavior-grounded framework for learning object
  • rdering concepts
  • The robot grounded three ordering concepts, “weight”,

“height”, and “width”

  • Future Work:

– Active action selection – Learn object ordering concepts in conjunction with object

categories, pairwise object relations, etc.

– Learn from humans (for a preview, see our next talk at Robotics

and Vision III)

slide-47
SLIDE 47

49

Thank you!

Jivko Sinapov Maxwell Svetlik Peter Stone

http://www.cs.utexas.edu/~larg/bwi_web/

Priyanka Khante