COMP 150: Developmental Robotics Instructor: Jivko Sinapov - - PowerPoint PPT Presentation

COMP 150: Developmental Robotics

Instructor: Jivko Sinapov www.cs.tufts.edu/~jsinapov

Language Acquisition

Something fun...

Announcements

Project Deadlines

• Project Presentations: Dec 5 and 7
• Final Report + Deliverables: Dec 11
• Deliverables:

– Presentation slides + videos – Final Report (PDF) – Source code (link to github repositories)

Presentation Guidelines

• 10 minutes talk + 5 min for questions
• Practice! Time your presentation when you

practice and use a timer during the actual presentation as well

• My advice: find another group and practice

to each other

• Format: Google Slides (so that we don’t

have to switch computers)

Presentation Schedule – Tue Dec 5

• Raina Galbiati, Doo-yun Her, and Cassie Collins
• Azmina Karukappadath, Sam Weiss, and Yuelin

Liu

• Timi Dayo-Kayode, Michael Edegware, and

Jong Seo Yoon

• Matt Ryan
• Meghan O'Brien, Tooba Ahsen, and Elizabeth

Lanzilla

Presentation Schedule – Thu Dec 7

• Julia Novakoff, Teddy Laurita, and George

Pesmazoglou

• Eric Chen, Matt Shenton, and Avi Block
• Ari Brown and Julie Jiang
• Christopher Hylwa, Sonal Chatter, and Brett

Gurman

• Brad Oosterveld, Tyler Frascav

Final Report Guidelines

• Approximately 8 pages + 1 page for references
• Default Google Doc template or default overleaf

LaTeX template

• A bit about the structure…
• May include appendix if you have a lot of visual

results

Last Readings and Homework

• See course website

Language Acquisition

The Turing Test

The Turing Test

The Turing Test

The First ChatBot (~1966)

ELIZA

• http://psych.fullerton.edu/mbirnbaum/psych101/

Eliza.htm

Discussion: what is missing from programs like ELIZA?

Natural Language Processing

• The study of algorithms and data structures

used to manipulate text and text-like data

• Applications in information retrieval, web

search, dialogue agents, text mining, etc.

• Traditionally, not concerned with connecting

semantic representations to the real world

Example: Computing Parse Trees

Example: Document Classification

https://abbyy.technology/_media/en:features:classification- scheme.png

Example: Word Embeddings

https://image.slidesharecdn.com/introductiontowordembeddings-160405062343/95/a-simple-introduction-to-word-embeddings-5-638.jpg?cb=1494520542

The Symbol Grounding Problem

“How can the semantic interpretation of a formal symbol system be made intrinsic to the system, rather than just parasitic on the meanings in our heads? How can the meanings of the meaningless symbol tokens, manipulated solely on the basis

• f their (arbitrary)shapes, be grounded in

anything but other meaningless symbols?”

• Steven Hamas, 1990

Deb Roy, “Grounding Language in the World: Schema Theory Meets Semiotics” (2005)

Circular Definitions

Grounding

Sensor Projections

Sensor Projections

INPUT IMAGE Color Histogram

Transformer Projection

Transformer Projection

Color Histogram Entropy of Histogram

Categorizer

Entropy of Histogram “Multicolored”

Action Projector

Schemas for Actions

Schemas for Objects

Spatial Relations

Deb Roy’s Definition of Grounding

• “I define grounding as a causal-predictive cycle

by which an agent maintains beliefs about its world.” (p. 8)

• “An agent’s basic grounding cycle cannot

require mediation by another agent.” (p. 9)

• “An autonomous robot simply cannot afford to

have a human in the loop interpreting sensory data on its behalf.” (p. 9)

• “Cyclic interactions between robots and their

environment, when well designed, enable a robot to learn, verify, and use world knowledge to pursue goals. I believe we should extend this design philosophy to the domain of language and intentional communication.” (p. 5)

• “causality alone is not a sufficient basis for

grounding beliefs. Grounding also requires prediction of the future with respect to the agent’s own actions.” (p. 10)

• “The problem with ignoring the predictive part of

the grounding cycle has sometimes been called the ”homunculus problem”.”

Take Home Message

Language should be grounded in terms of the robot’s own perceptual and sensorimotor capabilities

Thomason, J., Sinapov, J., Svetlik, M., Stone, P., and Mooney, R. (2016) Learning Multi-Modal Grounded Linguistic Semantics by Playing I, Spy In proceedings of the 2016 International Joint Conference on Artificial Intelligence (IJCAI)

43

Motivation: Grounded Language Learning

Robot, fetch me the green empty bottle

44

Exploratory Behaviors in our Robot

45

Video

46

Video

47

Video

48

Sensorimotor Feature Extraction

Time Joint Efforts (Haptics) . . . . . .

49

Sensorimotor Contexts

grasp lift hold lower drop

proprio- ception

push press

haptics

look

audio shape color VGG

50

Sensorimotor Contexts

grasp lift hold lower drop

proprio- ception

push press

haptics

look

audio shape color VGG

51

Feature Extraction: Color

Color Histogram (4 x 4 x 4 = 64 bins)

Object Segmentation

52

Feature Extraction: Shape

3D Object Point Cloud Histogram of Shape Features

53

Joint-Torque values for all joints Joint-Torque Features

Feature Extraction: Haptics

54

Feature Extraction: Audio

audio spectrogram Spectro-temporal Features

55

Feature Extraction: VGG

56

Feature Extraction: VGG

57

Data from a single exploratory trial

grasp lift hold lower drop

proprio- ception

push press

haptics

look

audio shape color VGG

x 5 per object

58

Category Recognition Overview

Category Recognition Models

Sensorimotor Feature Extraction Interaction with Object Category Estimates

. . . Empty? Red? Container?

Sinapov, J., Schenck, C., and Stoytchev, A. (2014). Learning Relational Object Categories Using Behavioral Exploration and Multimodal Perception In the Proceedings of the 2014 IEEE International Conference on Robotics and Automation (ICRA)

59

Key Questions

How can the robot learn object-related words from everyday human users? Do human users use non-visual object descriptors when referring to objects?

60

Object Exploration Dataset

32 common household and

• ffice items

Each object was explored a total of 5 times with 7 different behaviors The robot perceived objects using the visual, auditory, and haptic sensory modalities

Thomason, J., Sinapov, J., Svetlik, M., Stone, P., and Mooney, R. (2016). Learning Multi-Modal Grounded Linguistic Semantics by Playing I, Spy In proceedings of the 2016 International Joint Conference on Artificial Intelligence (IJCAI)

61

Our attempt: I-Spy game

62

Learning Words via Game-play

Human: “an empty metallic aluminum container”

63

Semantic Parsing

64

Example Words for an Object

65

Learning Words via Game-play

66

Learning Words via Game-play

Human: “a tall blue cylindrical container”

67

Learning Words via Game-play

Robot: “open half-full container”

68

Asking Verification Questions

69

Results

70

“can” “tall” “half-full” “pink” WORD F-measure improvement as a result of adding non- visual modalities 0.857 0.516 0.463

. . . . . . . .

71

Summary of Experiment

• The robot learned over 80 words through interactive

game play

• The robot's word representations were grounded in

multiple behaviors and sensory modalities

• Future Work:

– Active action selection when classifying a new object – Active action selection when learning a new words – Actively seek humans out for help with learning about

• bjects

72

“Opportunistic” Active Learning

Thomason, J., Padmakumar, A., Sinapov, J., Hart, J., Stone, P., and Mooney, R. (2017) Opportunistic Active Learning for Grounding Natural Language Descriptions In proceedings of the 1st Annual Conference on Robot Learning (CoRL 2017)

73

“Opportunistic” Active Learning

Thomason, J., Padmakumar, A., Sinapov, J., Hart, J., Stone, P., and Mooney, R. (2017) Opportunistic Active Learning for Grounding Natural Language Descriptions In proceedings of the 1st Annual Conference on Robot Learning (CoRL 2017)

74

What actions should the robot perform when learning a new word?

• Baseline: perform all actions on a set of labeled
• bjects and estimate which ones work well
• But can we do better?

75

Behavior Scores for Words

76

Word Embeddings

Thomason, J., Sinapov, J., Stone, P., and Mooney, R. (2018) Guiding Exploratory Behaviors for Multi-Modal Grounding of Linguistic Descriptions To appear in proceedings of the 32nd Conference of the Association for the Advancement of Artificial Intelligence (AAAI)

77

Word Embeddings

Thomason, J., Sinapov, J., Stone, P., and Mooney, R. (2018) Guiding Exploratory Behaviors for Multi-Modal Grounding of Linguistic Descriptions To appear in proceedings of the 32nd Conference of the Association for the Advancement of Artificial Intelligence (AAAI)

78

Word Embeddings

Thomason, J., Sinapov, J., Stone, P., and Mooney, R. (2018) Guiding Exploratory Behaviors for Multi-Modal Grounding of Linguistic Descriptions To appear in proceedings of the 32nd Conference of the Association for the Advancement of Artificial Intelligence (AAAI)

79

Results

80

Results

Discussion

• What are some of the limitations of these

approaches?

• When will they fail?

Project Breakout