Human-Robot Dialogue and Collaboration in Search and Navigation - - PowerPoint PPT Presentation

▶
human robot dialogue and collaboration
SMART_READER_LITE
LIVE PREVIEW

Human-Robot Dialogue and Collaboration in Search and Navigation - - PowerPoint PPT Presentation

Nov 24, 2022 313 likes •520 views

Human-Robot Dialogue and Collaboration in Search and Navigation Claire Bonial , Stephanie M. Lukin, Ashley Foots, Cassidy Henry, Matthew Marge, Kimberly A. Pollard, Ron Artstein, David Traum & Clare R. Voss. LREC 2018 7 May 2018 C. Bonial |

slide-1
SLIDE 1
  • C. Bonial | US Army Research Laboratory | UNCLASSIFIED

Human-Robot Dialogue and Collaboration in Search and Navigation

Claire Bonial, Stephanie M. Lukin, Ashley Foots, Cassidy Henry, Matthew Marge, Kimberly A. Pollard, Ron Artstein, David Traum & Clare R. Voss. LREC 2018 7 May 2018

1

slide-2
SLIDE 2
  • C. Bonial | US Army Research Laboratory | UNCLASSIFIED

Road Map

2

How do we get from actions expressed in unconstrained natural language to robot execution in the physical world?

  • 1. Intro: Motivation and Research

Overview

  • 2. Corpus Collection: Our series of

experiments

  • 3. Today’s Focus – Corpus Features:

– Annotations – Actions in the data

  • 4. Conclusions and Future Work
slide-3
SLIDE 3
  • C. Bonial | US Army Research Laboratory | UNCLASSIFIED
  • 1. Introduction

Motivation: robots must be able to communicate effectively with humans in shared tasks (e.g. search-and-rescue)

  • Ideally, two-way spoken dialogue

3

I see a hole in a brick wall… What do you see ahead?

slide-4
SLIDE 4
  • C. Bonial | US Army Research Laboratory | UNCLASSIFIED
  • 1. Introduction

How might people talk to a robot in a collaborative search/navigation task?

– Wizard-of-Oz (WoZ) methodology: human “wizard” stands in for automated components – Phased WoZ used in virtual human dialogue systems (DeVault et al. 2014, Artstein et al. 2015) – WoZ objectives: refine and evaluate the domain and provide training data for automated natural language understanding

4 Knepper et al. 2015

slide-5
SLIDE 5

5

Experiment 2 Automate Some “Wizard” Labor Beyond… Full Automation Of “Wizard” Experiment 1 Exploratory Data Collection

  • C. Bonial | US Army Research Laboratory | UNCLASSIFIED
  • 1. Introduction
slide-6
SLIDE 6

6

Experiment 2 Automate Some “Wizard” Labor Beyond… Full Automation Of “Wizard” Experiment 1 Exploratory Data Collection

  • C. Bonial | US Army Research Laboratory | UNCLASSIFIED
  • 1. Introduction
slide-7
SLIDE 7
  • C. Bonial | US Army Research Laboratory | UNCLASSIFIED
  • 2. Corpus Collection

7

Marge et al. IEEE RO-MAN - 2016

Human Commander VIEWS ROBOT (remote from Commander) VERBAL COMMANDS Commander Participant

slide-8
SLIDE 8
  • C. Bonial | US Army Research Laboratory | UNCLASSIFIED
  • 2. Corpus Collection
  • Dialogue Manager (DM-

Wizard) is the “brains”

  • f the robot in natural

language interactions

  • Robot Navigator

(experimenter) navigates robot based on instructions from DM- Wizard

8

Commander Participant VIEWS “Behind the scenes” RN MOVES ROBOT

DM-WIZARD Robot Navigator

VERBAL COMMANDS

slide-9
SLIDE 9
  • C. Bonial | US Army Research Laboratory | UNCLASSIFIED
  • 2. Corpus Collection

9

slide-10
SLIDE 10
  • 3. Corpus Details

10

  • C. Bonial | US Army Research Laboratory | UNCLASSIFIED

Preparing Freely Available Release of Exp 1, 2 data:

  • 20 Participants
  • 20 Hours of experimental interactions (audio transcribed

and aligned with text chat messages)

  • 3,573 Participant utterances totaling 18,336 words (tokens)
  • 13,550 Dialogue Manager Wizard words in text messages
slide-11
SLIDE 11
  • 3. Corpus: Annotations

11

  • C. Bonial | US Army Research Laboratory | UNCLASSIFIED
slide-12
SLIDE 12
  • 3. Corpus: Actions in the Data

12

  • C. Bonial | US Army Research Laboratory | UNCLASSIFIED

Most dialogue moves are commands:

  • Send-image
  • Rotate
  • Drive

Each clear, unambiguous instruction is realized in 3 ways:

  • 1. Spoken natural language instructions (from participant)
  • 2. Simplified text message instructions (DM Translation for

wizard navigator)

  • 3. Actual executed action (i.e. data from robot: turns, changes

in location, images sent) ➢ Current work: developing annotation schema relating these

slide-13
SLIDE 13
  • 3. Corpus: Actions in the Data

13

  • C. Bonial | US Army Research Laboratory | UNCLASSIFIED

Participant

(Audio Stream 1)

DM->Participant

(Chat Room 1)

DM->RN

(Chat Room 2)

RN

(Audio Stream 2)

turn ninety degrees to the left

  • k

turn left 90 degrees turning… done done

slide-14
SLIDE 14
  • 3. Corpus: Actions in the Data

14

  • C. Bonial | US Army Research Laboratory | UNCLASSIFIED

Participant

(Audio Stream 1)

DM->Participant

(Chat Room 1)

DM->RN

(Chat Room 2)

RN

(Audio Stream 2)

take a picture

  • f

what's behind you turn 180, photo executing... image sent

Move back to the wall behind you; Can you go around and take a photo behind the TV?

slide-15
SLIDE 15
  • 3. Corpus: Actions in the Data

15

  • C. Bonial | US Army Research Laboratory | UNCLASSIFIED

Participant

(Audio Stream 1)

DM->Participant

(Chat Room 1)

DM->RN

(Chat Room 2)

RN

(Audio Stream 2)

go into the center

  • f

the room in front

  • f

you and then take a picture at the <pause> east south west and north position move into the center

  • f

the room in front

  • f

you, take photos at east, south, west, north positions executing... done done go into the room behind you and do the same

slide-16
SLIDE 16
  • 4. Conclusions & Open Questions
  • Phased, Wizard-of-Oz approach allows us to develop

technology in a data-driven fashion — observing the range of language people will use when addressing a robot

  • This language can be translated into a tractable set

for an automated system while maintaining good coverage of the language in the domain – Preliminary Classifier: automating the DM- Wizard’s first response to participant instructions (either translating to the RN or clarifying to participant) – Uses string divergence measures

slide-17
SLIDE 17
  • C. Bonial | US Army Research Laboratory | UNCLASSIFIED
  • 4. Conclusions & Open Questions

17

Gervits et al. ACL-Demos 2018

slide-18
SLIDE 18

18

  • 4. Conclusions & Open Questions

Annotation schema under development will make explicit relation between NL instructions and physical execution; questions remain…

  • Can natural language — execution mapping be acquired

through machine learning?

  • Is a more complex spatial/dialogue model needed, and/or a

symbolic representation? We invite this community to utilize this data in considering aspects of action modeling in robots.

slide-19
SLIDE 19
  • C. Bonial | US Army Research Laboratory | UNCLASSIFIED

19

thank you

slide-20
SLIDE 20

Project Members at ARL Claire Bonial Linguistics (Adelphi) Ashley Foots Audiology (APG) Cory Hayes Human-Robot Interaction (Adelphi) Susan Hill Human-Robot Interaction (APG) Stephanie Lukin Computational Linguistics (ARL West) Matthew Marge Computational Linguistics (Adelphi) Kimberly Pollard Neurobiology (ARL West) Clare Voss Computer Sci., Linguistics (Adelphi) Cassidy Henry Linguistics (SMART Scholar) Project Members at USC/Institute for Creative Technologies Ron Artstein Linguistics Anton Leuski Computer Science David Traum Computational Linguistics And a host of interns!

Collaborators

20

More from David Traum: LREC May 9, Session O4: Dialogue