SLIDE 1
Li, Peltason and Wrede, March 2007
1
Li, Peltason and Wrede, March 2007
2
Why and how to model multi-modal interaction for a mobile robot - - PowerPoint PPT Presentation
Why and how to model multi-modal interaction for a mobile robot - - PowerPoint PPT Presentation
Why and how to model multi-modal interaction for a mobile robot companion Shuyin Li, Julia Peltason and Britta Wrede Bielefeld University Germany Li, Peltason and Wrede, March 2007 1 Outline Introduction to Human-Robot Interaction (HRI)
SLIDE 2
SLIDE 3
Li, Peltason and Wrede, March 2007
3
Robots should be aware of environmental changes
Introduction: HRI with a personal robot
Robot characteristics Requirements for the system
situated situation- awareness
SLIDE 4
Li, Peltason and Wrede, March 2007
4
Robots should be aware of environmental changes
Introduction: HRI with a personal robot
Robot characteristics Requirements for the system
Users expect human-like behaviors anthropomorphic social behaviors situated situation- awareness
SLIDE 5
Li, Peltason and Wrede, March 2007
5
Robots should be aware of environmental changes
Introduction: HRI with a personal robot
Robot characteristics Requirements for the system
Both users and robots have visual access to their interaction partener's body embodied multi-modal interaction Users expect human-like behaviors anthropomorphic social behaviors situated situation- awareness
SLIDE 6
Li, Peltason and Wrede, March 2007
6
Robots should be aware of environmental changes
Introduction: HRI with a personal robot
Robot characteristics Requirements for the system
Both users and robots have visual access to their interaction partener's body embodied multi-modal interaction Users expect human-like behaviors anthropomorphic social behaviors situated situation- awareness
SLIDE 7
Li, Peltason and Wrede, March 2007
7
Robots should be aware of environmental changes
Introduction: HRI with a personal robot
Robot characteristics Requirements for the system
Both users and robots have visual access to their interaction partener's body embodied multi-modal interaction Users expect human-like behaviors anthropomorphic social behaviors situated situation- awareness
SLIDE 8
Li, Peltason and Wrede, March 2007
8
Outline
Introduction to Human-Robot Interaction (HRI) Observations in a user study Observations in a user study
BIRON@Home BIRON@Home Quiet speakers Quiet speakers Meta-commentators Meta-commentators
A multi-modal interaction framework Summary
SLIDE 9
Li, Peltason and Wrede, March 2007
9
The user study: BIRON@Home
SLIDE 10
Li, Peltason and Wrede, March 2007
10
The user study: BIRON@Home
Experimental setup with BIRON
Non-task behaviors of BIRON:
- 1. situation awareness
- 2. social behavior:
14 subjects, each interaction about 7 min. Only output-modality of BIRON: speech
SLIDE 11
Li, Peltason and Wrede, March 2007
11
The user study: situation awareness
Face recognition sound source detection Face recognition Human leg detection
SLIDE 12
Li, Peltason and Wrede, March 2007
12
Observation I: quiet speakers
SLIDE 13
Li, Peltason and Wrede, March 2007
13
Observation I: quiet speakers
The problem
No means to communicate pre-interaction attention
Possible reason
inapproprateness of speech modality
(“legs detected, face detected, face lost again, face detected, ...”)
Solution
use non-verbal modalities
(because they are suitable to represent static information which is only
- ccasionally updated)
SLIDE 14
Li, Peltason and Wrede, March 2007
14
The user study: social behavior
... User: Follow me. BIRON: OK, I'm following you. User: This is a cup. BIRON: It's nice. You are really doing very well! User: (laugh) BIRON: Come here.
Performance remark
SLIDE 15
Li, Peltason and Wrede, March 2007
15
Observation II: meta-commentators
SLIDE 16
Li, Peltason and Wrede, March 2007
16
Observation II: meta-commentators
Problem
users reply to social comments using out-of-vocabulary words
Possible reason
reciprocity and obtrusiveness of the speech modality
Solution
making remarks using non-verbal modalities
(because they are unobtrusive and do not impose strong obligation to reply)
SLIDE 17
Li, Peltason and Wrede, March 2007
17
Outline
Introduction to Human-Robot Interaction (HRI) Observations in a user study A multi-modal interaction framework A multi-modal interaction framework
Currently popular approaches Currently popular approaches Our approach Our approach
Summary
SLIDE 18
Li, Peltason and Wrede, March 2007
18
Interaction framework: existing works
Currently popular approaches address differences between multi-modal information by grouping it into categories and handle different categories separately
SLIDE 19
Li, Peltason and Wrede, March 2007
19
Interaction framework: existing works
Cassell: generic architecture for embodied conversational agents
SLIDE 20
Li, Peltason and Wrede, March 2007
20
Interaction framework: existing works
Traum: dialog model for multi-modal, multi-party dialog in virtual world (based on information-state theory)
SLIDE 21
Li, Peltason and Wrede, March 2007
21
Interaction framework: multi-modal grounding
Our approach address a common feature of multi-modal information: evocative functions
SLIDE 22
Li, Peltason and Wrede, March 2007
22
Interaction framework: multi-modal grounding
Evocative functions of conversational behaviors (CBs) Definition: CBs evoke a reaction from the interaction partner Validity: for both propositional and interactional info
SLIDE 23
Li, Peltason and Wrede, March 2007
23
Interaction framework: multi-modal grounding
Grounding Definition: the process
- f
establishing shared understanding during a conversation. Basic idea: for each account (Presentation) issued in a conversation, there needs to be a feedback (Acceptance) from the interaction partner Application area: traditionally adopted to model evocative functions of propositional information
→ to be extended!
SLIDE 24
Li, Peltason and Wrede, March 2007
24
Interaction framework: multi-modal grounding
Our approach: extending grounding Modeling both propositional and interactional contribution with Interaction Unit Organizing IUs based on the principle of grounding
SLIDE 25
Li, Peltason and Wrede, March 2007
25
Interaction framework: multi-modal grounding
Interaction Unit (IU)
Motivation Layer Behavior Layer
SLIDE 26
Li, Peltason and Wrede, March 2007
26
Interaction framework: multi-modal grounding
Interaction Unit (IU)
(motivation conception)
verbal generator non-verbal generator
SLIDE 27
Li, Peltason and Wrede, March 2007
27
Interaction framework: multi-modal grounding
gesture speech speech speech facial expression ...
Grounding
Motivation Motivation Motivation Motivation Motivation gaze ...
SLIDE 28
Li, Peltason and Wrede, March 2007
28
Interaction framework: multi-modal grounding
Grounding models:
[Clark92] [Traum94] [Cahn&Brennan99]
Our approach: [Li2006]
S. Li, B. Wrede, and G. Sagerer. A computational model
- f
multi-modal
- grounding. In Proc. ACL SIGdial workshop
- n discourse and dialog, in conjunction with
COLING/ACL 2006, pages 153-160. ACL Press, 2006.
Discourse
IU 1 Ex1 Ex2 Ex3 IU_2 IU 3 IU 4 R(Ex4,Ex2) = support R(Ex3,Ex2) = default
SLIDE 29
Li, Peltason and Wrede, March 2007
29
Interaction framework: multi-modal grounding
Pre-interaction attention: solving the quiet-speaker-problem
User: (shows legs)
uninstantiated shows legs
unintentional motivation
U1
SLIDE 30
Li, Peltason and Wrede, March 2007
30
Interaction framework: multi-modal grounding
User: (shows legs) BIRON: (opens eyes)
uninstantiated shows legs
unintentional motivation
uninstantiated
- pens eyes
provides acceptance to user IU
U1 U2
Pre-interaction attention: solving the quiet-speaker-problem
SLIDE 31
Li, Peltason and Wrede, March 2007
31
Interaction framework: multi-modal grounding
User: (shows face) BIRON: (raises head)
uninstantiated shows face and legs
unintentional motivation
uninstantiated raises head
provides acceptance to user IU
U3 U4
Pre-interaction attention: solving the quiet-speaker-problem
SLIDE 32
Li, Peltason and Wrede, March 2007
32
Interaction framework: multi-modal grounding
User: This is a cup.
deictic gesture
shows BIRON a cup
U5
Making social comments: solving the meta-commentator-problem
“This is a cup”
SLIDE 33
Li, Peltason and Wrede, March 2007
33
Interaction framework: multi-modal grounding
User: This is a cup. BIRON: I beg your pardon?
“This is a cup” deictic gesture
shows BIRON a cup initiates conversational repair
U5 U6
Making social comments: solving the meta-commentator-problem
“I beg your pardon” uninstantiated
SLIDE 34
Li, Peltason and Wrede, March 2007
34
Interaction framework: multi-modal grounding
User: This is a cup. BIRON: I beg your pardon? BIRON: (looking embarrassed)
“This is a cup” deictic gesture
shows BIRON a cup
uninstantiated
initiates conversational repair
U5 U6 uninstantiated looking embarrassed
shows social awareness
U7
Making social comments: solving the meta-commentator-problem
“I beg your pardon”
SLIDE 35
Li, Peltason and Wrede, March 2007
35
Interaction framework: multi-modal grounding
Implemented on systems BIRON and BARTHOC BIRON BARTHOC
SLIDE 36
Li, Peltason and Wrede, March 2007
36