visual attention in spoken hri
play

Visual Attention in Spoken HRI Maria Staudte & Matthew Crocker - PowerPoint PPT Presentation

Aug 18, 2023 •142 likes •465 views

Visual Attention in Spoken HRI Maria Staudte & Matthew Crocker Saarland University, Germany 1 Gaze in HRI Robot gaze appears active,enjoyable,friendly (e.g. Kanda et al. 01, Kuno et al. 07) Robot gaze is different

  1. Visual Attention in Spoken HRI Maria Staudte & Matthew Crocker Saarland University, Germany 1

  2. Gaze in HRI Robot gaze appears “active”,“enjoyable”,“friendly” (e.g. Kanda et al. ’01, Kuno et al. ’07) Robot gaze is different from human behaviour Can human-like robot gaze be more than “active” or “friendly”? 2

  3. Gaze in HRI Can robot gaze fulfill similar functions as human gaze? coordinating dialogue (e.g. Cassell et al. ’99, Sidner et al. ‘05) information structure (e.g. Mutlu et al. ’06) signaling and checking understanding / joint attention visual reference 3

  4. People do... ...look at an object they are about to mention (about 1 sec before mentioning) ...look at an object that someone else mentions (about 200-500 msec after mentioning) ...look at where the speaker looks 4

  5. Should robots ... ...look at an object they are about to mention? (about 1 sec before mentioning) ...look at an object that someone else mentions? (about 200-500 msec after mentioning) ...look at where the speaker looks? 5

  6. Should robots ... ...look at an object they are about to mention? (about 1 sec before mentioning) 6

  7. Hypothesis ...look at an object they are about to mention? (about 1 sec before mentioning) Cognitively motivated robot gaze behaviour is beneficial for HRI People follow robot gaze This affects the comprehension of the robot utterance 7

  8. Design - Videos showing robot - Task: Right or wrong robot statement? - Measures: • Eye-tracking • Response time (button presses) 8

  9. Example: true - congruent * Thanks to the CoSy/CogX group at DFKI for robi. 9

  10. Conditions Spoken Sentence: Statement Gaze Gaze towards: The cone is taller than the heart that’s silver. TRUE congruent <cone> <silver heart> The cone is taller than the heart that’s silver . TRUE incongruent <cone> <green heart> The cone is taller than the heart that’s silver . TRUE no gaze ------------ ----------------- The cone is taller than the heart that’s green. FALSE congruent <cone> <green heart> The cone is taller than the heart that’s green. FALSE incongruent <cone> <silver heart> The cone is taller than the heart that’s green. FALSE no gaze ------------ ----------------- 10

  11. Conditions Spoken Sentence: Statement Gaze Gaze towards: The cone is taller than the heart that’s silver . TRUE congruent <cone> <silver heart> The cone is taller than the heart that’s silver . TRUE incongruent <cone> <green heart> The cone is taller than the heart that’s silver . TRUE no gaze ------------ ----------------- The cone is taller than the heart that’s green. FALSE congruent <cone> <green heart> The cone is taller than the heart that’s green. FALSE incongruent <cone> <silver heart> The cone is taller than the heart that’s green. FALSE no gaze ------------ ----------------- 11

  12. Conditions Spoken Sentence: Statement Gaze Gaze towards: The cone is taller than the heart that’s silver. TRUE congruent <cone> <silver heart> The cone is taller than the heart that’s silver . TRUE incongruent <cone> <green heart> The cone is taller than the heart that’s silver . TRUE no gaze ------------ ----------------- The cone is taller than the heart that’s green . FALSE congruent <cone> <green heart> The cone is taller than the heart that’s green . FALSE incongruent <cone> <silver heart> The cone is taller than the heart that’s green . FALSE no gaze ------------ ----------------- 12

  13. Conditions Spoken Sentence: Statement Gaze Gaze towards: The cone is taller than the heart that’s silver . congruent TRUE <cone> <silver heart> The cone is taller than the heart that’s silver . TRUE incongruent <cone> <green heart> The cone is taller than the heart that’s silver . TRUE no gaze ------------ ----------------- The cone is taller than the heart that’s green . FALSE congruent <cone> <green heart> The cone is taller than the heart that’s green. FALSE incongruent <cone> <silver heart> The cone is taller than the heart that’s green. FALSE no gaze ------------ ----------------- 13

  14. Conditions Spoken Sentence: Statement Gaze Gaze towards: The cone is taller than the heart that’s silver. TRUE congruent <cone> <silver heart> The cone is taller than the heart that’s silver . incongruent TRUE <cone> <green heart> The cone is taller than the heart that’s silver . TRUE no gaze ------------ ----------------- The cone is taller than the heart that’s green. FALSE congruent <cone> <green heart> The cone is taller than the heart that’s green . FALSE incongruent <cone> <silver heart> The cone is taller than the heart that’s green. FALSE no gaze ------------ ----------------- 14

  15. Example: true - incongruent 15

  16. Analysis “The cone is taller than ...” Interest Recorded Areas Fixations 16

  17. Measures Response Time : Button presses for right/wrong statements Fixations : Fixated objects during robot gaze and speech IP1 IP2 IP3 “The cone is taller than the heart that is silver.” 17

  18. Design Details 48 subjects 24 items, 48 fillers 72 trials 30 min. duration Repeated-Measures ANOVA: fixation proportions response times 18

  19. Results Fixations: robot gaze - following robot speech - following Response Time: congruent < no robot gaze < incongruent 19

  20. Results True - congruent True - incongruent 20

  21. Results Fixations: robot gaze - following robot speech - following Response Time: congruent < no robot gaze < incongruent 21

  22. Results 22

  23. Summary Human behaviour in HRI is consistent with that in HHI Utterance-mediated & gaze-mediated gaze Generality of results: Speech-oriented task Mechanic robot head/eyes Errors (incongruent behaviour) 23

  24. Conclusions People rely on cognitively motivated robot gaze as visual reference Visual reference is a cue for grounding and disambiguation People associate the robot’s visual attention with what it intends to say Inconsistent robot gaze behaviour leads to comprehension difficulties (slow-down) A robot should only make “sensible” head & eye movements that are consistent with its utterance! 24

  25. Outlook Investigating perception of “intended” referents (confirming previous results and assumptions) Investigating flexibility of temporal alignment with speech Extending work to other non-verbal dimensions 25

  26. More interesting stuff... 26

  27. Results RT Experiment 27

  28. Correction Experiment Task: Correct robot’s errors (utterances) Purpose: What did people think was the intended object? 28

  29. False - No Gaze “The cone is taller than the heart that’s green.” (Predicted) Correction: The cone is shorter than the heart that’s green. The cone is taller than the heart that’s silver. 29

  30. False - Congruent “The cone is taller than the heart that’s green.” (Predicted) Correction: The cone is shorter than the heart that’s green. The cone is taller than the heart that’s silver. 30

  31. False - No Gaze “The cone is taller than the heart that’s green.” (Predicted) Correction: The cone is shorter than the heart that’s green. The cone is taller than the heart that’s silver. 31

  32. Results Green heart Silver heart Gaze to: <Green heart> ----- <Silver heart> 32

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Visual Attention FEF V4 spatial attention: simultaneous neural recordings in V4

Visual Attention FEF V4 spatial attention: simultaneous neural recordings in V4

10/29/14 Visual Attention FEF V4 spatial attention: simultaneous neural recordings in V4 &amp; Frontal Eye Fields (monkeys) Gregoriou, Gotts, Zhou, Desimone (2010) object attention: MEG &amp; fMRI in FFA (faces),

238 views • 4 slides
Attention, Transformer and BERT Prof. Kuan-Ting Lai 2020/6/16 Attention is All You Need! A.

Attention, Transformer and BERT Prof. Kuan-Ting Lai 2020/6/16 Attention is All You Need! A.

Attention, Transformer and BERT Prof. Kuan-Ting Lai 2020/6/16 Attention is All You Need! A. Waswani et al., NIPS , 2017 Google Brain &amp; University of Toronto 2 Attention Visual attention and textual attention

628 views • 21 slides
Coordinated Interplay of Scene, Utterance, and World Knowledge - Comprehension of spoken

Coordinated Interplay of Scene, Utterance, and World Knowledge - Comprehension of spoken

Coordinated Interplay of Scene, Utterance, and World Knowledge - Comprehension of spoken utterances that relate to a visual scene - Eye movements in scenes during utterance comprehension - utterance can direct attention to an object in the scene

207 views • 10 slides
Eye-tracking for capturing human visual attention Eye-tracking for capturing human visual

Eye-tracking for capturing human visual attention Eye-tracking for capturing human visual

BubbleView : an interface for crowdsourcing image importance maps and tracking visual attention Nam Wook Zoya Michelle Krzysztof Aude Fredo Hanspeter Kim* Bylinskii* Borkin Gajos Oliva Durand Pfister Eye-tracking for capturing human

830 views • 69 slides
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Kelvin Xu, Jimmy

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Kelvin Xu, Jimmy

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, Yoshua Bengio Presented by Kathy Ge Motivation: Attention

567 views • 20 slides
Attention in NLP CS 6956: Deep Learning for NLP Overview What is attention Attention in

Attention in NLP CS 6956: Deep Learning for NLP Overview What is attention Attention in

Attention in NLP CS 6956: Deep Learning for NLP Overview What is attention Attention in encoder-decoder networks Various kinds of attention 2 Overview What is attention? Attention in encoder-decoder networks 3 Visual

971 views • 73 slides
Improving Image and Sentence Matching with Multimodal Attention and Visual Attributes Yan Huang

Improving Image and Sentence Matching with Multimodal Attention and Visual Attributes Yan Huang

National Lab of Institute of Automation Pattern Recognition Chinese Academy of Sciences Improving Image and Sentence Matching with Multimodal Attention and Visual Attributes

402 views • 38 slides
Cognitive Workload and Visual Attention Analyses of the Air Traffic Control Tower Flight Data

Cognitive Workload and Visual Attention Analyses of the Air Traffic Control Tower Flight Data

Cognitive Workload and Visual Attention Analyses of the Air Traffic Control Tower Flight Data Manager (TFDM) Prototype Demonstration* Kiran Lokhande &amp; Hayley J. Davison Reynolds MIT Lincoln Laboratory HFES 56 th Annual Meeting 26 October,

442 views • 25 slides
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Kelvin Xu, Jimmy Lei

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Kelvin Xu, Jimmy Lei

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Kelvin Xu, Jimmy Lei Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard S. Zemel, Yoshua Bengio. ICML 2015 Presented By: Sai Krishna Bollam

410 views • 18 slides
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Kelvin Xu*, Jimmy Ba

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Kelvin Xu*, Jimmy Ba

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Kelvin Xu*, Jimmy Ba , Ryan Kiros , Kyunghyun Cho*, Aaron Courville*, Ruslan Salakhutdinov , Richard Zemel , Yoshua Bengio* eal*/ University of Toronto

824 views • 46 slides
Neurocognitive Dysfunction in Schizophrenia Attention/Vigilance Immediate Memory Secondary

Neurocognitive Dysfunction in Schizophrenia Attention/Vigilance Immediate Memory Secondary

Neurocognitive Dysfunction in Schizophrenia Attention/Vigilance Immediate Memory Secondary Memory Verbal Fluency, Learning Spatial Working Memory Motor Reaction Time Processing Speed Executive Functions Visual-motor Skills Early Visual

246 views • 21 slides
Analyzing the Relationship Between Head Pose and Gaze to Model Driver Visual Attention Sumit Jha

Analyzing the Relationship Between Head Pose and Gaze to Model Driver Visual Attention Sumit Jha

Analyzing the Relationship Between Head Pose and Gaze to Model Driver Visual Attention Sumit Jha and Carlos Busso Multimodal Signal Processing (MSP) Laboratory Department of Electrical Engineering, The University of Texas at Dallas,

453 views • 24 slides
A spatiotemporal model with visual attention for video classification Mo Shan and Nikolay

A spatiotemporal model with visual attention for video classification Mo Shan and Nikolay

A spatiotemporal model with visual attention for video classification Mo Shan and Nikolay Atanasov Department of Electrical and Computer Engineering July 16, 2017 Outline Motivation Proposed model Experiment Conclusion Motivation Video

161 views • 12 slides
Show, Attend, and Tell Neural Image Caption Generation with Visual Attention Kelvin Xu, Jimmy

Show, Attend, and Tell Neural Image Caption Generation with Visual Attention Kelvin Xu, Jimmy

Show, Attend, and Tell Neural Image Caption Generation with Visual Attention Kelvin Xu, Jimmy Lei Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard S. Zemel, Yoshua Bengio University of Montreal and University of

481 views • 16 slides
Spoken Language Structure Hsin-min Wang References: - X. Huang et al., Spoken Language

Spoken Language Structure Hsin-min Wang References: - X. Huang et al., Spoken Language

Spoken Language Structure Hsin-min Wang References: - X. Huang et al., Spoken Language Processing, Chapter 2 Human Speech Communication Spoken language is used to communicate information from a speaker to a listener. Speech production and

730 views • 62 slides
The Attention Economy What is the attention economy? A business model where you (as the

The Attention Economy What is the attention economy? A business model where you (as the

The Attention Economy What is the attention economy? A business model where you (as the company) want to hold the users attention as much as possible. Attention is treat like a scarce resource What are ethical issues that have emerged

170 views • 3 slides
Cone Representations, Languages, and Compilers for Convex Optimization Stephen Boyd joint work

Cone Representations, Languages, and Compilers for Convex Optimization Stephen Boyd joint work

Cone Representations, Languages, and Compilers for Convex Optimization Stephen Boyd joint work with Michael Grant and Jacob Mattingley Electrical Engineering Department, Stanford University Berkeley Optimization Day, 3/6/2010 Outline

790 views • 49 slides
The crossing number of the cone of a graph C. A. Alfaro, A. Arroyo, M. Der n ar, B. Mohar

The crossing number of the cone of a graph C. A. Alfaro, A. Arroyo, M. Der n ar, B. Mohar

The crossing number of the cone of a graph C. A. Alfaro, A. Arroyo, M. Der n ar, B. Mohar GD2016, Athens, Greece, September 20 Alfaro-Arroyo-Dernar-M. cr(cone(G)) Cone of a graph Cone of G : C ( G ) . . . add a universal vertex (apex) C

301 views • 18 slides
Bounded H -calculus for Closed Extensions of Cone Differential Operators J org Seiler

Bounded H -calculus for Closed Extensions of Cone Differential Operators J org Seiler

Bounded H -calculus for Closed Extensions of Cone Differential Operators J org Seiler Universit` a di Torino (joint work with Elmar Schrohe) IWOTA 2017 , Technische Universit at Chemnitz, August 2017 Bounded H -calculus (sketch)

343 views • 18 slides
Xuefeng Mao Department of Mathematics, Shanghai University, xuefengmao@shu.edu.cn Fudan

Xuefeng Mao Department of Mathematics, Shanghai University, xuefengmao@shu.edu.cn Fudan

Cone length for DG modules and global dimension of DG algebras Xuefeng Mao Department of Mathematics, Shanghai University, xuefengmao@shu.edu.cn Fudan University, September 16, 2011 logo Motivation and inspiration Invariants on DG modules

1.67k views • 90 slides
r s tt

r s tt

r s tt r r tt r r

300 views • 18 slides
Mitigation of BGP Route Leaks ietf-idr-route-leak-detection-mitigation-06 (Route leak definition:

Mitigation of BGP Route Leaks ietf-idr-route-leak-detection-mitigation-06 (Route leak definition:

Methods for Prevention, Detection and Mitigation of BGP Route Leaks ietf-idr-route-leak-detection-mitigation-06 (Route leak definition: RFC 7908) K. Sriram, D. Montgomery, B. Dickson, K. Patel, and A. Robachevsky IDR Working Group Meeting,

244 views • 9 slides
Triangulations Of Polytopes and Algebraic Geometry Francisco Santos Universidad de Cantabria,

Triangulations Of Polytopes and Algebraic Geometry Francisco Santos Universidad de Cantabria,

Triangulations Of Polytopes and Algebraic Geometry Francisco Santos Universidad de Cantabria, Santander, SPAIN http://personales.unican.es/santosf Anogia, August 25 2005 Outline of the talk 1. Triangulations of polytopes (a brief overview).

1.21k views • 120 slides
Alex Suciu Representation Theory and Related Topics Seminar Northeastern University April 10,

Alex Suciu Representation Theory and Related Topics Seminar Northeastern University April 10,

A LGEBRAIC MODELS AND THE T ANGENT C ONE THEOREM Alex Suciu Representation Theory and Related Topics Seminar Northeastern University April 10, 2015 A LEX S UCIU A LGEBRAIC MODELS A PRIL 10, 2015 1 / 20 R ESONANCE VARIETIES R ESONANCE VARIETIES

416 views • 20 slides

More recommend