a framework for learning multimodal clarification
play

A Framework for Learning Multimodal Clarification Strategies Verena - PowerPoint PPT Presentation

Sep 19, 2023 •201 likes •933 views

Motivation Framework The Data Collection Performance modelling Future work Summary A Framework for Learning Multimodal Clarification Strategies Verena Rieser 1 Ivana Kruijff-Korbayov 1 Oliver Lemon 2 1 Department of Computational

  1. Motivation Framework The Data Collection Performance modelling Future work Summary A Framework for Learning Multimodal Clarification Strategies Verena Rieser 1 Ivana Kruijff-Korbayová 1 Oliver Lemon 2 1 Department of Computational Linguistics, Saarland University 2 School of Informatics, University of Edinburgh In affiliation with: TALK Project http://www.talk-project.org/

  2. Motivation Framework The Data Collection Performance modelling Future work Summary CRs in Spoken Dialogue Systems System: What city are you leaving from? User: Urbana Champaign. System: Sorry, I’m not sure I understood what you said. Where are you leaving from? User: Urbana Champaign. System: I’m still having trouble understanding you. . . . What city are you leaving from? User: Chicago. [ CMU Communicator – User-System ] → System performs badly and sounds quite artificial.

  3. Motivation Framework The Data Collection Performance modelling Future work Summary CRs in Human-Human Dialogue Cust: I guess getting a car in London will not do me much good in /uh/ Spain is that right? Agent: I’m sorry? Getting a car . . . ? Cust: Yeah I’ll need a car in Madrid. Agent: OK. Cust.: I’ll be returning on Thursday the fifth. Agent: The fifth of February? Cust.: /UHU/ [ CMU Communicator – Human-Human ] → How to convert these kinds of clarification strategies to dialogue systems?

  4. Motivation Framework The Data Collection Performance modelling Future work Summary Outline Motivation Previous work Framework The Learning Approach The Data Collection Experimental Setup Results form the WOZ study Performance modelling RL and Performance modelling Dialogue costs and multimodality Ambiguity and (sub-)task success Future work Policy Shaping User-centred rewards

  5. Motivation Framework The Data Collection Performance modelling Future work Summary Outline Motivation Previous work Framework The Learning Approach The Data Collection Experimental Setup Results form the WOZ study Performance modelling RL and Performance modelling Dialogue costs and multimodality Ambiguity and (sub-)task success Future work Policy Shaping User-centred rewards

  6. Motivation Framework The Data Collection Performance modelling Future work Summary Outline Motivation Previous work Framework The Learning Approach The Data Collection Experimental Setup Results form the WOZ study Performance modelling RL and Performance modelling Dialogue costs and multimodality Ambiguity and (sub-)task success Future work Policy Shaping User-centred rewards

  7. Motivation Framework The Data Collection Performance modelling Future work Summary Outline Motivation Previous work Framework The Learning Approach The Data Collection Experimental Setup Results form the WOZ study Performance modelling RL and Performance modelling Dialogue costs and multimodality Ambiguity and (sub-)task success Future work Policy Shaping User-centred rewards

  8. Motivation Framework The Data Collection Performance modelling Future work Summary Outline Motivation Previous work Framework The Learning Approach The Data Collection Experimental Setup Results form the WOZ study Performance modelling RL and Performance modelling Dialogue costs and multimodality Ambiguity and (sub-)task success Future work Policy Shaping User-centred rewards

  9. Motivation Framework The Data Collection Performance modelling Future work Summary Outline Motivation Previous work Framework The Learning Approach The Data Collection Experimental Setup Results form the WOZ study Performance modelling RL and Performance modelling Dialogue costs and multimodality Ambiguity and (sub-)task success Future work Policy Shaping User-centred rewards

  10. Motivation Framework The Data Collection Performance modelling Future work Summary Generating CRs in task-oriented dialogues [ Rieser and Moore ] Implications for generating clarification requests in task-oriented dialogues , ACL-05. • Form-function mappings • Human decision making on function features was influenced by dialogue type, modality and channel quality .

  11. Motivation Framework The Data Collection Performance modelling Future work Summary Generating CRs in task-oriented dialogues [ Rieser and Moore ] Implications for generating clarification requests in task-oriented dialogues , ACL-05. • Form-function mappings • Human decision making on function features was influenced by dialogue type, modality and channel quality .

  12. Motivation Framework The Data Collection Performance modelling Future work Summary Generating CRs in task-oriented dialogues [ Rieser and Moore ] Implications for generating clarification requests in task-oriented dialogues , ACL-05. • Form-function mappings → We know how to generate surface forms of CRs once we have the functions • Human decision making on function features was influenced by dialogue type, modality and channel quality .

  13. Motivation Framework The Data Collection Performance modelling Future work Summary Generating CRs in task-oriented dialogues [ Rieser and Moore ] Implications for generating clarification requests in task-oriented dialogues , ACL-05. • Form-function mappings → We know how to generate surface forms of CRs once we have the functions • Human decision making on function features was influenced by dialogue type, modality and channel quality .

  14. Motivation Framework The Data Collection Performance modelling Future work Summary Generating CRs in task-oriented dialogues [ Rieser and Moore ] Implications for generating clarification requests in task-oriented dialogues , ACL-05. • Form-function mappings → We know how to generate surface forms of CRs once we have the functions • Human decision making on function features was influenced by dialogue type, modality and channel quality . For dialogue systems we still don’t know:

  15. Motivation Framework The Data Collection Performance modelling Future work Summary Generating CRs in task-oriented dialogues [ Rieser and Moore ] Implications for generating clarification requests in task-oriented dialogues , ACL-05. • Form-function mappings → We know how to generate surface forms of CRs once we have the functions • Human decision making on function features was influenced by dialogue type, modality and channel quality . For dialogue systems we still don’t know: → How to set the function features?

  16. Motivation Framework The Data Collection Performance modelling Future work Summary Generating CRs in task-oriented dialogues [ Rieser and Moore ] Implications for generating clarification requests in task-oriented dialogues , ACL-05. • Form-function mappings → We know how to generate surface forms of CRs once we have the functions • Human decision making on function features was influenced by dialogue type, modality and channel quality . For dialogue systems we still don’t know: → How to set the function features? → How do these strategies perform?

  17. Motivation Framework The Data Collection Performance modelling Future work Summary Outline Motivation Previous work Framework The Learning Approach The Data Collection Experimental Setup Results form the WOZ study Performance modelling RL and Performance modelling Dialogue costs and multimodality Ambiguity and (sub-)task success Future work Policy Shaping User-centred rewards

  18. Motivation Framework The Data Collection Performance modelling Future work Summary Approach Assumptions • Clarification strategies involve complex decision making over a variety of contextual factors • and exhaustive planning towards maximising a desired outcome. → Apply reinforcement learning (RL) in the information state update (ISU) approach. What is RL?

  19. Motivation Framework The Data Collection Performance modelling Future work Summary Framework for learning multimodal CRs Overall approach: MDP = ( S , A , T , R ) 1. Collect data on possible strategies in WOZ experiment. → Extract { A , S , R } 2. Bootstrap an initial policy using supervised learning in the ISU approach. → Learn wizards’ decisions in context ( T ) 3. Optimise the learnt policy for dialogue systems using RL ( π * ≈ maxE [ � j ≥ i r ( d , j ) | s i , a ] ). → How can we improve online reward measures r ( d , j ) ?

  20. Motivation Framework The Data Collection Performance modelling Future work Summary Outline Motivation Previous work Framework The Learning Approach The Data Collection Experimental Setup Results form the WOZ study Performance modelling RL and Performance modelling Dialogue costs and multimodality Ambiguity and (sub-)task success Future work Policy Shaping User-centred rewards

  21. Motivation Framework The Data Collection Performance modelling Future work Summary The SAMMIE-2 1 Data Collection Figure: Multimodal Wizard-of-Oz data collection setup for an in-car music player application, using the Lane Change driving simulator. Top right: User, Top left: Wizard, Bottom: transcribers. 1 SAMMIE stands for Saarbrücken Multimodal MP3 Player Interaction Experiment (cf. for more details [ Kruijff-Korbayová et al. ] , ENLG 2005).

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Multimodal Machine Learning Louis-Philippe (LP) Morency CMU Multimodal Communication and Machine

Multimodal Machine Learning Louis-Philippe (LP) Morency CMU Multimodal Communication and Machine

Multimodal Machine Learning Louis-Philippe (LP) Morency CMU Multimodal Communication and Machine Learning Laboratory [MultiComp Lab] 1 CMU Course 11-777: Multimodal Machine Learning 2 Lecture Objectives What is Multimodal? Multimodal:

2.48k views • 71 slides
Multimodal Machine Learning Louis-Philippe (LP) Morency CMU Multimodal Communication and Machine

Multimodal Machine Learning Louis-Philippe (LP) Morency CMU Multimodal Communication and Machine

Multimodal Machine Learning Louis-Philippe (LP) Morency CMU Multimodal Communication and Machine Learning Laboratory [MultiComp Lab] 1 CMU Course 11-777: Multimodal Machine Learning 2 Lecture Objectives What is Multimodal? Multimodal:

1k views • 72 slides
Multimodal Machine Learning Main Goal Define a common taxonomy for multimodal machine learning

Multimodal Machine Learning Main Goal Define a common taxonomy for multimodal machine learning

Multimodal Machine Learning Main Goal Define a common taxonomy for multimodal machine learning and provide an overview of research in this area Introduction: Preliminary Terms Modality : the way in which something happens or is experienced

575 views • 26 slides
Multimodal Deep Learning Ahmed Abdelkader Design & Innovation Lab, ADAPT Centre Talk outline

Multimodal Deep Learning Ahmed Abdelkader Design & Innovation Lab, ADAPT Centre Talk outline

Multimodal Deep Learning Ahmed Abdelkader Design & Innovation Lab, ADAPT Centre Talk outline What is multimodal learning and what are the challenges? Flickr example: joint learning of images and tags Image captioning: generating

735 views • 48 slides
Retrieving Categorical Emotions using a Probabilistic Framework to Define Preference Learning

Retrieving Categorical Emotions using a Probabilistic Framework to Define Preference Learning

Retrieving Categorical Emotions using a Probabilistic Framework to Define Preference Learning Samples R EZA L OTFIAN AND C ARLOS B USSO Rela0ve Angry label score Ha Happy Multimodal Signal Processing (MSP) lab An Angry The University of

443 views • 20 slides
Using Multimodal Learning Analytics to Identify Aspects of Collaboration in Project-based Learning

Using Multimodal Learning Analytics to Identify Aspects of Collaboration in Project-based Learning

Using Multimodal Learning Analytics to Identify Aspects of Collaboration in Project-based Learning Daniel Spikol, Malm University, Sweden Emanuele Ruffaldi, Scuola Superiore Sant'Anna,Italy Mutlu Cukurova, University College London, UK CSCL

259 views • 22 slides
Multimodal Learning for Image Captioning and Visual Question Answering Xiaodong He Deep Learning

Multimodal Learning for Image Captioning and Visual Question Answering Xiaodong He Deep Learning

Multimodal Learning for Image Captioning and Visual Question Answering Xiaodong He Deep Learning T echnology Center Microsoft Research UC Berkeley, April 7 th , 2016 Knowledge Freebase Text Vision Barack Obama is an American politician

882 views • 50 slides
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of

Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of

Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information Sudha Rao 1 and Hal Daum III 1,2 1 University of Maryland College Park 2 Microsoft Research New York CODE + DATA :

1.54k views • 111 slides
Multimodal Framework Proposal Skip Cave Chief Scientist, Intervoice Inc. 1 Workshop Goals

Multimodal Framework Proposal Skip Cave Chief Scientist, Intervoice Inc. 1 Workshop Goals

Multimodal Framework Proposal Skip Cave Chief Scientist, Intervoice Inc. 1 Workshop Goals Identify & prioritize requirements for changes, extensions, and additions to the MMI architecture to better support Speech, GUI, Ink, and

188 views • 15 slides
Concentric Cable Tender (DD6739/TX) Clarification Presentation Purpose of the clarification

Concentric Cable Tender (DD6739/TX) Clarification Presentation Purpose of the clarification

Concentric Cable Tender (DD6739/TX) Clarification Presentation Purpose of the clarification To explain the requirements of this tender. Tender extensions by suppliers must be requested 5 days prior closing date. Eskom Standard

168 views • 13 slides
An online platform for student - other learning dyad interaction, multimodal feedback and

An online platform for student - other learning dyad interaction, multimodal feedback and

An online platform for student - other learning dyad interaction, multimodal feedback and assessment. Lunch and Learn for Education Focused academics, Feb 8 th 2019 Dr Silas Taylor Convenor of Clinical Skills, UNSW Medicine

398 views • 29 slides
Speaker and Emotion Recognition of TV-Series Data Using Multimodal and Multitask Deep Learning

Speaker and Emotion Recognition of TV-Series Data Using Multimodal and Multitask Deep Learning

Speaker and Emotion Recognition of TV-Series Data Using Multimodal and Multitask Deep Learning Sashi Novitasari 1 , Quoc Truong Do 1 , Sakriani Saktj 1,3 , Dessi Lestari 2 , Satoshi Nakamura 1,3 1 Graduate School of Informatjon Science, Nara

395 views • 16 slides
Visualizing Meaning: Modeling Communication through Multimodal Simulations James Pustejovsky

Visualizing Meaning: Modeling Communication through Multimodal Simulations James Pustejovsky

Creating Situated Grounding Multimodal Simulation Situated Communication Learning by Communication Visualizing Meaning: Modeling Communication through Multimodal Simulations James Pustejovsky Brandeis University COLING 2018 Santa Fe, New

1.04k views • 76 slides
Multimodal semi-supervised learning for image classification Matthieu Guillaumin, Jakob Verbeek,

Multimodal semi-supervised learning for image classification Matthieu Guillaumin, Jakob Verbeek,

Multimodal semi-supervised learning for image classification Matthieu Guillaumin, Jakob Verbeek, Cordelia Schmid LEAR team, INRIA Grenoble, France Motivation and goal Images often come with additional textual info. Videos with scripts and

501 views • 21 slides
Unsupervised learning of multimodal image registration using domain adaptation with projected

Unsupervised learning of multimodal image registration using domain adaptation with projected

Heinrich & Hansen: Learning multimodal registration: domain adaptation & projected Earth Movers discrepancies Unsupervised learning of multimodal image registration using domain adaptation with projected Earth Movers discrepancies

679 views • 5 slides
Multimodal Sentiment Analysis with Word-Level Fusion and Reinforcement Learning Minghai Chen*,

Multimodal Sentiment Analysis with Word-Level Fusion and Reinforcement Learning Minghai Chen*,

Multimodal Sentiment Analysis with Word-Level Fusion and Reinforcement Learning Minghai Chen*, Sen Wang*, Paul Pu Liang*, Tadas Baltrusaitis, Amir Zadeh, Louis-Philippe Morency Natural Computer Interaction Int Intel elligent gent Robo Ro

719 views • 27 slides
MPI Forum - Overview presented by Richard L. Graham Chairman Outline Goal Forum

MPI Forum - Overview presented by Richard L. Graham Chairman Outline Goal Forum

MPI Forum - Overview presented by Richard L. Graham Chairman Outline Goal Forum Structure Meeting Schedule Scope Voting Rules 2 Goal To produce new versions of the MPI standard that better serves the needs of the

452 views • 32 slides
Design and Analysis of Algorithms This Class Website and Contact Website www.cs.kent.edu/

Design and Analysis of Algorithms This Class Website and Contact Website www.cs.kent.edu/

Design and Analysis of Algorithms This Class Website and Contact Website www.cs.kent.edu/ aleitert/fall15/ Important information Slides Announcements Email aleitert@cs.kent.edu Feedback First time teaching this

316 views • 20 slides
Active Error Detection and Resolution for Speech-to-Speech (S2S) Translation Rohit Prasad Rohit

Active Error Detection and Resolution for Speech-to-Speech (S2S) Translation Rohit Prasad Rohit

Active Error Detection and Resolution for Speech-to-Speech (S2S) Translation Rohit Prasad Rohit Kumar Sankaranarayanan Ananthakrishnan Wei Chen Sanjika Hewavitharana Matthew Roy Frederick Choi Aaron Challenner Enoch Kan Arvind Neelakantan

447 views • 27 slides
Everything Second Example of . . . Third Example of . . . Is a Matter of Degree: Our

Everything Second Example of . . . Third Example of . . . Is a Matter of Degree: Our

Everything Is a Matter . . . Formulation of the . . . There Should be an . . . First Example of . . . Everything Second Example of . . . Third Example of . . . Is a Matter of Degree: Our Explanation of . . . First Explanation: . . . Second

356 views • 13 slides
Provision of Financial Advisory Services (Panel) For a Period of Three Years on an As and When

Provision of Financial Advisory Services (Panel) For a Period of Three Years on an As and When

Clarification Meeting: Provision of Financial Advisory Services (Panel) For a Period of Three Years on an As and When Required Basis - Corp 5142 Venue Online (Microsoft Teams Meeting) Date: 22 July 2020 Questions & Answers: SCOPE FOR

405 views • 5 slides
HMIS Lead Monitoring May 2020 Ryan Burger, ICF Mary Schwartz, Abt Associates 1 Webinar

HMIS Lead Monitoring May 2020 Ryan Burger, ICF Mary Schwartz, Abt Associates 1 Webinar

HMIS Lead Monitoring May 2020 Ryan Burger, ICF Mary Schwartz, Abt Associates 1 Webinar Instructions Webinar will last about 60 minutes Participants in listen only mode Submit questions in Question and Answer box on right

424 views • 29 slides
Clarifying Roles Jonathan Worthington German Perl Workshop 2007 Clarifying Roles The Perl 6

Clarifying Roles Jonathan Worthington German Perl Workshop 2007 Clarifying Roles The Perl 6

Clarifying Roles Jonathan Worthington German Perl Workshop 2007 Clarifying Roles The Perl 6 Object Model The Perl 6 object model attempts to improve on the Perl 5 one Nicer, more declarative syntax One way to do things, rather than

408 views • 37 slides
Clarifying the Value You Deliver! Presented By: C.

Clarifying the Value You Deliver! Presented By: C.

Clarifying the Value You Deliver! Presented By: C. Richard Weylman, CSP, CPAE Sponsored By: Why should I Buy

521 views • 32 slides

More recommend