Dialogue Systems Emerging interdisciplinary area since the early - PowerPoint PPT Presentation

Dialogue Systems • Emerging interdisciplinary area since the early 1990s • integration of speech technology, natural language processing, AI, dialogue / communication theory, human factors, … Advanced Dialogue Modeling for • scientific / academic – based research Practical Applications: • commercially driven R&D Introduction • achievements and challenges Ivana Kruijff-Korbayova Ivana Kruijff-Korbayová: Advanced Dialogue Modeling for Practical Applications Ivana Kruijff-Korbayová: Advanced Dialogue Modeling for Practical Applications Dialogues System Research Dialogue System Industry Ivana Kruijff-Korbayová: Advanced Dialogue Modeling for Practical Applications Ivana Kruijff-Korbayová: Advanced Dialogue Modeling for Practical Applications

Typical Pipeline Architecture Typical Pipeline Architecture (Multimodal) Fission & generation Dialogue Audio & management Video / GUI Back Modality-Specific Interpretation end recognition & Fusion Ivana Kruijff-Korbayová: Advanced Dialogue Modeling for Practical Applications Ivana Kruijff-Korbayová: Advanced Dialogue Modeling for Practical Applications Dialogue Control Finite State Systems • Finite state systems • Sequence of predefined steps (dialogue script) – Sequence of predefined steps (dialogue script) • Set of states and transitions Task complexity – State determines system utterance • Frame-based systems (form-filling) • Carefully designed prompts at each state – User utterance determines transition to next state (deterministic) – Task represented as a set of slots to fill (frame, template) • Speech recognition tailored to expected responses given the state • No recursion! (=no nested subdialogues) • Agent-based systems • System-driven interaction (system initiative) – Joint problem solving by collaborating agents • Rigid dialogue flow Ivana Kruijff-Korbayová: Advanced Dialogue Modeling for Practical Applications Ivana Kruijff-Korbayová: Advanced Dialogue Modeling for Practical Applications

FSM: Example 1 FSM: Example 1 (extended) U: Elevator? U: Elevator? Person>Floor # S: Hello. Which floor would you like to go to? S: Hello. Where would you like to go to? Floor_n U: Third floor. U: Prof. Barry. S: OK, I am taking you to the third floor. S: Prof. Barry is on the fourth floor. … I am taking you to the fourth floor. person floor n init init Floor_1 Go_floor Welcome Ask_floor welcome Ask_floor floor 1 floor # � States: … (variable for floor number) unknown unknown Init-State: … Alphabet: … Transition function: … Not_und Not_und Ivana Kruijff-Korbayová: Advanced Dialogue Modeling for Practical Applications Ivana Kruijff-Korbayová: Advanced Dialogue Modeling for Practical Applications FSM: Example 2 FSM: Example 3 bad # bad # unknown unknown States: … States: … Init-State: … Init-State: … Alphabet: … Alphabet: … Transition function: … Transition function: … repeat acct# repeat pin repeat depart repeat dest unknown unknown correct pin city good # city bad # unknown bad # unknown what what init init …. get_pin# Lookup dest_city Lookup get_acct# depart_city service info good # correct pin city city balance flight [McTear 2002] Ivana Kruijff-Korbayová: Advanced Dialogue Modeling for Practical Applications Ivana Kruijff-Korbayová: Advanced Dialogue Modeling for Practical Applications

Finite State Systems: Sum Up Frame-Based Models • Advantages • Task represented as a set of slots to fill – Fixed prompts can be pre-recorded – Frame (form): what info should be supplied by user – Speech recognition and input interpretation can be tuned for each state – Dialogue states: which slots are filled • Disadvantages • Strategies for selecting the next system action (given dialogue state) – Very rigid dialogue flow • Enables mixed initiative (“over-answering”) – Inhibiting user initiative • Somewhat more flexibility – In principle can make more flexible, but it quickly gets very complex – Only suitable for simple fixed tasks departure_city ? departure_date ? destination_city ? return_date ? … Ivana Kruijff-Korbayová: Advanced Dialogue Modeling for Practical Applications Ivana Kruijff-Korbayová: Advanced Dialogue Modeling for Practical Applications Frame-Based Models Frame-Based Models • Deciding what to do next S: What can I do for you? departure_city ? – Next unfilled slot (similar to FS-based) U: I want to fly to Paris departure_date ? – Slot-combination weighting destination_city Paris – Ontology-based coherence return_date ? • Database lookup ... – Delayed (typically; after certain slots filled) – Immediate (can be “expensive”, but helpful ) S: Where will you fly from? departure_city Berlin U: From Berlin on August 1 st . departure_date 1/8/05 destination_city Paris return_date ? ... Ivana Kruijff-Korbayová: Advanced Dialogue Modeling for Practical Applications Ivana Kruijff-Korbayová: Advanced Dialogue Modeling for Practical Applications

Ontology-based coherence Delayed vs. Immediate Lookup S: What is the patient’s sex? S: What can I do for you? U: Female with severe nipple discharge U: I want to fly from Saarbruecken to Luxembourg. S: What is the patient’s age? U: Fifty five S: When would you like to fly? S: Is the discharge bilateral? U: April 1st. U: No S: At what time would like to fly? S: What is the patient’s sex? U: Female with severe nipple discharge U: In the morning. S: Is the discharge bilateral? S: Sorry. There is no flight from Saarbruecken to Luxembourg on April U: No 1st in the morning. S: What is the patient’s age? U: Fifty five [Milward&Beveridge 2003] S: What can I do for you? U: I want to fly from Saarbruecken to Luxembourg. S: Sorry. There is no flight from Saarbruecken to Luxembourg. Ivana Kruijff-Korbayová: Advanced Dialogue Modeling for Practical Applications Ivana Kruijff-Korbayová: Advanced Dialogue Modeling for Practical Applications VoiceXML Frame-Based DM: Sum Up • VoiceXML is a web-based markup language for representing spoken dialogs • Advantages – Form: basic unit of functionality – Enables some user initiative – Field: prompts for and accepts user input – More flexible – Prompt: sequence of audio elements or TTS messages • Disadvantages – Audio: audio file or TTS message to play – Speech recognition more difficult, because user input less restricted – Filled: processes input, can pass control to other forms – Not every task can be represented by a frame • Form Interpretation Algorithm – Defines how fields in a form are filled in , and how the fill ordering can be modified • Global event handlers for (e.g., error handling, help) – Define behavior when predefined global conditions occur • VoiceXML application collects and processes info, and plays back info • VoiceXML assumes a voice browser – Info conveyed to user by audio output (synthesized and/or recorded) – Info received from user as audio input (voice and/or telephone keypad tones) – (analogical to HTML) Ivana Kruijff-Korbayová: Advanced Dialogue Modeling for Practical Applications Ivana Kruijff-Korbayová: Advanced Dialogue Modeling for Practical Applications

Agent-Based DM Collaboration • (Collaborative) problem solving by (rational) agents • Communication is a joint activity: agents collaborate to establish and achieve their goals • Dialogue involves recognition, construction and execution of plans • Neither agent can accomplish the task alone • Agents have knowledge about solving tasks --> mixed initiative – deciding on goals (objectives): adopt, select, defer, abandon, release • Need joint goals and mutual understanding – forming plans to achieve goals (recipes) – executing those plans (acting) • --> cooperation – revising decisions (re-planning, abandoning goals, etc.) • --> grounding • Agents communicate to establish common ground • Agents reason about beliefs and actions Ivana Kruijff-Korbayová: Advanced Dialogue Modeling for Practical Applications Ivana Kruijff-Korbayová: Advanced Dialogue Modeling for Practical Applications Intention Recognition Interleaved Planning&Acting Given: plan for getting a BA User: Send ambulance one to Parma right away. System: OK. [sends ambulance] U: I’ll take German 101 fall semester. System: Where should we take the victim once we pick them up? AcademicPlan GetBA(PolySci) User: Rochester General Hospital. System: OK. Math/LangReq GetGermanCredits German101(fall03) [Blaylock et al. 2003] Ivana Kruijff-Korbayová: Advanced Dialogue Modeling for Practical Applications Ivana Kruijff-Korbayová: Advanced Dialogue Modeling for Practical Applications