Language, Knowledge & Interaction Engineering (TKI: Taal, Kennis & Interactie) University of Twente Contact: Anton Nijholt, anijholt@cs.utwente.nl 1
TKI profile University of Twente, Department of Computer Science � Language, Knowledge & Interaction Engineering (TKI) TKI has activities in: � Multimodal Interaction � Agent Technology & Virtual Reality � Speech & Language Processing � Multimedia Retrieval � Usability, User Modeling & Personalization 2
(MultiMedia) Content Disclosure R&D portfolio: � (MultiMedia) Information retrieval & extraction � ASR applications for Dutch � Recommender systems � Web navigation � User belief modeling Participation in EU & national projects � 21, PopEye, Olive, Echo, Mumis, M4 � Senter & ‘breakthrough’ projects: IOP, Waterland, PIDGIN � Euregio: Computational Intelligence 3
Multimodal Interaction R&D portfolio � Agent platforms and agent communication � Designing & specifying virtual & multi-user environments � Navigation & multimodal interaction � Designing embodied conversational agents � Verbal and nonverbal interaction with agents � Generating embodied conversations � Affective computing Projects (selection) � AVEIRO (Agents in Virtual Environments) � Learning multimodal interaction (STW: submitted) � Modality choice in output generation (NWO: accepted) � FP5, FP6, ICES/KIS (in progress, submitted) 4
Ph.D. Students & Topics User Modeling/Adaptation � Jamroga, Herder, Biemans ‡ , van Setten ‡ Interaction Modeling � Jovanovic, Keizer, van Schooten VR/Animation/Visualization � Bui, Kiss, Malchanau Machine Learning � Otterlo, Snoek Multimedia Retrieval � Ordelman, Westerveld ‡ , Froon ‡ 5
Multimodality Small pilot projects on integration of modalities in interfaces � Mouse clicks, language, speech & windows � Cross-modal references M4 European project on multi-sensory perception FP6 European project proposals on multimodal interfaces 6
Multimodality Integration 7
Multimodality Integration 8
Multimodality Integration 9
Multimodality Integration M4: Multi Modal Meeting Manager IDIAP, Sheffield, EPFL, CTIT, etc. 10
Syntax Focus Recording of signals (auditory, visual) How people act and what they do (movements) Who is doing what (tracking, identification) Who is speaking (tracking, identification) 11
Interpretation Focus Semantics and pragmatics of verbal and nonverbal acts and events in meetings � Conversational acts (content and event related) � Facial expressions, gaze, body language (sleep) � Gestures (iconic, deictic, beat, ..) � Roles of participants � Dynamics of meetings � Culture, social relationships Integration of modalities and coordination among modalities � Integration of modalities in a semantic representation � Reasoning about modalities Multimedia presentation Generation � Summarization, retrieval, VR presentation 12
WP3: Start in december 2002 Natasa Jovanovic � Ph.D. Research in the context of M4 � Starting from december 2002 � Research Topic: Semantic integration of Information obtained from different media streams Added (Extra): Ruud Greven � Building a VR meeting room 13
Meetings Scripts, simulation and evaluation Natasa Jovanovic (2003 –2006) 14
Aspects of Meetings Organized group activity: participants Structured, dynamics � goal oriented Kinds of meetings (goals): discussion, presentation, evaluation, Communicative proces Roles of participants: competitive, or cooperative objectives. Social and affective relations between partners Stages, dynamics of meeting as social process 15
Scenes in Meetings A scene is a set of simultaneous events and actions A meeting scene may help to identify dialogue acts . Dynamics of meeting characterized by time-sequences of scenes. What are the boundaries of a scene ? Positions of partners, what are they doing? Do scene sequences characterize types of meetings, and stages of meetings ? What kinds of events/actions do simultaneously occur in a scene ? A scenario is a description of scenes. 16
evaluation schema Meeting scenario Recognition Evaluator Meeting script A/V-Observers Event/Action Recognizers Meeting 17
Dialogue act recognition (objectives) Recognize in a meeting Who is performing what kind of dialogue act ? � What is the topic ? � To whom is it adressed? � Who is he responding to? � What is the affective relation? (emotion) � What is the rhetorical relation? (coherence) � � What type of dialogue act is it ? content, control, referring to nonverbal actions 18
Dialogue act recognition (A/V data needed) Audio � Speech recognition: linguistic, prosodic features, � Other meaningfull sounds: e.g. laughter � Speaker/subject identification Video � Visible events: actions like walking, gesturing, 19
Dialogue act recognition Principle of the method Use whatever relevant (partial) knowledge you have to make an educated guess about what you want to predict Hence, higher-level knowledge about the structure and dynamics of meetings in general may help to better identify dialogue acts and vice versa. 20
Structure of Meetings P1 P2 P3 P4 P5 P6 time 21
Interaction patterns Social interaction patterns Talkativeness Interested/Uninterested 22
Meetings Designing a Virtual Meeting Room Ruud Greven (2003) 23
Virtual Meeting Room Virtual meeting room allows: � Browsing of meetings displayed in 3D � Displaying gestures and emotions � Zooming in/out, Fast Forward � Take the (physical) viewpoint from a participant � Modifying/adding verbal and nonverbal characteristics of participants � On-line participation 24
Virtual Meeting Room Virtual reality meeting room � Multimodal generation from annotated meeting notes and the semantic/pragmatic representations � 2003: One year project 25
Virtual Meeting Room Generating from text Accident description reports for insurance companies 26
Virtual Meeting Room 27
Turntaking, Gaze Behavior 28
Recommend
More recommend
