Language, Knowledge & Interaction Engineering (TKI: Taal, - - PowerPoint PPT Presentation

1

Language, Knowledge & Interaction Engineering

(TKI: Taal, Kennis & Interactie)

University of Twente Contact: Anton Nijholt, anijholt@cs.utwente.nl

2

TKI profile

University of Twente, Department of Computer Science

Language, Knowledge & Interaction

Engineering (TKI)

TKI has activities in:

Multimodal Interaction Agent Technology & Virtual Reality Speech & Language Processing Multimedia Retrieval Usability, User Modeling & Personalization

3

(MultiMedia) Content Disclosure

R&D portfolio:

(MultiMedia) Information retrieval & extraction ASR applications for Dutch Recommender systems Web navigation User belief modeling

Participation in EU & national projects

21, PopEye, Olive, Echo, Mumis, M4 Senter & ‘breakthrough’ projects: IOP, Waterland, PIDGIN Euregio: Computational Intelligence

4

Multimodal Interaction

R&D portfolio

Agent platforms and agent communication Designing & specifying virtual & multi-user environments Navigation & multimodal interaction Designing embodied conversational agents Verbal and nonverbal interaction with agents Generating embodied conversations Affective computing

Projects (selection)

AVEIRO (Agents in Virtual Environments) Learning multimodal interaction (STW: submitted) Modality choice in output generation (NWO: accepted) FP5, FP6, ICES/KIS (in progress, submitted)

5

Ph.D. Students & Topics

User Modeling/Adaptation

Jamroga, Herder, Biemans‡, van Setten‡

Interaction Modeling

Jovanovic, Keizer, van Schooten

VR/Animation/Visualization

Bui, Kiss, Malchanau

Machine Learning

Otterlo, Snoek

Multimedia Retrieval

Ordelman, Westerveld‡, Froon‡

6

Multimodality

Small pilot projects on integration of modalities in interfaces

Mouse clicks, language, speech & windows Cross-modal references

M4 European project on multi-sensory perception FP6 European project proposals on multimodal interfaces

7

Multimodality Integration

8

Multimodality Integration

9

Multimodality Integration

10

Multimodality Integration

M4:

Multi Modal Meeting Manager

IDIAP, Sheffield, EPFL, CTIT, etc.

11

Syntax Focus

Recording of signals (auditory, visual) How people act and what they do (movements) Who is doing what (tracking, identification) Who is speaking (tracking, identification)

12

Interpretation Focus

Semantics and pragmatics of verbal and nonverbal acts and events in meetings

Conversational acts (content and event related) Facial expressions, gaze, body language (sleep) Gestures (iconic, deictic, beat, ..) Roles of participants Dynamics of meetings Culture, social relationships

Integration of modalities and coordination among modalities

Integration of modalities in a semantic representation Reasoning about modalities

Multimedia presentation Generation

Summarization, retrieval, VR presentation

13

WP3: Start in december 2002

Natasa Jovanovic

Ph.D. Research in the context of M4 Starting from december 2002 Research Topic: Semantic integration of

Information obtained from different media streams

Added (Extra): Ruud Greven

Building a VR meeting room

14

Meetings

Scripts, simulation and evaluation

Natasa Jovanovic (2003 –2006)

15

Aspects of Meetings

Organized group activity: participants Structured, dynamics goal oriented Kinds of meetings (goals): discussion, presentation, evaluation, Communicative proces Roles of participants: competitive, or cooperative

• bjectives.

Social and affective relations between partners Stages, dynamics of meeting as social process

16

Scenes in Meetings

A scene is a set of simultaneous events and actions A meeting scene may help to identify dialogue acts. Dynamics of meeting characterized by time-sequences of scenes. What are the boundaries of a scene ? Positions of partners, what are they doing? Do scene sequences characterize types of meetings, and stages

• f meetings ?

What kinds of events/actions do simultaneously occur in a scene ? A scenario is a description of scenes.

17

evaluation schema

Meeting scenario Meeting script

Meeting

A/V-Observers Event/Action Recognizers

Recognition Evaluator

18

Dialogue act recognition (objectives)

Recognize in a meeting

• Who is performing what kind of dialogue act ?
• What is the topic ?
• To whom is it adressed?
• Who is he responding to?
• What is the affective relation? (emotion)
• What is the rhetorical relation? (coherence)

What type of dialogue act is it ?

content, control, referring to nonverbal actions

19

Dialogue act recognition (A/V data needed)

Audio

Speech recognition: linguistic, prosodic

features,

Other meaningfull sounds: e.g. laughter Speaker/subject identification

Video

Visible events: actions like walking, gesturing,

20

Dialogue act recognition

Principle of the method Use whatever relevant (partial) knowledge you have to make an educated guess about what you want to predict Hence, higher-level knowledge about the structure and dynamics of meetings in general may help to better identify dialogue acts and vice versa.

21

Structure of Meetings

time P1 P2 P3 P4 P5 P6

22

Interaction patterns

Social interaction patterns Talkativeness Interested/Uninterested

23

Meetings

Designing a Virtual Meeting Room

Ruud Greven (2003)

24

Virtual Meeting Room

Virtual meeting room allows:

Browsing of meetings displayed in 3D Displaying gestures and emotions Zooming in/out, Fast Forward Take the (physical) viewpoint from a participant Modifying/adding verbal and nonverbal

characteristics of participants

On-line participation

25

Virtual Meeting Room

Virtual reality meeting room

Multimodal generation from annotated meeting notes and

the semantic/pragmatic representations

2003: One year project

26

Virtual Meeting Room

Generating from text Accident description reports for insurance companies

27

Virtual Meeting Room

28

Turntaking, Gaze Behavior