A plan-based approach for affective sports commentary in real-time - - PowerPoint PPT Presentation

Embodied Presentation Teams: A plan-based approach for affective sports commentary in real-time

Author: Ivan Gregor Supervisor: Michael Kipp

Virtual Agents

• Multimodal user interfaces
• Entertaining, Enjoyable
• Example applications

– computer games – tutoring systems – commentary agents

2

GALA Challenge

• Commentary on a continuous sports event

– Complex behaviour – Affective – Real-time

• Horse race
• Tennis game

3

GALA is a part of the IVA (Intelligent Virtual Agents) conference

GALA 2009 - Input

• Video & ANVIL file
• Timestamped events

4

Player events Ball events throw shot serve cross net forehand hit_ net backhand hit _tape forehand-volley bounce Backhand volley fault smash

• ut

miss

Position side Position long. Position lateral Position height server net left low receiver mid court middle middle baseline right high

Related Work - ERIC

• Won GALA 2007
• Horse race reporter
• Rule-based
• Template-based NLG
• Domain independent

– horse race – tank battle game

5

Related Work - Spectators

• GALA 2009
• Small set of rules
• No commentary

6

Related Work - STEVE

• Tutoring system
• HTN planning
• Virtual environment
• User questions
• No emotions

7

Presentation Teams (André, Rist)

• Generate presentations on the fly
• Choice of a presentation team
• 2 ≤ distinct virtual agents
• Roles (expertise)
• Interest
• Personality profiles
• Dialogues
• Opposing roles (entertaining, understanding)

8

Our system – Aims (1)

• Behaviourally complex, affective commentary
• n a tennis game (GALA 2009)
• Real-time
• Presentation Team

– Different roles – Attitudes to the players – Personality profiles

9

Our system – Aims (2)

• Dialogue planning (HTN planning)
• Interruptions
• Interaction (user pre-defined questions)
• Background knowledge

10

Video – Features of the system

11

Dialogue Planning

12

HTN Planner (JSHOP) INPUT:

• Facts describing the current state of the tennis game
• Commentators` attitudes to the players
• Background facts

OUTPUT: possible plans (dialogues)

Dialogue Schemes (1)

Dialogue Scheme Example of a Generated Dialogue A: argument for/against X B: contrary A: “That serve was really phenomenal!“ B: “Well, that is a little exaggerated! “ A: argument for/against X B: contrary A: override A: “Blake is in great shape as usual.“ B: “But he already produced several unforced errors.“ A: “Still, he is the best player on the court.“ A: argue for X B: elaborate on X A: “Excellent return by Safin.“ B: “Unreachable for Blake.“ A: background fact X B: evidence of X A: “The brother of Blake Thomas is a well known player.“ B: “His best ranking was the 141st place in 2002." A: background fact X B: consequence of X A: “Roddick has been 4 times injured recently.“ B: “It will be hard to break through today.“

13

Dialogue schemes were introduced by Elisabeth André and Thomas Rist

Dialogue Schemes (2)

14

Planning Large Dialogue Contributions

15

Planning Tree (1)

16

• Hierarchy of dialogues
• Root = goal task
• Internal node = compound task
• Leaf = primitive task or reference to an

internal node

Planning Tree (2)

17

Affect

• Lexical selection

– Choice of a dialogue scheme

• Gestures

– Utterances with the gesture annotation

• Facial expression

– Emotion Module

18

Affect – Planning with Attitude

• Lexical selection (Choice of a dialogue scheme)
• Gestures (Gesture tags in utterances)

19

Appraisal Dialogue Scheme Example of a Generated Dialogue A: positive B: positive A: argue for X B: support X A: “Outstanding ace by Blake!” B: “Blake hits blistering serve down the line!” A: positive B: negative A: argue for X B: play down X A: ”Excellent forehand by Safin!” B: “That`s a bit overstated.” A: negative B: positive A: point out fault X B: excuse X A: ”Safin failed to get the ball over the net.” B: “Safin just overhits the serve.” A: neutral B: negative A: convey fact X B: consequence of X A: “The score is already 30:0.” B: “Safin and Ferrer are real losers as usual!” A: neutral B: neutral A: convey fact X B: elaborate on fact X A: “Deuce again.” B: “Safin and Ferrer got back on board.”

Affect – OCC Generated Emotions (1)

• Facial Expression

– Emotion module (Jess) – Simulate 8 basic OCC emotions

20

OCC Emotion Description JOY Something happened that I wanted to happen. DISTRESS Something happened that I did not want to happen. HOPE Something may happen that I really want to occur. FEAR Something may happen that I wish to never occur. RELIEF Something bad did not happen. DISAPPOINTMENT Something did not happen that I really wanted to occur. SATISFACTION Something happened that I really wanted to occur. FEAR-CONFIRMED Something bad did actually happen. OCC (Ortony, Collins, Clore) Cognitive model of emotions

Affect – OCC Generated Emotions (2)

• Initialization: personality
• Input: Facts + Attitudes
• Functionality

– EEC – Initial intensity – Emotion decay

• Output: Vector of emotion intensities

21

Personality Trait Optimistic Choleric Extravert Neurotic Social

EEC definitions (Jess) Emotion + Initial Intensity Facts from the tennis game & Attitudes

Background Knowledge

• CSV files with background facts about

– players – tournaments

22

Background knowledge Examples of deduced fact Player`s details A sister of a player is also a tennis professional. Ranking A player is leading the ATP score. Style A player is playing risky as usual. Injury A player has been four times injured recently. Player`s results A player won two matches in a row. Tournaments details The tournament is played in London on grass.

System Architecture

23

Jess JSHOP FSM

Deduction of High-level facts

24

Tennis Simulator Event Manager Discourse Planner

Mapping: plan → utterances

Plan: a list of plan operators Operator head:

briskly_returned_serve ?server ?receiver ?receiver_shot

Template:

briskly_returned_serve ?server ?receiver ?receiver_shot

contains 1-3 utterances: 1.) {EmotionSurprise} {Play} ?receiver generated a ?receiver_shot

{Look} return that was out of ?server's reach. 2.) …

substitution:

?server := Safin {EmotionSurprise} := $(Emotion,surprise,0.9,500,1000,3000) ?receiver := Federer {Look} := $(Motion,presentation/look/lookto_right02,400,500,0,1200,0.8) ?receiver_shot := forehand {Play} := $(Motion,interaction/bye/bye01,400,500,0,10000,1.5)

25

Operator (variables) Template (slots) Annotated utterance Command to the Avatar Engine

Video – Commentary Excerpt

26

28

Conclusion

• Two virtual agents engaged in dialogues to comment
• n a tennis game given as GALA 2009
• Affect

– Lexical selection – Facial expression – Gestures (synchronized with the speech)

• Customized commentary

– Commentators` attitudes to the players – User question (more engaging)

• Background Knowledge

28

Future Work

• EMBR (A Realtime Animation Engine for

Interactive Embodied Agents)

• Prosody module
• Base dialogue schemes on OCC Emotions
• Dynamic Replanning
• Other domains

– tutoring systems – tourist guides – guide for the blind

29

EMBR

QUESTIONS

Thankyou to the EMBOTS group, DFKI, and Charamel GmbH

30