Emergent Verbal Behaviour in Human-Robot Interaction Kristiina - - PowerPoint PPT Presentation

Emergent Verbal Behaviour in Human-Robot Interaction

Kristiina Jokinen & Graham Wilcock University of Helsinki

2 CogInfoCom, Budapest, 2011 Jokinen & Wilcock

Pyro: Python Robotics

Open source Python robotics toolkit

http://pyrobotics.org

For teaching and research

Simulators and real robots

Artificial intelligence and robotics

Reinforcement learning, fuzzy decisions,

neural networks, genetic algorithms, ...

3 CogInfoCom, Budapest, 2011 Jokinen & Wilcock

Pyspeech: Python Speech

Open source Python speech interface

http://code.google.com/p/pyspeech

Speech input and output

Speech recognition functions Text-to-speech functions

For Windows computers

Uses Microsoft Speech Engine

4 CogInfoCom, Budapest, 2011 Jokinen & Wilcock

Emergent Verbal Behaviour 1

Autonomous non-verbal behaviour

Robot performs actions silently

Autonomous verbal behaviour

Robot explains its actions

Interactive verbal behaviour

Human requests specific actions

5 CogInfoCom, Budapest, 2011 Jokinen & Wilcock

Non-verbal behaviour

• Autonomous behaviour
• Wander randomly
• Avoid obstacles
• Follow a wall
• etc.
• Robot acts silently

Jokinen & Wilcock CogInfoCom, Budapest, 2011 6

Verbal behaviour

• Robot explains its
• wn actions
• ”object on right”
• (therefore) ”turn left”
• Monologue
• One-way info
• Can be irritating
• ”clear, clear, clear...”

Jokinen & Wilcock CogInfoCom, Budapest, 2011 7

Cooperative Verbal Behaviour

Autonomous verbal behaviour

Robot explains its actions

Interactive verbal behaviour

Human requests ”talk less”, ”talk more”

Cooperative verbal behaviour

Robot changes its verbosity level No repeating, only says new things

8 CogInfoCom, Budapest, 2011 Jokinen & Wilcock

Emergent Verbal Behaviour 2

Human-initiated verbal interaction

Human asks robot to do something ”Book me a flight to Paris”

Robot-initiated verbal interaction

Goal: find out what the human wants Method: the robot asks the questions ”Classical” spoken dialogue systems

9 CogInfoCom, Budapest, 2011 Jokinen & Wilcock

10 CogInfoCom, Budapest, 2011 Jokinen & Wilcock

Spoken Dialogue Systems

Example: flight reservations system

Find out origin city, destination city ... Find out depart day, return day ...

Use finite state transitions

Well-known in spoken dialogue systems

Pyro includes finite state machines

Example implemented directly in Pyro

11 CogInfoCom, Budapest, 2011 Jokinen & Wilcock

Fixed-domain Dialogues

”Classical” dialogue systems Fixed-domain database

Flights, cities, days Easily add new flights, new cities

Cannot easily switch domains

Jokinen & Wilcock CogInfoCom, Budapest, 2011 12

Open-domain Dialogues

Web-based information retrieval

Open-domain, any topic Example: Wikipedia articles

Question-answering systems Not yet ”natural” conversations

Jokinen & Wilcock CogInfoCom, Budapest, 2011 13

Emergent Verbal Behaviour 3

Human-initiated open-domain dialogue

Human decides the topic ”Tell me about Shakespeare”

Robot continues with the topic

Gets Wikipedia article about Shakespeare Reads out the first paragraph How to continue the conversation?

14 CogInfoCom, Budapest, 2011 Jokinen & Wilcock

15 CogInfoCom, Budapest, 2011 Jokinen & Wilcock

Topic and NewInfo

Jokinen PhD thesis (1994)

Response Generation in Information-seeking Dialogues

Jokinen & Wilcock (2003)

Adaptivity and Response Generation in a Spoken Dialogue System

16 CogInfoCom, Budapest, 2011 Jokinen & Wilcock

Topic and NewInfo

Topic

Need to know the current topic Need to keep track of topic shifts

NewInfo

Gives some new information about Topic Dialogue response is based on NewInfo Topic link may be explicit or implicit

17 CogInfoCom, Budapest, 2011 Jokinen & Wilcock

Topics in Wikipedia

Articles

Article titles identify major topics Explicit disambiguation of similar titles

Sections

Section headings identify sub-topics Topic trees created by human authors

18 CogInfoCom, Budapest, 2011 Jokinen & Wilcock

19 CogInfoCom, Budapest, 2011 Jokinen & Wilcock

Topics in Spoken Wikipedia

Dialogue, not monologue

Don’t read out entire article (monologue) This would be irritating (need ”talk less”)

Avoid inappropriate sections

”Would you like to know about his Life, his

Plays or his Poems?”

”Would you like to know about his See

also, his Notes or his References?”

20 CogInfoCom, Budapest, 2011 Jokinen & Wilcock

NewInfo in Wikipedia

Text paragraphs

Give new information about subtopics Typically, one subtopic per paragraph

Hypertext links

Links identify major NewInfos Links might become new Topics Clicking on a link causes a topic shift

21 CogInfoCom, Budapest, 2011 Jokinen & Wilcock

22 CogInfoCom, Budapest, 2011 Jokinen & Wilcock

NewInfo in Spoken Wikipedia

Hyperlinks, NewInfos and topic shifts

”Shakespeare was born and raised in

Stratford-upon-Avon” (NewInfo)

”Stratford-upon-Avon?” (topic shift) ”Stratford-upon-Avon is a market town and

civil parish in Warwickshire, England”

”Warwickshire?”

23 CogInfoCom, Budapest, 2011 Jokinen & Wilcock

Feedback in Spoken Wikipedia

Text paragraphs

Speak a paragraph, stop for feedback Positive or negative? Interested or not?

Feedback may be non-verbal

Eye gaze Facial expression Body language

24 CogInfoCom, Budapest, 2011 Jokinen & Wilcock

Emergent Verbal Behaviour 4

Robot-initiated open-domain dialogue

Robot suggests a topic If human interested, robot continues

”Interesting” topics from Wikipedia

Wikipedia front page layout ”On this day” articles ”Did you know ...?” articles

25 CogInfoCom, Budapest, 2011 Jokinen & Wilcock

26 CogInfoCom, Budapest, 2011 Jokinen & Wilcock