Query Clarification in Voice Search The case of ambiguous terms and - - PowerPoint PPT Presentation

Query Clarification in Voice Search

The case of ambiguous terms and false memories Master’s Thesis by Arefeh Bahrami

Refrees:

• Prof. Dr. Benno Stein
• Prof. Dr. Matthias Hagen

Advisor: Johannes Kiesel

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

1

Motivation

Clarification in text-based search

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

2

Motivation

Clarification in text-based search

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

3

Motivation

How about voice-based search?

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

4

Motivation

The Scenario

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

5

Motivation

The Scenario

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

6

Motivation

The Scenario

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

7

Motivation

The Scenario

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

8

Motivation

The Scenario

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

9

Motivation

The Scenario

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

10

Motivation

The Scenario

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

11

Motivation

The Scenario

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

12

Motivation

The Scenario

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

13

Motivation

The Scenario

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

14

Motivation

The Scenario

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

15

Motivation

The Scenario

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

16

Motivation

What was the first option?

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

17

Motivation

Research questions

❒ RQ I: Does the user background affect user satisfaction when experiencing query clarification? ❒ RQ II: Do the length and the number of the clarifying options affect the user satisfaction? ❒ RQ III: What is the best way of clarifying the user’s intent?

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

18

Study 1: Clarifying Ambiguous Queries

Section 2 Study 1: Clarifying Ambiguous Queries

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

19

Study 1: Clarifying Ambiguous Queries

Response types & Topics

❒ Direct ❒ 3-meanings ❒ 3-categories ❒ Many meanings (5-meanings) ❒ Longer explanations (3-long-meanings) ❒ Verify

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

20

Study 1: Clarifying Ambiguous Queries

Response types & Topics

❒ Direct ❒ 3-meanings ❒ 3-categories ❒ Many meanings (5-meanings) ❒ Longer explanations (3-long-meanings) ❒ Verify

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

20

Study 1: Clarifying Ambiguous Queries

Study setup

Create mock-up skill for Amazon Alexa ❒ Natural set-up ❒ Real-time answer

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

21

Study 1: Clarifying Ambiguous Queries

Study setup

Task example:

Scenario: You want to surprise your Irish partner with an Irish cocktail called B-52, but you don’t know how to make it. Interaction start: ❒

• Alexa. Find!

❒ How to do a B-52? After interaction: Agree Neutral Disagree Don’t know The system answered my question

✷ ✷ ✷ ✷ ✷ ✷

The system behaved as I expected

✷ ✷ ✷ ✷ ✷ ✷

The system was easy to hear/understand

✷ ✷ ✷ ✷ ✷ ✷

The system was pleasant to use

✷ ✷ ✷ ✷ ✷ ✷

❒ ❒ ❒

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

22

Study 1: Clarifying Ambiguous Queries

Study setup

Task example:

Scenario: You want to surprise your Irish partner with an Irish cocktail called B-52, but you don’t know how to make it. Interaction start: ❒

• Alexa. Find!

❒ How to do a B-52? After interaction: Agree Neutral Disagree Don’t know The system answered my question

✷ ✷ ✷ ✷ ✷ ✷

The system behaved as I expected

✷ ✷ ✷ ✷ ✷ ✷

The system was easy to hear/understand

✷ ✷ ✷ ✷ ✷ ✷

The system was pleasant to use

✷ ✷ ✷ ✷ ✷ ✷

❒ 14 participants (9 proficient, 5 intermediate), 13 main tasks ❒ 182 interaction phases ❒ 728 ratings

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

22

Study 1: Clarifying Ambiguous Queries

Research questions

❒ RQ I: Does the user background affect user satisfaction when experiencing query clarification? ❒ RQ II: Do the length and the number of the clarifying options affect the user satisfaction? ❒ RQ III: Does the user satisfaction decrease when the system ask for clarification?

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

23

Study 1: Clarifying Ambiguous Queries

Results

Understandability and pleasantness by English fluency level

The system was easy to hear/understand Ratings density 0.0 0.2 0.4 0.6 0.8 1.0 agree neutral disagree English proficiency intermediate proficient The system was pleasant to use Ratings density 0.0 0.2 0.4 0.6 0.8 1.0 agree neutral disagree English proficiency intermediate proficient Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

24

Study 1: Clarifying Ambiguous Queries

Results

Predictability and understandability by frequency of a participant’s voice interface usage:

The system behaved as I expected Ratings density 0.0 0.2 0.4 0.6 0.8 1.0 agree neutral disagree Voice assistant usage never rarely The system was easy to hear/understand Ratings density 0.0 0.2 0.4 0.6 0.8 1.0 agree neutral disagree Voice assistant usage never rarely Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

25

Study 1: Clarifying Ambiguous Queries

Research questions

❒ RQ I: Does the user background affect user satisfaction when experiencing query clarification? ❒ RQ II: Do the length and the number of the clarifying options affect the user satisfaction? ❒ RQ III: Does the user satisfaction decrease when the system ask for clarification?

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

26

Study 1: Clarifying Ambiguous Queries

Results

Distribution and mean (❳) of pleasantness for participants with intermediate English level by response method:

• 5−meanings

3−categories verify 3−meanings−long 3−meanings agree neutral disagree The system was pleasant to use

3-meaning response method (µp 1.4)

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

27

Study 1: Clarifying Ambiguous Queries

Research questions

❒ RQ I: Does the user background affect user satisfaction when experiencing query clarification? ❒ RQ II: Do the length and the number of the clarifying options affect the user satisfaction? ❒ RQ III: Does the user satisfaction decrease when the system ask for clarification?

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

28

Study 1: Clarifying Ambiguous Queries

Results

Response-specific ratings for predictability and pleasantness:

The system behaved as I expected Ratings density 0.0 0.2 0.4 0.6 0.8 1.0 agree neutral disagree Response methods direct (hit) verify (hit) verify (miss) direct (miss) The system was pleasant to use Ratings density 0.0 0.2 0.4 0.6 0.8 1.0 agree neutral disagree Response methods direct (hit) verify (hit) verify (miss) direct (miss) Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

29

Study 2: Clarifying Queries with False Memories

Section 3 Study 2: Clarifying Queries with False Memories

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

30

Study 2: Clarifying Queries with False Memories

Known-item search

Re-finding previously seen/heard items is not that easy!

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

31

Study 2: Clarifying Queries with False Memories

False memories in question

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

32

Study 2: Clarifying Queries with False Memories

Movie”..Maggie Smith plays a creepy woman...”

Question

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

33

Study 2: Clarifying Queries with False Memories

Movie”..Maggie Smith plays a creepy woman...”

Question Actual known-item

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

33

Study 2: Clarifying Queries with False Memories

Hmm...

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

34

Study 2: Clarifying Queries with False Memories

Research questions

❒ RQ I: Does language fluency affect user satisfaction? ❒ RQ II: Do wrong clarifications degrade user satisfaction? ❒ RQ III: How to best clarify false memories?

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

35

Study 2: Clarifying Queries with False Memories

Response types & Topics

❒ None no clarification (“Sorry, I don’t know that one!”) ❒ Direct answer query for a changed detail without informing users. ❒ Negatively Clarified (”Sorry, I don’t know that one, but...”) ❒ Positively Clarified (“You probably mean...”)

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

36

Study 2: Clarifying Queries with False Memories

Response types & Topics

❒ None no clarification (“Sorry, I don’t know that one!”) ❒ Direct answer query for a changed detail without informing users. ❒ Negatively Clarified (”Sorry, I don’t know that one, but...”) ❒ Positively Clarified (“You probably mean...”)

annotations for 2,755 questions

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

36

Study 2: Clarifying Queries with False Memories

Study setup

Task example:

Scenario: You try to remember the title of a controversial book that came out back in the 1990s and claimed scientific evidence that whites are genetically superior to blacks. You think it was called something like ”The something Factor.” Interaction start: Alexa. Explore! What is the title of the book from the 1990s that claimed superiority of Whites and is called ”The something Factor” ? Post-interaction questions: Agree Neutral Disagree Don’t know The system was helpful

✷ ✷ ✷ ✷ ✷ ✷

The system behaved as I expected

✷ ✷ ✷ ✷ ✷ ✷

The system was easy to hear/understand

✷ ✷ ✷ ✷ ✷ ✷

The system was pleasant to use

✷ ✷ ✷ ✷ ✷ ✷

❒ ❒ ❒

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

37

Study 2: Clarifying Queries with False Memories

Study setup

Task example:

Scenario: You try to remember the title of a controversial book that came out back in the 1990s and claimed scientific evidence that whites are genetically superior to blacks. You think it was called something like ”The something Factor.” Interaction start: Alexa. Explore! What is the title of the book from the 1990s that claimed superiority of Whites and is called ”The something Factor” ? Post-interaction questions: Agree Neutral Disagree Don’t know The system was helpful

✷ ✷ ✷ ✷ ✷ ✷

The system behaved as I expected

✷ ✷ ✷ ✷ ✷ ✷

The system was easy to hear/understand

✷ ✷ ✷ ✷ ✷ ✷

The system was pleasant to use

✷ ✷ ✷ ✷ ✷ ✷

❒ 12 participants (7 proficient, 5 intermediate), 14 main tasks ❒ 168 interaction phases ❒ 672 ratings

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

37

Study 2: Clarifying Queries with False Memories

Research questions

❒ RQ I: Does language fluency affect user satisfaction? ❒ RQ II: Do wrong clarifications degrade user satisfaction? ❒ RQ III: How to best clarify false memories?

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

38

Study 2: Clarifying Queries with False Memories

Results

Effectiveness and predictability by English fluency:

The system behaved as I expected Ratings density 0.0 0.2 0.4 0.6 0.8 1.0 agree neutral disagree English fluency intermediate proficient Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

39

Study 2: Clarifying Queries with False Memories

Results

Understandability and pleasantness by English fluency

The system was easy to hear/understand Ratings density 0.0 0.2 0.4 0.6 0.8 1.0 agree neutral disagree English fluency intermediate proficient The system was pleasant to use Ratings density 0.0 0.2 0.4 0.6 0.8 1.0 agree neutral disagree English fluency intermediate proficient Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

40

Study 2: Clarifying Queries with False Memories

Research questions

❒ RQ I: Does language fluency affect user satisfaction? ❒ RQ II: Do wrong clarifications degrade user satisfaction? ❒ RQ III: How to best clarify false memories?

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

41

Study 2: Clarifying Queries with False Memories

Results

Distribution and mean (❳) of pleasantness by response method and changed detail:

• Positively clarified
• Negatively clarified
• Direct
• None
• ther detail

false memory

• ther detail

false memory

• ther detail

false memory agree neutral disagree The system was pleasant to use

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

42

Study 2: Clarifying Queries with False Memories

Research questions

❒ RQ I: Does language fluency affect user satisfaction? ❒ RQ II: Do wrong clarifications degrade user satisfaction? ❒ RQ III: How to best clarify false memories?

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

43

Study 2: Clarifying Queries with False Memories

Results

Response-specific ratings for predictability and pleasantness when the false memory detail is corrected:

The system behaved as I expected Ratings density 0.0 0.2 0.4 0.6 0.8 1.0 agree neutral disagree Response type direct negatively clarified positively clarified The system was pleasant to use Ratings density 0.0 0.2 0.4 0.6 0.8 1.0 agree neutral disagree Response methods direct negatively clarified positively clarified Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

44

Conclusion and Future Work

Summary

❒ First step towards the task of voice search query clarification. ❒ 1st Study: clarifying ambiguous queries, 7 response types, mock-up skill for Amazon Alexa, results analysis. ❒ 2nd Study: clarifying queries with false memories, 4 response types, mock-up skill, results analysis.

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

45

Conclusion and Future Work

Conclusion & Future work

❒ Users won’t mind be asked for clarification. ❒ Consider English fluency level. ❒ Use positive tone in the response. ❒ Three clarification options work best. ❒ Test alternative response methods. ❒ Use crowdsourcing platforms.

✱

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

46

Conclusion and Future Work

Conclusion & Future work

❒ Users won’t mind be asked for clarification. ❒ Consider English fluency level. ❒ Use positive tone in the response. ❒ Three clarification options work best. ❒ Test alternative response methods. ❒ Use crowdsourcing platforms.

Thank you! ✱

Arefeh Bahrami Query Clarification in Voice Search October 26, 2018

46