Query Clarification in Voice Search The case of ambiguous terms and false memories Master’s Thesis by Arefeh Bahrami Refrees: Prof. Dr. Benno Stein Prof. Dr. Matthias Hagen Advisor: Johannes Kiesel Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 1
Motivation Clarification in text-based search Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 2
Motivation Clarification in text-based search Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 3
Motivation How about voice-based search? Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 4
Motivation The Scenario Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 5
Motivation The Scenario Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 6
Motivation The Scenario Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 7
Motivation The Scenario Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 8
Motivation The Scenario Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 9
Motivation The Scenario Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 10
Motivation The Scenario Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 11
Motivation The Scenario Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 12
Motivation The Scenario Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 13
Motivation The Scenario Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 14
Motivation The Scenario Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 15
Motivation The Scenario Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 16
Motivation What was the first option? Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 17
Motivation Research questions ❒ RQ I: Does the user background affect user satisfaction when experiencing query clarification? ❒ RQ II: Do the length and the number of the clarifying options affect the user satisfaction? ❒ RQ III: What is the best way of clarifying the user’s intent? Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 18
Study 1: Clarifying Ambiguous Queries Section 2 Study 1: Clarifying Ambiguous Queries Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 19
Study 1: Clarifying Ambiguous Queries Response types & Topics ❒ Direct ❒ 3-meanings ❒ 3-categories ❒ Many meanings (5-meanings) ❒ Longer explanations (3-long-meanings) ❒ Verify Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 20
Study 1: Clarifying Ambiguous Queries Response types & Topics ❒ Direct ❒ 3-meanings ❒ 3-categories ❒ Many meanings (5-meanings) ❒ Longer explanations (3-long-meanings) ❒ Verify Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 20
Study 1: Clarifying Ambiguous Queries Study setup Create mock-up skill for Amazon Alexa ❒ Natural set-up ❒ Real-time answer Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 21
❒ ❒ ❒ Study 1: Clarifying Ambiguous Queries Study setup Task example: Scenario: You want to surprise your Irish partner with an Irish cocktail called B-52, but you don’t know how to make it. Interaction start: Alexa. Find! ❒ How to do a B-52? ❒ After interaction: Agree Neutral Disagree Don’t know ✷ ✷ ✷ ✷ ✷ ✷ The system answered my question ✷ ✷ ✷ ✷ ✷ ✷ The system behaved as I expected ✷ ✷ ✷ ✷ ✷ ✷ The system was easy to hear/understand ✷ ✷ ✷ ✷ ✷ ✷ The system was pleasant to use Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 22
Study 1: Clarifying Ambiguous Queries Study setup Task example: Scenario: You want to surprise your Irish partner with an Irish cocktail called B-52, but you don’t know how to make it. Interaction start: Alexa. Find! ❒ How to do a B-52? ❒ After interaction: Agree Neutral Disagree Don’t know ✷ ✷ ✷ ✷ ✷ ✷ The system answered my question ✷ ✷ ✷ ✷ ✷ ✷ The system behaved as I expected ✷ ✷ ✷ ✷ ✷ ✷ The system was easy to hear/understand ✷ ✷ ✷ ✷ ✷ ✷ The system was pleasant to use ❒ 14 participants (9 proficient, 5 intermediate), 13 main tasks ❒ 182 interaction phases ❒ 728 ratings Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 22
Study 1: Clarifying Ambiguous Queries Research questions ❒ RQ I: Does the user background affect user satisfaction when experiencing query clarification? ❒ RQ II: Do the length and the number of the clarifying options affect the user satisfaction? ❒ RQ III: Does the user satisfaction decrease when the system ask for clarification? Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 23
Study 1: Clarifying Ambiguous Queries Results Understandability and pleasantness by English fluency level 1.0 1.0 English proficiency English proficiency intermediate intermediate proficient proficient 0.8 0.8 Ratings density Ratings density 0.6 0.6 0.4 0.4 0.2 0.2 0.0 0.0 agree neutral disagree agree neutral disagree The system was easy to hear/understand The system was pleasant to use Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 24
Study 1: Clarifying Ambiguous Queries Results Predictability and understandability by frequency of a participant’s voice interface usage: 1.0 1.0 Voice assistant usage Voice assistant usage never never rarely rarely 0.8 0.8 Ratings density Ratings density 0.6 0.6 0.4 0.4 0.2 0.2 0.0 0.0 agree neutral disagree agree neutral disagree The system behaved as I expected The system was easy to hear/understand Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 25
Study 1: Clarifying Ambiguous Queries Research questions ❒ RQ I: Does the user background affect user satisfaction when experiencing query clarification? ❒ RQ II: Do the length and the number of the clarifying options affect the user satisfaction? ❒ RQ III: Does the user satisfaction decrease when the system ask for clarification? Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 26
Study 1: Clarifying Ambiguous Queries Results Distribution and mean ( ❳ ) of pleasantness for participants with intermediate English level by response method: ● 3−meanings ● 3−meanings−long ● verify ● 3−categories ● 5−meanings agree neutral disagree The system was pleasant to use 3-meaning response method ( µ p � 1 . 4) Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 27
Study 1: Clarifying Ambiguous Queries Research questions ❒ RQ I: Does the user background affect user satisfaction when experiencing query clarification? ❒ RQ II: Do the length and the number of the clarifying options affect the user satisfaction? ❒ RQ III: Does the user satisfaction decrease when the system ask for clarification? Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 28
Study 1: Clarifying Ambiguous Queries Results Response-specific ratings for predictability and pleasantness: 1.0 1.0 Response methods Response methods direct (hit) direct (hit) verify (hit) verify (hit) 0.8 0.8 verify (miss) verify (miss) direct (miss) direct (miss) Ratings density Ratings density 0.6 0.6 0.4 0.4 0.2 0.2 0.0 0.0 agree neutral disagree agree neutral disagree The system behaved as I expected The system was pleasant to use Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 29
Study 2: Clarifying Queries with False Memories Section 3 Study 2: Clarifying Queries with False Memories Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 30
Study 2: Clarifying Queries with False Memories Known-item search Re-finding previously seen/heard items is not that easy! Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 31
Study 2: Clarifying Queries with False Memories False memories in question Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 32
Study 2: Clarifying Queries with False Memories Movie”..Maggie Smith plays a creepy woman...” Question Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 33
Study 2: Clarifying Queries with False Memories Movie”..Maggie Smith plays a creepy woman...” Question Actual known-item Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 33
Study 2: Clarifying Queries with False Memories Hmm... Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 34
Study 2: Clarifying Queries with False Memories Research questions ❒ RQ I: Does language fluency affect user satisfaction? ❒ RQ II: Do wrong clarifications degrade user satisfaction? ❒ RQ III: How to best clarify false memories? Query Clarification in Voice Search October 26, 2018 Arefeh Bahrami 35
Recommend
More recommend
