A Mixed-Method Analysis of Text and Audio Search Interfaces with Varying Task Complexity Edith Law Horaţiu Bota Sa Sash sha Vt Vtyurina University of Waterloo University of Waterloo Johanne Trippas Charlie Clarke University of Waterloo University of Melbourne
Voice assistants How many grams are in Will it rain tomorrow? one ounce? Turn on kitchen lights When is Easter? How tall is Carle Ray Add ice cream to Jepsen? my shopping list Tell me about golden Set an alarm for eight a.m. retrievers 2
Voice assistants o Correct answer exists o Direct and concise How many grams are in o Single source needed Will it rain tomorrow? one ounce? Turn on kitchen lights When is Easter? How tall is Carle Ray Add ice cream to Jepsen? my shopping list Tell me about golden Set an alarm for eight a.m. retrievers 3
Voice assistants o Correct answer exists o Direct and concise How many grams are in o Single source needed Will it rain tomorrow? one ounce? Turn on kitchen lights When is Easter? o Multiple possible answers o Multiple possible sources o A lot of information available How tall is Carle Ray Add ice cream to Jepsen? my shopping list Tell me about golden Set an alarm for eight a.m. retrievers 4
Search Engine Results Page (SERP) 5
Search Engine Results Page (SERP) 6
Search Engine Results Page (SERP) 7
Search Engine Results Page (SERP) 8
Search Engine Results Page (SERP) 9
- Alexa, tell me about golden retrievers. - The Golden Retriever is a medium-large gun dog that was bred to retrieve shot waterfowl, such as ducks and upland game birds, during hunting and shooting parties. The name “retriever” refers to the breed’s ability to retrieve shot game undamaged due to their soft mouth. 10
How can we present SERP through a voice-only channel? 11
Plan Two interfaces: Search results from Two studies: AMT and LAB 6 search tasks Audio and Text Google Search API 12
Plan Two interfaces: Search results from Two studies: AMT and LAB 6 search tasks Audio and Text Google Search API 13
Search Tasks o How tall is CN tower in Toronto? o Which planet was researched by spacecraft Magellan? 2 2 X 2 2 X o ... health and benefits of seaweed and algae... o ... scientific expeditions in 2 2 X Antarctica... o ... Hubble telescope achievements... o ... new hydroelectric projects... 14
Two interfaces: Search results from Two studies: AMT and LAB 6 search tasks Audio and Text Google Search API 15
Search results Result # 5 Result # 1 Text Result # 10 Result # 5 Go Google le interface shuffle Search Se Se Sear arch Result # 1 Result # 10 quer query API AP Result # 100 Result # 50 Audio interface Result # 50 Result # 100 16
Two interfaces: Search results from Two studies: AMT and LAB 6 search tasks Audio and Text Google Search API 17
Interfaces: Text Dataset at: 18 github.com/sashavtyurina/audio-serp-ictir-2020
Interfaces: Audio Dataset at: 19 github.com/sashavtyurina/audio-serp-ictir-2020
Interfaces: Audio Dataset at: 20 github.com/sashavtyurina/audio-serp-ictir-2020
Interfaces: Audio Dataset at: 21 github.com/sashavtyurina/audio-serp-ictir-2020
Interfaces: Audio 22
Interfaces: Text and Audio 23
Two interfaces: Search results from Two studies: AMT and LAB 6 search tasks Audio and Text Google Search API 24
Two studies Amazon Mechanical Turk (AMT) Laboratory study (LAB) - 69 participants - 36 participants - Identify general trends - Develop deep understanding - USD 3.50 per task Choose best, 2 nd best, and the least useful results - Choose best, 2 nd best, and the least useful - - NASA TLX results - Verbal interview TLX TLX x3 + & & Interview Interview 25
Differences in overall ranking Search results Possible user selections Bootstrap average difference of correct Result #1 selections between Text and Audio conditions Result #1 2 correct Result #5 Result #10 Result #100 Result #10 Result #1 Result #5 Result #50 3 correct Result #100 Result #100 Result #10 1 correct Result #50 Result #100 26
Perceived workload Temporal Mental Text interface scores significantly lower Effort on NASA TLX than Audio interface Performance Frustration 27
Qualitative observations “The first one is Zimbabwe one, and... I think I clicked the Philadelphia one.” Navigation “The best one was the brief history one” shortcuts “I chose the NASA one as the best one, and then the one from “the weathernetwork” as the second best one” “The best one was from a travel website” 28
Qualitative observations “The URLs and the sources they kind of like blended in to actual information” Uncertainty “ I couldn’t tell when it was going to stop... It’s why Instagram videos suck — you wrt structure can’t see how far along you are in the video” Uncertainty wrt duration “Just give me the name of the website, just say ‘Wikipedia’, “It was very monotone, just say ‘NASA’, whatever it was, I don’t need the URL” washing over me” Abbreviations Monotonicity “This one on the ScienceDirect using algae “He said the URL, or something like that, and marine vegetation looked like it could and then he repeated the title which was have been promising, but then it cuts off, so the exact same thing as the URL” Truncated not sure” Repetitions sentences 29
Qualitative observations “I had to carefully listen to the audio. And when I’m listening to audio, I feel like this is the only chance I’m listening to it” “I can browse through the results quicker visually. And I’m able Cognitive load to pick out key-words” 30
Discussion o Interface (Text or Audio) has a significant effect on the result selection and perceived workload. o We did not find an effect of the task complexity o A voice-based search system should: o be aware of the content it is returning o clearly indicate constituent parts o use prosodic features to avoid monotone voice o avoid abbreviations o avoid repetition o avoid truncated sentences 31
A Mixed-Method Analysis of Text and Audio Search Interfaces with Varying Task Complexity Edith Law Horaţiu Bota Sa Sash sha Vt Vtyurina University of Waterloo University of Waterloo Johanne Trippas Charlie Clarke University of Waterloo University of Melbourne
Recommend
More recommend
