A Mixed-Method Analysis of Text and Audio Search Interfaces with - - PowerPoint PPT Presentation

▶
a mixed method analysis of text and audio search
SMART_READER_LITE
LIVE PREVIEW

A Mixed-Method Analysis of Text and Audio Search Interfaces with - - PowerPoint PPT Presentation

Mar 26, 2024 276 likes •604 views

A Mixed-Method Analysis of Text and Audio Search Interfaces with Varying Task Complexity Edith Law Horaiu Bota Sa Sash sha Vt Vtyurina University of Waterloo University of Waterloo Johanne Trippas Charlie Clarke University of Waterloo

slide-1
SLIDE 1

A Mixed-Method Analysis of Text and Audio Search Interfaces with Varying Task Complexity

Sa Sash sha Vt Vtyurina

University of Waterloo

Charlie Clarke

University of Waterloo

Edith Law

University of Waterloo

Johanne Trippas

University of Melbourne

Horaţiu Bota

slide-2
SLIDE 2

Voice assistants

Will it rain tomorrow? Turn on kitchen lights Set an alarm for eight a.m. When is Easter? How many grams are in

  • ne ounce?

Tell me about golden retrievers How tall is Carle Ray Jepsen? Add ice cream to my shopping list

2

slide-3
SLIDE 3

Voice assistants

  • Correct answer exists
  • Direct and concise
  • Single source needed

Will it rain tomorrow? Turn on kitchen lights Set an alarm for eight a.m. When is Easter? How many grams are in

  • ne ounce?

Tell me about golden retrievers How tall is Carle Ray Jepsen? Add ice cream to my shopping list

3

slide-4
SLIDE 4

Voice assistants

  • Correct answer exists
  • Direct and concise
  • Single source needed
  • Multiple possible answers
  • Multiple possible sources
  • A lot of information available

Will it rain tomorrow? Turn on kitchen lights Set an alarm for eight a.m. When is Easter? How many grams are in

  • ne ounce?

Tell me about golden retrievers How tall is Carle Ray Jepsen? Add ice cream to my shopping list

4

slide-5
SLIDE 5

Search Engine Results Page (SERP)

5

slide-6
SLIDE 6

Search Engine Results Page (SERP)

6

slide-7
SLIDE 7

Search Engine Results Page (SERP)

7

slide-8
SLIDE 8

Search Engine Results Page (SERP)

8

slide-9
SLIDE 9

Search Engine Results Page (SERP)

9

slide-10
SLIDE 10
  • Alexa, tell me about golden retrievers.
  • The Golden Retriever is a medium-large gun

dog that was bred to retrieve shot waterfowl, such as ducks and upland game birds, during hunting and shooting parties. The name “retriever” refers to the breed’s ability to retrieve shot game undamaged due to their soft mouth.

10

slide-11
SLIDE 11

How can we present SERP through a voice-only channel?

11

slide-12
SLIDE 12

Plan

12

6 search tasks Search results from Google Search API Two interfaces: Audio and Text Two studies: AMT and LAB

slide-13
SLIDE 13

Plan

13

6 search tasks Search results from Google Search API Two interfaces: Audio and Text Two studies: AMT and LAB

slide-14
SLIDE 14

Search Tasks

14

2 2 X 2 2 X 2 2 X

  • How tall is CN tower in Toronto?
  • Which planet was researched

by spacecraft Magellan?

  • ... health and benefits of

seaweed and algae...

  • ... scientific expeditions in

Antarctica...

  • ... Hubble telescope achievements...
  • ... new hydroelectric projects...
slide-15
SLIDE 15

15

6 search tasks Search results from Google Search API Two interfaces: Audio and Text Two studies: AMT and LAB

slide-16
SLIDE 16

Search results

Text interface Audio interface

16

Go Google le Se Sear arch AP API

Se Search quer query Result # 1 Result # 5 Result # 10 Result # 50 Result # 100 shuffle Result # 5 Result # 10 Result # 1 Result # 100 Result # 50

slide-17
SLIDE 17

17

6 search tasks Search results from Google Search API Two interfaces: Audio and Text Two studies: AMT and LAB

slide-18
SLIDE 18

Interfaces: Text

18

Dataset at: github.com/sashavtyurina/audio-serp-ictir-2020

slide-19
SLIDE 19

Interfaces: Audio

19

Dataset at: github.com/sashavtyurina/audio-serp-ictir-2020

slide-20
SLIDE 20

Interfaces: Audio

20

Dataset at: github.com/sashavtyurina/audio-serp-ictir-2020

slide-21
SLIDE 21

Interfaces: Audio

21

Dataset at: github.com/sashavtyurina/audio-serp-ictir-2020

slide-22
SLIDE 22

Interfaces: Audio

22

slide-23
SLIDE 23

Interfaces: Text and Audio

23

slide-24
SLIDE 24

24

6 search tasks Search results from Google Search API Two interfaces: Audio and Text Two studies: AMT and LAB

slide-25
SLIDE 25

Two studies

Laboratory study (LAB) Amazon Mechanical Turk (AMT)

  • 69 participants
  • Identify general trends
  • USD 3.50 per task
  • Choose best, 2nd best, and the least useful

results

  • 36 participants
  • Develop deep understanding
  • Choose best, 2nd best, and the least useful results
  • NASA TLX
  • Verbal interview

TLX & Interview TLX & Interview

x3 +

25

slide-26
SLIDE 26

Differences in overall ranking

Possible user selections

Result #1 Result #5 Result #50 Result #10 Result #100

Search results

Result #1 Result #10 Result #100 2 correct Result #1 Result #5 Result #100 3 correct Result #10 Result #50 Result #100 1 correct Bootstrap average difference of correct selections between Text and Audio conditions

26

slide-27
SLIDE 27

Perceived workload

Temporal Mental Effort Performance Frustration Text interface scores significantly lower

  • n NASA TLX than Audio interface

27

slide-28
SLIDE 28

Qualitative observations

“The first one is Zimbabwe one, and... I think I clicked the Philadelphia one.” “The best one was the brief history one” “I chose the NASA one as the best one, and then the one from “the weathernetwork” as the second best one” “The best one was from a travel website”

28

Navigation shortcuts

slide-29
SLIDE 29

Qualitative observations

“The URLs and the sources they kind of like blended in to actual information” “I couldn’t tell when it was going to stop... It’s why Instagram videos suck — you can’t see how far along you are in the video” “It was very monotone, washing over me” “Just give me the name of the website, just say ‘Wikipedia’, just say ‘NASA’, whatever it was, I don’t need the URL” “This one on the ScienceDirect using algae and marine vegetation looked like it could have been promising, but then it cuts off, so not sure” “He said the URL, or something like that, and then he repeated the title which was the exact same thing as the URL”

29

Uncertainty wrt structure Uncertainty wrt duration Monotonicity Abbreviations Truncated sentences Repetitions

slide-30
SLIDE 30

Qualitative observations

“I had to carefully listen to the audio. And when I’m listening to audio, I feel like this is the only chance I’m listening to it” “I can browse through the results quicker visually. And I’m able to pick out key-words”

30

Cognitive load

slide-31
SLIDE 31

Discussion

  • Interface (Text or Audio) has a significant effect on the result selection

and perceived workload.

  • We did not find an effect of the task complexity
  • A voice-based search system should:
  • be aware of the content it is returning
  • clearly indicate constituent parts
  • use prosodic features to avoid monotone voice
  • avoid abbreviations
  • avoid repetition
  • avoid truncated sentences

31

slide-32
SLIDE 32

A Mixed-Method Analysis of Text and Audio Search Interfaces with Varying Task Complexity

Sa Sash sha Vt Vtyurina

University of Waterloo

Charlie Clarke

University of Waterloo

Edith Law

University of Waterloo

Johanne Trippas

University of Melbourne

Horaţiu Bota