Yours is better! Participant Response Bias in HCI Nicola Dell 1 - - PowerPoint PPT Presentation

▶
yours is better
SMART_READER_LITE
LIVE PREVIEW

Yours is better! Participant Response Bias in HCI Nicola Dell 1 - - PowerPoint PPT Presentation

Mar 24, 2023 468 likes •968 views

Yours is better! Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of Washington 2 San Jose State University 3 Microsoft Research India HCI is expanding and

slide-1
SLIDE 1

“Yours is better!”

Participant Response Bias in HCI

Nicola Dell1 Vidya Vaidyanathan2 Indrani Medhi3 Edward Cutrell3 William Thies3

1University of Washington 2San Jose State University 3Microsoft Research India

slide-2
SLIDE 2

HCI is expanding and diversifying

2

slide-3
SLIDE 3

HCI is expanding and diversifying

There are large differences between the people being studied and those studying them

3

slide-4
SLIDE 4

Demand characteristics are aspects of a study that convey the investigator’s hypothesis to participants

4

slide-5
SLIDE 5

Demand characteristics may result in participant response bias

5

slide-6
SLIDE 6

Demand characteristics may result in participant response bias

6

slide-7
SLIDE 7

Demand characteristics may result in participant response bias

Response bias can influence the outcome of studies with human participants

7

slide-8
SLIDE 8

What about demand characteristics in HCI?

8

slide-9
SLIDE 9

How do social and demographic factors influence demand characteristics?

9

slide-10
SLIDE 10

Let’s design an experiment to explore demand characteristics in HCI

10

slide-11
SLIDE 11

Let’s design an experiment to explore demand characteristics in HCI

11

Completely identical video clips

slide-12
SLIDE 12

Observe participants interacting with technological devices

12

slide-13
SLIDE 13

Interview Script

“Thank you for participating in my experiment. I am a computer scientist and I'm trying to improve video players on mobile phones. I want you to watch a short video on these two phones and tell me which

  • ne looks better, or if they look the same. The same

video will play on both phones, but this phone uses my new player [indicate phone]. Please tell me your honest opinion and please concentrate because I will play each video only once. Do you have any questions? Ok, watch this one first. This one uses my new player [play video]. Now watch this one [play video]. Which one do you think looks better or do they look the same? Why? Thanks very much!”

13

slide-14
SLIDE 14

Two different interviewers

Foreign interviewer and local interviewer

14

slide-15
SLIDE 15

Participants: Auto rickshaw drivers

250 male participants aged 19 to 72 (M=37, SD=11.2)

15

slide-16
SLIDE 16

Conditions tested with the identical video clip

Rickshaw Drivers Foreign Interviewer (+ translator) 50 participants Local Interviewer 50 participants

16

slide-17
SLIDE 17

Participant Responses

17

35 7 8

10 20 30 40 50

Number of people

Foreign Interviewer

Preferred interviewer’s video Preferred other video Thought videos looked the same

slide-18
SLIDE 18

Participant Responses

18

19 9 22

10 20 30 40 50

35 7 8

10 20 30 40 50

Number of people

Foreign Interviewer Local Interviewer

Number of people

Preferred interviewer’s video Preferred other video Thought videos looked the same

slide-19
SLIDE 19

Demand characteristics can result in substantial participant response bias

19

slide-20
SLIDE 20

Demand characteristics can result in substantial participant response bias Social and demographic factors influence the amount of response bias

20

slide-21
SLIDE 21

21

What happens if the two clips are different?

slide-22
SLIDE 22

What happens if the two clips are different?

21

slide-23
SLIDE 23

What happens if the two clips are different?

21

slide-24
SLIDE 24

What happens if the two clips are different?

21

Interviewer always associated to the low-quality clip

slide-25
SLIDE 25

Conditions tested with the degraded video clip

Rickshaw Drivers Without association 50 participants Foreign Interviewer (+ translator) 50 participants Local Interviewer 50 participants

22

slide-26
SLIDE 26

4 38 8

10 20 30 40 50

Preferred low-quality video Preferred high-quality video Thought videos looked the same

Participant Responses

23

Number of people

Without association

slide-27
SLIDE 27

Preferred low-quality video Preferred high-quality video Thought videos looked the same

Participant Responses

23

Number of people 27 19 4

10 20 30 40 50

Foreign Interviewer

Number of people 4 38 8

10 20 30 40 50

Without association

slide-28
SLIDE 28

The characteristics of the interviewer can determine the outcome of the study

24

slide-29
SLIDE 29

Explaining participant responses

25

slide-30
SLIDE 30

Explaining participant responses

“The quality of the background color and figures is too light in that player while the quality of the color and graphics is better in [your] one. Will [your] new player be introduced in the market?”

26

slide-31
SLIDE 31

Explaining participant responses

“I prefer this one (associated) because the other

  • ne wasn't clear at all. This one was fully clear. I

can definitely show you the difference if we watch them together. I've given you my honest opinion, so please don't be cross with me if it wasn't the right one.”

27

slide-32
SLIDE 32

Detailed and insightful participant comments can’t be taken at face value

28

slide-33
SLIDE 33

What about other participant groups?

200 male university students aged 19 to 41 (M=25, SD=3.8)

29

slide-34
SLIDE 34

Responses with university students

30

  • Identical videos: 2:1 response bias
slide-35
SLIDE 35

Responses with university students

30

  • Identical videos: 2:1 response bias
  • Degraded videos: order of magnitude difference
slide-36
SLIDE 36

Responses with university students

30

  • Identical videos: 2:1 response bias
  • Degraded videos: order of magnitude difference
  • “I feel that in the newer version which you have

coded, whenever there was a significant color contrast between two parts of an image, your version was somewhat smoother and less pixelated.”

slide-37
SLIDE 37

There is still substantial response bias in studies with peers

32

slide-38
SLIDE 38

Limitations

33

slide-39
SLIDE 39

Limitations

  • Presence of a translator for some interviews

34

slide-40
SLIDE 40

Limitations

  • Presence of a translator for some interviews
  • The effects of social status and ethnicity will

vary from culture to culture

35

slide-41
SLIDE 41

Limitations

  • Presence of a translator for some interviews
  • The effects of social status and ethnicity will

vary from culture to culture

  • Many other aspects of demand characteristics

36

slide-42
SLIDE 42

What can we do to minimize demand characteristics in HCI?

37

slide-43
SLIDE 43

What can we do to minimize demand characteristics in HCI?

  • Be aware that response bias affects all studies

38

slide-44
SLIDE 44

What can we do to minimize demand characteristics in HCI?

  • Be aware that response bias affects all studies
  • Dissociate from a particular design or solution

39

slide-45
SLIDE 45

What can we do to minimize demand characteristics in HCI?

  • Be aware that response bias affects all studies
  • Dissociate from a particular design or solution
  • Minimize the differences in social status

between investigators and participants

40

slide-46
SLIDE 46

What can we do to minimize demand characteristics in HCI?

  • Be aware that response bias affects all studies
  • Dissociate from a particular design or solution
  • Minimize the differences in social status

between investigators and participants

  • Use triangulation to validate data collected

41

slide-47
SLIDE 47

Demand characteristics influence the

  • utcome of studies with human participants

42

slide-48
SLIDE 48

“Yours is better!”

Participant Response Bias in HCI

Nicola Dell1 Vidya Vaidyanathan2 Indrani Medhi3 Edward Cutrell3 William Thies3

1University of Washington 2San Jose State University 3Microsoft Research India

nixdell@uw.edu http://cs.washington.edu/homes/nixdell