EMPIRICAL USER-STUDIES human-computer interaction CSE 440 WINTER - - PowerPoint PPT Presentation

▶
empirical user studies
SMART_READER_LITE
LIVE PREVIEW

EMPIRICAL USER-STUDIES human-computer interaction CSE 440 WINTER - - PowerPoint PPT Presentation

Oct 24, 2023 682 likes •1.28k views

Maya Cakmak, Matt Kay, Brad Jacobson, King Xia EMPIRICAL USER-STUDIES human-computer interaction CSE 440 WINTER 2015 University of FEB 19 - WEEK 7 - THURSDAY Washington Methods for observing interaction hmmmm blah blah blah bla Passive

slide-1
SLIDE 1

University of Washington

human-computer interaction

CSE 440 WINTER 2015

FEB 19 - WEEK 7 - THURSDAY

EMPIRICAL USER-STUDIES

Maya Cakmak, Matt Kay, Brad Jacobson, King Xia

slide-2
SLIDE 2

University of Washington

Methods for observing interaction

2

Passive observation Think-aloud protocol

hmmmm blah blah blah bla

Comparative study

Last week

slide-3
SLIDE 3

University of Washington

Methods for observing interaction

2

Passive observation Think-aloud protocol

hmmmm blah blah blah bla

Comparative study

Last week “Empirical user study” “Controlled experiment” Today

slide-4
SLIDE 4

University of Washington

Evaluation Techniques (re-cap)

  • Asking users

–Questionnaires, interviews, focus groups

  • Observing users

–Passive observation, think-aloud protocol, ethnography, empirical user studies

  • Make users observe themselves

–Diaries, experience sampling

  • Ask experts

–Heuristic evaluation, cognitive walkthrough

3

slide-5
SLIDE 5

University of Washington

Evaluation Techniques (re-cap)

  • Asking users

–Questionnaires, interviews, focus groups

  • Observing users

–Passive observation, think-aloud protocol, ethnography, empirical user studies

  • Make users observe themselves

–Diaries, experience sampling

  • Ask experts

–Heuristic evaluation, cognitive walkthrough

3

slide-6
SLIDE 6

University of Washington

Designing an empirical study

4

slide-7
SLIDE 7

University of Washington

Designing an empirical study

  • What is being compared?

–Independent variables

4

slide-8
SLIDE 8

University of Washington

Designing an empirical study

  • What is being compared?

–Independent variables

  • What are they being compared in?

–Dependent variables (“metrics”)

4

slide-9
SLIDE 9

University of Washington

Designing an empirical study

  • What is being compared?

–Independent variables

  • What are they being compared in?

–Dependent variables (“metrics”)

  • What (else) is being varied? What is kept constant?

–Extraneous variables

4

slide-10
SLIDE 10

University of Washington

Designing an empirical study

  • What is being compared?

–Independent variables

  • What are they being compared in?

–Dependent variables (“metrics”)

  • What (else) is being varied? What is kept constant?

–Extraneous variables

4

slide-11
SLIDE 11

University of Washington

What is being compared?

5

“conditions”

slide-12
SLIDE 12

University of Washington

What is being compared?

5

Independent variable

interval

  • rdinal

categorical

Continuous values Ordered discrete values Unordered discrete values

“conditions”

slide-13
SLIDE 13

University of Washington

What is being compared?

6

  • Example: Interval independent variable

–What is the effect of height on telepresence systems?

Rae et al.

slide-14
SLIDE 14

University of Washington

Robotic telepresence

7

slide-15
SLIDE 15

University of Washington

What is being compared?

8

  • Example: Interval independent variable

–What is the effect of height on telepresence systems?

Rae et al.

slide-16
SLIDE 16

University of Washington

What is being compared?

9

  • Example: Ordinal independent variable

–What is the effect of educational background on acceptance of robots in the workplace?

Rae et al.

high school < college < graduate degree

slide-17
SLIDE 17

University of Washington

What is being compared?

10

  • Example: Categorical independent variable

–What is the effect of input modality on telepresence systems?

Rae et al.

–keyboard –mouse –joystick

slide-18
SLIDE 18

University of Washington

Within-subject vs. between subject

11

Same participant Participant-1 Participant-2

within between

slide-19
SLIDE 19

University of Washington

Within-subject vs. between subject

11

Same participant Participant-1 Participant-2

within between

+ allows comparison + requires less participants

  • subject to ordering effects
slide-20
SLIDE 20

University of Washington

Within-subject vs. between subject

11

Same participant Participant-1 Participant-2

within between

+ allows comparison + requires less participants

  • subject to ordering effects

> Order counterbalancing

slide-21
SLIDE 21

University of Washington

Designing an empirical study

  • What is being compared?

–Independent variables

  • What are they being compared in?

–Dependent variables (“metrics”)

  • What (else) is being varied? What is kept constant?

–Extraneous variables

12

slide-22
SLIDE 22

University of Washington

Independent vs. dependent variable

13

  • Example:

–What is the effect of height on telepresence systems?

Rae et al.

in terms of what?

slide-23
SLIDE 23

(subjective)

(objective)

Data Source Data type

University of Washington

What to measure or observe?

14

slide-24
SLIDE 24

(subjective)

(objective)

Data Source Data type

University of Washington

What to measure or observe?

14

How accurately is information remembered? How highly do participants rate the system? What frustrated the participants? What were the communication challenges?

slide-25
SLIDE 25

University of Washington

Dependent variables

15

what people do.. what people say..

slide-26
SLIDE 26

University of Washington

What is being measured?

16

  • Example: Interval dependent variable

–What is the effect of height on conversation control?

Rae et al.

  • ratio of time speaking
  • ratio of decisions influenced
  • self assessment of control

...

slide-27
SLIDE 27

University of Washington

What is being measured?

17

  • Example: Ordinal dependent variable

–What is the effect of height on user preference?

Rae et al.

  • user rating of the system
slide-28
SLIDE 28

University of Washington

What is being measured?

18

  • Example: Categorical dependent variable

–What is the effect of height on conversation control?

Rae et al.

  • choose one:

“I felt like the leader” “I felt like the follower”

slide-29
SLIDE 29

University of Washington

Designing an empirical study

  • What is being compared?

–Independent variables

  • What are they being compared in?

–Dependent variables (“metrics”)

  • What (else) is being varied?
  • (What is kept constant?)

–Extraneous variables

19

slide-30
SLIDE 30

University of Washington

Extraneous variables

  • Similar to independent variables but we are not

looking for an effect

–What is the effect of on conversation control?

20

  • things that vary unless you control for them

gender, age, background of participants

  • things that you explicitly vary to demonstrate lack of effect

tasks performed using the system

slide-31
SLIDE 31

University of Washington

Interpreting the results

  • What is being compared?

–Independent variables

  • What are they being compared in?

–Dependent variables (“metrics”)

21

slide-32
SLIDE 32

University of Washington

Interpreting the results

  • What is being compared?

–Independent variables

  • What are they being compared in?

–Dependent variables (“metrics”)

21

Does <independent variable> cause differences in <dependent variable>? Main question:

slide-33
SLIDE 33

University of Washington

Interpreting the results

22

Does height effect ratio of time speaking?

Yes/No?

slide-34
SLIDE 34

University of Washington

Analyzing the data

  • Factors

–Within vs. between groups –Number of variables –Type of dependent variables –Type of independent variables

23

slide-35
SLIDE 35

University of Washington

A common case: A/B testing

  • Two categorical independent variables (A vs. B)
  • One interval dependent variable

–key performance indicator

24

A: control B: treatment A B

key performance indicator

T-Test

slide-36
SLIDE 36

University of Washington

(Student’s) T-tests

  • Check if two means (averages) are reliably

different from each other

–t = (variance between groups)/(variance within groups) –Large t means different groups –Small t means similar groups

25

slide-37
SLIDE 37

University of Washington

(Student’s) T-tests Example

26

https://www.youtube.com/watch?v=0Pd3dc1GcHc

slide-38
SLIDE 38

University of Washington

(Student’s) T-tests Example

27

slide-39
SLIDE 39

University of Washington

(Student’s) T-tests Example

28

t = 2/6

slide-40
SLIDE 40

University of Washington

(Student’s) T-tests

29

p-value: probability that our data could be produced randomly

slide-41
SLIDE 41

University of Washington

(Student’s) T-tests

29

p-value: probability that our data could be produced randomly

p<0.05

slide-42
SLIDE 42

University of Washington

(Student’s) T-tests

29

p-value: probability that our data could be produced randomly

p<0.05

This means that there is only a 5% chance that there is no real difference between the two groups.

slide-43
SLIDE 43

University of Washington

(Student’s) T-tests

30

p-value: probability that our data could be produced randomly

–depends on number of participants

slide-44
SLIDE 44

University of Washington

(Student’s) T-tests

30

p-value: probability that our data could be produced randomly

bigger samples help but with diminishing returns

–depends on number of participants

slide-45
SLIDE 45

University of Washington

Types of t-tests

31

“independent” “unpaired” “between samples” “dependent” “paired” “within subjects” “repeated measures”

slide-46
SLIDE 46

University of Washington

Limitations of t-tests

  • Generalizes to similar population
  • Assumes that your data has Normal (Gaussian)

distribution

  • Sample size should be roughly the same
  • All data should be independent/ not influenced by

each other

  • Interval type variables (will not work for rankings)

32

slide-47
SLIDE 47

University of Washington

Lots of statistical tools available

33

http://www.graphpad.com/quickcalcs/ttest1.cfm

slide-48
SLIDE 48

University of Washington

Which statistical test to use?

34

http://www.ats.ucla.edu/stat/mult_pkg/whatstat/

slide-49
SLIDE 49

University of Washington

Comparisons in observational studies

35

Observational study Comparative study Think-aloud protocol

hmmmm blah blah blah bla

Post-hoc analysis

slide-50
SLIDE 50

University of Washington

A/B testing

36

slide-51
SLIDE 51

University of Washington

A/B testing example

37

A: No recommendations at checkout B: Recommendations based on cart content

Pro: cross-sell more items Con: distract people at check out

B wildly successful!

slide-52
SLIDE 52

University of Washington

A/B testing example

38

A B

Solitaire Poker

slide-53
SLIDE 53

University of Washington

A/B testing example

38

A B

Solitaire Poker A is 61% better!

slide-54
SLIDE 54

University of Washington

A/B testing example

39

A B

Ask why by default Ask why if user gives rating

slide-55
SLIDE 55

University of Washington

A/B testing example

39

A B

Ask why by default Ask why if user gives rating More than double response rate!

slide-56
SLIDE 56

University of Washington

A/B testing example

40

C

Ask a different question based on step1

slide-57
SLIDE 57

University of Washington

A/B testing example

40

C

Ask a different question based on step1 C outperforms B by a factor of 3.5!

slide-58
SLIDE 58

University of Washington

Limitations of A/B testing

  • Hill climbing, will not re-invent anything

41