Maya Cakmak, Matt Kay, Brad Jacobson, King Xia EMPIRICAL USER-STUDIES human-computer interaction CSE 440 WINTER 2015 University of FEB 19 - WEEK 7 - THURSDAY Washington
Methods for observing interaction hmmmm blah blah blah bla Passive observation Think-aloud protocol Comparative study Last week University of 2 Washington
Methods for observing interaction hmmmm blah blah blah bla Passive observation Think-aloud protocol Comparative study “Empirical user study” Last week “Controlled experiment” Today University of 2 Washington
Evaluation Techniques (re-cap) •Asking users –Questionnaires, interviews, focus groups •Observing users –Passive observation, think-aloud protocol, ethnography, empirical user studies •Make users observe themselves –Diaries, experience sampling •Ask experts –Heuristic evaluation, cognitive walkthrough University of 3 Washington
Evaluation Techniques (re-cap) •Asking users –Questionnaires, interviews, focus groups •Observing users –Passive observation, think-aloud protocol, ethnography, empirical user studies •Make users observe themselves –Diaries, experience sampling •Ask experts –Heuristic evaluation, cognitive walkthrough University of 3 Washington
Designing an empirical study University of 4 Washington
Designing an empirical study •What is being compared? –Independent variables University of 4 Washington
Designing an empirical study •What is being compared? –Independent variables •What are they being compared in? –Dependent variables (“metrics”) University of 4 Washington
Designing an empirical study •What is being compared? –Independent variables •What are they being compared in? –Dependent variables (“metrics”) •What (else) is being varied? What is kept constant? –Extraneous variables University of 4 Washington
Designing an empirical study •What is being compared? –Independent variables •What are they being compared in? –Dependent variables (“metrics”) •What (else) is being varied? What is kept constant? –Extraneous variables University of 4 Washington
What is being compared? “conditions” University of 5 Washington
What is being compared? interval Continuous values Independent variable ordinal Ordered discrete values categorical Unordered discrete values “conditions” University of 5 Washington
What is being compared? •Example: Interval independent variable –What is the effect of height on telepresence systems? Rae et al. University of 6 Washington
Robotic telepresence University of 7 Washington
What is being compared? •Example: Interval independent variable –What is the effect of height on telepresence systems? Rae et al. University of 8 Washington
What is being compared? •Example: Ordinal independent variable –What is the effect of educational background on acceptance of robots in the workplace? high school < college < graduate degree Rae et al. University of 9 Washington
What is being compared? •Example: Categorical independent variable –What is the effect of input modality on telepresence systems? – keyboard – mouse – joystick Rae et al. University of 10 Washington
Within-subject vs. between subject Participant-1 Participant-2 Same participant within between University of 11 Washington
Within-subject vs. between subject Participant-1 Participant-2 Same participant within between + allows comparison + requires less participants - subject to ordering effects University of 11 Washington
Within-subject vs. between subject Participant-1 Participant-2 Same participant within between + allows comparison + requires less participants - subject to ordering effects > Order counterbalancing University of 11 Washington
Designing an empirical study •What is being compared? –Independent variables •What are they being compared in? –Dependent variables (“metrics”) •What (else) is being varied? What is kept constant? –Extraneous variables University of 12 Washington
Independent vs. dependent variable •Example: –What is the effect of height on telepresence systems? in terms of what? Rae et al. University of 13 Washington
What to measure or observe? (objective) Data Source (subjective) Data type University of 14 Washington
What to measure or observe? What were the How accurately is communication information (objective) challenges? remembered? Data Source How highly do What frustrated participants rate the participants? the system? (subjective) Data type University of 14 Washington
Dependent variables what people do.. what people say.. University of 15 Washington
What is being measured? •Example: Interval dependent variable –What is the effect of height on conversation control? -ratio of time speaking -ratio of decisions influenced -self assessment of control ... Rae et al. University of 16 Washington
What is being measured? •Example: Ordinal dependent variable –What is the effect of height on user preference? -user rating of the system Rae et al. University of 17 Washington
What is being measured? •Example: Categorical dependent variable –What is the effect of height on conversation control? -choose one: “I felt like the leader” “I felt like the follower” Rae et al. University of 18 Washington
Designing an empirical study •What is being compared? –Independent variables •What are they being compared in? –Dependent variables (“metrics”) •What (else) is being varied? •(What is kept constant?) –Extraneous variables University of 19 Washington
Extraneous variables •Similar to independent variables but we are not looking for an effect –What is the effect of on conversation control? - things that vary unless you control for them gender, age, background of participants - things that you explicitly vary to demonstrate lack of effect tasks performed using the system University of 20 Washington
Interpreting the results •What is being compared? –Independent variables •What are they being compared in? –Dependent variables (“metrics”) University of 21 Washington
Interpreting the results •What is being compared? –Independent variables •What are they being compared in? –Dependent variables (“metrics”) Main question: Does <independent variable> cause differences in <dependent variable>? University of 21 Washington
Interpreting the results Does height effect ratio of time speaking? Yes/No? University of 22 Washington
Analyzing the data •Factors –Within vs. between groups –Number of variables –Type of dependent variables –Type of independent variables University of 23 Washington
A common case: A/B testing •Two categorical independent variables (A vs. B) •One interval dependent variable –key performance indicator A: control key performance indicator B: treatment T-Test A B University of 24 Washington
(Student’s) T-tests •Check if two means (averages) are reliably different from each other –t = (variance between groups)/(variance within groups) –Large t means different groups –Small t means similar groups University of 25 Washington
(Student’s) T-tests Example https://www.youtube.com/watch?v=0Pd3dc1GcHc University of 26 Washington
(Student’s) T-tests Example University of 27 Washington
(Student’s) T-tests Example t = 2/6 University of 28 Washington
(Student’s) T-tests p-value: probability that our data could be produced randomly University of 29 Washington
(Student’s) T-tests p-value: probability that our data could be produced randomly p<0.05 University of 29 Washington
(Student’s) T-tests p-value: probability that our data could be produced randomly p<0.05 This means that there is only a 5% chance that there is no real difference between the two groups. University of 29 Washington
(Student’s) T-tests p-value: probability that our data could be produced randomly –depends on number of participants University of 30 Washington
(Student’s) T-tests p-value: probability that our data could be produced randomly –depends on number of participants bigger samples help but with diminishing returns University of 30 Washington
Types of t-tests “independent” “dependent” “unpaired” “paired” “between samples” “within subjects” “repeated measures” University of 31 Washington
Limitations of t-tests •Generalizes to similar population •Assumes that your data has Normal (Gaussian) distribution •Sample size should be roughly the same •All data should be independent/ not influenced by each other •Interval type variables (will not work for rankings) University of 32 Washington
Lots of statistical tools available http://www.graphpad.com/quickcalcs/ttest1.cfm University of 33 Washington
Which statistical test to use? http://www.ats.ucla.edu/stat/mult_pkg/whatstat/ University of 34 Washington
Recommend
More recommend
