CMSC 20370/30370 Winter 2020 Evaluation Quantitative Methods Case - - PowerPoint PPT Presentation

CMSC 20370/30370 Winter 2020 Evaluation – Quantitative Methods Case Study: Accessibility

Jan 22, 2020

Quiz Time (5-7 minutes).

Quiz on Sound Awareness for Deaf and Hard

• f Hearing Users

Principles of Good Design

Administrivia

• IA2 due on Friday
• IA2 rubric on Piazza and course website
• Tentative schedule for group proposal

presentations will be posted on Friday

• More information on presentation format

provided on Friday

– Everyone is expected to attend each session – Everyone is expected to present in at least part of the presentation

Today’s Agenda

• Evaluating your design/prototype/system

– Usability testing – Inspection methods – Qualitative techniques

USER NEEDS DESIGN/PROTOTYPE IMPLEMENT EVALUATE USER-CENTERED DESIGN

Case Study: Sound Awareness

• Deaf and hard of hearing users
• Based on interviews with 12 DHH users in Study 1
• In Study 2, conducted Wizard of Oz lab study with 10

DHH users and three initial sound awareness prototypes

• Recommendations for sound awareness for DHH

– E.g. integrate into daily routines – Shared space – Uncertainty – Form factors

• Limitations

From Dhruv Jain’s website

From Dhruv Jain’s website

What home sound awareness do DHH users desire?

How do DHH users react to sound awareness prototypes?

These questions require empirical evaluation – testing with real users

We could run experiments!

Case Study: Sound Awareness Setup

• Used faculty and student lounge to look

like a home (had kitchen area, bathroom, dining room, lounge area, and windows to outside)

• 1 hour session
• Background questionnaire
• Initial prototype demos
• Thematic scenarios
• Semi-structured interview

From Dhruv Jain’s website

From Dhruv Jain’s website

Initial demo

• 3 sets of everyday actions

– Starts microwave and does dishes – Knocks on door, opens, greets, sits down, door closes – Makes coffee, pours liquid, bird chirs

Thematic scenarios

• Bathroom scenario (privacy)
• Babysitter scenario (activity tracking)
• Movie scenario (information overload)
• Each designed to gather feedback on specific

aspect for design

• Also looked at uncertainty using mockups

Post-study

• Semi-structured interview
• Data analysis – thematic coding
• Design implications and discussion

From Lazar, Research Methods in HCI

What does that mean?

• In HCI, lab studies are not always strictly

experiments

– E.g. the case study is a lab study but not a controlled experiment – In these cases, there is no null hypothesis or alternative hypothesis – Instead it is exploratory – Still useful for inclusive technology – why?

In Class Exercise

• How would we turn this into a lab

experiment assuming we had 2 systems

– An existing sound awareness prototype – Our sound awareness prototype

Hypothesis

• Precise problem statement that can be

tested with empirical investigation

• Our prototype will improve sound

awareness for DHH users in the home

Control?

• If you’re claiming your system improves
• ver the current state of the art, you have

to benchmark against it

• Or benchmark against having no

intervention to show your system makes a difference

• Control could be: no sound awareness

condition

Sample Size?

• In HCI, people run sometimes statistical

tests with very small sample sizes

• However, statistical power increases with

sample size

• Use calculator
• Depends on resource, no of conditions e.g.

Control, existing system, prototype

– Three conditions, 10 users each, 30 users total

Within or Between Subjects?

• Within – each user experiences all conditions

– Each user tries control, existing system, and sound awareness system,

• Between – each user in only 1 condition,

compared against different users in other conditions

– Users either try control, existing system, or sound awareness

• Pros and Cons?

Randomization?

• Avoid Priming Effects

– Exposure to one stimulus influences response to another stimulus

• Avoid Practice Effects

– Might perform better in within subjects if by third condition you are more familiar with the system

• Assign to different conditions

Dependent and Independent Variables?

• Dependent variable is what you’re

measuring

– how accurately can users locate or identify sounds or become aware of sounds

• Independent variable is what you are

alternating e.g. type of system

Confounding factors?

• Factors influencing dependent and

independent variables

– Age, DHH spectrum, environment is not like familiar home

Designing Experiments

• Hypothesis
• Control
• Sample Size
• Within Subjects vs Between Subjects
• Randomization

– Avoid Priming Effects – Avoid Practice Effects – Assign to different conditions

• Dependent + Independent variables
• Confounding factors

Hawthorne Effect

• Participants may behave differently in lab-

based experiments due to being observed or rewards for participation

• “Hawthorne Effect”
• Landsberger 1958 – study of workers caused

improvement in worker productivity that slumped when study ended

• Assumed to be because of motivation from

being observed and interest in them

HCI and Stats 101

• Significance testing – checking is observed difference occurring by

chance?

• Compare means of 2 groups

– T-tests, ANOVA (between or within group tests or both)

• Identify relationships

– Correlation (2 variables) – Regression (1 dependent variable and multiple independent variables)

• Non-parametric measures for categorical or ordinal data etc

– CHI-squared – relationship between variables – Mann-Whitney – between groups – Wilcoxon signed-rank – within groups – Kruskal-Wallis one way ANOVA – three or more sets of data – Friedmans two ANOVA

Experiments: Pros and Cons?

Other ways to evaluate systems empirically

• Many HCI studies use Amazon Mechanical

Turk or Prolific

– Not as well suited for many underserved or marginalized users – May require more information about context

• f use/user in context

– may not be target users

Case Study: HomeSound @ CHI 2020

Summary

• In HCI, we use both controlled and laboratory studies
• Need to consider study design and goals carefully
• Need to use right sample size and statistical

analysis

Coming up next class

• Project team discussions
• Come to class

– Will share tentative presentation schedule and information about presentation format – Ensure that your group checks in with one of the TAs on your project progress – TAs have a short checkpoint form – Q&A with TAs

• Turn in IA 2

Get in touch:

Office hours: Fridays 2-4pm (Sign up in advance) or by appointment JCL 355 Email: marshini@uchicago.edu