Surveys, interviews, and diary studies Michelle Mazurek (some - - PowerPoint PPT Presentation

1

Surveys, interviews, and diary studies

Michelle Mazurek

(some slides adapted from Blase Ur, Lorrie Cranor, and Rich Shay)

2

Today’s class

• Logistics: project groups and proposals
• Surveys
• Crowdsourcing (Mechanical Turk)
• Interviews
• Diary and ESM studies

3

Logistics

• Project groups
• Project proposals:

https://myelms.umd.edu/courses/1134544/ pages/course-project

• “Talk talk” this afternoon: 2pm CSIC 3117

4

SURVEYS

5

Why a survey?

• A little bit of data (each) from a lot of people
• Quantitative results

– Generalizable if done correctly

• Quick, easy, unobtrusive, relatively cheap
• Shallow data

– Multiple choice, short free-response

• Biases: self-reported, question/answer order, etc.

6

Survey best practices

• Pilot, pilot, pilot!

– Ensure questions are neutral, are not ambiguous – Test different question wordings

• Consider your sample
• Include attention checks
• Don’t make it too long

– No shortcuts (branch questions equally)

• Offer option not to answer (avoid lying)

7

Try it!

In groups of 2-3, write a 5-question survey about privacy for student records.

8

CROWDSOURCED STUDIES

Mechanical Turk and friends

9

Why crowdsourcing?

• Many participants, geographically distributed

– More diverse than students *

• Easy to recruit, screen, assign conditions, pay
• Most popular: Mechanical Turk

– Others: Crowdflower, Crowdsource.com, Samasource

10

How it works

Task Requester Worker Worker Worker $ $ $

✕

http://upload.wikimedia.org/wikipedia/commons/4/40/Turk-with-person.jpg

11

Limitations and risks

• Can’t observe participants or follow up

– Piloting is especially important

• Some users enter garbage

– Collect lots of data – Pay more than average – Don’t provide a “shortcut path” – Use quality checks: trivial, nonsense, repeats

12

Limitations and risks

• Population: Young, tech-savvy, many from India

– Buhr 2011, Ipeirotis 2010, Ross 2010, others – Can restrict to American IPs when necessary

• Measures to prevent repeat participants

– Cookies, IP tracking, MTurk ID list – Especially if you pay well

• Turker discussion boards

– If your study is game-able, will be reported to others

13

Details and procedures

• Short recruitment text
• Often, link to external task

– Built-in features are limited – Survey in qualtrics (https://umd.az1.qualtrics.com/) – Custom task site you built – CMU management infrastructure: SHELF

• You still need a consent form

– I have a sample

14

Paying participants

• When the participant has finished, you notify

MTurk and the participant is paid

– Important for your homeworks!

• Payment is taken from prepaid MTurk account.

15

Other useful features

• Screen and reject workers

– Location, quality rating

• Send notifications (e.g. to come back for part 2)
• Prevent repeated workers in the same task

– May need multiple tasks per study

• On average, 100 participants / day

– Starts faster, slows down, repost

16

Twitter regrets (Sleeper et al.)

• Mturk survey of 1,221 participants
• Compared conversational and Twitter regrets
• Emotional state, awareness, repair strategies

17

Twitter regrets

• Note the research questions in the introduction
• Why did they screen for Twitter users age 18+ in

the USA?

• Is conversational regret the right parallel?
• How was Mturk quality control done?
• How was the data coded?
• Limitations

18

INTERVIEWS

19

Why an interview

• Rich data (from fewer people)
• Good for exploration

– When you aren’t sure what you’ll find – Helps identify themes, gain new perspectives

• Usually cannot generalize quantitatively
• Potential for bias (conducting, analyzing)
• Structured vs. semi-structured

20

Interview best practices

• Make participants comfortable
• Avoid leading questions
• Support whatever participants say

– Don’t make them feel incorrect or stupid

• Know when to ask a follow-up
• Get a broad range of participants (hard)

21

DIARY STUDIES

22

Why do a diary study?

• Rich longitudinal data (from a few participants)

– In the field … ish

• Natural reactions and occurences

– Existence and quantity of phenomena – User reactions in the moment rather than via recall

• Lots of work for you and your participants
• On paper vs. technology-mediated

23

Experience sampling

• Kind of a prompted diary
• Send participants a stimulus when they are in

their natural life, not in the lab

24

Diary / ESM best practices

• When will an entry be recorded?

– How often? Over what time period?

• How long will it take to record an entry?

– How structured is the response?

• Pay well

– Pay per response, but don’t create bias

25

Facebook regrets (Wang et al.)

• Online survey, interviews, diary study, 2nd survey
• What do people regret posting? Why?
• How do users mitigate?

26

FB regrets – Interviews

• Semi-structured, in-person, in-lab
• Recruiting via Craigslist

– Why pre-screen questionnaire? – 19/301

• How were they coded?

27

FB regrets – Diary study

• 12 of 19 participants from the interview

participated at least one day

• Facebook activities, incidents
• Online form, open-ended questions

– “Have you changed anything in your privacy settings? What and why?” – “Have you posted something on Facebook and then regretted doing it? Why and what happened?” – 22+ days of entries: $15

28

Location-sharing (Consolvo et al.)

• Whether and what about location to disclose

– To people you know

• Preliminary interview

– Buddy list, expected preferences

• Two-week ESM (simulated location requests)
• Final interview to reflect on experience

29

ESM study

• Whether to disclose or not, and why

– Customized askers, customized context questions – If so, how granular? – Where are you and what are you doing? – One-time or standing request

• $60-$250 to maximize participation
• Average response rate: above 90%