Surveys, interviews, and diary studies Michelle Mazurek (some slides adapted from Blase Ur, Lorrie Cranor, and Rich Shay) 1
Today’s class • Logistics: project groups and proposals • Surveys • Crowdsourcing (Mechanical Turk) • Interviews • Diary and ESM studies 2
Logistics • Project groups • Project proposals: https://myelms.umd.edu/courses/1134544/ pages/course-project • “Talk talk” this afternoon: 2pm CSIC 3117 3
SURVEYS 4
Why a survey? • A little bit of data (each) from a lot of people • Quantitative results – Generalizable if done correctly • Quick, easy, unobtrusive, relatively cheap • Shallow data – Multiple choice, short free-response • Biases: self-reported, question/answer order, etc. 5
Survey best practices • Pilot, pilot, pilot! – Ensure questions are neutral, are not ambiguous – Test different question wordings • Consider your sample • Include attention checks • Don’t make it too long – No shortcuts (branch questions equally) • Offer option not to answer (avoid lying) 6
Try it! In groups of 2-3, write a 5-question survey about privacy for student records. 7
Mechanical Turk and friends CROWDSOURCED STUDIES 8
Why crowdsourcing? • Many participants, geographically distributed – More diverse than students * • Easy to recruit, screen, assign conditions, pay • Most popular: Mechanical Turk – Others: Crowdflower, Crowdsource.com, Samasource 9
How it works Requester Worker $ Worker Task $ Worker ✕ $ http://upload.wikimedia.org/wikipedia/commons/4/40/Turk-with-person.jpg 10
Limitations and risks • Can’t observe participants or follow up – Piloting is especially important • Some users enter garbage – Collect lots of data – Pay more than average – Don’t provide a “shortcut path” – Use quality checks: trivial, nonsense, repeats 11
Limitations and risks • Population: Young, tech-savvy, many from India – Buhr 2011, Ipeirotis 2010, Ross 2010, others – Can restrict to American IPs when necessary • Measures to prevent repeat participants – Cookies, IP tracking, MTurk ID list – Especially if you pay well • Turker discussion boards – If your study is game-able, will be reported to others 12
Details and procedures • Short recruitment text • Often, link to external task – Built-in features are limited – Survey in qualtrics (https://umd.az1.qualtrics.com/) – Custom task site you built – CMU management infrastructure: SHELF • You still need a consent form – I have a sample 13
Paying participants • When the participant has finished, you notify MTurk and the participant is paid – Important for your homeworks! • Payment is taken from prepaid MTurk account. 14
Other useful features • Screen and reject workers – Location, quality rating • Send notifications (e.g. to come back for part 2) • Prevent repeated workers in the same task – May need multiple tasks per study • On average, 100 participants / day – Starts faster, slows down, repost 15
Twitter regrets (Sleeper et al.) • Mturk survey of 1,221 participants • Compared conversational and Twitter regrets • Emotional state, awareness, repair strategies 16
Twitter regrets • Note the research questions in the introduction • Why did they screen for Twitter users age 18+ in the USA? • Is conversational regret the right parallel? • How was Mturk quality control done? • How was the data coded? • Limitations 17
INTERVIEWS 18
Why an interview • Rich data (from fewer people) • Good for exploration – When you aren’t sure what you’ll find – Helps identify themes, gain new perspectives • Usually cannot generalize quantitatively • Potential for bias (conducting, analyzing) • Structured vs. semi-structured 19
Interview best practices • Make participants comfortable • Avoid leading questions • Support whatever participants say – Don’t make them feel incorrect or stupid • Know when to ask a follow-up • Get a broad range of participants (hard) 20
DIARY STUDIES 21
Why do a diary study? • Rich longitudinal data (from a few participants) – In the field … ish • Natural reactions and occurences – Existence and quantity of phenomena – User reactions in the moment rather than via recall • Lots of work for you and your participants • On paper vs. technology-mediated 22
Experience sampling • Kind of a prompted diary • Send participants a stimulus when they are in their natural life, not in the lab 23
Diary / ESM best practices • When will an entry be recorded? – How often? Over what time period? • How long will it take to record an entry? – How structured is the response? • Pay well – Pay per response, but don’t create bias 24
Facebook regrets (Wang et al.) • Online survey, interviews, diary study, 2 nd survey • What do people regret posting? Why? • How do users mitigate? 25
FB regrets – Interviews • Semi-structured, in-person, in-lab • Recruiting via Craigslist – Why pre-screen questionnaire? – 19/301 • How were they coded? 26
FB regrets – Diary study • 12 of 19 participants from the interview participated at least one day • Facebook activities, incidents • Online form, open-ended questions – “Have you changed anything in your privacy settings? What and why?” – “Have you posted something on Facebook and then regretted doing it? Why and what happened?” – 22+ days of entries: $15 27
Location-sharing (Consolvo et al.) • Whether and what about location to disclose – To people you know • Preliminary interview – Buddy list, expected preferences • Two-week ESM (simulated location requests) • Final interview to reflect on experience 28
ESM study • Whether to disclose or not, and why – Customized askers, customized context questions – If so, how granular? – Where are you and what are you doing? – One-time or standing request • $60-$250 to maximize participation • Average response rate: above 90% 29
Recommend
More recommend
