Field and diary studies Michelle Mazurek Some slides adapted from - - PowerPoint PPT Presentation

field and diary studies
SMART_READER_LITE
LIVE PREVIEW

Field and diary studies Michelle Mazurek Some slides adapted from - - PowerPoint PPT Presentation

Field and diary studies Michelle Mazurek Some slides adapted from Lorrie Cranor, Blase Ur, Vibha Sazawal 1 Administrative HW1 almost done grading HW2 out soon Course project discussion soon! (2/28) 2 FIELD STU FI TUDIES 3 What


slide-1
SLIDE 1

1

Field and diary studies

Michelle Mazurek

Some slides adapted from Lorrie Cranor, Blase Ur, Vibha Sazawal

slide-2
SLIDE 2

2

Administrative

  • HW1 almost done grading
  • HW2 out soon
  • Course project discussion soon! (2/28)
slide-3
SLIDE 3

3

FI FIELD STU TUDIES

slide-4
SLIDE 4

4

What is a field study?

  • Study (observation) vs. (quasi) experiment

(treatment)

  • Observations from uncontrolled but real(istic)

conditions

slide-5
SLIDE 5

5

Why a field study?

  • Observation:

– As inductive hypothesis generation

  • Experiment:

– Better ecological validity – Validate a lab study result

  • Because you can’t get the data any other way!
slide-6
SLIDE 6

6

Why not a field study?

  • Logistically difficult
  • Limited piloting / not easy to adjust

– One shot at your participant pool

  • Expensive (money and time)
  • Researcher may influence outcomes

Plan extremely carefully!

slide-7
SLIDE 7

7

Field observation

  • Thorough, systematic classification of events +

relationships

  • Ethnography: Rich, qualitative description;
  • quotes. ``Embedding.”
  • Enumeration: Counting things (e.g., web-scale
  • measurements. Could also do in person.)
slide-8
SLIDE 8

8

Important questions

  • Who/what to observe, and why?

– Expectation of interesting-ness – How is your observation biased (who/what don’t you

  • bserve?)
  • For how long?

– Trad. Ethnography: years of embedding – HCI: “rapid” or “mini” ethnography

  • Researcher role?

– Observer. Participant? (Consider e.g. dark web)

slide-9
SLIDE 9

9

Researcher as participant

  • Not expert or adviser!
  • Observe before involving/asking too much
  • Detailed field notes!

– Who, what, where, when – Quotes – Anything confusing

  • … ethnography, lots of details we won’t get into

in terms of how to do it correctly

slide-10
SLIDE 10

10

Field experiments

  • Logistically, harder to pull off

– Unless you are for example Facebook – Can find smaller instances (usually quasi) – Occasional before/after instances (software patching, encryption stuff)

  • Consider impact of being observed (if known)
  • Consider biases in who/what is measured
slide-11
SLIDE 11

11

PhishGuru in the real world

  • Anti-phishing training delivered when users

follow a phishing link

  • Training, phishing, legitimate emails delivered to

300 employees in a Portuguese company

– Over several weeks

  • Why a field study, is this necessary?

Kumaraguru et al, eCrime Summit 2008

slide-12
SLIDE 12

12

Logistical problems

  • Didn’t include a legitimate email before training

to compare click rates

  • Control and experimental not 100% parallel
  • Participants talked to each other, sharing the

training materials

  • No one turned in the post-study questionnaire!
  • How could these have been avoided?
slide-13
SLIDE 13

13

DIA DIARY ST STUDIE DIES

Another flavor of field study … ish

slide-14
SLIDE 14

14

Why do a diary study?

  • Rich longitudinal data (from a few participants)

– In the field … ish

  • Natural reactions and occurences

– Existence and quantity of phenomena – User reactions in the moment rather than via recall – Less social desirability, maybe

slide-15
SLIDE 15

15

How to do a diary study?

  • Interval-contingent

– Will they remember?

  • Signal-contingent

– Reminder but intrusive

  • Event-contingent

– Also memory issues – Can be best depending on topic at hand

  • Paper vs technology-mediated

– Collect some things automatically!

slide-16
SLIDE 16

16

Experience sampling (ESM)

  • A kind of signal-contingent diary
  • Can be a reminder to create an entry; can create

an event to respond to

  • Send participants a stimulus when they are in

their natural life, not in the lab

slide-17
SLIDE 17

17

Lots of work!

  • Often training and briefing/debriefing
  • Consistent work over time
  • Requires commitment from participants
slide-18
SLIDE 18

18

Diary / ESM best practices

  • When will an entry be recorded?

– How often? Over what time period?

  • How long will it take to record an entry?

– How structured is the response?

  • How long is the commitment?
  • Stress motivation / completeness
  • Pay well

– Pay per response? But don’t create bias

slide-19
SLIDE 19

19

Screen time study (from HW1)

  • What worked and what didn’t?
  • Thoughts about experimental design?
slide-20
SLIDE 20

20

Other diary references

  • Facebook regrets -- Wang et al, SOUPS 2011
  • Location sharing (ESM) – Consolvo et al, CHI

2005

slide-21
SLIDE 21

26

Activity

  • In small groups

– Design a field or diary study – Think about logistical/planning issues