CyLab A Case Study on the Role of Usability Studies in Developing - - PowerPoint PPT Presentation

cylab
SMART_READER_LITE
LIVE PREVIEW

CyLab A Case Study on the Role of Usability Studies in Developing - - PowerPoint PPT Presentation

CyLab A Case Study on the Role of Usability Studies in Developing Public Engineering & Policy Public Policy Rebecca Balebako, Richard Shay, Lorrie Faith Cranor y & c S a e v c i u r P r i t e y l b L a a s


slide-1
SLIDE 1

1

C y L a b U s a b l e P r i v a c y & S e c u r i t y L a b

  • r

a t

  • r

y H T T P : / / C U P S . C S . C M U . E D U

Engineering & Public Policy

CyLab

A Case Study on the
 Role of Usability Studies in Developing Public
 Policy Rebecca Balebako, Richard Shay, Lorrie Faith Cranor

slide-2
SLIDE 2

WANTED: USABILITY EXPERTS

  • Usability experts are needed to help create and

evaluate public policy

  • Voting machines
  • Accessibility
  • Privacy and Security
  • I offer some lessons learned

2

slide-3
SLIDE 3

RECENT POLICY: WHITE HOUSE

3

slide-4
SLIDE 4

NTIA: MOBILE APPLICATION TRANSPARENCY

4

slide-5
SLIDE 5

MULTI-STAKEHOLDER PROCESS (MSHP)

  • Open meetings
  • Monthly
  • Stakeholders
  • App development companies
  • Consumer-advocate non-profits
  • Privacy lawyers

5

slide-6
SLIDE 6

NTIA CODE OF CONDUCT

  • Goal: Short-form privacy notice for apps
  • Inform app users about data collection
  • Improve transparency
  • Standardized notice

6

slide-7
SLIDE 7

NTIA CODE OF CONDUCT

  • Short form notice must inform users about
  • 7 Data Types
  • 8 Third-Party Entities

7

slide-8
SLIDE 8
  • Biometrics (information about your body, including fingerprints, facial

recognition, signatures and/or voice print.)

  • Browser History and Phone or Text Log (A list of websites visited, or the calls or

texts made or received.)

  • Contacts (including list of contacts, social networking connections or their

phone numbers, postal, email and text addresses.)

  • Financial Information (Includes credit, bank and consumer-specific financial

information such as transaction data.)

  • Health, Medical or Therapy Information (including health claims and

information used to measure health or wellness.)

  • Location (precise past or current location and history of where a user has

gone.)

  • User Files (files stored on the device that contain your content, such as

calendar, photos, text, or video.)

DATA TYPES

8

slide-9
SLIDE 9
  • Biometrics (information about your body, including fingerprints, facial

recognition, signatures and/or voice print.)

  • Browser History and Phone or Text Log (A list of websites visited, or the calls or

texts made or received.)

  • Contacts (including list of contacts, social networking connections or their

phone numbers, postal, email and text addresses.)

  • Financial Information (Includes credit, bank and consumer-specific financial

information such as transaction data.)

  • Health, Medical or Therapy Information (including health claims and

information used to measure health or wellness.)

  • Location (precise past or current location and history of where a user has

gone.)

  • User Files (files stored on the device that contain your content, such as

calendar, photos, text, or video.)

DATA TYPES

9

slide-10
SLIDE 10
  • Biometrics (information about your body, including fingerprints, facial

recognition, signatures and/or voice print.)

  • Browser History and Phone or Text Log (A list of websites visited, or the calls or

texts made or received.)

  • Contacts (including list of contacts, social networking connections or their

phone numbers, postal, email and text addresses.)

  • Financial Information (Includes credit, bank and consumer-specific financial

information such as transaction data.)

  • Health, Medical or Therapy Information (including health claims and

information used to measure health or wellness.)

  • Location (precise past or current location and history of where a user has

gone.)

  • User Files (files stored on the device that contain your content, such as

calendar, photos, text, or video.)

DATA TYPES

10

slide-11
SLIDE 11

THIRD-PARTY ENTITIES

  • Ad Networks (Companies that display ads to you through apps.)
  • Carriers (Companies that provide mobile connections.)
  • Consumer Data Resellers (Companies that sell consumer information to other

companies for multiple purposes including offering products and services that may interest you.)

  • Data Analytics Providers (Companies that collect and analyze your data.)
  • Government Entities (Any sharing with the government except where required
  • r expressly permitted by law.)
  • Operating Systems and Platforms (Software companies that power your

device, app stores, and companies that provide common tools and information for apps about app consumers.)

  • Other Apps (Other apps of companies that the consumer may not have a

relationship with)

  • Social Networks (Companies that connect individuals around common

interests and facilitate sharing.)

11

slide-12
SLIDE 12

THIRD-PARTY ENTITIES

  • Ad Networks (Companies that display ads to you through apps.)
  • Carriers (Companies that provide mobile connections.)
  • Consumer Data Resellers (Companies that sell consumer information to other

companies for multiple purposes including offering products and services that may interest you.)

  • Data Analytics Providers (Companies that collect and analyze your data.)
  • Government Entities (Any sharing with the government except where required
  • r expressly permitted by law.)
  • Operating Systems and Platforms (Software companies that power your

device, app stores, and companies that provide common tools and information for apps about app consumers.)

  • Other Apps (Other apps of companies that the consumer may not have a

relationship with)

  • Social Networks (Companies that connect individuals around common

interests and facilitate sharing.)

12

slide-13
SLIDE 13

FRAGILE AGREEMENT

13

slide-14
SLIDE 14

USABILITY TEST SUBGROUP

  • There was no consensus in the usability group with

regard to the following:

  • Is any of the actual language of the Code subject

to testing for consumer comprehension?

14

slide-15
SLIDE 15

EXPERIMENT TO EVALUATE THE UNDERSTANDING OF THE CODE OF CONDUCT TERMS

Rebecca Balebako, Rich Shay, Lorrie Faith Cranor

15

slide-16
SLIDE 16

ONLINE SURVEY

  • 10 randomized app scenarios
  • Users selected the data and entities shared in each

scenario

  • 2 conditions – with and without parentheticals

16

slide-17
SLIDE 17

SCENARIO EXAMPLE

17

slide-18
SLIDE 18

PARENTHETICAL CONDITION

18

slide-19
SLIDE 19

SURVEY PARTICIPANTS

  • 791 participants from Amazon mturk
  • 51% female
  • Age 18-73 years (mean 33, std 11)
  • 82% own a smartphone
  • Total cost: $913.35

19

slide-20
SLIDE 20

WHAT IS THE RIGHT ANSWER?

  • Ask the Experts – NTIA MSHP participants
  • 4 participated
  • Low agreement amongst experts
  • All 4 agreed on 8/19 entities
  • All 4 agreed on 16/34 data types

20

slide-21
SLIDE 21

PARTICIPANT RESULTS

  • Used ‘common understanding’
  • Winning term
  • High common understanding:
  • >60% of participants agreed on the winning term
  • Low common understanding
  • <60% of participants agreed

21

slide-22
SLIDE 22

COMMON UNDERSTANDING THIRD PARTIES

22

SuperTax: State Agency

slide-23
SLIDE 23

COMMON UNDERSTANDING THIRD PARTIES

23

slide-24
SLIDE 24

24

slide-25
SLIDE 25

25

SuperTax: Photo of W2

With parenthetical

User Files (files stored on the device that contain your content, such as calendar, photos, text, or video.)

slide-26
SLIDE 26

USER STUDY RESULTS

  • Parenthetical text helped sometimes
  • Third-Party entities are poorly understood.
  • Better definitions are needed

26

slide-27
SLIDE 27

IMPACT

  • Technical report released July 17, 2013
  • Final NTIA MSHP meeting July 25th, 2013

27

slide-28
SLIDE 28

PUBLIC POLICY FOR USABILITY EXPERTS

  • Disagreement about what ‘usability’ is.
  • Cost of usability studies impacts what gets studied

and when.

  • Process fatigue; the timeline to solve a problem is

different than in academia.

  • Engage early.

28

slide-29
SLIDE 29

29

B A L E B A K O @ C M U . E D U

QUESTIONS?

slide-30
SLIDE 30

LIMITATIONS

30

  • No ground truth
  • Did not test better or alternative wording
  • Not part of the typical flow for users
  • Short form was not actually tested
  • Final Code of Conduct was announced one week

after tech report was released

slide-31
SLIDE 31

PROTOTYPE

31

slide-32
SLIDE 32

CURRENT INTERFACES

32

slide-33
SLIDE 33

COMMON UNDERSTANDING THIRD PARTIES

33

slide-34
SLIDE 34

COMMON UNDERSTANDING DATA TYPES

34

slide-35
SLIDE 35

COMMON UNDERSTANDING DATA TYPES

35

slide-36
SLIDE 36

COMMON UNDERSTANDING DATA TYPES

36

SuperTax: Photo of W2

With parenthetical

User Files (files stored on the device that contain your content, such as calendar, photos, text, or video.)

slide-37
SLIDE 37

PROTOTYPE FROM ACT

37