A Multiple Method Approach to Testing a Complex Webbased - - PowerPoint PPT Presentation

A Multiple Method Approach to Testing a Complex Web‐based Establishment Survey Instrument

Aryn Hernandez Temika Holland Rebecca Keegan Amy Anderson Riemer

U.S. Census Bureau

July 12, 2019

1

Disclaimer: Any views expressed are those of the author and not necessarily those of the U.S. Census Bureau. The Census Bureau has reviewed this data product for unauthorized disclosure of confidential information and has approved the disclosure avoidance practices applied. (Approval ID: CBDRB‐FY19‐ESMD‐B00016)

Overview

• Economic Census Instrument Design Challenges
• Economic Census Instrument and Respondent Portal Features
• Multi‐year, Multi‐method Testing Approach Overview
• Usability Testing Highlights
• Economic Census Instrument
• Respondent Portal
• How‐to Videos
• Ongoing and Future Research

2

The Economic Census (EC)

• Every five years, referencing years ending in 2 and 7
• Mandatory, self‐administered establishment‐based survey
• Requests data about nearly 4 million establishments
• Massive multinational companies with 1000s of establishments, single location businesses, and

everything in between

• All industries
• All levels of computer literacy
• Collects a wide range of financial and production data
• For example: payroll, employment, revenue by product type, expenses, etc…
• Questions vary by industry classification
• Answering the survey can involve multiple people within a business to complete and typically

requires access to business records

• Data are used for economic benchmarks and by policy makers, federal agencies, trade

associations, businesses, and others

• Between 2012 and 2017, the survey instrument for the EC was radically transformed

from a downloaded software application to an entirely web‐based platform

3

The Centurion Instrument

4

• Retains much of the

downloadable software’s functionality, but…

• Entirely web‐based
• Cleaner, more modern
• More obvious navigation
• Clearly defined reporting steps

Any company data shown is dummy data.

Design Challenges (cont.)

• Concurrent with the development of the Centurion Instrument
• Issues with current economic survey methods:
• Assigning individual userIDs and passwords for each survey
• Different URL for each survey
• Evolving Security Standards
• Solution:
• One URL for all surveys
• One set of login credentials

5

The Respondent Portal

6

Any company data shown is dummy data.

The Respondent Portal

• Respondents…
• must create their own accounts
• must link the survey to their accounts
• can view survey Information
• can request extensions
• can access secure messaging center
• can share survey access with others in
• rganization

7

Before Portal After Portal

Survey Requests Multiple Survey Requests Business Respondent Portal, One Login One Portal, One Login Survey Requests Multiple Survey Requests Business Respondent Websites, Multiple Logins Multiple Websites, Multiple Logins

Multi‐Year, Multi‐Method Testing Overview

• Methodologies:
• Task analysis
• Cognitive testing
• Usability testing
• Paper prototypes, high‐fidelity prototypes
• Contact strategy experiments
• Respondent debriefings
• Paradata evaluations

8

Usability Testing Highlights

• Evaluated the web instrument and portal to identify issues and assess

the site’s ease of use

• In person, at respondents’ place of business
• Respondents provided feedback on components of the 2017 EC

electronic reporting system:

• 1. CPASS (Respondent Portal registration process)
• 2. Respondent Portal Prototype
• 3. Centurion prototype

9

Usability Testing Highlights

• Methodology:
• Introductions/Informed Consent
• Demographic Questionnaire
• User tasks
• Think aloud
• Probes
• Satisfaction survey (SUS)
• Debriefing

10

• Metrics:
• Effectiveness
• Percentage of successful completions for

each task

• Efficiency
• Task completion time
• Subjective satisfaction
• Participants’ satisfaction ratings
• Verbal and non‐verbal behavior

Usability Testing Highlights

• Centurion User Tasks:
• 1. Logging into the survey
• 2. Updating locations lists (Add/Delete Locations)
• 3. Locating printable worksheet
• 4. Beginning reporting
• 5. Downloading/Uploading the spreadsheet (Multi‐locations only)
• 6. Reviewing entered data and resolving issues
• 7. Submitting the completed survey

11

• Metrics:
• Effectiveness
• Efficiency
• Subjective satisfaction
• Verbal and non‐verbal

behavior

Usability Testing Highlights

• Modified System Usability Scale

1. I think that I would like to use the site to complete the Economic Census 2. I found the site to be simple 3. I thought the site was easy to use 4. I think that I could use the site without the support of a technical person 5. I found that the various functions in the site were well put together 6. I thought there was a lot of consistency in the site 7. I would imagine that most people would learn to use the site very quickly 8. I found the site to be very intuitive 9. I felt very confident using the site

• 10. I could use the site without having to learn anything

12

• Metrics:
• Effectiveness
• Efficiency
• Subjective satisfaction
• Verbal and non‐verbal

behavior

*Multi‐location respondent results (n=29)

Usability Testing Highlights

• Respondent Portal Tasks:

1) Create an account 2) Link a survey using authentication code 3) Locate and use the ‘Share Survey’ feature 4) Locate survey information 5) Locate the ‘Help’ button 6) Locate the secure messaging form 7) Request an extension 8) Check filing status

13

How‐To Videos

• Usability testing, debriefings, and paradata analysis revealed that

respondents do not always use available tools and features in the instrument

• To modernize our support and enhance user experience, we tested

short 'How‐to' videos for specific features and user actions

• Topics for these videos were chosen from the most difficult user tasks from usability

testing as well as the most common call center questions

• About 2 minutes or less in length
• Screen recordings with narration

14

Video Topics

• 1. Dashboard Overview
• 6. Messages Feature
• 2. Organizing Dashboard
• 7. Upload Spreadsheet
• 3. Print Preview
• 8. Reviewing Data
• 4. Add/Delete
• 9. Submitting Survey
• 5. Download Spreadsheet
• 10. Creating a Portal Account

How‐To Video Testing

• Respondents completed tasks similar to usability testing
• Respondents then viewed and rated the how‐to videos
• Finally, debriefing questions were asked
• Included internet experience questionnaire asking about average usage of how‐to

videos

• Videos were well liked and thought to be useful to future respondents
• Selected results from Internet Experience Questionnaire:
• How often do you seek out how‐to videos when you need assistance with a task?
• Average response: Somewhat often (4.3/7)
• How likely would you be to access a how‐to video if you’re learning a new skill?
• Average response: Pretty likely (5.3/7)
• Respondents reported watching an average of 1 ‐2 help videos over the past month.

15

Ongoing and Future Testing

• Just because the 2017 Economic Census data collection period has

ended does not mean testing ends!

• Centurion Instrument is still the home of ASM/COS
• More surveys being moved into Respondent Portal
• Looking towards 2022!
• Starting to include more demographic questions in our interviews to

get a better understanding of our actual end users and not just the companies themselves

• Continuing to analyze paradata to see respondent behaviors outside
• f an interview setting
• Incorporating eye‐tracking

16

Thank you!

Contact Info

Aryn Hernandez Aryn.Hernandez@census.gov 301‐763‐7982 Rebecca Keegan Rebecca.Keegan@census.gov 301‐763‐6003

17

Temika Holland Temika.Holland@census.gov 301‐763‐5241 Amy Anderson Riemer Amy.E.Anderson.Riemer@census.gov 301‐763‐7544