Research: The Game Changers Mark Schulman, Ph.D. AAPOR Roundtable - - PowerPoint PPT Presentation

Evolving Survey Research: The Game Changers

Mark Schulman, Ph.D.

AAPOR Roundtable 67th Annual Conference May 2012

Abt Associates | pg 2

From the great sage himself…

“The future ain't what it used to be.” Yogi Berra US baseball player, coach, & manager (1925 - )

Abt Associates | pg 3

Challenges and Opportunities

• Difficulties of securing cooperation in our busy and diverse

society

• Growing recognition of shortcomings in traditional self-reports

research

• Emergence of new technologies which are redefining what we

can do and collect

• Vast stores of real-time administrative records may be utilized

by govt. in place of or augmentiing contracted surveys

• Need for more efficient sample designs, particularly for rare

populations

• Probable decline of federal/state research budgets, boosting

search for efficiencies

Abt Associates | pg 4

Digital Revolution Opens New Doors

• Smartphones, iPads, other digital devices and social media provide real-

time interactions to capture consumer behavior, preferences, underlying motivational structures, diffusion, and even photographs.

• Require new perspectives in the conduct of surveys, well beyond the stale

"online vs. phone vs. household versus intercept."

• Require multiple data collection modes to reach our diverse population and

better understanding of mode effects

• Need to combine multiple software platforms
• Real data capture, not just self-reports on questionnaires!

Abt Associates | pg 5

Smartphones will soon overtake ordinary cell phones

Abt Associates | pg 6

Even Smarter Smartphones!

• Talk to Siri as you would to a
• person. Say something like “Tell my

wife I’m running late.” “Remind me to call the vet.” “Any good burger joints around here?” Siri does what you say, finds the information you need, then answers you. It’s like you’re having a conversation with your iPhone.

Abt Associates | pg 7

How do we harness apps?*

• Smartphones equipped with peripherals give

researchers new ways to automate data collection and potentially reduce:

• recall bias
• noncompliance/item nonresponse
• verall measurement error

*(Buskirk, “Is your Health Survey Research as Smart as your Phone,” 2011)

Abt Associates | pg 8

• Harvard’s iPhone Swine Flu Tracking app

(Rao, 2009)

• Apps to improve patient/provider

communications (Patrick, et al. 2008)

• Wellness diary apps for health promotion

(Koskinen and Salminen, 2007)

• Diabetes maintenance and smoking

cessation apps (Logan et al., 2007 and Abroms, et al., 2011)

*See, Trent Buskirk, “Is your Health Survey Research as Smart as your Phone?,” 2011

Changing World Example: Prevention and Treatment Apps in Public Health – How do we harness?*

Abt Associates | pg 9

Example: Harvard Swine Flu App

Includes videos, animations and text that allow you to learn:

• how to reduce the risk to you and your family
• how to prepare your business for the pandemic
• real-time updates and news from Harvard Med about H1N1.
• “HealthMap” about the state of the epidemic in your current

location and other locations. Take away points for surveys?

Abt Associates | pg 10

Smartphone Prevention and Treatment Apps

• Wahoo’s Fitness Packs
• ThinkLab’s Electronic Stethoscope

(and Heart rate Monitoring)

Abt Associates | pg 11

Digital measurement: new frontiers

• Abt SRBI Transportation/Urban Planning Group

pioneering in application of GIS to Household Travel Surveys

• GIS devices supplementing paper-and-pencil diaries

to measure all household travel

• Old self-report diaries omitted much household travel

Abt Associates | pg 12

Other Innovative GIS Applications

• Agency for Healthcare Research and Quality (AHRQ). Abt SRBI providing GIS

services to Abt Associates to develop a Learning Network and Technical Assistance Center to support principal investigators and their research.

• Neighborhood Stabilization Tracking Panel (NSP). Abt SRBI GIS is working with Abt to

manage all GIS mapping and analysis for the visual tracking survey component. Measuring potential impacts of a program designed to stabilize neighborhoods with high rates of foreclosures or vacancies.

• NYC Survey of Youths. Abt SRBI GIS is developing live spatial analysis services for real time screening in

CATI systems of survey participants. These live GIS services allow for customized geographies to be utilized screening by home locations in designated complex survey target areas.

• Environmental Analysis - Digital Elevation Models (DEM), topography imagery

analysis, view shed and watershed modeling, etc. Example: Advisory Base Flood Elevations (ABFE). Abt SRBI GIS supported Carlos Martín and Tom McCall of Abt Associates in Bethesda, MD in determining the Advisory Base Flood Elevations (ABFE) for 3,626 properties throughout Texas, Louisiana and Mississippi.

Abt Associates | pg 13

Sampling/Stat: New Frontiers

• Increasing use of efficient sample designs, such as dual ABS -

List

• Current early-stage explorations of modeling techniques for

non-probability samples: probability of selection not known -- such as web panels

• Early attempts: Propensity Scoring: Other adjustment and

modeling techniques being explored to make inferences from non-prob samples.

• Growing use of imputation to substitute for missing data.
• Growing use of paradata and responsive design: Data about the process by

which the survey data were collected. Paradata includes the times of day interviews were conducted, how long the interviews took, how many times there were contacts with each interviewee or attempts to contact the interviewee, the reluctance of the interviewee, and the mode of communication (such as phone, Web, email, or in person). Responsive design to optimize sample.

• Respondent-driven sample: variant of snowball sampling. Not in

favor.

Abt Associates | pg 14

Data Collection from Administrative Records

• Government records contain huge amounts of data which may some

day supplant the need for contracted surveys.

• Govt. data bases include Medicare, Medicaid, Social Security,

birth/death records, etc.

• Netherlands census does not conduct a total enumeration, but relies
• n administrative records and limited sample surveys
• U.S. Census examining this approach. See Bob Groves’s blogs
• Very major privacy issues and “big brother” political issues here.
• May be appropriate mianly for govt.-conducted surveys.
• SRBI has a medical records coding staff in place.

Abt Associates | pg 15

Biospecimen Collection

• Growing use of biological specimen collection in

surveys.

• Combine biological specimens with social science

data

• Includes blood, saliva, etc.
• SRBI collecting specimens for surveys of military

veterans, urine specimens, etc.

Abt Associates | pg 16

Biospecimen Collection*

• Combining biospecimens with social surveys adds

complex legal/ethical/privacy issues and cost to projects.

• Storing specimens adds additional issues, including

storage, retrieval, and sharing of data.

• Informed consent becomes much more complex.

(See Committee on National Statistics Report, 2010)

Abt Associates | pg 17

Market Research: Changing Landscape

• Social media, smart phones and other digital devices provide real-time

interactions to capture consumer behavior, preferences, their underlying motivational structures, and diffusion.

• Tweets becoming social sensors, measuring everything from purchase

behavior and movie preferences to influenza rates

• Applications when don’t need accurate estimation, such as qualitative
• My new years resolution: Never be afraid to be kicked in the teeth. Let the blood and the

bruises define your legacy. - Lady Gaga 2012 1 Jan

Abt Associates | pg 18

Google Flu Trends 2007-2008

• Close relationship between how many people search

for flu-related topics and how many people actually have flu symptoms (CDC).

Blue: CDC Yellow: Google Estimate

Abt Associates | pg 19

Web Spying!

• Web surveillance, with strategic partners, tracks and

analyzes millions of online tweets, postings, chat rooms, etc. http://www.radian6.com/see-

demo/?utm_source=google&utm_medium=adwords&utm_campaign=socialmediamonitoring&gclid=CPCW 54eZt60CFUOQ7QodI2TFoA

Abt Associates | pg 20

Bottom Line

• Need to leverage growing technological innovation for data

collection.

• More development and application of efficient sampling
• More expansive definition of “data collection,” including

biospecimens

• New technologies and apps require broader thinking
• Explore greater use of administrative records in place of or in

conjunction with surveys for some government projects

Abt Associates | pg 21

Discussion