using Big Data to Understand Human Systems Ryan Kennedy University - - PowerPoint PPT Presentation

Opportunities and Challenges in using Big Data to Understand Human Systems

Ryan Kennedy University of Houston

How Big is Big Data?

• In the 3rd Century BC, the Library of Alexandria was thought to

contain the entire sum of human knowledge.

• Today, there is enough digitally stored information to give every

person alive 320 times as much information as we think were stored in that library.

Opportunities in Big Data

• Microtargeting
• Real-time feedback
• Information on hard-to-find groups
• Data on social systems

Big Data Hype

• (Mayer-Schönberger and Cukier 2014):
• Approaching “N = all”
• Correlation is enough (no need for theory)

Problems in Big Data

• Big Data Hubris
• Overfitting
• Vulnerability to Artifacts
• Non-ideal Users
• Blue Team Issues
• Red Team Issues

Big Data Hubris

• The belief that volume can solve all problems.
• Don’t get me wrong, it can solve some (e.g. Xbox project).
• But there are several things that have to be clear:
• Sampling frame
• Convenience samples are still convenience samples
• Generalizability still has the standard limitations
• Motivations for uses of technology not always clear
• Behavioral analogue must be clear (and often requires small data)

Can Twitter Predict Elections?

Party Election Results Share of Twitter Mentions (Original Study) Share of Twitter Mentions (Replication) Christian Democrats (CDU) 28.4% 30.1% 18.6% Christian Social Democrats (CSU) 6.8% 5.6% 3.0% Social Democratic Party (SPD) 24.0% 26.6% 14.7% Free Democratic Party (FDP) 15.2% 17.3% 11.2% The Left (Die Linke) 12.4% 12.4% 8.3% Green Party (Grüne) 11.1% 8.0% 9.3% Pirate Party (Piraten) 2.1%

• 34.8%

Can Twitter Predict Elections?

Tweets mentioning Chavez Tweets mentioning Capriles

Better Small Data = Better Big Data

Using Big Data to Supplement Small Data

Overfitting

• When there is a lot of data to fit to a relatively small number of data

points, the number of strong correlations that will be found by chance alone increases dramatically.

• This danger is made even worse by modern algorithms that can find

very non-linear and highly-interactive relationships.

• Out-of-sample prediction helps, but it doesn’t completely solve the

problems.

• Causality is still important.

Google Flu Trends (GFT)

• Examine 50m search terms.
• Utilized those most heavily

correlated with flu prevalence, as measured by CDC regional reports, but curated to weed out non-flu-related searches.

• Released in 2008, updated in

2009, updated again in 2013.

Big data means big danger of overfitting…

• In Google Flu the search terms

are identified through “brute force” (50 million search terms fit to 1152 data points).

• Similar problems for other “big

data” resources (e.g. Twitter).

#feelingalittlesick #nyquilfixeseverything #blowingmyownbrainsout #fallingapart #shamblingnose #YOLO?

Vulnerability to Artifacts

• Need to know how your system turns unstructured data into

quantitative data.

• Examples:
• Google NGram project
• Stanford Sentiment Analyzer

(http://nlp.stanford.edu:8080/sentiment/rntnDemo.html)

Vulnerability to Artifacts

Vulnerability to Artifacts

Vulnerability to Artifacts

Non-Ideal Users

• Often an ideal-user assumption, but:
• Users often create multiple accounts
• Users will sometimes not answer particular questions
• Users will modify behavior if know are observed
• Users will modify behavior due to irrelevant events

Trends: abdominal pain on my right side

Blue Team Issues

• “Data exhaust” not designed for analysis.
• Process generating data always changing and geared towards goals
• ther than collecting accurate data.
• This means that the data-generating process can change without

warning and in unpredictable ways (e.g. Google’s Search Algorithm).

• Even if the data-generating process remains relatively stable, it can

still be idiosyncratic (e.g. event data).

Changes that could affect Google Flu…

• Improved Trends geolocation.
• Recommended searches.
• Related search listings.
• Diagnosis related to search

terms for symptoms.

• Customized search results by

time and location.

• Popularization of particular

terms.

• And many many more.

Red Team Issues

• As we become better at monitoring systems, individuals have

incentives to manipulate those signals.

• Bots and puppets.
• Purchased support.

Red Team Issues

Methods to Address Issues

• Strong ground truth measures (need small data to have good big

data).

• Causality still matters.
• Critically evaluate the sample from which big data is derived.
• Dynamic re-estimation.
• Understanding data generating process and critically tracking system

changes.

• Anticipating manipulation and setting up early warnings.
• “Rosetta Stone” for linking digital behavior to specific social context.

Dynamic re-estimation…

Santillana et al. Forthcoming. American Journal of Preventative Medicine.

Can we reach Asymptopia?

Understanding Causality and the Data Generating Process

Linking Digital Behavior to Context

Thank You

Ryan Kennedy University of Houston