Quasi-experimental Designs for Assessing Response on Social Media to - - PowerPoint PPT Presentation

Quasi-experimental Designs for Assessing Response on Social Media to Policy Changes

Yijun Tian1, Rumi Chunara2,3

1: NYU Courant Institute of Mathematical Sciences; Computer Science 2: NYU Tandon School of Engineering; Computer Science & Engineering 3: NYU College of Global Public Health; Biostatistics

SCHOOL OF GLOBAL PUBLIC HEALTH TANDON SCHOOL OF ENGINEERING COURANT INSTITUTE OF MATHEMATICAL SCIENCES

1

• Tobacco continues to be a global public health threat, killing more than five million

people each year [1]. Moreover, the landscape of tobacco products is evolving and policy is changing.

• Understanding public sentiment on tobacco products is important given the rapidly

changing regulation efforts [2].

• Social media systems can serve as an informational platform to understand human

preferences, sentiments, and reactions [3], compared to survey-based methods. However there are major concerns with drawing population-level conclusions from social media because of: the opt-in nature of contributing data (affecting types of populations represented), temporal confounders, and differences between places that are not explicitly measured [4].

Motivation

[1]: World Health Organization. 2010. Technical manual on tobacco tax administration. [2]: CDC. 2019. Smoking and tobacco use. https://www.cdc.gov/tobacco/. [3]: De Choudhury, M.; Gamon, M.; Counts, S.; and Horvitz, E. 2013. Predicting depression via social media. In ICWSM. [4]: Chunara, R.; Wisk, L. E.; and Weitzman, E. R. 2017. Denominator issues for Personally Generated Data in Population Health

• Monitoring. American journal of preventive medicine 52(4):549–553.

2

Tobacco Discussion on Social Media

• Tobacco is a well-discussed topic on social media (Lazard, Allison et al. 2016).
• Majority of posts on tobacco tend to center on experience sharing (Krauss, Melissa et al. 2015).

Social Media and Policy

• Public opinion can help inform the design of policies (Latimer, William W et al. 2003).
• Social media has been used to examine public discourse around ideological issues that also overlap

with policy (Sharma et al. 2017; Zhang and Counts 2016).

Quasi-experimental Designs (QEDs)

• QEDs are often used in circumstances when random assignment of treatment is either impossible or

infeasible (Shadish, Cook, and Campbell 2002).

• QEDs have previously been used with social media data to rule out threats to validity (Oktay, Taylor,

and Jensen 2010). Introduction

Related Work

3

• 1. Is there an effect on online sentiment in San

Francisco (SF) for different tobacco products based on different stages of a tobacco flavor ban?

• 2. Upon implementation of a state-level e-cigarette

tax, is there an effect on online e-cigarette sentiment in other states?

Questions we want to answer

4

Introduction

Data

Twitter

• Collect geo-located tweets through Twitter public API from April 2016 to April 2019.
• Use keyword lists and classifier [6] to collect:
• San Francisco tweets for Q1: E-cigarette tweets, Tobacco tweets, Flavored tobacco

tweets.

• State-level E-cigarette tweets for Q2.
• To ensure quality: we analyze tweeting frequency / bot accounts / commercial situation.

Reddit

• Collect data through Pushshift API from subreddit ‘electronic cigarette’ (most relevant

subreddit to ‘SF flavor ban’ with 188K members).

• Filter posts and comments relevant to ‘SF flavor ban’.

[6]: Huang, T.; Elghafari, A.; Relia, K.; and Chunara, R. 2017. High-resolution temporal representations of alcohol and tobacco behaviors from social media data. Proceedings of the ACM on human-computer interaction 1(CSCW):54.

5

Q1: Is there an effect on online sentiment in San Francisco for different tobacco products based on different stages of a tobacco flavor ban?

• San Francisco Prohibits the Sales of Flavored Tobacco Products (Health

Code Article 19Q) ○ Proposal: June 20, 2017 ○ Approval: June 5, 2018 ○ Enforcement: Jan 1, 2019

Policy Background

Q1: Is there an effect on online sentiment in San Francisco for different tobacco products based on different stages of a tobacco flavor ban?

6

Trends Overview

Q1: Is there an effect on online sentiment in San Francisco for different tobacco products based on different stages of a tobacco flavor ban?

• 3 categories: E-cigarette, Tobacco, Flavored

Tobacco.

• Sentiment analysis: Vader, which is a rule-

based model for sentiment analysis of social media text (Hutto and Gilbert 2014)

• To understand if the trends significantly

changed: Interrupted Time Series Analysis, in which we observe an outcome variable for a certain time interval, ∆t, before a treatment and after the treatment.

• To evaluate the difference between outcome

variables in different ∆t: two-sample t-test.

Tweet Trends Sentiment Trends

7

Tweet Trends

Q1: Is there an effect on online sentiment in San Francisco for different tobacco products based on different stages of a tobacco flavor ban?

• People are talking more about E-cigarette and Tobacco online after proposal.

○ The denominator of percentage is SF tweets.

• Significant differences in slope and intercept for all events except one.

E-cigarette Tobacco

8

Tweet Trends

Q1: Is there an effect on online sentiment in San Francisco for different tobacco products based on different stages of a tobacco flavor ban?

Flavored Tobacco

• Differing trend from E-cigarette and Tobacco:

○ The percentage is already consistently high.

• Flavored Tobacco tweets compose 7% - 10% of

the Tobacco tweets.

9

Q1: Is there an effect on online sentiment in San Francisco for different tobacco products based on different stages of a tobacco flavor ban?

Sentiment Trends

Q1: Is there an effect on online sentiment in San Francisco for different tobacco products based on different stages of a tobacco flavor ban?

• After proposal, positive decrease and negative increase.
• Positive and negative tweets have significant slope and

intercept differences for all events except a few.

E-cigarette Tobacco

Flavored Tobacco

10

Q1: Is there an effect on online sentiment in San Francisco for different tobacco products based on different stages of a tobacco flavor ban?

Content Analysis

Q1: Is there an effect on online sentiment in San Francisco for different tobacco products based on different stages of a tobacco flavor ban?

Tweets

• People tend to be negative (sarcastic) about the ban.

○ “So there’s gonna be a tobacco ban in the city of San Francisco.... sounds like the 1920’s..”

• Even a few increase in positive is generally positive on tobacco products.

○ “SF is voting on whether to ban flavored tobacco, including menthol cigarettes. Wow.”

Topics concordance in Reddit using Empath [7]

• Top two common topics: ‘negative emotion’, ‘business’.
• negative emotion:

○ Reddit: The last thing you ever want to hear, ”I’m from the government and I’m here to help”. People are unbelievably stupid these days. ○ Twitter: The FDA is trying to kill vaping because it’s going to do a better job reducing lung cancer than they ever have.

• business:

○ Reddit: I’ve stated this since the beginning, it’s not about the flavors, or the packaging, or the kids... It’s about control. ○ Twitter: Name an industry that has multiple years of 70+% growth, made countless good jobs, and improved the health of millions. #vape #ecig #vaping.

[7]: Fast, E.; Chen, B.; and Bernstein, M. S. 2016. Empath: Understanding topic signals in large-scale text. CoRR abs/1602.06979.

11

Q1: Is there an effect on online sentiment in San Francisco for different tobacco products based on different stages of a tobacco flavor ban?

• From 2017 to 2018, 4 States implemented e-cigarette tax policies:

○ California (CA): April 1, 2017 ○ Kansas (KS): July 1, 2017 ○ Delaware (DE): January 1, 2018 ○ New Jersey (NJ): September 29, 2018

• [Example] Neighbouring states of Kansas:

○ Nebraska (NE) ○ Missouri (MO) ○ Oklahoma (OK) ○ Colorado (CO)

Policy Background

Q2: Upon implementation of a state-level e-cigarette tax, is there an effect on online e-cigarette sentiment in other states?

12

Q1: Is there an effect on online sentiment in San Francisco for different tobacco products based on different stages of a tobacco flavor ban?

• Natural experiment is a condition with an exogenous change (tax enactment)

appeared, which approximates a randomized experiment.

• We compare the after-to-before ratios of e-cigarette discussion between

neighbouring states using a 𝛙2 analysis.

Natural Experiment Design

Q2: Upon implementation of a state-level e-cigarette tax, is there an effect on online e-cigarette sentiment in other states?

Before ecig tweets After ecig tweets After-to-before ratio State with tax A neighbouring State

13

• Kansas and Delaware had low numbers of e-cigarette tweets and the lowest tax per milliliter

rate, thus results in those states and adjacent ones are not strong enough to interpret.

After-to-Before Comparison

Q2: Upon implementation of a state-level e-cigarette tax, is there an effect on online e-cigarette sentiment in other states?

positive decrease negative increase significant neutral indicate a decrease in overall polarization.

14

Q1: Is there an effect on online sentiment in San Francisco for different tobacco products based on different stages of a tobacco flavor ban?

• Positive: the frequency before is significant.
• Negative: the enactment of Tax policy is significant, as well as the frequency of tweets and posters.

Regression Analysis

Q2: Upon implementation of a state-level e-cigarette tax, is there an effect on online e-cigarette sentiment in other states?

Outcome variable: frequency of e-cigarettes tweets in different sentiment after the tax.

15

Q1: Is there an effect on online sentiment in San Francisco for different tobacco products based on different stages of a tobacco flavor ban?

Content Analysis

Q2: Upon implementation of a state-level e-cigarette tax, is there an effect on online e-cigarette sentiment in other states?

E-cigarette tweets increase in neighbouring states after tax policy enactment

• After Delaware tax:

○ New Jersey: “Juuls are about to be cancelled bc big tobacco is raising the prices so marbolo lights I will be back for you soon” ○ Pennsylvania: “Do I really wanna waste $40 on a juul?”, “Yeo who’s selling juuls for the lowski I lost mine last night and I’m not paying 60 bucks for another one, can someone venmo me 50 dollars i wanna buy a juul”.

• After California tax:

○ Nevada: “BUY 1 GET 1 FREE on all 60ml’s and 120ml’s”

16

Harvineet Singh Vishwali Mhasawade Kunal Relia Nabeel Rehman, PhD Rina Singh, PhD

Satellite Imagery: Methods

Acknowledgements

17

Takeaways

• Our study focused on assessing the response on social media to offline policy changes,
• ffering new opportunities for understanding hidden populations.

○ Traditional survey methods are limited due to the hard to access properties.

• Overall, negative sentiment and discussion about the policies dominated discussion, and

increased directly after the policy implementation. ○ While traditional surveys show strong public support for policies.

• No clear evidence for increased polarization online in places adjacent to where a tax was

implemented ○ however, within the same state, the enactment of tax policy was significant predictor for negative tweets after the tax.

eddie.tian@nyu.edu

http://tianyijun.com 18