Quasi-experimental Designs for Assessing Response on Social Media to - PowerPoint PPT Presentation

Quasi-experimental Designs for Assessing Response on Social Media to Policy Changes Yijun Tian 1 , Rumi Chunara 2,3 1: NYU Courant Institute of Mathematical Sciences; Computer Science 2: NYU Tandon School of Engineering; Computer Science & Engineering 3: NYU College of Global Public Health; Biostatistics SCHOOL OF COURANT INSTITUTE TANDON SCHOOL GLOBAL OF MATHEMATICAL SCIENCES OF ENGINEERING PUBLIC HEALTH 1

Motivation ● Tobacco continues to be a global public health threat, killing more than five million people each year [1]. Moreover, the landscape of tobacco products is evolving and policy is changing. ● Understanding public sentiment on tobacco products is important given the rapidly changing regulation efforts [2]. ● Social media systems can serve as an informational platform to understand human preferences, sentiments, and reactions [3], compared to survey-based methods. However there are major concerns with drawing population-level conclusions from social media because of: the opt-in nature of contributing data (affecting types of populations represented), temporal confounders, and differences between places that are not explicitly measured [4]. [1]: World Health Organization. 2010. Technical manual on tobacco tax administration. [2]: CDC. 2019. Smoking and tobacco use. https://www.cdc.gov/tobacco/. [3]: De Choudhury, M.; Gamon, M.; Counts, S.; and Horvitz, E. 2013. Predicting depression via social media. In ICWSM. [4]: Chunara, R.; Wisk, L. E.; and Weitzman, E. R. 2017. Denominator issues for Personally Generated Data in Population Health 2 Monitoring. American journal of preventive medicine 52(4):549–553.

Introduction Related Work Tobacco Discussion on Social Media ● Tobacco is a well-discussed topic on social media (Lazard, Allison et al. 2016). ● Majority of posts on tobacco tend to center on experience sharing (Krauss, Melissa et al. 2015). Social Media and Policy ● Public opinion can help inform the design of policies (Latimer, William W et al. 2003). ● Social media has been used to examine public discourse around ideological issues that also overlap with policy (Sharma et al. 2017; Zhang and Counts 2016). Quasi-experimental Designs (QEDs) ● QEDs are often used in circumstances when random assignment of treatment is either impossible or infeasible (Shadish, Cook, and Campbell 2002). ● QEDs have previously been used with social media data to rule out threats to validity (Oktay, Taylor, and Jensen 2010). 3

Questions we want to answer 1. Is there an effect on online sentiment in San Francisco (SF) for different tobacco products based on different stages of a tobacco flavor ban? 2. Upon implementation of a state-level e-cigarette tax, is there an effect on online e-cigarette sentiment in other states? 4

Introduction Data Twitter • Collect geo-located tweets through Twitter public API from April 2016 to April 2019. • Use keyword lists and classifier [6] to collect: • San Francisco tweets for Q1: E-cigarette tweets, Tobacco tweets, Flavored tobacco tweets. • State-level E-cigarette tweets for Q2. • To ensure quality: we analyze tweeting frequency / bot accounts / commercial situation. Reddit • Collect data through Pushshift API from subreddit ‘electronic cigarette’ (most relevant subreddit to ‘SF flavor ban’ with 188K members). • Filter posts and comments relevant to ‘SF flavor ban’. [6]: Huang, T.; Elghafari, A.; Relia, K.; and Chunara, R. 2017. High-resolution temporal representations of alcohol and tobacco behaviors from social media data. Proceedings of the ACM on human-computer interaction 1(CSCW):54. 5

Q1: Is there an effect on online sentiment in San Francisco for different tobacco products based on different stages of a tobacco flavor ban? Policy Background ● San Francisco Prohibits the Sales of Flavored Tobacco Products (Health Code Article 19Q) ○ Proposal: June 20, 2017 ○ Approval: June 5, 2018 ○ Enforcement: Jan 1, 2019 6 Q1: Is there an effect on online sentiment in San Francisco for different tobacco products based on different stages of a tobacco flavor ban?

Q1: Is there an effect on online sentiment in San Francisco for different tobacco products based on different stages of a tobacco flavor ban? Trends Overview ● 3 categories: E-cigarette, Tobacco, Flavored Tobacco. ● Sentiment analysis: Vader, which is a rule- based model for sentiment analysis of social media text (Hutto and Gilbert 2014) ● To understand if the trends significantly changed: Interrupted Time Series Analysis, in which we observe an outcome variable for a certain time interval, ∆t, before a treatment and after the treatment. ● To evaluate the difference between outcome variables in different ∆t: two-sample t-test. Tweet Trends Sentiment Trends 7

Q1: Is there an effect on online sentiment in San Francisco for different tobacco products based on different stages of a tobacco flavor ban? Tweet Trends Tobacco E-cigarette ● People are talking more about E-cigarette and Tobacco online after proposal. ○ The denominator of percentage is SF tweets. ● Significant differences in slope and intercept for all events except one. 8

Q1: Is there an effect on online sentiment in San Francisco for different tobacco products based on different stages of a tobacco flavor ban? Tweet Trends Flavored Tobacco ● Differing trend from E-cigarette and Tobacco: ○ The percentage is already consistently high. ● Flavored Tobacco tweets compose 7% - 10% of the Tobacco tweets. 9

Q1: Is there an effect on online sentiment in San Francisco for different tobacco products based on different stages of a tobacco flavor ban? Sentiment Trends Tobacco E-cigarette Flavored Tobacco ● After proposal, positive decrease and negative increase. ● Positive and negative tweets have significant slope and intercept differences for all events except a few. 10 Q1: Is there an effect on online sentiment in San Francisco for different tobacco products based on different stages of a tobacco flavor ban?

Q1: Is there an effect on online sentiment in San Francisco for different tobacco products based on different stages of a tobacco flavor ban? Content Analysis Tweets ● People tend to be negative (sarcastic) about the ban. ○ “So there’s gonna be a tobacco ban in the city of San Francisco.... sounds like the 1920’s..” ● Even a few increase in positive is generally positive on tobacco products. ○ “SF is voting on whether to ban flavored tobacco, including menthol cigarettes. Wow.” Topics concordance in Reddit using Empath [7] ● Top two common topics: ‘negative emotion’, ‘business’. ● negative emotion: ○ Reddit: The last thing you ever want to hear, ”I’m from the government and I’m here to help”. People are unbelievably stupid these days. ○ Twitter: The FDA is trying to kill vaping because it’s going to do a better job reducing lung cancer than they ever have. ● business: ○ Reddit: I’ve stated this since the beginning, it’s not about the flavors, or the packaging, or the kids... It’s about control. ○ Twitter: Name an industry that has multiple years of 70+% growth, made countless good jobs, and improved the health of millions. #vape #ecig #vaping. 11 [7]: Fast, E.; Chen, B.; and Bernstein, M. S. 2016. Empath: Understanding topic signals in large-scale text. CoRR abs/1602.06979. Q1: Is there an effect on online sentiment in San Francisco for different tobacco products based on different stages of a tobacco flavor ban?

Q2: Upon implementation of a state-level e-cigarette tax, is there an effect on online e-cigarette sentiment in other states? Policy Background ● From 2017 to 2018, 4 States implemented e-cigarette tax policies: ○ California ( CA ): April 1, 2017 ○ Kansas ( KS ): July 1, 2017 ○ Delaware ( DE ): January 1, 2018 ○ New Jersey ( NJ ): September 29, 2018 ● [Example] Neighbouring states of Kansas: ○ Nebraska ( NE ) ○ Missouri ( MO ) ○ Oklahoma ( OK ) ○ Colorado ( CO ) 12 Q1: Is there an effect on online sentiment in San Francisco for different tobacco products based on different stages of a tobacco flavor ban?

Q2: Upon implementation of a state-level e-cigarette tax, is there an effect on online e-cigarette sentiment in other states? Natural Experiment Design ● Natural experiment is a condition with an exogenous change (tax enactment) appeared, which approximates a randomized experiment. ● We compare the after-to-before ratios of e-cigarette discussion between neighbouring states using a 𝛙 2 analysis. Before ecig tweets After ecig tweets After-to-before ratio State with tax A neighbouring State 13 Q1: Is there an effect on online sentiment in San Francisco for different tobacco products based on different stages of a tobacco flavor ban?

Q2: Upon implementation of a state-level e-cigarette tax, is there an effect on online e-cigarette sentiment in other states? After-to-Before Comparison positive decrease negative increase significant neutral indicate a decrease in overall polarization. ● Kansas and Delaware had low numbers of e-cigarette tweets and the lowest tax per milliliter rate, thus results in those states and adjacent ones are not strong enough to interpret. 14