Through Information Disclosure: A Field Quasi-Experiment BCSI 2013 - - PowerPoint PPT Presentation

Improving Internet Security Through Information Disclosure: A Field Quasi-Experiment

BCSI 2013

Qian Tang (School of Information Systems, SMU) Leigh Linden (Economics Department, UT Austin) John S. Quarterman (Quarterman Creations) Andrew Whinston (McCombs School of Business, UT Austin)

Reality Check

97% (of data beaches) were avoidable, without the need for organizations to resort to difficult or expensive countermeasures … A partner’s lax security practices and poor governance—often outside the victim’s control or expertise—are frequent catalysts in security incidents. (Verizon 2012 Data Breach Investigation Report, p.3, 23)

2

Challenges in Internet Security

• Cost concern

– Information security costs are difficult to justify when things are going well.

• No transparency

– No knowledge or internal policies to recognize or deal with security threats – Not to reveal info to protect company reputation – No legislative enforcement

• Negative externalities

– Someone else’s problem

3

Information Disclosure

• Disclosure makes information transparent.
• Disclosure can internalize negative

externalities of insecurity as reputational and financial loss.

• Current security info disclosure regulations

have mainly been guidelines and suggestions rather than legislation.

• Problems of voluntary reporting

4

What We Propose

• Mandatory information disclosure

– steer management away from any temptation to suppress unfavorable information – produce standardized data across different industries and companies

• Using existing data on security vulnerabilities

– minimize corporate reporting costs

• Straightforward information presentation

– deliver comparable and understandable information

5

Literature Review

• Economics of information security

– Anderson 2001, Anderson and Moore 2006, Schneier 2002, Kanich et al. 2008, Stone-Gross et al. 2011, Caballero et al. 2011, Rao and Reiley 2012

• Corporate information disclosure

– Davis et al. 2008, Balakrishnan et al. 2008, Loughran and McDonald 2011 – Villiers 2012, Verrecchia 1983, Field et al. 2005, Rogers et al. 2010, Jorgensen and Kirschenheiter 2003

• Security information disclosure

– Arora et al. 2004, Gal-Or and Ghose 2005, Gordon et al. 2003, Romanosky et al. 2011, Campbell et al. 2003 – Lenard and Rubin 2005, Cate 2009, Kannan et al. 2007

6

Research Hypotheses

• Awareness effect

– Provide the company with its security information

• Reputational effect

– Put the company subject to public scrutiny

• Overall effect of security info disclosure

– H1: Disclosure can improve overall internet security. – H2: The more security issues a company has, the more likely it will improve under disclosure.

7

Research Hypotheses

• Social comparison effect

– Companies compare themselves against other companies for self-evaluation and tend to behave in consistency with other companies (peer influence). – H3: Companies are more likely to improve their security when the worst security companies are disclosed to have less problems.

8

Experimental Setting

• Outgoing spam

– A severe security issue worldwide – Common symptom of security problems

9

FireEye, Spamhaus, ISPs took down Grum botnet on 19 July 2012

Experimental Setting

• Outgoing spam

– A severe security issue worldwide – Common symptom of security problems – Data available from blocklists – Measurability – Extreme negative externalities

• Key terms: Botnet, AS/ASN, Blocklist

10

Activity Spam Pollution Property crime Externality ratio 100:1 1:10 7:30

Sources: Stone-Gross et al. 2011, Kanich et al. 2008, Caballero, et al. 2011, Rao and Reiley 2012

Data collection and processing

• IP level spam data: blocklists

– Text files from CBL (Composite Blocking List) and PSBL (Passive Spam Block List) everyday – Each text file consists of millions of lines like this:

“1.0.17.248,AS2519,1.0.16.0/23,JP,vectant.ne.jp,,,1349617 960,spamsalot,39”

11

IP address, ASN, Netblock, Country code, Domain,,, Time, Botnet, Volume

• Mapping data: Team Cymru and CBL

– BGP routing data – Aggregate from IP address to netblock to AS and to

• rganization

SpamRankings.net

12

Quasi-Experimental Design

• Treatment: info disclosed on SpamRankings.net every month
• Clustered assignment: spam rankings for each country
• Period: Jan 2011-Jan 2012, with treatment started for US in

May, CA in June, BE and TR in July

• Matched pair design

13

Traffic to SpamRankings.net

14

Distribution of Internet traffic to SpamRankings.net

Within North America and Europe:

• US: 29.2%
• Canada: 11%
• Belgium: 0.1%
• Turkey: 1.4%

Quasi-experimental data

15

• Monthly outbound spam data on the top 250

spamming ASNs from Jan 2011 to Jan 2012 for the eight selected countries.

Statistical models

• Basic model
• With covariates
• With interaction effects

– including interaction by country pair

16

𝑍

𝑗𝑑𝑞𝑢 = 𝜄0 + 𝜄1𝐸𝑑𝑞 + 𝜁𝑗𝑑𝑞𝑢 (1)

𝑍

𝑗𝑑𝑞𝑢 is the outcome of interest for AS 𝑗 in country 𝑑 pair 𝑞 at time 𝑢

𝐸𝑑𝑞𝑢 is treatment indicator for whether information is available for country 𝑑 pair 𝑞 at time 𝑢 𝑍

𝑗𝑑𝑞𝑢 = 𝜄0 + 𝜄1𝐸𝑑𝑞 + 𝜄2𝑌𝑗𝑑𝑞 + 𝜕𝑞 + 𝜗𝑗𝑑𝑞𝑢 (2)

𝑌𝑗𝑑𝑞 is a vector of pre-treatment AS characteristics 𝜕𝑞 is country pair fixed effects 𝑍

𝑗𝑑𝑞𝑢 = 𝜄0 + 𝜄1𝐸𝑑𝑞 + 𝜄2𝑌𝑗𝑑𝑞 + 𝜄3𝑌𝑗𝑑𝑞 ∗ 𝐸𝑑𝑞 + 𝜕𝑞 + 𝜁𝑗𝑑𝑞𝑢 (3)

Standard errors

• Within country correlations (ρ=0.004**)

– Cluster-robust standard errors: cluster by countries to correct for heteroskedasticity and within-cluster error correlation (Bertrand et al. 2004)

• Small number of clusters (J=8)

– Bootstrap-t procedure: correct for over-rejection problem with few clusters (Cameron et al. 2008)

17

Manipulation Check: Country Level

• Trend in total outgoing spam volumes for the treated

group and the control group

18

Manipulation Check: Company Level

19

Selection bias does exist. ASes in the treatment group tend to generate less spam and have more IP addresses. The treated group have better-security and large- sized companies.

Covariates

20

Treatment effect is significantly negative and robust to log transformation and asymptotic refinement. H1 is supported.

+Covariates +Covariates

Overall Treatment Effect of Disclosure

21

Treatment Effect by AS Characteristics

Treatment effect is stronger on ASes with more

• utgoing spam.

H2 is supported.

Treatment Effect by Country Pair

22

Treatment effect is significantly negative for all country pairs except for Belgium-Netherlands pair.

Robustness Check: Treatment effect over time

23

Treatment effect is consistent over time.

Robustness check: Serial Correlation

24

Collapse the time series into “pre”- and “post”- periods can take into account the effective sample size. (Bertrand et al. 2004)

+Covariates

25

The more spam

• bserved from
• ther ASes, the

less likely an AS would reduce its

• wn spam. H3 is

supported.

ct ct

Social Comparison Effect

Conclusions

• Mandatory info disclosure on info security with

straightforward info presentation can lead to improvement in info security.

– We found an average of 15.9% reduction in spam volume of the treated group, compared to control group, due to spam info disclosure.

• Awareness, reputational, and social comparison

effects can be all leveraged to incentivize

• rganizations.
• Limitations

– Country pair – Covariates – Awareness effect vs. reputational effect

26

Future Research: Field Experimentation

27

• Use US organizations to design randomized field

experiment

– Collect covariate data

• Industry
• Geographic location
• Baseline spam volume

– Stratify by industry

• Randomize organizations to treatment and control groups

within each industry section

• Check the balance in other covariates between the treated

and the control

• Rerandomize if the balance is not acceptable

Next version SpamRankings.net

28

Next version SpamRankings.net

29

Thank you Questions?

qiantang@smu.edu.sg

Model development

• Level-1: ASN (Individual)
• Level-2: Country (Cluster)
• Level-3: Country pair (assuming fixed pair effects)
• Full model

31

𝑍

𝑗𝑑𝑞 = 𝜌0𝑑𝑞 + 𝑓𝑗𝑑𝑞,

𝑓𝑗𝑑𝑞~𝑂(0, 𝜏𝑑

2)

𝜌0𝑑𝑞 = 𝛾00𝑞 + 𝛾01𝑞𝐸𝑑𝑞 + 𝑠0𝑑𝑞, 𝑠0𝑑𝑞~𝑂(0, 𝜐𝑞

2)

𝛾00𝑞 = 𝛿000 + 𝑣00𝑞 𝛾01𝑞 = 𝛿010 + 𝑣01𝑞 𝑍

𝑗𝑑𝑞 = 𝛿000 + 𝑣00𝑞 + (𝛿010 + 𝑣01𝑞)𝐸𝑑𝑞 + 𝜁𝑗𝑑𝑞, 𝜁𝑗𝑑𝑞= 𝑠0𝑑𝑞+𝑓𝑗𝑑𝑞

𝑗 ASN 𝑑 Country 𝑞 Country pair Country pair fixed effects Treatment-by-country-pair interaction Average treatment effect Errors correlated within country