Cost-benefit analysis Tyler Moore CSE 5/7338 Computer Science - - PDF document

Cost-benefit analysis

Tyler Moore

CSE 5/7338 Computer Science & Engineering Department, SMU, Dallas, TX

Lecture 10

Review of security investment so far

Metrics for quantifying security benefits

1

ALE0: expected loss without security investment

2

ALEs: expected loss with security investment

3

EBISs: ALE0 − ALEs

4

ENBISs: ALE0 − ALEs − c

High-level investment metrics

1

ROSI

2

NPV

3

IRR

2 / 26

Security investment questions worth answering

Q: Should we invest in security? A: Yes, if ENBIS > 0 Q: Should we invest in defense A or B? A: Choose the one with higher ROSI (or NPV if considering longer time horizons) Q: How much should we invest? A: Security investment models (e.g., Gordon-Loeb) say to invest until marginal cost of added security equals marginal benefit Q: Is a security investment cost-effective? A1: Yes, if ENBIS > 0 A2: Probably, if the minimum probability of attack required to break even is high enough

3 / 26

Cost-benefit analysis (CBA)

Used widely in public policy to justify expenditures Quite similar to the security metrics presented earlier, especially ENBIS Emphasis placed on making best-effort estimates of key figures

1

Costs of insecurity (ALE0)

2

Costs of security countermeasures (c)

3

Probability of attack (p0)

4

Risk reduction r = p0−ps

p0

In CBA, a security investment is considered cost-effective if ENBIS > 0. CBA exercises estimate the above figures and use the findings as evidence when deciding whether or not to adopt (or continue spending money on) a countermeasure When there is uncertainty over some figures, a range of values is considered

4 / 26

Notes Notes Notes Notes

ENBIS using risk reduction

ENBIS equations from earlier presentations using Bernoulli loss assumptions used p0 and the improved probability ps We can equivalently express this in terms of reduced risk ENBIS = (p0 − ps) · λ − c ENBIS = p0 · p0 − ps p0 · λ − c ENBIS = p0 · r · λ − c

5 / 26

ENBIS for multiple sources of loss

Up to now, we have assumed that there is a single financial loss λ associated with an attack In fact, losses can take many forms, each with its own magnitude and probability of occurrence Ideally, we would like to account for each type of loss independently and combine into an aggregate measure Suppose there are n loss types. We can calculate the ENBIS as follows: ENBIS = p0 · r · λ − c ENBIS = p0 · r ·

n

• i=1

(P(λi|attack) · λi) − c

6 / 26

Cost-benefit analysis tasks

Estimate p0 using available data (sometimes hard) Enumerate the loss types, estimate their cost and conditional probability of occurring once an attack takes place Estimate (or take as input) security costs c Estimate (or take as input) risk-reduction rate r We discuss cost-benefit efforts for two examples: terrorist attacks targeting highway bridges (reading 1) and sewer

• verflows at wastewater facilities (reading 2)

7 / 26

Case 1: terrorist attacks targeting highway bridges

Estimate p0 using available data (sometimes hard)

No known instances in past, so assign small probability (p0 = 10−4)

Enumerate the loss types, estimate their cost and conditional probability of occurring once an attack takes place

1

Bridge replacement: $40 million (average of replacement costs for prior collapses), cond. prob. = 1.0

2

Loss of life: 80 lives with actuarial value $6.3M each, occurring with cond. prob. 0.2 (estimated from prior collapses)

Estimate (or take as input) security costs c

NPV of 20% of bridge-replacement value amortized over 25 years = $260,000

Estimate (or take as input) risk-reduction rate r

Taken to be r = 0.9 High value selected to give benefit the best possible chance of exceeding costs

8 / 26

Notes Notes Notes Notes

Case 1: terrorist attacks targeting highway bridges

ENBIS = p0 · r ·

n

• i=1

(P(λi|attack) · λi) − c ENBIS = Fill in the equation ENBIS = −247K Based on this calculation, the security investment does not seem to be justified.

9 / 26

Case 2: sewage overflows at wastewater facilities

Estimate p0 using available data

Original goal: estimate probability of malicious attack triggering large overflows, but there have only been a few publicly reported attacks Revised goal: estimate probability of large sewage overflows triggered by accident or attack, since both can be detected and sometimes prevented by incident detection system California Water Board reported 46 large overflows in one year in state They separately reported that facilities cover 110,593 sewer miles Hence the number of overflows can be expressed as

46 110593 = 4.16 × 10−4 × # miles.

Cities with population over 100,000 have an average of 1,300 sewer miles in their facilities Hence p0 = 0.541

Note that p0 is more accurately interpreted here as the expected number of overflows during the time period

10 / 26

Case 2: sewage overflows at wastewater facilities

Enumerate the loss types, estimate their cost and conditional probability of occurring once an attack takes place Loss category Data? Direct losses Cleanup costs yes Property damage yes Regulatory costs yes Lost business for victims no Victim health costs no Indirect losses Lost business for non-victims no Broader environmental impact no Psychological distress no We can estimate the costs for the categories we have to arrive at a lower bound for the total cost

11 / 26

Case 2: sewage overflows at wastewater facilities

Enumerate the loss types, estimate their cost and conditional probability of occurring once an attack takes place i Loss category λi P(λi|SO) Comments 1 Cleanup costs 22K 1 Likely underestimate 2 Property damage 1.4M 0.25 no data for cond. prob. 3 EPA fine 2.89M 0.01

46 violations 2000–2011

46 SOs in CA in 2012 12.1% of US pop. in CA ×12 yrs 12 / 26

Notes Notes Notes Notes

Case 2: sewage overflows at wastewater facilities

Estimate (or take as input) security costs c City Cost factor Cost/year Sewer miles Reference 1 20K 1300 Atlanta 2 39K 2125 DC 3 59K 1800 San Francisco 6 118K 993 New Orleans 8 157K 1600 Estimate (or take as input) risk-reduction rate r

Taken to be r = 0.4 Argued that some overflows couldn’t be prevented, but some should be

13 / 26

Case 2: sewage overflows at wastewater facilities

ENBIS = p0 · r ·

n

• i=1

(P(λi|attack) · λi) − c ENBIS = Fill in the equation ENBIS = 67K Based on this calculation, the security investment is justified for the “average” city.

14 / 26

Case 2: sewage overflows at wastewater facilities

Recall that security investment costs and the expected number of large overflows vary by city City Cost/year Sewer miles ENBIS Reference 20K 1300 67K Atlanta 39K 2125 103K DC 59K 1800 62K San Francisco 118K 993

• 51K

New Orleans 157K 1600

• 50K

15 / 26

Case 2: sewage overflows at wastewater facilities

16 / 26

Notes Notes Notes Notes

What if we are uncertain about the accuracy of estimates?

When we are uncertain about one or more of the estimated parameters, we can do a breakeven analysis to identify the value a parameter must take for ENBIS = 0. The best parameter to vary is the one that is most uncertain Often, this is p0, the probability of attack without security investment

17 / 26

Cybersecurity is not the only discipline where estimating probabilities of rare events is difficult

The assessment of the probabilities that adversaries will choose courses of action should be the outputs of analysis, not required input parameters Quote is from National Academies of Science report on bioterrorism risks What does this mean for cost-benefit analysis?

18 / 26

Breakeven analysis with probability of attack as output

ENBIS = (p0 − ps) · λ − c ENBIS = p0 · p0 − ps p0 · λ − c ENBIS = p0 · r · λ − c Setting ENBIS to 0 and solving for p0: p0 = c r · λ We can then see for a range of parameter values what the corresponding breakeven probability of attack must be to justify security investment

19 / 26

Breakeven analysis for case 1

p0 = c r · λ p0 = Fill in the equation p0 = 0.002

20 / 26

Notes Notes Notes Notes

Breakeven probabilities (as percentages) for case 1

Source: http://politicalscience.osu.edu/faculty/jmueller/CIP.pdf 21 / 26

Breakeven analysis for case 2

p0 = c r · λ p0 = c r · (22K · 1 + 1.4M · 0.25 + 2.89M · 0.0101) p0 = c r · 401K

22 / 26

Breakeven probability of sewage overflow for case 2

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.5 1.0 1.5 2.0 2.5 Risk reduction probability p_0 (Expected # overflows) c=20K c=50K c=100K 23 / 26

Breakeven analysis with risk reduction as output

ENBIS = (p0 − ps) · λ − c ENBIS = p0 · p0 − ps p0 · λ − c ENBIS = p0 · r · λ − c Setting ENBIS to 0 and solving for r: r = c p0 · λ We can then see for a range of parameter values what the corresponding breakeven risk reduction must be to justify security investment

24 / 26

Notes Notes Notes Notes

Breakeven risk reduction for case 2

100 200 300 400 500 0.0 0.2 0.4 0.6 0.8 1.0 Cost ($K) Breakeven risk reduction probability 1300 sewer miles 500 sewer miles 3000 sewer miles 25 / 26

R code to generate plot

br <−f u n c t i o n ( c , l , p) c /( l ∗p) c o s t s <− seq (10 ,500 , by=1) p over <− f u n c t i o n ( m i l e s =1300) 46/110593∗ m i l e s pdf ( ’ cbrr −sewer . pdf ’ ) p l o t ( x=costs , y=br ( costs ,401 , p over (1300)) , type =’ l ’ , ylab = ’ Breakeven r i s k r e d u c t i o n p r o b a b i l i t y ’ , xlab =’Cost ($K) ’ , lwd=2, ylim=c (0 ,1)) l i n e s ( x=costs , y=br ( costs ,401 , p over (500)) , l t y =’dashed ’ , lwd=2) l i n e s ( x=costs , y=br ( costs ,401 , p over (3000)) , l t y =’dotted ’ , lwd=2) legend (” bottomright ” , legend=c (”1300 sewer m i l e s ” ,”500 sewer m i l e s ” ,”3000 sewer m i l e s ”) , l t y=c (” s o l i d ” ,” dashed ” ,” dotted ”) , lwd=2) dev . o f f ()

26 / 26

Notes Notes Notes Notes