Optimal filter and Cost-Benefit Analysis Tyler Moore CSE 7338 - - PDF document

Optimal filter and Cost-Benefit Analysis

Tyler Moore

CSE 7338 Computer Science & Engineering Department, SMU, Dallas, TX

Lecture 3

Outline

1

Information security risk management

2

Optimal filter configuration

3

Cost-benefit analysis

2 / 53 Information security risk management

Information security risk management

Just as it can be useful to translate infosec risks and defenses into the language of investment (ROSI, NPV, etc.), one must also be aware of terminology from risk management As IT becomes essential to many businesses, border between information security investment and general risk management has blurred

4 / 53 Information security risk management

Risk management terminology overview

Risk analysis identification quantification Risk management acceptance mitigation avoidance transfer Risk monitoring validation documentation Cyberinsurance

5 / 53

Notes Notes Notes Notes

Information security risk management Risk acceptance

Risk acceptance

After risks are identified and quantified, they must be “managed” The simplest option is to do nothing Such “risk acceptance” is prudent when:

1

Worst-case loss is small enough to be paid from proceeds or reserves

2

Probability of occurrence is smaller than other business risks that threaten the organization’s survival

This is why the security policies for start-ups are often weaker than for entrenched firms

6 / 53 Information security risk management Risk mitigation

Risk mitigation

If risk is too big and probable to be accepted, risk mitigation aims to reduce the probability and severity of a loss This is where security investment comes in Recall that the optimal level of investment normally leaves residual risk that must be dealt with using acceptance, avoidance, or transfer

7 / 53 Information security risk management Risk avoidance

Risk avoidance

Aims to reduce the probability and severity of loss, as in risk mitigation However, rather than use technology, here one forgoes risky activities This introduces opportunity costs of lost business opportunities Example: online merchant refusing overseas orders due to high fraud risk Example: company disconnects database with customers’ personal information online Question: what are the opportunity costs in these cases?

8 / 53 Information security risk management Risk avoidance

Risk transfer

The final option is to buy an insurance contract to recover any future losses incurred This is only available in limited circumstances Why has the cyber-insurance market remained small?

Difficulty in quantifying losses Even when possible, many firms would rather keep quiet than share with an insurance company Externalities mean that the costs of insecurity are often borne by others Correlated risk is prevalent

9 / 53

Notes Notes Notes Notes

Information security risk management Risk avoidance

Risk management example: credit card issuers

Credit card issuers regularly manage fraud

1 Risk acceptance: fraud is paid from the payment fees charged to

merchants

2 Risk mitigation: install anti-fraud technology (raises costs of security) 3 Risk avoidance: downgrade high-risk cardholders to debit or require

• nline verification (leads to lost business)

4 Risk transfer: structure consumer credit risk and sell it on the market 10 / 53 Optimal filter configuration

Domain-specific models

Up to now we have modeled security investment at a very high level Map costs to benefits, assume diminishing marginal returns to investment, etc. Useful for when justifying security budgets compared to non-security expenditures Not useful for deciding how best to allocate a given security budget Today, we discuss a model for a tactical security investment decision: configuring a filter to balance false positives and negatives

12 / 53 Optimal filter configuration ROC curves

Binary classification is a recurring problem in CS

Common task: distill many observations to a binary signal

{0, 1}: communications theory S = {undervalued, overvalued}: stock trading S = {reject, accept}: research hypothesis S = {benign, malicious}: security filter

Such simplification inevitably leads to errors compared to reality (aka ground truth)

13 / 53 Optimal filter configuration ROC curves

Filter defense mechanism

Reality Signal no attack attack benign 1 − α β malicious α 1 − β α: false positive rate, β: false negative rate

14 / 53

Notes Notes Notes Notes

Optimal filter configuration ROC curves

Receiver operating characteristic

Detection rate 1 − β 1 False positive rate α 1

45◦ 15 / 53 Optimal filter configuration ROC curves

Receiver operating characteristic

Detection rate 1 − β 1 False positive rate α 1

45◦

α = β

EERsolid EERdashed

15 / 53 Optimal filter configuration An economic model of optimal filter configuration

Model for optimal filter configuration

Binary classifiers are imperfect Finding the optimal trade-off, say for an IDS or spam filter, is hard Can be framed as an economic trade-off between opportunity cost of false positives and losses incurred by false negatives

16 / 53 Optimal filter configuration An economic model of optimal filter configuration

Model for optimal filter configuration

We can see from ROCs that β can be expressed as a function of α. β : [0, 1] → [0, 1] defines the false negative rate as a function of the false positive rate α β(0) = 1, β(1) = 0 We assume β′(x) < 0 and β′′(x) ≥ 0

17 / 53

Notes Notes Notes Notes

Optimal filter configuration An economic model of optimal filter configuration

Model for optimal filter configuration

Suppose we rely on a filter to scan incoming email attachments for malware a: cost of false positive (blocking a benign email) b: cost of false negative (delivering malicious email) p: probability of email containing malware Cost C(α) = p · β(α) · b + (1 − p) · α · a

Suppose p = 0.1, a = $250, b = $500, α = 0.1, β = .2 C(α) = 0.1 · 0.2 · 500 + 0.9 · 0.1 · 250 = $32.50

18 / 53 Optimal filter configuration An economic model of optimal filter configuration

Optimal filter configuration: exercise 1

Suppose we rely on a filter to scan incoming email attachments for

• malware. Suppose the cost of dealing with a false negative event is

$400, and the cost of dealing with a false positive is $200. 20% of incoming email has malware. You can choose between two configurations

• Config. A: 10% false positive rate and 30% false negative rate
• Config. B: 25% false positive rate and 15% false negative rate

Your task: compute the expected costs for both configurations, and state which configuration you prefer.

19 / 53 Optimal filter configuration An economic model of optimal filter configuration

Model for optimal filter configuration

α∗ = arg min

α p · β(α) · b + (1 − p) · α · a

which has first-order condition (FOC) 0 = δα

• p · β(α∗) · b + (1 − p) · α∗ · a
• after rearranging, we obtain:

β′(α∗) = −1 − p p · a b

20 / 53 Optimal filter configuration An economic model of optimal filter configuration

Optimal filter configuration (continuous ROC curves)

Detection rate 1 − β 1 False positive rate α 1

Indifference curves

(1−p)a p·b

α∗

B

α∗

A

21 / 53

Notes Notes Notes Notes

Optimal filter configuration An economic model of optimal filter configuration

Optimal filter configuration (continuous ROC curves)

Detection rate 1 − β 1 False positive rate α 1

45◦

B A α = β EERA = EERB AUCA = AUCB

21 / 53 Optimal filter configuration An economic model of optimal filter configuration

Optimal filter configuration (continuous ROC curves)

Detection rate 1 − β 1 False positive rate α 1

45◦

B A

(1−p)a p·b

α∗

B

α∗

A

21 / 53 Optimal filter configuration An economic model of optimal filter configuration

Optimal filter configuration (discrete ROC curves)

Detection rate 1 − β 1 False positive rate α 1

45◦

(1−p)a p·b

C F E α∗D

22 / 53 Optimal filter configuration An economic model of optimal filter configuration

Optimal filter configuration example (discrete ROC curves)

Detection rate 1 − β 1 False positive rate α 1

0.2 0.7 0.4 0.9 0.2 0.4 s l

• p

e 2 0.5 0.5 s l

• p

e 1 0.3 0.1 slope 1/3

(1−p)a p·b

C F E α∗D α∗ = 0.2 if 1 ≤ (1−p)a

p·b

≤ 2

23 / 53

Notes Notes Notes Notes

Optimal filter configuration An economic model of optimal filter configuration

Optimal filter configuration: exercise 2

Suppose we rely on a filter to scan incoming email attachments for

• malware. Suppose the cost of dealing with a false negative event is

$400, and the cost of dealing with a false positive is $200. 20% of incoming email has malware. You can choose between two configurations

• Config. A: 10% false positive rate and 30% false negative rate
• Config. B: 25% false positive rate and 15% false negative rate

Your task

1

Draw the ROC curve for configurations A and B (plus (0% FP, 100% FN) and (100% FP, 0% FN))

2

Calculate the slope of the indifference curve for the optimal configuration

3

Select the optimal point for the ROC curve

24 / 53 Cost-benefit analysis

Review of security investment so far

Metrics for quantifying security benefits

1

ALE0: expected loss without security investment

2

ALEs: expected loss with security investment

3

EBISs: ALE0 − ALEs

4

ENBISs: ALE0 − ALEs − c

High-level investment metrics

1

ROSI

2

NPV

3

IRR

26 / 53 Cost-benefit analysis

Security investment questions worth answering

Q: Should we invest in security? A: Yes, if ENBIS > 0 Q: Should we invest in defense A or B? A: Choose the one with higher ROSI (or NPV if considering longer time horizons) Q: How much should we invest? A: Security investment models (e.g., Gordon-Loeb) say to invest until marginal cost of added security equals marginal benefit Q: Is a security investment cost-effective? A1: Yes, if ENBIS > 0 A2: Probably, if the minimum probability of attack required to break even is high enough

27 / 53 Cost-benefit analysis

Cost-benefit analysis (CBA)

Used widely in public policy to justify expenditures Quite similar to the security metrics presented earlier, especially ENBIS Emphasis placed on making best-effort estimates of key figures

1

Costs of insecurity (ALE0)

2

Costs of security countermeasures (c)

3

Probability of attack (p0)

4

Risk reduction r = p0−ps

p0

In CBA, a security investment is considered cost-effective if ENBIS > 0. CBA exercises estimate the above figures and use the findings as evidence when deciding whether or not to adopt (or continue spending money on) a countermeasure When there is uncertainty over some figures, a range of values is considered

28 / 53

Notes Notes Notes Notes

Cost-benefit analysis

ENBIS using risk reduction

ENBIS equations from earlier presentations using Bernoulli loss assumptions used p0 and the improved probability ps We can equivalently express this in terms of reduced risk ENBIS = (p0 − ps) · λ − c ENBIS = p0 · p0 − ps p0 · λ − c ENBIS = p0 · r · λ − c

29 / 53 Cost-benefit analysis

ENBIS for multiple sources of loss

Up to now, we have assumed that there is a single financial loss λ associated with an attack In fact, losses can take many forms, each with its own magnitude and probability of occurrence Ideally, we would like to account for each type of loss independently and combine into an aggregate measure Suppose there are n loss types. We can calculate the ENBIS as follows: ENBIS = p0 · r · λ − c ENBIS = p0 · r ·

n

• i=1

(P(λi|attack) · λi) − c

30 / 53 Cost-benefit analysis

Cost-benefit analysis tasks

Estimate p0 using available data (sometimes hard) Enumerate the loss types, estimate their cost and conditional probability of occurring once an attack takes place Estimate (or take as input) security costs c Estimate (or take as input) risk-reduction rate r We discuss cost-benefit efforts for two examples: terrorist attacks targeting highway bridges (reading 1) and sewer overflows at wastewater facilities (reading 2)

31 / 53 Cost-benefit analysis

Case 1: terrorist attacks targeting highway bridges

Estimate p0 using available data (sometimes hard)

No known instances in past, so assign small probability (p0 = 10−4)

Enumerate the loss types, estimate their cost and conditional probability of occurring once an attack takes place

1

Bridge replacement: $40 million (average of replacement costs for prior collapses), cond. prob. = 1.0

2

Loss of life: 80 lives with actuarial value $6.3M each, occurring with

• cond. prob. 0.2 (estimated from prior collapses)

Estimate (or take as input) security costs c

NPV of 20% of bridge-replacement value amortized over 25 years = $260,000

Estimate (or take as input) risk-reduction rate r

Taken to be r = 0.9 High value selected to give benefit the best possible chance of exceeding costs

32 / 53

Notes Notes Notes Notes

Cost-benefit analysis

Case 1: terrorist attacks targeting highway bridges

ENBIS = p0 · r ·

n

• i=1

(P(λi|attack) · λi) − c ENBIS = Fill in the equation ENBIS = −247K Based on this calculation, the security investment does not seem to be justified.

33 / 53 Cost-benefit analysis

Case 2: sewage overflows at wastewater facilities

Estimate p0 using available data

Original goal: estimate probability of malicious attack triggering large

• verflows, but there have only been a few publicly reported attacks

Revised goal: estimate probability of large sewage overflows triggered by accident or attack, since both can be detected and sometimes prevented by incident detection system California Water Board reported 46 large overflows in one year in state They separately reported that facilities cover 110,593 sewer miles Hence the number of overflows can be expressed as

46 110593 = 4.16 × 10−4 × # miles.

Cities with population over 100,000 have an average of 1,300 sewer miles in their facilities Hence p0 = 0.541

Note that p0 is more accurately interpreted here as the expected number of overflows during the time period

34 / 53 Cost-benefit analysis

Case 2: sewage overflows at wastewater facilities

Enumerate the loss types, estimate their cost and conditional probability of occurring once an attack takes place Loss category Data? Direct losses Cleanup costs yes Property damage yes Regulatory costs yes Lost business for victims no Victim health costs no Indirect losses Lost business for non-victims no Broader environmental impact no Psychological distress no We can estimate the costs for the categories we have to arrive at a lower bound for the total cost

35 / 53 Cost-benefit analysis

Case 2: sewage overflows at wastewater facilities

Enumerate the loss types, estimate their cost and conditional probability of occurring once an attack takes place i Loss category λi P(λi|SO) Comments 1 Cleanup costs 22K 1 Likely underestimate 2 Property damage 1.4M 0.25 no data for cond. prob. 3 EPA fine 2.89M 0.01

46 violations 2000–2011

46 SOs in CA in 2012 12.1% of US pop. in CA ×12 yrs 36 / 53

Notes Notes Notes Notes

Cost-benefit analysis

Case 2: sewage overflows at wastewater facilities

Estimate (or take as input) security costs c City Cost factor Cost/year Sewer miles Reference 1 20K 1300 Atlanta 2 39K 2125 DC 3 59K 1800 San Francisco 6 118K 993 New Orleans 8 157K 1600 Estimate (or take as input) risk-reduction rate r

Taken to be r = 0.4 Argued that some overflows couldn’t be prevented, but some should be

37 / 53 Cost-benefit analysis

Case 2: sewage overflows at wastewater facilities

ENBIS = p0 · r ·

n

• i=1

(P(λi|attack) · λi) − c ENBIS = Fill in the equation ENBIS = 67K Based on this calculation, the security investment is justified for the “average” city.

38 / 53 Cost-benefit analysis

Case 2: sewage overflows at wastewater facilities

Recall that security investment costs and the expected number of large overflows vary by city City Cost/year Sewer miles ENBIS Reference 20K 1300 67K Atlanta 39K 2125 103K DC 59K 1800 62K San Francisco 118K 993

• 51K

New Orleans 157K 1600

• 50K

39 / 53 Cost-benefit analysis

Case 2: sewage overflows at wastewater facilities

40 / 53

Notes Notes Notes Notes

Cost-benefit analysis

What if we are uncertain about the accuracy of estimates?

When we are uncertain about one or more of the estimated parameters, we can do a breakeven analysis to identify the value a parameter must take for ENBIS = 0. The best parameter to vary is the one that is most uncertain Often, this is p0, the probability of attack without security investment

41 / 53 Cost-benefit analysis

Cybersecurity is not the only discipline where estimating probabilities of rare events is difficult

The assessment of the probabilities that adversaries will choose courses of action should be the outputs of analysis, not required input parameters Quote is from National Academies of Science report on bioterrorism risks What does this mean for cost-benefit analysis?

42 / 53 Cost-benefit analysis

Breakeven analysis with probability of attack as output

ENBIS = (p0 − ps) · λ − c ENBIS = p0 · p0 − ps p0 · λ − c ENBIS = p0 · r · λ − c Setting ENBIS to 0 and solving for p0: p0 = c r · λ We can then see for a range of parameter values what the corresponding breakeven probability of attack must be to justify security investment

43 / 53 Cost-benefit analysis

Breakeven analysis for case 1

p0 = c r · λ p0 = Fill in the equation p0 = 0.002

44 / 53

Notes Notes Notes Notes

Cost-benefit analysis

Breakeven probabilities (as percentages) for case 1

Source: http://politicalscience.osu.edu/faculty/jmueller/CIP.pdf 45 / 53 Cost-benefit analysis

Breakeven analysis for case 2

p0 = c r · λ p0 = c r · (22K · 1 + 1.4M · 0.25 + 2.89M · 0.0101) p0 = c r · 401K

46 / 53 Cost-benefit analysis

Breakeven probability of sewage overflow for case 2

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.5 1.0 1.5 2.0 2.5 Risk reduction probability p_0 (Expected # overflows) c=20K c=50K c=100K

47 / 53 Cost-benefit analysis

Breakeven analysis with risk reduction as output

ENBIS = (p0 − ps) · λ − c ENBIS = p0 · p0 − ps p0 · λ − c ENBIS = p0 · r · λ − c Setting ENBIS to 0 and solving for r: r = c p0 · λ We can then see for a range of parameter values what the corresponding breakeven risk reduction must be to justify security investment

48 / 53

Notes Notes Notes Notes

Cost-benefit analysis

Breakeven risk reduction for case 2

100 200 300 400 500 0.0 0.2 0.4 0.6 0.8 1.0 Cost ($K) Breakeven risk reduction probability 1300 sewer miles 500 sewer miles 3000 sewer miles

49 / 53 Cost-benefit analysis

R code to generate plot

br <−f u n c t i o n ( c , l , p) c /( l ∗p) c o s t s <− seq (10 ,500 , by=1) p over <− f u n c t i o n ( m i l e s =1300) 46/110593∗ m i l e s pdf ( ’ cbrr −sewer . pdf ’ ) p l o t ( x=costs , y=br ( costs ,401 , p over (1300)) , type =’ l ’ , ylab = ’ Breakeven r i s k r e d u c t i o n p r o b a b i l i t y ’ , xlab =’Cost ($K) ’ , lwd=2, ylim=c (0 ,1)) l i n e s ( x=costs , y=br ( costs ,401 , p over (500)) , l t y =’dashed ’ , lwd=2) l i n e s ( x=costs , y=br ( costs ,401 , p over (3000)) , l t y =’dotted ’ , lwd=2) legend (” bottomright ” , legend=c (”1300 sewer m i l e s ” ,”500 sewer m i l e s ” ,”3000 sewer m i l e s ”) , l t y=c (” s o l i d ” ,” dashed ” ,” dotted ”) , lwd=2) dev . o f f ()

50 / 53 Cost-benefit analysis

Exercise: CBA for patient data breaches

Suppose that the Acme hospital chain is considering investing in controls to reduce the likelihood of suffering a breach of personal health records Security improvements will cost $2 million per year, and Acme estimates it would lose $50 million from a successful breach of its records Acmes risk management team estimates that protection would reduce its risk to suffering a breach by 40% Problem 1: Calculate the break-even annual probability of a breach

• ccurring.

51 / 53 Cost-benefit analysis

Exercise: Cost-benefit analysis for patient data breaches

Problem 1: Calculate the break-even annual probability of a breach

• ccurring.

Solution: Set ENBIS to 0 and solve for p0, we get the following:

52 / 53

Notes Notes Notes Notes

Cost-benefit analysis

Exercise: Cost-benefit analysis for patient data breaches

Suppose instead that it is determined that the breach probability is 5%. Problem 2: Based on this updated information, calculate the risk reduction that would be required of security mechnismsm in order to break even. Solution: set ENBIS to 0 and solve for r, we get the following:

53 / 53

Notes Notes Notes Notes