Applied Quantitative Cyber Risk Analysis Michael Rich, OSCP, CISSP - - PowerPoint PPT Presentation

Applied Quantitative Cyber Risk Analysis

Michael Rich, OSCP, CISSP Director of IT Security, Infrastructure & Operations Motion Picture Industries Pension & Health Care Plan

| 2 |

Disclaimer for those reading from the ISACA link

• My talks are image and slide-build heavy.
• So they don’t “print” well.
• Sorry about that.

| 3 |

Agenda

• Seek Beyond Your Interest

– “Risk Analysis: Don’t Just Trust Your Gut” – Evan Wheeler @ BSidesLA 2016

• The Idea:

– What is a Risk? – The Calibration of the Experts – Monte Carlo Risk simulation – A Cyber Risk Model Example

• The Application:

– Risk Decomposition – Gedanken Experiments – “The SHOCKING truth about probability they don’t want you to know!!!” – Snowflakes and Monte Carlo – Equivalent Life Event Probabilities

• Now What?

| 4 |

The Idea

| 5 |

What is a Risk?

• An event that has some chance of happening and causes effects we don’t

want.

Qualitative Analysis

Source credit: https://www.risklens.com/blog/4-steps-to-a-smarter-risk-heat-map

Quantitative Analysis

| 6 |

What is a Risk?

• Probability of Occurrence

– Numerically-expressed probability – Can be a range to express uncertainty

• i.e.: 9-14% chance
• Impact (Loss)

– Numerically expressed range:

• Upper bound
• Lower bound
• 90% confidence

– Used with a log-normal distribution

• 5% values are < Lower bound
• 5% of values are > Upper bound
• Black Swans!

Log-normal distribution example

| 7 |

Log Normal – In Real Life

Image from Blackline.com

| 8 |

What is a Risk?

• Estimated over given time period
• A basic risk:

– Tomorrow, if a traffic incident occurs during my morning commute, I will be delayed – Probability of occurrence: 30% – Impact (90% confidence): 5 – 60 minute delay from normal commute time

| 9 |

Subjective Range Estimation AKA The Calibration of the Experts

• The Equivalent Bet: for 1000 Imperial Credits would you rather

– See if the answer is in your interval – Spin the dial?

Win it all Win nothing

• What is the stated capacity of Wembley Stadium in London?

Capacity: 90,000

This slide covered on purpose so we don’t ruin the fun at the event!!

| 10 |

Monte Carlo Simulation

• Iterate over probability of occurrence and generate random impacts
• Many times (100K+)

Probability: 30% Impact, Upper bound: 60 Impact, Lower bound: 5 Number Trials 10001 Trial Delay 1 2 14.55244 3 17.37702 4 16.64968 5 6 7 8 9 10 49.68741

Example:

| 11 |

Sim Results and the Loss Exceedance Curve

| 12 |

Reducing Loss Exceedance Curves

• Curves are pretty, but I need a number!

– Ranking – Comparison – Mitigation effectiveness

• In insurance world:

– Average Annual Loss = Premium – “Area under the curve”

• For Commute:

– Average Event Impact – 6.8 minutes…. But…

241 Minute MAX impact

| 13 |

Methodology Demonstration – The Shared Home Computer

Cost chosen as impact only for purposes of this example

Banking Trojan Probability 5% Max Impact $25,000 ($35,000) Min Impact $500 Ransomware Probability 10% Max Impact $3000 Min Impact $200 Creepy Spyware Probability 2% Max Impact $2000 ($5000) Min Impact $300 Clumsy Cat Probability 5% Max Impact $3000 Min Impact $750 Amazon Spree Probability 30% Max Impact $750 Min Impact $150

Risks over next 6 months

| 15 |

Simulation Results (100K iterations)

Use Case: Ranking Risks

Total Expected Average Loss $638 Banking Trojan $317 Amazon Spree $112 Ransomware $110 Clumsy Cat $80 Creepy Spyware $19

| 16 |

The Application

| 17 |

Risk Decomposition

• Break your risk effects down into chunks

– Measureable and observable – Company dependent

• Manpower Costs

– Business Departments – Leadership

• Remediation Costs

– IR Retainer – Legal – Hardware – Software

| 18 |

Risk Decomposition

LB UB Cap LB UB Cap LB UB Cap LB UB Cap LB UB Cap LB UB Cap $/Hr Time Security Active? Time $/Hr IT Leadership Active? Time $/Hr IT Ops Active? LB UB Cap LB UB Cap LB UB Cap LB UB Cap LB UB Cap LB UB Cap Time $/Hr Retirements Active? Time $/Hr PSC Active? Accounting Active? Time $/Hr

LB UB Cap LB UB Cap LB UB Cap LB UB Cap IR Retainer Active? Cost Legal Active? Cost Active? Cost Hardware Software Active? Cost

| 19 |

Gedanken Experiments

| 20 |

The ONE SHOCKING Truth About Probability

• Aggregate probability is a bitch…
• 2 times in 120 days, I escalated a security event to the CIO
• What are the odds I have to escalate an issue any given day:

– Odds: 2/120 – Probability [Odds/(1+Odds)]: 1.64%

• What is the probability (p) I’ll have an event in the next 6 months I have

to escalate?

• Well:

– Probability (p-not) of it not happening [1-p]: 98.4% – Probability of it not happening for 120 consecutive days [(p-not)^120]: 14.4% – Probability of an escalated event in 120 days [1-(not happening)]: 85.6%

| 21 |

Is Monte Carlo a Precious Snowflake?

(Sensitivity Analysis)

• 3 independent variables. How sensitive is the Average Event Loss?

Probability Lower Bound Upper Bound

| 22 |

Monte Carlo IS a Precious Snowflake.. Probably

| 23 |

Ooof.. It’s Even Worse Than I Thought

| 24 |

Handling the Snowflake

• Must include uncertainty in your probability estimate (i.e. a range)
• Instead of 1% probability, let’s make it 0.5% - 1.5% (50% error bar)

Test AEL($) 1% Fixed $72 1% +/- .5% $70

| 25 |

Beta Distribution

• Single: $71.79
• Uniform: $71.15
• Beta: $71.63

Test EAL ($) 1% fixed $71.79 1% +/- 0.5% $71.15 1% Beta $71.63

| 26 |

Some More Experiments

Test EAL ($) 5% fixed $367 5% +/- 4% $355 5% Beta $356

| 27 |

Some More Experiments

Test EAL ($) 5% fixed $350 5% +/- 4% $349 4% +/- 3% $293 4% fixed $277

| 29 |

Statistically Equivalent Probabilities

• 100% - 50%
• 50% - 10%
• 10%
• 3%
• 1.5%
• 1%
• 0.8%
• 0.02%

| 30 |

Beta Distribution: Establish Probability from Test Cases

• If you have a set of cases, you can get a probability distribution

| 32 |

Using Probability for Complicated Scenarios

• Calibrate expert
• Ask expert to assess probability of the event given no other data

– “What is the probability of an adversary managing to inject code on the target system in the next 6 months?”

• Ask expert to re-assess given various conditions

– “What if the firewalls are discovered to be misconfigured?” – “What if a Cooperative Vulnerability Inspection team demonstrates code injection?” – “What if a black-box adversarial assessment team demonstrates it?”

• Use Log-Odds-Ratio

– Statistically valid method for combining the effects of multiple conditions on a final probability

| 33 |

Log Odds Ratio Example

Use Case: Using expert knowledge

Initial Prob: P(E) 1.0% Firewall CVI Finding Adversarial Software Adversary Intel Rogue Network Device Rogue USB P(E|X1) Correct Config Code Injection Code Injection Updated 1 Hop away Detected Detected P(E|X2) Incorrect Config No Injection No Injection Not Updated 2 Hops away Not detected Not Detected P(E|X3) 3+ Hops away P(E|X4) P(E|X1) 0.5% 5.0% 10.0% 1.0% 10.0% 1.0% 1.0% P(E|X2) 2.0% 0.5% 0.5% 5.0% 3.0% 15.0% 15.0% P(E|X3) 1.0% P(E|X4) Condition State Which Applies

Correct Config No Injection Code Injection Updated 1 Hop away Detected Detected

Conditional Probability

23.2%

Conditions

| 34 |

Now What?

• For Me

– Solidify my risk decompositions – Identify my events to analyze – Calibrate my team – Model and Simulate – Submit Blackhat ‘18 paper

• For You

– Go read Hubbard’s book – Go get my code: https://github.com/richmr/QuantitativeRiskSim – Think about your decompositions – Identify your events – Model and Simulate – Come watch my Blackhat ‘18 presentation

| 35 |

Summary

• Quantitative risk modeling can be a reality in Cybersecurity

– Use Case: Risk ranking and prioritization – Use Case: Assessing control audit results – Use Case: Mitigation comparison – Use Case: Quantifying expert knowledge on complex systems – Use Case: Test planning

• Networks can improve its cybersecurity… Measurably!
• Python Simulation Code available at:

– https://github.com/richmr/QuantitativeRiskSim

| 36 |