Applied Quantitative Cyber Risk Analysis Michael Rich, OSCP, CISSP - PowerPoint PPT Presentation

Applied Quantitative Cyber Risk Analysis Michael Rich, OSCP, CISSP Director of IT Security, Infrastructure & Operations Motion Picture Industries Pension & Health Care Plan

| 2 | Disclaimer for those reading from the ISACA link  My talks are image and slide-build heavy.  So they don’t “print” well.  Sorry about that.

| 3 | Agenda  Seek Beyond Your Interest – “Risk Analysis: Don’t Just Trust Your Gut” – Evan Wheeler @ BSidesLA 2016  The Idea: – What is a Risk? – The Calibration of the Experts – Monte Carlo Risk simulation – A Cyber Risk Model Example  The Application: – Risk Decomposition – Gedanken Experiments – “The SHOCKING truth about probability they don’t want you to know!!!” – Snowflakes and Monte Carlo – Equivalent Life Event Probabilities  Now What?

| 4 | The Idea

| 5 | What is a Risk?  An event that has some chance of happening and causes effects we don’t want. Qualitative Analysis Quantitative Analysis Source credit: https://www.risklens.com/blog/4-steps-to-a-smarter-risk-heat-map

| 6 | What is a Risk?  Probability of Occurrence Log-normal distribution example – Numerically-expressed probability – Can be a range to express uncertainty  i.e.: 9-14% chance  Impact (Loss) – Numerically expressed range:  Upper bound  Lower bound  90% confidence – Used with a log-normal distribution  5% values are < Lower bound  5% of values are > Upper bound  Black Swans!

| 7 | Log Normal – In Real Life Image from Blackline.com

| 8 | What is a Risk?  Estimated over given time period  A basic risk: – Tomorrow, if a traffic incident occurs during my morning commute, I will be delayed – Probability of occurrence: 30% – Impact (90% confidence): 5 – 60 minute delay from normal commute time

| 9 | Subjective Range Estimation AKA The Calibration of the Experts  What is the stated capacity of Wembley Stadium in London?  The Equivalent Bet: for 1000 Imperial Credits would you rather Win nothing – See if the answer is in your interval – Spin the dial? This slide covered on purpose so we don’t ruin the fun at the event!! Win it all Capacity: 90,000

| 10 | Monte Carlo Simulation  Iterate over probability of occurrence and generate random impacts  Many times (100K+) Example: Probability: 30% Impact, Upper bound: 60 Impact, Lower bound: 5 Number Trials 10001 Trial Delay 1 0 2 14.55244 3 17.37702 4 16.64968 5 0 6 0 7 0 8 0 9 0 10 49.68741

| 11 | Sim Results and the Loss Exceedance Curve

| 12 | Reducing Loss Exceedance Curves  Curves are pretty, but I need a number! – Ranking – Comparison – Mitigation effectiveness  In insurance world: – Average Annual Loss = Premium – “Area under the curve”  For Commute: – Average Event Impact – 6.8 minutes…. But… 241 Minute MAX impact

| 13 | Methodology Demonstration – The Shared Home Computer Cost chosen as impact only for purposes of this example Banking Trojan Clumsy Cat Probability 5% Probability 5% Max Impact $3000 Max Impact $25,000 ($35,000) Min Impact $750 Min Impact $500 Amazon Spree Ransomware Probability 30% Probability 10% Risks over next 6 months Max Impact $750 Max Impact $3000 Min Impact $150 Min Impact $200 Creepy Spyware Probability 2% Max Impact $2000 ($5000) Min Impact $300

| 15 | Simulation Results (100K iterations) Use Case: Ranking Risks Total Expected Average $638 Loss Banking Trojan $317 Amazon Spree $112 Ransomware $110 Clumsy Cat $80 Creepy Spyware $19

| 16 | The Application

| 17 | Risk Decomposition  Break your risk effects down into chunks – Measureable and observable – Company dependent  Manpower Costs – Business Departments – Leadership  Remediation Costs – IR Retainer – Legal – Hardware – Software

| 18 | Risk Decomposition Active? Active? Active? LB LB LB Time UB Time UB Hardware Cost UB PSC Cap Security Cap LB LB Cap $/Hr UB $/Hr UB Active? Cap Cap LB Active? Active? Software Cost UB LB LB Time UB Time UB Cap Retirements Cap IT Ops Cap Active? LB LB LB $/Hr UB $/Hr UB IR Retainer Cap Cost UB Cap Active? Active? Cap LB LB Active? UB Time Time UB LB Cap Accounting IT Leadership Cap Legal LB LB Cost UB $/Hr UB $/Hr UB Cap Cap Cap

| 19 | Gedanken Experiments

| 20 | The ONE SHOCKING Truth About Probability  Aggregate probability is a bitch…  2 times in 120 days, I escalated a security event to the CIO  What are the odds I have to escalate an issue any given day: – Odds: 2/120 – Probability [Odds/(1+Odds)]: 1.64%  What is the probability (p) I’ll have an event in the next 6 months I have to escalate?  Well: – Probability (p-not) of it not happening [1-p]: 98.4% – Probability of it not happening for 120 consecutive days [(p-not)^120]: 14.4% – Probability of an escalated event in 120 days [1-(not happening)]: 85.6%

| 21 | Is Monte Carlo a Precious Snowflake? (Sensitivity Analysis)  3 independent variables. How sensitive is the Average Event Loss? Probability Lower Bound Upper Bound

| 22 | Monte Carlo IS a Precious Snowflake.. Probably

| 23 | Ooof.. It’s Even Worse Than I Thought

| 24 | Handling the Snowflake  Must include uncertainty in your probability estimate (i.e. a range)  Instead of 1% probability, let’s make it 0.5% - 1.5% (50% error bar) Test AEL($) 1% Fixed $72 1% +/- .5% $70

| 25 | Beta Distribution  Single: $71.79  Uniform: $71.15  Beta: $71.63 Test EAL ($) 1% fixed $71.79 1% +/- 0.5% $71.15 1% Beta $71.63

| 26 | Some More Experiments Test EAL ($) 5% fixed $367 5% +/- 4% $355 5% Beta $356

| 27 | Some More Experiments Test EAL ($) 5% fixed $350 5% +/- 4% $349 4% +/- 3% $293 4% fixed $277

| 29 | Statistically Equivalent Probabilities  100% - 50%  50% - 10%  10%  3%  1.5%  1%  0.8%  0.02%

| 30 | Beta Distribution: Establish Probability from Test Cases  If you have a set of cases, you can get a probability distribution

| 32 | Using Probability for Complicated Scenarios  Calibrate expert  Ask expert to assess probability of the event given no other data – “What is the probability of an adversary managing to inject code on the target system in the next 6 months?”  Ask expert to re-assess given various conditions – “What if the firewalls are discovered to be misconfigured?” – “What if a Cooperative Vulnerability Inspection team demonstrates code injection?” – “What if a black-box adversarial assessment team demonstrates it?”  Use Log-Odds-Ratio – Statistically valid method for combining the effects of multiple conditions on a final probability

| 33 | Log Odds Ratio Example Use Case: Using expert knowledge Initial Prob: P(E) 1.0% Conditions Firewall CVI Finding Adversarial Software Adversary Intel Rogue Network Device Rogue USB P(E|X1) Correct Config Code Injection Code Injection Updated 1 Hop away Detected Detected P(E|X2) Incorrect Config No Injection No Injection Not Updated 2 Hops away Not detected Not Detected P(E|X3) 3+ Hops away P(E|X4) P(E|X1) 0.5% 5.0% 10.0% 1.0% 10.0% 1.0% 1.0% P(E|X2) 2.0% 0.5% 0.5% 5.0% 3.0% 15.0% 15.0% P(E|X3) 1.0% P(E|X4) Condition State Which Applies Correct Config No Injection Code Injection Updated 1 Hop away Detected Detected Conditional Probability 23.2%

| 34 | Now What?  For Me – Solidify my risk decompositions – Identify my events to analyze – Calibrate my team – Model and Simulate – Submit Blackhat ‘18 paper  For You – Go read Hubbard’s book – Go get my code: https://github.com/richmr/QuantitativeRiskSim – Think about your decompositions – Identify your events – Model and Simulate – Come watch my Blackhat ‘18 presentation

| 35 | Summary  Quantitative risk modeling can be a reality in Cybersecurity – Use Case: Risk ranking and prioritization – Use Case: Assessing control audit results – Use Case: Mitigation comparison – Use Case: Quantifying expert knowledge on complex systems – Use Case: Test planning  Networks can improve its cybersecurity… Measurably!  Python Simulation Code available at: – https://github.com/richmr/QuantitativeRiskSim

| 36 |