When using a mathematical model, careful atuention must be given - PowerPoint PPT Presentation

Uncertainty in risk engineering: concepts Eric Marsden <eric.marsden@risk-engineering.org> ‘‘ When using a mathematical model, careful atuention must be given to uncertainties in the model. – Richard Feynman

▷ Epistemic uncertainty ▷ Decision uncertainty benefjts? model is a “correct” formulation of the problem • cannot be reduced • example: wind speed at Toulouse airport 100 days from now • related to lack of knowledge or precision of a model parameter • model uncertainty: lack of confjdence that the mathematical determine parameter exactly • parameter uncertainty: scientifjc knowledge insuffjcient to • related to the real variability of a • in general, reducible with suffjcient investment • presence of ambiguity or controversy about how to quantify or compare social objectives • which risk metrics, which acceptance criteria? • how to aggregate the utilities of individuals? • how to discount delayed benefjts against short-term population or a physical property 2 / 23 uncertainty model stochastic variability temporal variability spatial variability epistemic uncertainty uncertainty parameter uncertainty decision uncertainty goals & objectives values & preferences Types of uncertainty ▷ Stochastic (or aleatory) uncertainty

▷ Decision uncertainty ▷ Stochastic (or aleatory) uncertainty benefjts? model is a “correct” formulation of the problem population or a physical property • cannot be reduced • example: wind speed at Toulouse airport 100 days from now • related to lack of knowledge or precision of a model parameter • model uncertainty: lack of confjdence that the mathematical determine parameter exactly • parameter uncertainty: scientifjc knowledge insuffjcient to • in general, reducible with suffjcient investment • presence of ambiguity or controversy about how to quantify or compare social objectives • which risk metrics, which acceptance criteria? • how to aggregate the utilities of individuals? • how to discount delayed benefjts against short-term • related to the real variability of a 2 / 23 uncertainty model stochastic variability temporal variability spatial variability epistemic uncertainty uncertainty preferences parameter uncertainty decision uncertainty goals & objectives values & Types of uncertainty ▷ Epistemic uncertainty

▷ Epistemic uncertainty ▷ Stochastic (or aleatory) uncertainty benefjts? population or a physical property • cannot be reduced • example: wind speed at Toulouse airport 100 days from now • related to lack of knowledge or precision of a model parameter • model uncertainty: lack of confjdence that the mathematical • parameter uncertainty: scientifjc knowledge insuffjcient to model is a “correct” formulation of the problem determine parameter exactly • in general, reducible with suffjcient investment • presence of ambiguity or controversy about how to quantify or compare social objectives • which risk metrics, which acceptance criteria? • how to aggregate the utilities of individuals? • how to discount delayed benefjts against short-term • related to the real variability of a 2 / 23 uncertainty uncertainty stochastic variability temporal variability spatial variability epistemic model preferences uncertainty parameter uncertainty decision uncertainty goals & objectives values & Types of uncertainty ▷ Decision uncertainty

3 / 23 Epistemic uncertainty and linguistic imprecision Communication relies on shared context, but terms used for discussing likelihood are very subjective and “fuzzy”

Source: github.com/zonination/perceptions 4 / 23 Epistemic uncertainty and linguistic imprecision

Forecast from US National Intelligence Estimate 29-51 Probability of an Invasion of Yugoslavia (1951): ‘‘ Although it is impossible to determine which course the Kremlin is likely to adopt, we believe that the extent of Satellite military and propaganda preparations indicates that an atuack on Yugoslavia in 1951 should be considered a serious possibility. Authors of the report were asked “what odds they had had in mind when they agreed to that wording”. Their answers ranged from 1:4 to 4:1. Image: Podgarić monument, former Yugoslavia 5 / 23 Illustration of linguistic imprecision

Bank of England projection of various macroeconomic indicators use “fan charts” to illustrate the level of uncertainty in their predictions (probability mass in each colored band is 30%, 10% probability that outcomes lie outside of the colored area). Note that there is also uncertainty about data concerning the past. Figure source: bankofengland.co.uk 6 / 23 Uncertainty does not only concern the future

He who knows and knows he knows, He is wise — follow him; He who knows not and knows he knows not, He is a child — teach him; He who knows and knows not he knows, He is asleep — wake him; He who knows not and knows not he knows not, He is a fool — shun him. Ancient arabic proverb 7 / 23 Treatment of uncertainty

‘‘ As we know, there are known knowns. Tiere are things we know we know. We also know there are known unknowns. Tiat is to say we know there are some things we do not know. But there are also unknown unknowns, the ones we don’t know, we don’t know. – Donald Rumsfeld, February 2002, US DoD news briefjng Image source: US DoD, public domain 8 / 23 Types of uncertainty

Aims of quantitative uncertainty assessments: • help prioritize any additional measurement, modeling or R&D efgorts • “this is of suffjcient quality for this purpose” choice of a maintenance policy, an operation or the design of the system criteria or regulatory thresholds • examples: nuclear or environmental licensing, aeronautical certifjcation… 9 / 23 Goal of uncertainty modelling ▷ understand the infmuence of uncertainties ▷ to qualify or accredit a model or a method of measurement ▷ to infmuence design : compare relative performance and optimize the ▷ compliance : to demonstrate the system’s compliance with explicit

Hazard identifjcation ⓿ Worst case approach ❶ Quasi worst case ❷ Best estimates ❸ Probabilistic risk analysis ❹ Adapted from Uncertainties in global climate change estimates , E. Paté-Cornell, Climatic Change , 1996:33:145-149 10 / 23 Five levels of integration of uncertainty in risk assessment

• hazard is clearly defjned and solution is simple and inexpensive • hazard is poorly known and would have catastrophic impact, so benefjts of available solutions would dwarf the costs in any case 11 / 23 Integration level 0 ▷ Undertake hazard identifjcation ▷ Example: product is carcinogenic (yes/no) ▷ Suitable approach where no numerical tradeofg required:

victims in a specifjed event?” a reasonable solution to address the worst case parameters, someone can still highlight an “even worse” case which would require even more safety investment Image: The Great Wave ofg Kanagawa , K. Hokusai, ≈ 1825, public domain 12 / 23 Integration level 1 ▷ Worst-case approach ▷ Example: “What is the maximum number of potential ▷ Suitable approach when the worst case is clear and there is ▷ Typical approach used for emergency planning ▷ Problem : no matter how conservative you are concerning

13 / 23 • how to be coherent between “maximum probable fmood” & ⁇ fairly • can’t guarantee that people in difgerent locations are treated • diffjcult to assess resulting level of safety “maximum credible earthquake”? distribution of the potential loss distribution “maximum credible earthquake” in this area?” • insurance industry is concerned with maximum forseeable loss Integration level 2 ▷ Quasi worst-case and plausible upper bounds ▷ Example: “What is the “maximal probable fmood” or the ▷ Fundamentally, we are truncating the probability ? ⁇ ? Loss ▷ Problems:

parameters’ probability distributions of an accident or of losses in an accident in a chemical plant?” undesirable consequences) will be ignored in this approach 14 / 23 Integration level 3 ▷ Best estimates , using point values at the median of the ▷ Example: “What is the ‘most credible’ estimate of the probability ▷ Problem : a low probability outcome (even with hugely

15 / 23 levels of losses in difgerent degrees of failure of a probabilities or future frequencies of events • estimate probability distribution of each input parameter • propagate uncertainty through model to obtain distribution of outputs of interest • stochastic “Monte Carlo” methods particular dam?” Integration level 4 ▷ Probabilistic risk analysis based on mean output of interest parameter A model y’ = f(y, t ) parameter B parameter C ▷ Example: “What is the probability of exceeding specifjed See slides on Monte Carlo methods at risk-engineering.org