Risk metrics Eric Marsden <eric.marsden@risk-engineering.org> “You can’t manage what you don’t measure”
▷ A measure is an operation for assigning a number to something ▷ A metric is our interpretation of the assigned number ▷ Tiere may be several measures (measurement methods) for one metric ▷ Example risk metrics: • deaths per passenger kilometre (transportation) • probability of failure on demand (systems reliability) • value at risk (fjnance) ▷ In these slides we focus on metrics for safety , rather than for fjnancial risks 2 / 41 Terminology “When you cannot measure, your knowledge is meager and unsatisfactory.” — Lord Kelvin
basis of continual improvement ▷ Safety performance is diffjcult to measure • success results in the absence of an outcome (injuries, losses, health impacts) • “Safety is elusive because it is a dynamic nonevent — what produces the stable outcome is constant change rather than continuous repetition” [K. Weick] • low incident rates, even over several years, do not guarantee that major accident hazards are controlled ▷ Tiere is no single and easy to measure indicator for safety 3 / 41 Context ▷ Measurement is a key step in any management process and forms the
Note: management’s obsession with metrics, people’s behaviour, can have a negative safety impact ▷ many facets of safety performance can be managed using good professional judgment, without quantitative measures ▷ Goodhart’s law: “When a measure becomes a target, it ceases to be a good measure” 4 / 41 and the resulting biases they introduce into
5 / 41 Source: dilbert.com Metrics and modern management
Source: Paul Slovic Q : Is coal mining getting safer? 6 / 41 Illustration: diffjculties in measuring safety Accidental deaths per Accidental deaths per thousand coal mine million tons of coal employees in USA mined in USA
▷ Individual risk : risk to any particular individual, either a worker or a member of the public ▷ Location-based risk : risk that a person who is continually present and unprotected at a given location will die as a result of an accident within the site ▷ Societal risk : risk to society as a whole • example: the chance of a large accident causing a defined number of deaths or injuries • product of the total amount of damage caused by a major accident and the probability of this happening during some specifjed period of time 7 / 41 Expressing risk to people
Probability that a specifjc individual (for example the most exposed individual in the population) should sufger a fatal accident during the period over which the averaging is carried out (usually a 12-month period). Individual risk [NORSOK Z-013N] ▷ Metric: individual risk per annum ( irpa ): probability that an individual is killed during one year of exposure ▷ Measure: observed fatality count number of people exposed 8 / 41 Metrics for individual risk
Suggested reading: Acceptance criteria in Denmark and the EU , Dutch Environmental Protection Agency Annual probability that an unprotected, permanently present individual dies due to an accident at a hazardous site ▷ is a property of the location, not of the individual ▷ mostly used in land-use planning ▷ ofuen represented with iso-risk contours (see fjgure) Location-specifjc individual risk 9 / 41 Metrics for individual risk
exposure • can be expressed per million hours worked, per plane takeofg, per km transported, per hour transported worked ▷ Potential loss of life ( pll ): statistically expected number of fatalities within a specifjed population during a specifjed period of time • note: when all members of a population are exposed to the same level of risk, 𝑄𝑀𝑀 = 𝑜 × 𝐽𝑆𝑄𝐵 10 / 41 Metrics for societal risk ▷ Fatal accident rate ( far ): expected number of fatalities per unit of • typical formulation: number of company/contractor fatalities per 10 8 hours
11 / 41 A Farmer diagram or F-N curve shows frequency and number of deaths for difgerent accident scenarios Note: ▷ drawn with a logarithmic scale on both axes ▷ a lower curve is better Metrics for societal risk 10 -2 Cumulative frequency per year, F large number of accidents -3 10 with few victims -4 10 -5 10 -6 10 -7 10 small number of accidents -8 10 with many victims -9 10 10 -10 -11 10 1 10 100 1000 10000 Number of fatalities, N
F-N diagram indicating acceptable risks, alarp zone and non-acceptable risks 12 / 41 Example F-N diagram Number of events per year Number of fatalaties per event
13 / 41 Source: Transport fatal accidents and FN-curves , HSE RR073, hse.gov.uk/research/rrpdf/rr073.pdf F-N diagram for transport accidents 10000 FN-curves for road, rail and air transport, 1967-2001 1000 Road 1969-2001 Accidents per year with N or more fatalities 100 10 1 Rail 1967-2001 A viat on i 1967-2001 0.1 0.01 1 10 100 1000 Number of fatalities, N
14 / 41 Source: Channel Tunnel Safety Case (1994) F-N diagram used in a safety case
F-N diagram for difgerent socially accepted activities Source: Risk and Safety in Engineering , course notes by M. Faber, ETHZ 15 / 41
• number of hours ofg work per 200 000 employee working hours (including work-related illness) ▷ ltif : Lost Time Injury Frequency • number of lost time injuries (fatalities + lost work day cases) per 1 000 000 work hours ▷ Also used in shareholder reporting on “industrial risk” (as the sole indicator…) • this is unfortunate since process safety metrics are equally (or more!) important for indicating the level of safety • occupational safety metrics are not correlated with process safety metrics (though there is a widely held view that they are) 16 / 41 Typical occupational safety metrics ▷ ltrir : Lost Time Reportable Incident Rate
17 / 41 Source: BP’s Sustainability Review, 2018, from bp.com Company safety indicators: example
18 / 41 Source: BP’s Sustainability Review, 2018, from bp.com Company safety indicators: example
• number of accidents per million fmights • number of fatal accidents per million fmights • number of people killed per year • number of hull losses per million fmights ▷ Most widely published metrics concern public air transport operations in scheduled operations, using Western-built aircrafu ▷ Accident rates tend to be higher for: • private or military fmights • cargo operations, test fmights • non-scheduled operations • aircrafu built in former Eastern-block countries 19 / 41 Illustration in civil aviation ▷ Typical metrics:
20 / 41 Weight ( mtow ) > 5700 kg Source: ICAO Annex 13, icao.int counted as an accident ▷ Destruction using military weapons ( e.g. MH 17 over Ukraine in 2014) not declared a hull loss. 10% of the aircrafu’s hull reserve value, whichever is lower, or has been • Tie aircrafu has sustained major structural damage exceeding 1 million usd or • Tie aircrafu is turbine powered and has a certifjcated Maximum Take-Ofg operations, training, maintenance/test fmights are all excluded. specifjcally scheduled/charter passenger or cargo service. Executive jet • Tie intention of the fmight is limited to normal commercial aviation activities, crew or passengers) • Person(s) have boarded the aircrafu with the intention of fmight (either fmight event where all of the following criteria are satisfjed: IATA defjnition of an accident ▷ iata (trade association for the major airlines) defjnes an accident as an
21 / 41 Source: IATA safety report for 2018, iata.org IATA safety indicators for civil aviation
22 / 41 Source: IATA safety fact sheet for 2015, iata.org IATA safety indicators for civil aviation All Accident Rate - Industry vs. IATA This rate includes accidents for all aircraft: it includes Substantial Damage and Hull Loss accidents for jets and turboprops. The All Accident rate is calculated as the number of accidents per million sectors. This is the most comprehensive of the accident rates calculated by IATA. Trend 2009-2013 2009 2010 2011 2012 2013 2014 Industry 2.71 2.77 2.63 2.11 2.24 1.92 2.48 IATA Member Airlines 1.78 1.49 1.87 0.74 1.60 0.94 1.49
indicators-2013-data/ Source: OGP report on Safety performance indicators – 2013 data , iogp.org/bookstore/product/safety-performance- 23 / 41 Illustration: safety performance metrics in oil & gas industry
24 / 41 Source: UK RSSB annual safety report, 2015 Illustration: safety performance metrics in railway transport Chart 9 Trend in SPAD risk Underlying risk (annual m ov ing av erage) S P ADs (annual m ov ing total) 200 400 ep 2006) 150 300 SPAD: Signal Passed At S at S P ADs S eptem ber 2006 baseline = 100% Danger (“burned red (percentage of risk light”) 100 200 isk 50 100 R 0 0 6 7 8 9 0 1 2 3 4 7 8 9 0 1 2 3 4 5 0 0 0 0 1 1 1 1 1 0 0 0 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 t r t r t r t r t r t r t r t r t r p a p a p a p a p a p a p a p a p a e e e e e e e e e M M M M M M M M M S S S S S S S S S
25 / 41 Source: UK RSSB annual safety report, 2015 Illustration: safety performance metrics in railway transport
Recommend
More recommend
