C-18 Model Risk in Financial Systems Casualty Actuarial Society - - PowerPoint PPT Presentation

▶
c 18 model risk in financial systems
SMART_READER_LITE
LIVE PREVIEW

C-18 Model Risk in Financial Systems Casualty Actuarial Society - - PowerPoint PPT Presentation

Apr 25, 2023 101 likes •411 views

C-18 Model Risk in Financial Systems Casualty Actuarial Society Annual Meeting November 6-9, 2011 Parr Schoolman, FCAS, MAAA, CERA Sr. Managing Director Aon Benfield Analytics Agenda Physics Envy Section 1 Insurance Risk Checklist and

slide-1
SLIDE 1

C-18 Model Risk in Financial Systems

Casualty Actuarial Society Annual Meeting November 6-9, 2011

Parr Schoolman, FCAS, MAAA, CERA

  • Sr. Managing Director

Aon Benfield Analytics

slide-2
SLIDE 2

1

Agenda

Section 1 Physics Envy Section 2 Insurance Risk Checklist and Implications for Model Risk Section 3 Conclusion

slide-3
SLIDE 3

Section 1: Physics Envy

slide-4
SLIDE 4

3

Model Risk and Uncertainty

Physics Envy

  • “Physics Envy Can be Hazardous to Your Wealth”, (Lo, Mueller 2010)

– The goal: to be able to build models of economic systems that are as predictive as those used in physics – The problem: people are less predictable than electrons – Risk and the Taxonomy of Uncertainty

  • Level 1: Complete Certainty
  • Level 2: Risk without Uncertainty
  • Level 3: Fully Reducible Uncertainty
  • Level 4: Partially Reducible Uncertainty
  • Level 5: Irreducible Uncertainty

– Sixth Level: Zen Uncertainty…”attempts to understand uncertainty are mere illusions”

slide-5
SLIDE 5

4

Risk vs. Uncertainty

Physics Envy and Financial Modeling

Source: “Physics Envy Can be Hazardous to Your Wealth”, (Lo, Mueller 2010)

Partially Reducible Uncertainty Fully Reducible Uncertainty Risk without Uncertainty Complete Certainty Irreducible Uncertainty All states of the system are determined exactly F=mA Newtonian Physics Randomness governed by a known probability distribution Casino Randomness governed by an unknown, but constant probability distribution Casino with no

  • dds posted

Randomness governed by an unknown probability distribution, with time varying parameters Model Risk Uncertainty beyond the reach of probabilistic analysis Known Unknown Unknowns Black Swans Mathematics, Physics Statistics, Chemistry, Biology Economics Philosophy

slide-6
SLIDE 6

5

Risk vs. Uncertainty

Physics Envy and Financial Modeling

Partially Reducible Uncertainty Fully Reducible Uncertainty Risk without Uncertainty Complete Certainty Irreducible Uncertainty All states of the system are determined exactly F=mA Newtonian Physics Randomness governed by a known probability distribution Casino Randomness governed by an unknown, but constant probability distribution Casino with no

  • dds posted

Randomness governed by an unknown probability distribution, with time varying parameters Model Risk Uncertainty beyond the reach of probabilistic analysis Known Unknown Unknowns Black Swans Mathematics, Physics Economics Philosophy

Source: “Physics Envy Can be Hazardous to Your Wealth”, (Lo, Mueller 2010)

slide-7
SLIDE 7

6

Risk vs. Uncertainty

Physics Envy and Financial Modeling

Partially Reducible Uncertainty Fully Reducible Uncertainty Risk without Uncertainty Complete Certainty Irreducible Uncertainty All states of the system are determined exactly F=mA Newtonian Physics Randomness governed by a known probability distribution Casino Randomness governed by an unknown, but constant probability distribution Casino with no

  • dds posted

Randomness governed by an unknown probability distribution, with time varying parameters Model Risk Uncertainty beyond the reach of probabilistic analysis Known Unknown Unknowns Black Swans Mathematics, Physics Statistics, Chemistry, Biology Economics Philosophy

Source: “Physics Envy Can be Hazardous to Your Wealth”, (Lo, Mueller 2010)

slide-8
SLIDE 8

7

Risk vs. Uncertainty

Physics Envy and Financial Modeling

Partially Reducible Uncertainty Fully Reducible Uncertainty Risk without Uncertainty Complete Certainty Irreducible Uncertainty All states of the system are determined exactly F=mA Newtonian Physics Randomness governed by a known probability distribution Casino Randomness governed by an unknown, but constant probability distribution Casino with no

  • dds posted

Randomness governed by an unknown probability distribution, with time varying parameters Model Risk Uncertainty beyond the reach of probabilistic analysis Known Unknown Unknowns Black Swans Mathematics, Physics Statistics, Chemistry, Biology Economics Philosophy

Source: “Physics Envy Can be Hazardous to Your Wealth”, (Lo, Mueller 2010)

slide-9
SLIDE 9

8

Risk vs. Uncertainty

Physics Envy and Financial Modeling

Partially Reducible Uncertainty Fully Reducible Uncertainty Risk without Uncertainty Complete Certainty Irreducible Uncertainty All states of the system are determined exactly F=mA Newtonian Physics Randomness governed by a known probability distribution Casino Randomness governed by an unknown, but constant probability distribution Casino with no

  • dds posted

Randomness governed by an unknown probability distribution, with time varying parameters Model Risk Uncertainty beyond the reach of probabilistic analysis Known Unknown Unknowns Black Swans Mathematics, Physics Statistics, Chemistry, Biology Economics Philosophy

Source: “Physics Envy Can be Hazardous to Your Wealth”, (Lo, Mueller 2010)

slide-10
SLIDE 10

9

Risk vs. Uncertainty

Physics Envy and Financial Modeling

Partially Reducible Uncertainty Fully Reducible Uncertainty Risk without Uncertainty Complete Certainty Irreducible Uncertainty All states of the system are determined exactly F=mA Newtonian Physics Randomness governed by a known probability distribution Casino Randomness governed by an unknown, but constant probability distribution Casino with no

  • dds posted

Randomness governed by an unknown probability distribution, with time varying parameters Model Risk Uncertainty beyond the reach of probabilistic analysis Known Unknown Unknowns Black Swans Mathematics, Physics Statistics, Chemistry, Biology Economics Philosophy

Insurance Risk Space

slide-11
SLIDE 11

10

Insurance Risk Uncertainty Continuum

  • Key characteristics to consider in the evaluation of insurance risk uncertainty and its

impact upon model risk: – Is the key loss driver a physical (earth, wind, fire, …) or behavioral process (lawsuits, market price volatility, regulatory actions…)? – Do we have a structural or statistical model of the underlying process? – How stable are historical trends, correlations? Is past performance relevant for prospective risk?

slide-12
SLIDE 12

Section 3: Insurance Risk Checklist & Implications for Model Risk

  • Asset Risk
  • Reserve Risk
  • Catastrophe Risk
slide-13
SLIDE 13

12

Monthly Change in Corporate Credit Spreads

  • 0.5
  • 0.4
  • 0.3
  • 0.2
  • 0.1

0.1 0.2 0.3 0.4 0.5 Dec-91 Dec-92 Dec-93 Dec-94 Dec-95 Dec-96 Dec-97 Dec-98 Dec-99 Dec-00 Dec-01 Dec-02 Dec-03 Dec-04 Dec-05 Dec-06

BBB 3-5 Yr

Credit Crisis Lessons

Asset Risk Modeling Challenges LTCM Crisis 9/11

Standard Deviation 8.4 basis points

Source: Bloomberg, Aon Benfield Analytics

slide-14
SLIDE 14

13

Monthly Change in Corporate Credit Spreads

  • 1
  • 0.5

0.5 1 1.5 2

Dec-91 Dec-92 Dec-93 Dec-94 Dec-95 Dec-96 Dec-97 Dec-98 Dec-99 Dec-00 Dec-01 Dec-02 Dec-03 Dec-04 Dec-05 Dec-06 Dec-07 Dec-08

BBB 3-5 Yr

Credit Crisis Lessons

Asset Risk Modeling Challenges

October 2008 spread change 147.2 basis points 17x Standard Deviation result

Source: Bloomberg, Aon Benfield Analytics

slide-15
SLIDE 15

14

Insurance Model Risk Characteristics Checklist

Asset Risk

  • Risk driven by behavioral “animal spirits”
  • Risk management models (trading VaR models) statistical rather than structural in nature
  • Underlying trends and correlations unstable

– Input assumptions too often confused with model output

Insurance Model Risk Characteristics Checklist: Asset Risk Risk Process

Physical Behavioral

Model of Process

Structural Statistical

Parameter Stability

Stable Unstable

Partially Reducible Uncertainty Fully Reducible Uncertainty Irreducible Uncertainty

slide-16
SLIDE 16

15

Reserve Risk

  • What is Reserve Risk?
  • Management: How much will my actuary’s reserve estimate change this year?
  • Actuary: How far off could my estimate of ultimate losses be?

SCHEDULE P - PART 2 - SUMMARY

1 2 3 4 5 6 7 8 9 10 11 12 Years in Which INCURRED NET LOSSES AND DEFENSE AND COST CONTAINMENT EXPENSES REPORTED AT YEAR END ($000 OMITTED) DEVELOPMENT Losses One Two Were Incurred 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 Year Year 1 Prior 241.8 262.5 279.0 295.7 308.0 315.5 323.1 326.6 331.3 336.5 5.2 9.9 2 2001 234.1 235.4 237.7 240.7 242.5 243.6 244.3 244.7 244.9 245.0 0.1 0.3 3 2002 XXX 231.4 226.7 228.4 231.3 232.3 232.5 232.5 232.5 232.6 0.1 0.1 4 2003 XXX XXX 244.8 234.3 231.9 230.4 229.3 229.2 228.5 228.1

  • 0.4
  • 1.0

5 2004 XXX XXX XXX 259.9 246.1 241.3 237.9 235.3 233.8 232.8

  • 1.0
  • 2.4

6 2005 XXX XXX XXX XXX 277.8 267.4 262.5 258.8 256.3 254.8

  • 1.5
  • 3.9

7 2006 XXX XXX XXX XXX XXX 262.9 256.3 257.6 248.1 244.1

  • 4.1
  • 13.6

8 2007 XXX XXX XXX XXX XXX XXX 283.7 284.5 274.4 272.5

  • 1.8
  • 12.0

9 2008 XXX XXX XXX XXX XXX XXX XXX 325.7 322.2 318.8

  • 3.4
  • 6.9

10 2009 XXX XXX XXX XXX XXX XXX XXX XXX 291.9 288.2

  • 3.7

XXX 11 2010 XXX XXX XXX XXX XXX XXX XXX XXX XXX 289.9 XXX XXX 12 Totals XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX

  • 10.5
  • 29.5

Management View Actuarial View

slide-17
SLIDE 17

16

Reserve Risk

Initial Booked Loss Ratio Volatility – Commercial Auto Liability

1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

  • 6.0%
  • 4.0%
  • 2.0%

0.0% 2.0% 4.0% 6.0% 8.0% 10.0% 12.0% 14.0% 55% 60% 65% 70% 75% 80% 85% Calendar Year Loss Reserve Development Accident Year Ultimate Loss Ratio at 12 Months

Source: SNL US Industry Aggregate Schedule P, Aon Benfield Analytics

slide-18
SLIDE 18

17

1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008

  • 6.0%
  • 4.0%
  • 2.0%

0.0% 2.0% 4.0% 6.0% 8.0% 10.0% 12.0% 14.0% 55% 60% 65% 70% 75% 80% 85% 90% 95% Calendar Year Loss Reserve Development Accident Year Ultimate Loss Ratio at 12 Months & Current

Reserve Risk

Most Recent Booked Loss Ratio Volatility – Commercial Auto Liability

Subsequent Development

2009 2010 Source: SNL US Industry Aggregate Schedule P, Aon Benfield Analytics

slide-19
SLIDE 19

18

The Cycle: A Structural View

  • 15%
  • 10%
  • 5%

0% 5% 10% 15% 55% 60% 65% 70% 75% 80% 85% 90% 95% Accident Year Ultimate Loss Ratio at 12 Months Calendar Year Loss Reserve Development

Coming off the hard market detect decrease in pricing, terms & conditions and increase loss pick Hit barrier of “maximum bookable loss ratio”, find reasons why it will be different this time; adequacy

  • f initial loss pick decreases,

reserve deficiencies build up Will ERM/SOX etc. improve rigor of initial loss picks this cycle? Dam breaks, market hardens, substantial increase in rate leads to drop in current AY loss pick concurrent with prior year development Fill soft market reserve hole booking conservative initial loss ratios

slide-20
SLIDE 20

19

Insurance Model Risk Characteristics Checklist

Reserve Risk

  • Risk influenced by behavioral claims handling process and framing bias
  • Reserve risk methods statistical in nature, but use of predictive modeling techniques

increasing

  • Underlying trends and correlations are more stable than for asset risks, still not

Insurance Model Risk Characteristics Checklist: Reserve Risk Risk Process

Physical Behavioral

Model of Process

Structural Statistical

Parameter Stability

Stable Unstable

Partially Reducible Uncertainty Fully Reducible Uncertainty Irreducible Uncertainty

slide-21
SLIDE 21

20

Catastrophe Risk

Catastrophe Model

  • Historical data available regarding event

frequencies

  • Combined with property damage estimates

by event type and property characteristics

  • Result: modeled loss cost ratios

by peril

Historical Hurricane Paths by Category Hurricane Loss Cost Ratio Earthquake Loss Cost Ratio Tornado Frequency

slide-22
SLIDE 22

21

Catastrophe Risk

Exposure Concentration

  • Loss cost ratio by peril combined

with exposure data

  • Location level data quality is

important for improving reliability

  • f modeled results
  • Catastrophe loss severity is

amplified by exposure concentrations – Correlation driven by event size and exposure concentrations

slide-23
SLIDE 23

22

Catastrophe Risk

Reasonable Tail Estimates?

US P&C Industry: Hurricane Occurrence PML's

Model 1 Model 2 Model 3 Model 4 Model 5 Model 6 AAL 11 12 13 15 19 21 SD 25 27 31 32 39 41 100 Yr PML 120 124 129 141 162 177

# St Deviations 4.7 4.6 4.1 4.4 4.1 4.3

250 Yr PML 183 198 191 216 235 258

# St Deviations 7.2 7.3 6.1 6.8 6.0 6.3 ($Billions)

slide-24
SLIDE 24

23

Insurance Model Risk Characteristics Checklist

Catastrophe Risk

  • Risk process is physical in nature…with some regulatory behavioral risk
  • Catastrophe models are structural in nature
  • Correlations are driven by geographic concentrations of the fixed locations of insured

property; frequency and loss trends have uncertainty related to limited number of

  • bservations

Insurance Model Risk Characteristics Checklist: Cat Risk Risk Process

Physical Behavioral

Model of Process

Structural Statistical

Parameter Stability

Stable Unstable

Partially Reducible Uncertainty Fully Reducible Uncertainty Irreducible Uncertainty

slide-25
SLIDE 25

Section 3: Conclusion

slide-26
SLIDE 26

25

Model Risk and Implications for Capital Modeling:

Capital Modeling Theory and Practice

  • Theory presents objective, “scientific” process

Data Parameters Model Capital RAROC & Target CR

slide-27
SLIDE 27

26

Model Risk and Implications for Capital Modeling:

Capital Modeling Theory and Practice

Data Parameters Model Capital RAROC & Target CR

Flexibility Flexibility Huge Flexibility Flexibility

  • In practice broad enough “range of reasonableness” at multiple modeling junctures to

justify or rationalize very broad range of outcomes

  • Modeling often does not capture all of the risks management must consider

– Distribution channel – Regulatory, rate filing issues – Competitor actions and reactions – Residual market & pool exposures

slide-28
SLIDE 28

27

Model Specification Error Model Granularity Total Error

Synthesis Misspecification Marginal Misspecification

Model Risk and Implications for Capital Modeling:

Balance Complexity And Accuracy

Pressure for Greater Granularity

slide-29
SLIDE 29

28

Model Risk Recommendations for Better Decision-making

  • Acknowledge the limits of data available to support the model

– Build structural models where possible, but recognize more complex ≠ more accurate – Maintain humility to recognize when information is not available to support structural model assumptions – Do not confuse model assumptions with model results

  • Stress test key assumptions

– How sensitive are results to correlation assumptions? – Maintain multi-model perspective – Maintain realistic disaster scenario set with control limit targets regardless of the estimated probability of event

  • History can be as useful as the math when thinking about risk

– Bad things have happened in the past… what if they happened again? – “History doesn’t repeat itself, but it often rhymes” – Mark Twain