INVENTORY Lloyds Banking Group Scotiabank TIAA NYU Tandon School - - PowerPoint PPT Presentation
INVENTORY Lloyds Banking Group Scotiabank TIAA NYU Tandon School of Financial Risk Engineering THE MANY CHALLENGES OF MODEL INVENTORY Can Any Financial Firm Truthfully Claim to Have a Complete and Accurate Model Inventory? Jon R. Hill, Ph.
Lloyds Banking Group Scotiabank TIAA NYU Tandon School of Financial Risk Engineering
Jon R. Hill, Ph. D. Professor of Financial Risk Engineering, NYU Independent Consultant in Model Risk Management jh7050@nyu.edu jonhill@optonline.net
Jon Hill, Ph. D., is a Subject Matter Expert in Model Risk Management and is an independent consultant to financial institutions.
and Risk Engineering.
awareness of model risk to the broader risk and financial communities and to provide a forum for topical discussion of model risk management challenges and regulatory requirements. Jon is a former Managing Director at Credit Suisse with over twenty years of experience in various areas of quantitative finance. As head of the Global Head of Model Risk Standards at Credit Suisse he led a team comprised of 14 model risk managers in New York London, Zurich, Mumbai and Singapore. Jon’s team had responsibility for the
Prior to joining Credit Suisse in 2017, Jon founded and led the Validation team for Market and Operational Risk Models at Morgan Stanley for 6 ½ years. Prior to Morgan Stanley Jon performed hands-on model validations at Solomon Smith-Barney ( which later became Citigroup) was a member of a quantitative finance research team.
the US and Europe.
comprehensive guidance for model risk management
discussed by today’s session panelists
In April, 2011 the FRB & OCC Jointly Issued SR11- 7/OCC2011-12.* This 21-page Document Set the Bar for Model Risk Management (MRM) at All Conforming Firms
“For the purposes of this document, the term model refers to a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates.” “A model consists of three components: an information input component, which delivers assumptions and data to the model; a processing component, which transforms inputs into estimates; and a reporting component, which translates the estimates into useful business information.”
* ‘Supervisory guidance on model risk management’, 4th April, Supervisory and Regulatory Letter SR11-7. Document may be downloaded from: https://www.federalreserve.gov/supervisionreg/srletters/sr1107.htm
SR11-7 Established Requirements for Firm Wide Aggregation of Model Risk, Posing of Effective Challenge (by MRM), and Tiering Models by Risk Rating
“Banks should consider risk from individual models and in the
dependencies among models; reliance on common assumptions, data, or methodologies; and any other factors that could adversely affect several models and their outputs at the same time. With an understanding of the source and magnitude of model risk in place, the next step is to manage it properly. A guiding principle for managing model risk is "effective challenge" of models, that is, critical analysis by objective, informed parties who can identify model limitations and assumptions and produce appropriate changes. The range and rigor of validation activities conducted prior to first use of a model should be in line with the potential risk presented by use of the model.
SR11-7 Established A Formal Requirement for Firm-Wide Model Inventories with Comprehensive Information About Models in Use or Recently Retired as well as Their Intended Use and Any Upstream Data or Model Dependencies.
Banks should maintain a comprehensive set of information for models implemented for use, under development for implementation, or recently retired. While each line of business may maintain its own inventory, a specific party should also be charged with maintaining a firm-wide inventory of all models, which should assist a bank in evaluating its model risk in the
validation should be included as a separate model and cross- referenced with other variations. The inventory should describe the purpose and products for which the model is designed, actual or expected usage, and any restrictions on use. It is useful for the inventory to list the type and source of inputs used by a given model and underlying components (which may include other models), as well as model outputs and their intended use.
understanding of inter-dependencies within the firm’s model ecosystem
attestations (voluntary declarations) by model owners and stakeholders (developers, supervisors and users).
have a complete and accurate model inventory.
attestation
Model Development vs. Model Risk Management
➢ As a result model developers tend to have little interest in modifying their models to accommodate the requirements of MRM. ✓ Model Risk Managers are tasked with identifying and mitigating the holistic model risks that reside in a firm’s model ecosystem fabric. ✓ Model developers are tasked with designing and implementing and testing models that efficiently and accurately convert input data into useful outputs. ✓ These two groups tend to work completely independently within separate silos at most financial firms.
Today All Leading Financial Firms Engage in a Process of Assigning Their Models into High, Medium and Low Risk Tiers
This practice is partly motivated by a single line in SR11-7 that Provides enough wiggle room for a Potential Escape Clause for Model Owners: “The range and rigor of validation activities conducted prior to first use of a model should be in line with the potential risk presented by use of the model.” Model risk tiering allows firms to substantially reduce validation requirements for the lower risk model tiers. As a result, some model owners may look for creative ways to have their software ‘widgets’ bucketed into the lowest risk tier. This is one of the reasons that at most firms a rigorous model discovery process will invariably reveal a set of potential models that have been ‘hiding in the shadows’.
Model Risk Managers at many firms that strive to be SR11-7 compliant have developed procedures to classify all quantitative software implementations into one of three categories: model, non-model (calculator) or something in- between (near-model, tool, etc.).
FLOD: would prefer not to have anything they own classified as a model to avoid the overhead of a rigorous, SR11-7 compliant full validation. SLOD: has to answer to senior management and regulators about why a software implementation was or was not classified as a model and therefore subject or not subject to the rigors of a full SR11-7 model validation. These opposing priorities sometimes result in ‘rather animated conversations’ between first and second LODs over which cloud domain a particular implementation should be assigned to: true model, near-model or non-model.
1) Definition of a model. How to distinguish models from near models from calculators (or tools). What criteria may be useful? 2) Should a firms create a single inventory for all its models, or allow a multiple, distributed inventory approach (i.e. independent inventories for different classes of models)? 3) How can a firm be certain that’s its model inventory is both accurate and complete (a key requirement of SR11-7)? Is manual attestation by model
4) Should quantitative non-models also be stored in inventory? 5) Aggregation of model risk to a firm wide level requires an understanding of model and data inter-dependencies throughout a firm’s model ecosystem. How can such dependency data be entered into model inventories? 6) What are the most important data fields that should be present in any firm’s model inventory? Should this include data about model usage? 7) How can firm’s track how, when and where their models are being used throughout the year? 8) What to do about EUC (end user controlled) models, many of which may be ‘hiding in the shadows’ from MRM?
Of the Many Challenges Confronting Today’s Model Risk Managers, Eight of the Most Daunting Are Related to Inventory:
Of those eight challenges listed in the previous slide, the panelists have selected three to be examined in greater depth during today’s session
#7) On the dynamics of model usage. How can firms track how, when and where their models are being used throughout the year?
Lloyds Banking Group #3) How can a firm be certain that its model inventory is both complete and accurate? Creating and maintaining a complete and accurate inventory of the firm’s models. Why are so many models ‘hiding in the shadows’? - Discussion led by Nikolai Kukharkin, Managing Director, Head of Model Risk Management, TIAA #5) How can model and data inter-dependencies within a firm’s model ecosystem be determined in order to support aggregation of model risk? - Discussion led by Ajeeth Sankaran, US Head of Model Risk, Scotiabank
A Complete Picture of the Upstream/Downstream Model and Data Inter- Dependencies Within a Firm’s Entire Model Ecosystem Can be Very
Diagrams are a Useful Way of Capturing these Complexities.
This network graphic was created with GEPHI – an
Recent Publications by This Speaker That Address Some of the Many Challenges of Model Risk Management
Request Copies From: jh7050@nyu.edu or jonhill@optonline.net
Inventory, risk-tiering, model inter-dependencies and other topics in model risk management are pursued in greater detail in the following refereed journal articles: Hill, J. R. (2018) “Shouldn’t A Model ‘Know’ Its Own ID?”, The Journal of Structured Finance, Fall, pp. 89-98 Hill, J. R. (2019) “The 14 Top Challenges for Today’s Model Risk Managers: Has the Time Come to Think About Going Beyond SR11-7?”, The Journal Of Risk Management In Financial Institutions, Spring, Vol. 12, 2, pp. 145-167, ISSN 1752-8887 Hill, J. R. (2020) “A Smarter Model Risk Management Follows From Making Smarter Models: An Abbreviated Guide for Building the Next Generation of Smart Models”, The Journal Of Risk Management In Financial Institutions, a model risk edition, Winter, Vol. 13, No.1, pp. 24-34, ISSN 1752-8887.
1) It’s not a model because it is just a calculation 2) It’s not a model because we are still building it; this is just the first draft 3) It’s not a model because it has a minimal impact 4) It’s not a model because I say it isn’t 5) It’s not a model because an (upstream) input into a (downstream) model cannot itself be a model 6) It’s not a model because it’s a vendor-supplied solution, and vendor solutions are the vendor’s responsibility, not ours. 7) It’s not a model because it is only used for accounting, and is subject to a high level of Qualitative Adjustments 8) It’s not a model because we can’t explain it to senior management 9) It’s not a model because we only use it for reporting to regulators 10) It’s not a model because we don’t have resources and we don’t want to increase our model count and hence we really prefer not to consider this to be a model And finally, the lamest of all reasons for “why it’s not a model”: 11) It’s not a model because it is just an EUC spreadsheet, and spreadsheets, by definition, are not models!
Model Owners Often Go to Extraordinary Lengths to Convince MRM About “Why It’s Not a Model” Here are 11 Choice Evasions That Were Submitted by Model Risk Managers at a Recent MRM Conference All of These Are Absurd …. Although Some Are Funny
Suman Datta Head of Portfolio Quantitative Research Lloyds Banking Group Ajeeth Sankaran US Head of Model Risk Scotiabank Nikolai Kukharkin Managing Director, Head of Model Risk Management TIAA Just Group Jon Hill Professor of Model Risk Management NYU Tandon School of Financial Risk Engineering
Suman Datta, Head of Portfolio Quantitative Research, Lloyds Banking Group Ajeeth Sankaran, US Head of Model Risk, Scotiabank Nikolai Kukharkin, Managing Director, Head of Model Risk Management, TIAA Jon Hill, Professor of Model Risk Management, NYU Tandon School of Financial Risk Engineering
Ask your questions using the question icon on the bottom right hand-side of your screen.