SHOULD A MODEL KNOW ITS OWN ID? Some Thoughts About 7 TH ANNUAL - PowerPoint PPT Presentation

SHOULD A MODEL ‘KNOW’ ITS OWN ID? Some Thoughts About 7 TH ANNUAL RISK AMERICAS CONFERENCE Mitigating Inventory MAY 17-18, 2018 NEW YORK CITY Risk By Accurately Tracking Model P RESENTED BY J ON H ILL , P H . D. FORMER MANAGING DIRECTOR Usage GLOBAL MODEL RISK GOVERNANCE CREDIT SUISSE NEWYORK JONHILL @ OPTONLINE . NET

Disclaimer All of the ideas, opinions, suggestions, notions or asides offered in this presentation are entirely the opinions of the speaker and should not be construed to represent in any way those of Credit Suisse, Morgan Stanley, Citigroup or any other previous employers. Furthermore, any anecdotes, cautionary tales or war stories that may insinuate themselves into this presentation shall be understood to have occurred at a mythical institution that will only be identified as “Retro Bank”, unless otherwise specifically identified by the speaker.

• What is inventory risk? • What types of questions about model usage and inventory are difficult to answer with today’s database inventorIes? • How could these questions be answered accurately? • Description of the necessary functionality to support a Model Transponder Function. • A two-step phased implementation approach can Contents minimize disruption and production overhead. • Summary: Pros and Cons

SOME THOUGHTS ON My Definition of Inventory Risk (adapted from SR11-7): MITIGATING Inventory risk is the risk resulting INVENTORY RISK BY from incomplete or inaccurate quantitative model inventories, the IMPROVING MODEL use of models that have previously been retired or remain USAGE unvalidated or the use of models TRANSPARENCY that have not be entered into inventory.

Introduction Model Inventory Attestation Is Still Primarily a Manual Process! Why Is That? o One of the more daunting challenges facing model risk managers at major financial firms is the task of ascertaining that the model inventory, however it is implemented and maintained, is complete and accurate. 1 o At almost every firm this is accomplished through a manual process called attestation: model managers or functional heads for every asset class and business unit are asked to sign off on the complete set of models that fall within their domain of ownership and responsibility. o Such attestations are typically done via email requiring the a model risk manager to a the list of models that inventory indicates is owned and maintained by the model supervisor or functional and obtaining a confirmation by return email. Such a process can be both clumsy and error-prone – some models may simply be overlooked in the process (the technical term is “falling through the cracks”), some may be ‘orphans’, models mis -assigned due to staff turnover or re-allocation of responsibilities and therefore without owners; some orphans may no longer be in use. 1 None of the firms I have worked at (Salomon Smith-Barney, Citigroup, Morgan Stanley, Credit Suisse) or with as a consultant have any accurate quantitative way of answering these types of questions other than to query model owners/developers or their downstream users and receiving qualitative estimates. It is also un uncomfortable fact that model supervisors/owners/developers do not always know a who all of their downstream users are.

Introduction Model Inventory Attestation Is Still Primarily a Manual Process! Why Is That? o Resolving these discrepancies can require numerous iterations of the attestation process to determine the current correct ownership of orphan models. And of course, it is not an uncommon experience that some models have no owner assigned at all due to staff turnover. Particularly problematic are upstream and downstream dependencies between models. We rely on model owners to identify all upstream models, but very often they will have complete knowledge of all downstream models, models that receive other models’ output as their input. o Particularly problematic are upstream and downstream dependencies between models. While model owners should be aware of upstream dependencies (these can be traced by following all model inputs back to their source 2 ), but very often they will not have complete knowledge of all downstream models, models that receive other models’ output as their input. These would be best known to the downstream model users. 1 Note: the role of model risk manager is relatively new and complements the role of model validator in mitigating model risk 2 If they cannot there are larger problems in model development management.

In a Bank Exam, Could Your Firm Give Accurate Answers to the Following Seven Basic Questions Involving Model Inventory? 1 1) What is the exact number of different models that have been used over the last year? 2) How often has each model been executed, by day, by month, by year? 3) Where are the firm’s models being used? Business unit, legal entity, geographic regions? 4) Are there any models in your inventory that were not executed during the last year? 5) Are there any models that were executed on any of your firm’s computers that do not appear in inventory? Please provide a full listing. 6) Are you able to provide a full list of the IDs of models that exhibit significant seasonality? If so, what are the peak and trough’s of seasonal model usage. 7) Were there any instances of a retired model still being executed during the last year? 1 There are likely other types of questions regarding model inventory that are difficult to answer accurately. These seven are the most important questions I can think of. Perhaps you can think of some others.

Let’s Call it “Inventory Risk” Inability to answer the previous questions regarding model usage is indicative of a form of model risk that is that is not often identified or analyzed in its totality because it belongs to a class of seldom recognized risks that reside outside of and between models. 1 What are the sorts of liabilities that may arise from model inventory risk? Here are a few …. o Regulatory risk arising from incomplete or inaccurate model inventories (i.e. CCAR bank exams) o Financial and regulatory risks arising from the use of unvalidated or retired models o Difficulty in identifying models still in inventory but no longer in use o Inability to enforce model risk management practices uniformly across all models, asset classes, regions and legal entities. o Manual inventory attestation processes are error-prone and invariably result in errors of omission o Incomplete understanding of upstream and downstream model dependencies o Lack of transparency into firmwide model usage, regionality, seasonality, etc. In an age of automation, machine learning and big data we really should ask ourselves if we cannot find better ways to make firmwide model usage more transparent and in doing so help to automate the model attestation process 1 A tip of the hat to Martin Goldberg for his seminal 2017 paper entitled “Much of Model Risk Does Not Come From Any Model”, Th e Journal of Structured Finance, Spring, 2017, pp. 32-37. Although not described in this paper, inventory risk is clearly from the class of less well-recognized model risks that are external to models. Martin is currently working at Bloomberg on credit risk models.

This section will attempt to identify a single underlying reason why few if any firms can THE HEART OF THE answer the 7 questions MATTER on the previous slide with a high degree of confidence. This is the true source of most model usage opacity and inventory risk

The Heart of the Matter Is This: Models Do Not ‘know’ Their Own IDs The root cause of model usage opacity may be traced to this single surprising blind spot in most firms’ model risk management framework. 1 Let’s try to put this into perspective by comparing to some other familiar technologies: • my smart phone ‘knows’ its unique serial number (it’s embedded in the permanent onboard memory that stays with the phone for life). • My washing machine knows its own serial number too, so does my automobile. These are embedded in the onboard electronics that control these devices. • Even before electronics, serial numbers were stamped on the frames of every automobile that Henry Ford produced and somewhere on almost all manufactured products of any significance. Today, Tesla can track every car they’ve ever made, its location, travel speed, level of charge, etc. Most important financial models are assigned Model IDs as a convenient lookup index into the automated model databases that almost all firms have to maintain today. These databases typically house all of the relevant documentation for each model such as development and validation documents, and in some rare cases, even source code. Yet the models themselves do not ‘know’ their own IDs in the sense that the ID number is embedded in the model’s source code. In the next section I will introduce the concept of a Transponder Function, which if added to every model in a firm’s inventory, can go a long way towards improving the transparency of model usage and mitigating many of the risks listed in slide #8. 1 At first blush this may not seem to be a true root cause. This presentation will endeavor to convince any doubters that this is indeed the case.

Creating models that ‘know’ their own IDs by embedding them in their TWO STEPS TO source code is a simple MITIGATING MODEL yet necessary first step. But this alone is not INVENTORY RISK sufficient. The real heart of the matter is what we might do with that embedded information.