Model Risk Management: Model Validation

An Introduction to Model Validation

Model validation is a critical part of the MRM lifecycle; at this stage, the independent model validation function confirms that models are fit for purpose and that they can be put into use.

According to the FED and the OCC’s SR 11-7 guidance definitions of the validation criteria:

“model validation is the set of processes and activities intended to verify that models are performing as expected, in line with their design objectives and business uses. All model components, including input, processing, and reporting, should be subject to validation; this applies equally to models developed in-house and to those purchased from or developed by vendors or consultants. The rigor and sophistication of validation should be commensurate with the bank’s overall use of models, the complexity and materiality of its models, and the size and complexity of the bank’s operations.”

 This definition introduces a sense of flexibility with regard to model validation. However, GreySpark observes that only a handful of mature CIBs adapted their validation requirements to date on the basis of the complexity, of the risk level, of the use and of the type of model being validated. For example, a model classified as ‘high-risk’ under the SR 11-7 guidelines would require a bank to carry out  more rigorous testing than a low risk model.

In general, most banks struggle to adapt their validation processes to the different parameters that define a model. As such, validation steps are fastidious, and the documentation supporting the process is burdensome for model developers. Additionally, in-adapted validation requirements often lead to uncertainty as to the validity of the testing requirements and, as a consequence, to the appropriateness of the risk-mitigating measures associated with each model. In this article, GreySpark explores these challenges further and details existing best practices currently used by mature CIBs to effectively cope with them.

The Challenges of Model Validation

Model validation relies mostly on the results of the multiple tests that are performed on the model being validated. However, many CIBs do not adapt testing requirements to different types of models. The previous article in this series focused on that particular challenge, and it identified the best practices that can be implemented to improve the validity of testing requirements to ensure that they are fit for purpose.

CIBs experience the same issue with regard to documentation requirements, often relying on a ‘one template fits all’ philosophy, with little regard of whether the validation template is fit for purpose for the validation of a specific model or not. These development templates are used to document information on the model – for example, in the documentation of assumptions, justifications or testing results –  and they are completed by model developers or model owners and are reviewed by the independent validation function. When these templates are not adapted or adaptable to different types or models, then the risk that models are not validated in the context for which they were built is enhanced. Moreover, model owners and developers may be confused as to exactly which information should be provided as all fields in the template are not always relevant to the model at hand.

This lack of flexibility in the validation process creates a challenge that is reinforced by the fact that some CIBs have not yet taken the time to train and recruit the right personnel in the rigours of the model validation process. GreySpark observes that, in those cases where personnel are not adequately trained in the appropriate processes, the validation function then does not have the technical knowledge to appropriately assign testing requirements or to validate test results. This then causes the validation process to extend over weeks or even months and, by the end of the process, the model’s validation may have taken such a long time that it becomes obsolete, thus creating a loss of business opportunities.

A Flexible Approach to Validation

 GreySpark believes that, in order to reduce the current time frames associated with model validation, CIBs should allow flexibility in their process and the documentation supporting it. The end goal of model validation is to mitigate the risk associated with each model.

This means that the model validation stage should not be excluded from the risk-based approach taken at other steps of the model lifecycle. Therefore, testing and documentation requirements should be adapted and adaptable to each type of model or categories of models (see Figure 1).

Figure 1: A Flexible Approach to Risk Model Validation

Source: GreySpark analysis

A proposed approach to model documentation includes creating different development templates, differentiated on the basis of (non-exhaustive):

• the model’s risk level;
• the model’s asset class; and
• where applicable, the activity of the algorithm that the model feeds into.

For example, where CIBs observed that different asset classes would require different fit-for-purpose types of tests, the development template should be adapted on a per asset class basis to request only the identified tests specific to such asset classes.

However, having in place adapted documentation does not mean that more or less information should be shared depending on the model, but rather that the specifics of the required test results and information are tweaked to be fit-for-purpose. In fact, the model validation function could not perform its role appropriately without enough information on the model.

Therefore, in the most mature CIBs, all information that enabled the development of the model is shared from the model development team to the validation team. This information includes:

• a comprehensive description of the model;
• of its logical foundations;
• methodology
• limitations
• source codes
• test results
• key assumptions, and
• interdependencies with other models or algorithms.

To support and enable GreySpark’s proposed flexible approach to model validation, the most mature banks invested in automated and workflow support tools.  These tools enable the digital safekeeping and sharing of the development templates, and they quicken validation processes by allowing different validation lines to provide their approval of the models directly on the system without having to meet. These automated tools can also be set up to automatically assign the appropriate testing requirements to each model.

These best practices should improve transparency between lines of defence, fasten the validation process and increase the quality of the review of all risk management models. However, the validation process relies heavily on the quality and on the efficiency of the information sharing between stakeholders. The final article of this series will examine the contingencies for efficient information sharing and the best practices to enable it.

The fourth article in the series of five will examine the challenges and best practices around model change management.