Are You Searching For Validity?

A new generation of models lies at the heart of modern banking, including commercial credit rating models, credit portfolio models, capital adequacy models, and operational risk models. Validating these models is becoming critical to sound bank risk management because:
- the models are ever more frequently used as the basis for important internal bank decisions; and
- third parties such as regulators, rating agencies and investors increasingly demand that banks prove the strength of their risk and capital modeling.

Taken together, these trends will soon force the banking industry to put in place more formal strategies for validating models, whether the models are built internally or purchased from vendors. These strategies can be thought of in terms of the four separate activities set out in Figure 1.

Some banks know a lot about validating market risk models as result of complying with the 1996 amendment to the 1988 Basel Capital Accord. But the banking industry’s new generation of models are based on less rich data than market risk models – and the new validation strategies will accordingly take a different shape.
To develop a useful framework for validating the new breed of models across their enterprise, banks need to tackle four questions: 
1. What do regulators say about the validation of credit and other complex models where data is scarce?  
2. What methodologies can be used to validate models beyond statistical backtesting?
3. How can improved model processes and governance support validation?
4. Do users of model results, such as risk committee members, understand the models?

Stepping through the regulatory minefield

Since around 2000, regulators have developed an intense love/hate relationship with risk and capital models.

On the one hand, regulators love the way that models push banks to be rigorous, objective and quantitative about capital adequacy. Key pieces of regulation such as Basel II and SR99-18 in the US put quantitative risk models and their validation right at the heart of risk management and regulatory compliance.

On the other hand, regulators are increasingly wary that banks might become too dependent on models that are not up to the job, or that decision makers will misinterpret model results.

The result is a series of recent regulatory rules, papers and advisory notes on risk model validation. [1]

These papers stress that the responsibility for validation lies with senior bank management and lay out some broad principles, but they avoid setting out prescriptive procedures for validating specific models [2] – which means they can’t be used as a kind of route map for banks.

A couple of things are clear from these regulator comments, however, and the first is that the regulatory interest in models is not going to go away – regulators will continue to be interested in how a bank builds and uses models for all its material risks including interest rate risk and concentration risk (Pillar II of Basel II) as well as market, credit and operational risks (Pillar I).

Second, banks cannot expect, and probably should not want, a detailed route map for model validation from regulators around the world. Instead, they must use the regulators’ high-level principles to build their own map.
 
Beyond backtesting

For the new generation of risk models, backtesting often has to be supplemented with other quantitative procedures, such as benchmarking, and the replication of models. For example, where internal historical data is scarce in the case of low-default portfolios benchmarking tools might include:
– comparison of ratings and rating migration matrices to the ratings and migration matrices of rating agencies;
– the use of internal or external expert judgments; and
– comparison to market-based proxies for credit quality, eg, premiums for credit derivatives, bond spreads, and credit ratings on securitization tranches.

Benchmarking is not the only way to supplement backtesting. Models can also be subjected to replication – independent builds of the model to see if the same results are generated – and the testing of key results, as well as direct mechanical checks such as taking the model apart to check each component such as the equations, the coding, and the data quality.

An important point here is that regulators accept there is no single validation methodology that can be applied to all the different kinds of bank risk portfolio.

Indeed, for problematic portfolios and risk types, banks must triangulate between multiple methodologies using their judgment. They will also have to document this complex model building and validation process so that it can be verified by internal audit and by third parties.

Model process and governance

This all means that it is doubly important that modeling and validation are conducted within a framework of transparent processes and rigorous governance. The framework must extend far beyond the model itself to include decisions about model design, data selection, judgmental adjustments, result reporting formats and so on.

For reference, Figure 2 sets out how Basel regulators see this wider framework of validation in the case of a bank’s internal rating system, including benchmarking and backtesting. In addition, Table 1 sets out some questions that senior managers can use to prompt wide-ranging thinking about models and model validation.

Table 1. Key model validation questions for senior managers

I. Data
a. Is the data time series appropriate and rich enough?
b. Is the data cleansing process applied consistently?
c. What checks/balances exist to ensure that the data reflects the portfolio risk being modeled?
d. Is data security adequate?

 

II. Model design 
a. Are all assumptions documented?
b. Have other theories been robustly argued and reasons for rejection objectively analyzed?
c. Do senior management understand the principle of the model and associated reports and have they approved the model development project?

III. Process
a. Does the independent validation team have the right skills and resources?
b. Feedback mechanism to update the model if the results are misleading?
c. Is the model validation process itself documented?

IV. Governance
i. Model risk policy, eg, correct documentation, sign off, discussion of assumptions, theoretical background for theory?
ii. Validation focus: Not all models need to be validated, eg, models for financial reporting may have separate reconciliation process
iii. Restrictions: eg, use prototypes for given period only, eg, six months?
iv. Validation team independent of the developers?
v. Are models and validation process audited by Internal/External audit?
vi. What is the correct frequency for model review, eg, annually, or more frequently if there is a change to the bank’s product mix or client base?

The table starts off with familiar questions about data integrity, but also includes process and governance issues such as the need to ensure that those validating a model are independent from the builders of the model. Making sure that each stage of a modeling process is explicit and documented is not simply a compliance exercise – it’s also best practice for business reasons.

One of the great strengths of risk models is that any weaker components of the model can be identified and strengthened over time. But it can be difficult to improve complex risk and capital models unless the role of judgments is defined and documented, eg, in the case of credit models, judgments about qualitative risk factors such as management strength, or judgmental overrides of probability of default estimates in the final rating.

The bank should also set expectations in advance for how models should perform, eg, the level of concordance between a new credit rating model’s output and the order rankings produced under traditional risk rating systems.

Regulators will demand clear documentation because it makes their job easier, and because documentation mitigates the effect of the bank losing key staff who understand how the bank’s models work.

An interesting question here is the degree of transparency expected for vendor models, compared to internally built models. Regulators understand that banks often have to make a trade-off between transparency and convenience when they decide to use vendor models, but stress this does not absolve banks of responsibility for validation. For example, OCC-2000-16 says that “within the limit that vendors will not reveal proprietary information, banks [should] require that the vendors provide information on how the vendor built and validated the model”.

Regulators might also ask banks to demonstrate that they understand the key concepts behind any vendor model and how it should be applied, and that the bank has independently tested key results such as bank portfolio values or parameter values under some worst-case scenario.

Are users in the loop?

Another challenge is the relationship between model builders and the users of model results such as senior management.

It’s easy for model designers to over-interpret the desires of managers who commission and use risk models. For example, they may assume that the user wants to hear a (potentially misleading) single result, rather than a more complex-sounding range of numbers and an explanation of the modeling assumptions that underpin this range.

Misunderstandings can also creep in as risk modelling becomes part of a complex, organizational process and as model results begin to be adapted for many different purposes from risk management to risk pricing and incentive compensation.

One way to combat this is to make sure model users are involved in the model building process. This ensures that modelers are made aware of the user’s business insights and intuitions, as well as encouraging users to buy into the modeling approach. In addition, formal validations of the model should include checks on whether the model results are being applied to appropriate purposes and how model results are viewed at the top of the bank.

Senior management and members of the risk committee must have a general understanding of the bank’s key risk and capital models, and a detailed understanding of the relevant reports and how they can be used. Otherwise, they will not be able to apply the results to key questions of capital adequacy and strategy, or communicate the results meaningfully to third parties.
 
Conclusion

Model validation is often thought of as a rather technical and mathematical exercise. However, bank losses from model risk are often caused by poor governance of the wider modeling process, or by a poor understanding of the assumptions and limitations surrounding the model results, rather than by errors in equations.

The growing importance of models in helping executives answer some of banking’s most critical questions – from compliance and capital adequacy to business performance and risk-adjusted compensation – suggests that model validation is too important to be narrowly defined or left to the “quants”.

For both best practice and regulatory compliance reasons, senior bank executives must begin to take a more commanding role in ensuring that model validation is aligned with the overall interests of the bank – and that the bank’s investment in sound risk modeling can be easily communicated and proved to third parties.

This article was contributed by Mark Fogarty, Managing Director at BancWare ERisk, who welcomes your comments at MFogarty@erisk.com.

Footnotes:
1) For example, Studies on the Validation of Internal Rating Systems, Working Paper No. 14, Basel Committee on Banking Supervision, May 2005; Update on Work of the AIG related to Validation Under the Basel II Framework. Number 4, January 2005; Forum on Validation of Consumer Credit Risk Models, Federal Reserve Bank of Philadelphia, November 2004; It’s Not Just About the Models: Recognizing the Importance of Qualitative Factors, Susan Schmidt Bies, Federal Reserve, December 2004; OCC 2000-16: Risk Modeling – Model Validation, May 30, 2000
2) A possible exception is Consultation Paper 10 published by Committee of European Banking Supervisors, which some industry bodies criticized at draft stage for offering too much prescriptive detail on validating IRB credit models and operational risk models.

©2010 Sungard. All rights reserved. Legal Information