The Big Four and The Advent of Model Risk

6/24/2019 - Monocle Journal

An example of the extent to which audit firms have had a certain disproportionate degree of influence not only over their banking clients, but also over the regulators, is something that is known in the South African industry as the “long form”.

Prior to the 2008 financial crisis, and somewhat ironically, simultaneous to the build-up of leverage and risk that led to the crisis, the Basel Committee for Banking Supervision (BCBS) developed a documented set of proposed rules called Basel II. This set of regulations had long been baking in the oven.

It was recognised as early as the nineties that the rules governing banking globally were woefully inadequate. The Basel rules, until the first few years of the new millennium, included a very basic method for calculating credit risk, and had adopted – somewhat conversely – an extremely complex method for calculating market risk. The credit risk calculations were so simple in fact that there was significant opportunity for regulatory arbitrage. The market risk methods on offer, on the other hand, required significant mathematical prowess, complex non-linear equations, and were very difficult to implement.

The Committee decided in the late nineties to overhaul the entire set of rules, regularly publishing consultative papers for industry feedback and conducting what are known as quantitative impact studies. The rules became effective in 2005, or 2006, depending on how long it took for each respective in-country local regulator to adopt Basel II as law.

In any case, the laws came in too late, the U.S. dragged its heels in signing up to the Basel agenda, and the crisis was inevitable anyway given the leverage that was allowed to build up through the use of structures such as collateralised debt obligations and credit default swaps.

Of the many regulatory failures that were blamed post-crisis for the upending of the banking world and the economic fortunes of many, was the idea of what became known as model risk. The BCBS’ insistence that banks move away from using overly-simplistic methods for calculating credit risk and move towards an IRB-Approach (Internal Ratings-Based Approach) meant that every bank needed to construct quantitative – and statistically-accurate – models for measuring the credit risk inherent in each and every one of their asset-side portfolios.

As an example, this implied that there should be separate and distinct models for the retail bank assets, the corporate bank assets, the business bank assets and the trading assets of any particular diversified bank. Also, for each of these divisions within the bank, it was recognised that the credit risk inherent in a mortgage portfolio, for example, was significantly different to the credit risk inherent in an unsecured short-term loan portfolio, as another example.

In fact, the Basel regulators created a methodology for the breakdown of banking assets. There would ideally be homogeneity between banks, in terms of how they classified their assets for credit risk measurement purposes. This would ensure consistency in treatment between banks, and would conversely ensure that each stipulated portfolio within a particular bank was distinct and non-overlapping with their other asset-side portfolios. This was called the asset classification process and was required to be – and to this day remains – an automated and regulated process.

The asset classification process begins with distinguishing between retail, retail SME (Small/Medium Enterprise), corporate, public service entity, and sovereign customers within a bank, using customer revenue as one of the determining factors, and using counterparty type as another. The theory is – based on comprehensive international studies conducted by the BCBS in tandem with several academic institutions – that the correlation of credit risk events occurring simultaneously in different counterparty groups, is less than one.

This means that the Merton Contingent Claims Hypo   thesis that was used by the BCBS in converting internally-determined credit risk calculation metrics into risk weights for each asset exposure, needed to be distinct per portfolio type. The risk weights would, in turn, determine the amount of equity and other capital that need to be held against the credit risk of the balance sheet of a bank – and this needed to be fair in its treatment and consistent worldwide.

The Contingent Claims Hypothesis had been proposed by Robert Merton in the 1970s and was based on option pricing theory for which Fischer Black and Myron Scholes had won the Nobel Prize decades later – commonly known as the Black-Scholes equation. In Merton’s version of it, he hypothesised that the bondholders of a hypothetical firm’s debt had an optional claim over that firm’s assets, since they could always choose to liquidate these assets to recover their principal investment and accrued interest. Merton pointed out that the value of this implied put option – since it gave the bondholders the right, but not the obligation, to exercise a fire-sale of the assets of the firm – was one that would increase in value non-linearly as the value of the firm’s equity declined in context to the debt the firm held.

Whilst Merton’s equation has been heavily criticised over the past three decades, it was the best proxy that the BCBS could come up with to adopt as a consistent and implementable method for converting credit risk characteristics of an asset into distinctive risk weights.

Naturally, the likelihood of unsecured short-term loans defaulting is far higher than the likelihood of mortgage loans defaulting, since empirically this had been observed in a range of studies, but it is also self-evident that the obligor – the customer or borrower – would rather pay their monthly mortgage than pay off their credit-card debt. On the one hand they could lose their house, on the other hand they would potentially only lose the ability to borrow further against their credit card. Retail assets were therefore further broken down between those that are secured by collateral versus those that are unsecured – home and car loans versus unsecured student loans and current accounts as the most common differentiation.

On the corporate side, the breakdown begins with differentiating between corporate SME customers and larger corporates, and then splits between vanilla term lending and what is known as specialised lending.

Specialised lending was developed by banks as a sophisticated method by which projects – for example the construction of a toll road, or the building of a maximum-security prison – could be financed. It is an especially important category of bank financing and is essential for economic growth since it allows banks to lend in instances where there clearly is no single existing entity wishing to have its balance sheet held up as security against the loan, and where that which is being financed has not yet been constructed but is expected, years later, to generate sufficient cashflows to pay down the accrued interest and principal of the loan.

The BCBS decided upon five categories of specialised lending, and for all banks worldwide, each specialised lending deal they engage in needs to be categorised as such. Recall that banks were required now – under the Basel II rules for the IRB-Approach – not only to categorise their asset-side using these asset classification rules, but they needed to do so in an automated fashion.

Also, for each portfolio identified as distinct within the bank – and recall that credit risk inherent within a corporate loan portfolio in South Africa could be distinct to credit risk within a corporate loan portfolio in Zambia – the bank needed to be able to calculate the underlying credit risk characteristics required to be inputted into the Merton Risk-Weighted Asset calculation stipulated by the BCBS.

There are five distinct inputs to this equation per portfolio and these are: the probability of default on the portfolio anticipated over the next twelve months (the PD), the anticipated value of the exposure to the bank at the time of default (the EAD), the loss-given default, i.e. the anticipated loss expected to be incurred (after seizing of collateral and potential sequestration of the customer) should default occur (the LGD); and two other factors less statistically complex.

These first three characteristics however are essentially values per portfolio per asset exposure, expected to be updated regularly to anticipate changing market conditions, that are statistical in nature and that require non-linear complex statistical models based on rich and accurate data, to calculate. In many banks – even within the four largest South African banks – this meant that, given the range of customer types, product types and geographical diversity of lending taking place, there could be over a hundred models to build and implement. The trouble was that these banks – as well as sophisticated banks worldwide which had decided to adopt the more complex IRB-Approach – shortly after completing the gargantuan task of building these complex models, having them validated by a team of statisticians from the Reserve Bank, and then having implemented them into their reporting and decisioning systems, found themselves in the teeth of the 2008 financial crisis.

Far more complex statistical models measuring the credit risk within tranches of collateralised debt obligations that had been constructed by investment banks in the U.S., for the purpose of pricing credit default swaps, as well as for the purpose of trading these same CDO securities between investors and banks, had completely and utterly failed to anticipate the risk. Whilst teams of highly-educated statisticians and mathematicians were burrowed away working on problems of joint distributions – attempting to address the problem of joint distributions and non-Gaussian distributions using state-of-the-art copula functions – borrowers in places like Las Vegas, Nevada, had substantially over-exposed themselves with mortgages, often not only on one house but on several.

The Michael Lewis book chronicling this irrationality and monumental myopia on the part of the banking industry as a whole, The Big Short, at one point introduces a Vegas-based stripper who uses a self-certification certificate (a certificate declaring one’s monthly income and expenses especially designed at the time for “self-employed” borrowers), to buy not one, but three properties for investment, using one hundred percent loan to value debt from banks.

The contrast was mind-numbing: on the one hand the smartest analysts in the world working on credit risk models, on the other hand the most audacious level of stupidity in banking history since the 1920s. The result was not to reject credit risk modelling as a pursuit, but rather to invoke one of the most telling of euphemisms that came out of the crisis – model risk.

The central idea behind model risk is not that risk modellers are fraudulent – anything but, in fact – but rather that they can make mistakes. Firstly, they can make statistical mistakes, and secondly, the data they use to build and validate the models could often be incorrect representations of the portfolio in question. Model risk is particularly intractable, since, despite the fact that it is scientific in nature, like all science, there are many opinions, often held by substantially bright and educated, and sometimes egotistical modellers, and often this led to methodological arguments that could unravel an entire implementation project.

Even more disturbingly, however, when analysts at banks were asked to reproduce the results that they had reported for the calculation of a particular risk weight – the probabilities of default on a range of assets in their corporate term loan portfolio, for example – they were unable to do so. It became clear to regulators, and to the South African regulator in particular, that analysts would often extract data directly out of a source system, create a model using sophisticated software and idiosyncratic methods, generate results, and have the Information Technology (IT) teams implement the model. There was a lack of documentation, oversight, internal validation, and proof.

In response to this, the South African regulator of banks, the Reserve Bank, decided to introduce an extra layer of validation, in which banks on an annual basis needed to prove that each and every one of their models were accurate by insisting that they be replicated from beginning to end, using the same source data. This is what became known as the “long form”, since the standard form that the SARB designed detailing the specific checks that need to be performed at each step of this replication process, is particularly long. To put it mildly, this is a significant task.

Recall that, as part of the model development process, banks would have already had to comply with the requirement of having their models internally validated. This implied that, in addition to banks’ already extensive internal validation processes, external parties would have to be contracted to perform this entire replication process for every credit risk model in use by the bank. In the South African context, and not generally worldwide, the Big Four audit firms argued that it was they who should be tasked with this extra level of compliance, and they alone. Naturally, this made some sense since it is somewhat of an audit-like check that one is performing. The SARB agreed, and so it became an additional cost to banks to have their auditors perform these replication checks on an annual basis.

The challenge here is that these are statistical tests, often requiring deep statistical knowledge – the models plagued with potential quantitative or methodological errors. The assumptions made in an LGD model – for example, on the discount rate to be applied to cashflows expected to emanate from collateral liquidation at an estimated future point in time – can be riddled with statistical or fundamental errors, and these assumptions themselves cannot easily be “checked” by even the most experienced data statistician. The audit firms’ personnel can, at best, in the context of replicating credit models from scratch, perform logical checks and balances – but one questions the value of this in the context of the problem that motivated for this kind of work in the first place.

Remember: the world’s most important rating agents, Standard & Poor’s (S&P), Moody’s and Fitch, all three of which employ statistical financial analysts in droves, did not get the measurement of bank-issuer risk, nor for that matter CDO-tranche risk a little bit wrong. They massively and dubiously underestimated the risk in orders of magnitude. They frequently rated CDO securities as triple-A, implying a one in a thousand-year probability of failure, and in certain cases double-digit percentages of these portfolios went bust in a single year.

The Big Four auditors in South Africa motivated for, and received, sole and exclusive right to perform annual and tedious checks on every single credit risk model within each of the banks domiciled in this country, despite the enormous volume of work they are already responsible for within the banking world, and the tremendous accountability that comes with that. This is despite the fact that the majority of their employees on the audit side are candidate chartered accountants. Where they have recruited statistical skills to be able to perform the “long form” work, they have done so on the advisory side, which is not regulated under IRBA, and which is therefore not purely an audit function.

To be clear, IRBA does lay out the requirements necessary for an auditor to rely on the work of an expert, but it is wishful thinking to imagine that any auditor could possibly have the necessary knowledge to adequately ascertain the veracity of the work. It therefore is counterintuitive that this work is limited only to these four audit firms. In fact, in doing so, the SARB may have inadvertently increased systemic risk in the South African banking system, especially in view of the erosion of trust in the KPMG brand.


Recent Posts


13th Floor, Greenpark Corner, 3 Lower Road, Morningside, Sandton, South Africa

Phone: +27 11 263 5800
Fax: +27 11 263 5811


301 New Cumberland, 163 Beach Road, Mouille Point, Cape Town, South Africa

Phone: +27 (0) 82 952 1415


4 Lombard Street, London, EC3V 9HD, England

Phone: +44 (0) 2071 902 990




Copyright © 2019 Monocle Solutions. All Right Reserved.