9.5 Model Validation

VaR models have become the primary method for calculating MRR, so regulators have set rigorous guidelines for the internal backtesting of these models. The first part of this section describes the backtesting methodology that is outlined in the 1996 Amendment and the translation of backtesting results into risk capital requirements. The second part discusses why different VaR models often give quite different results for the same portfolio, highlighting the various sources of error in each method. VaR estimates can be very sensitive to the choice of historical data period and assumptions about model parameters such as the covariance matrix. Assessment of a VaR model should therefore include an analysis of the sensitivity of results to these decisions and assumptions.

9.5.1 Backtesting Methodology and Regulatory Classification

A 1% 1-day VaR measure gives the level of loss that would be exceeded in normal market circumstances one day in every 100, if the portfolio is left unmanaged. That is, the unrealized P&L, also called the theoretical P&L, of the portfolio will show a loss greater than the 1% 1-day VaR estimate, on average, one day in every 100. So if an accurate VaR model is tested over a period of 1000 days one would expect 10 losses that exceed the level of VaR if the model is accurate. However, if the model is predicting a VaR that is too low, more than 10 exceptional losses will be observed. These exceptional losses are illustrated in Figure 9.8. They form the basis of the VaR model backtests that were proposed in the 1996 Amendment to the Basel Accord.

The total number of exceptional losses may be regarded as a random variable that has a binomial distribution. The probability of success (an exceptional loss) is p - 0.01 for a 1 % VaR, and the number of trials n is the number of days for the backtest, say n - 1000. Then the expected number of exceptional losses

P&L

Figure 9.8 Backtests.

is up =10 and the variance of the exceptional losses is np(\ - p) - 9.9. Therefore the standard deviation is a/9.9 = 3.146 and, using the fact that a binomial distribution is approximately normal when n is large and p is small, a 99% confidence interval for the number of exceptional losses, assuming that the VaR model is accurate, is approximately

~~ -0.005 /(/2/>(1 -p)), tip + Z0M5s/(np(l -P))

(see §A.2.1). In this example it becomes

(10 - 2.576 x 3.146, 10 + 2.576 x 3.146) = (1.896, 18.104).

That is, we are about 99% sure that we will observe between 2 and 18 exceptional losses if the VaR model is correct. Similarly, if the VaR model is accurate, we are 90% confident that between 5 and 15 exceptional losses will be observed in the backtest, because

(np - Z0(m>/(np(\ -p)), np + Z0MSy/{np(l -p)) = (10 - 1.645 x 3.146, 10+ 1.645 x 3.146) = (4.825, 15.175).

The 1996 Amendment to the Basel accord describes the form of backtests that must be undertaken by firms wishing to use a VaR model for the calculation of MRR. Regulators recommend using the last 250 days of P&L data to backtest the 1% 1-day VaR that is predicted by an internal model. The model should be backtested against both theoretical and actual P&L.19 Whether or not actual P&L gives rise to more exceptions during backtests than theoretical P&L will depend on the nature of trading. If the main activity is hedging one should expect fewer exceptions, but if traders are undertaking more speculative trades that increase P&L volatility, then the opposite will be observed.

For each area of operations in the firm, such as equity derivatives trading, a backtest is performed by first choosing a candidate portfolio that reflects the type of positions normally taken. The portfolio is held fixed and for each of the last 250 days the VaR is compared with the P&L of the portfolio. The number of exceptional losses over the past 250 days is then recorded.

From the binomial model, the standard error for a 1 % VaR for a backtest on 250 days is (250 x 0.01 x 0.99) = /2.475 = 1.573, so a 90% confidence interval for the number of exceptions observed if the VaR model is accurate (2.5 ± 1.645 x 1.573). That is, one is approximately 90% confident that no more than 5 exceptions will occur when the VaR model is accurate. Thus regulators will accept that VaR models which give up to 4 exceptional losses during backtests are performing their function with an appropriate accuracy. These models are labeled green zone models and have a multiplier of - 3 for

19 Actual includes the P&L from positions taken during the day (even when they are closed out at the end of the day), fees, commissions and so forth. Theoretical P&L is the P&L that would have been obtained if the position had been left unchanged.

Whether or not actual P&L gives rise to more exceptions during backtests than theoretical P&L will depend on the nature of trading

Regulators will not necessarily adhere to these rules in a hard-and-fast fashion

Table 9.4: Multipliers for the calculation of market risk requirements

the calculation of MRR.2U If more than four exceptions are recorded the multiplier increases up to a maximum of 4 as shown in Table 9.4 (some red zone models may in fact be disallowed). Regulators will not necessarily adhere to these rules in a hard-and-fast fashion. Some allowance may be made if markets have been particularly turbulent during the backtesting period, particularly if the model has performed well in previous backtests. It should be expected that the results of the backtests will depend on the data period chosen.

9.5.2 Sensitivity Analysis and Model Comparison

VaR estimates are highly sensitive to the assumptions of the VaR model and the other decisions that will need to be made. For example, covariance VaR models and Monte Carlo VaR models both require a covariance matrix, the generation of which requires many assumptions about the nature of asset or risk factor returns. If the same covariance matrix is employed to calculate the covariance VaR and the Monte Carlo VaR of a linear portfolio using both of these methods one should obtain the same result. The reader can verify that this is the case using the VaR spreadsheets on the CD. Differences will only arise if one drops the assumption of normality in one or other of the models (as in §10.3.1, for example), or if too few simulations are being employed in the Monte Carlo VaR.

Historical VaR models employ neither covariance matrices nor normality assumptions, so it is often the case that historical VaR estimates differ substantially from covariance or Monte Carlo VaR estimates. If this is found to be the case one should test whether the assumption of normality is warranted (§10.1). One should also investigate the accuracy of the covariance matrix forecasts, as far as this is possible (§5.1). If neither the covariance matrix nor the normality assumptions are thought to be a problem, then it is possible that it is the historical VaR estimate that is inaccurate, and this will most likely be due an inappropriate choice of historical period. It may be too short, so the

It is often the case that historical VaR measures differ substantially from covariance or Monte Carlo VaR measures

20 The market risk capital charge will be either yesterdays VaR measure or times the average of the last 60 days VaR measures, whichever is greater.