Figure A.10 Regression lines with: (a) shift dummy; (b) slope dummy; (c) shift and slope dummies.

Appendix 5 Prediction

Model specification procedures can leave much to subjective choice (§8.4). It is common that many different model specifications will be considered. Some real historic data should be retained for the purpose of evaluating the predictive performance of a model, to evaluate which is the best specification. This process is called model validation. One should evaluate predictions for a variety of different market conditions. A thorough backtest of a certain model specification would typically involve very many predictive tests. This appendix illustrates how point and confidence interval predictions are generated from a regression model. The general framework for backtesting is outlined in §A.5.2, and an overview of the statistical and operational forecast evaluation methods that are commonly employed is presented in §A.5.3.

A.5.1 Point Predictions and Confidence Intervals

A regression model generates point predictions of the expected value of the dependent variable for given values of the explanatory variables. To see how this is done, consider the general linear model (A. 1.7):

Y, = B,*,, + B2A-2r + . . . + BA + e,

A point forecast Y of Fis given by estimating the parameters by some method to obtain Pi, P2, . . ., p\ and then substituting certain values of the explanatory variables Xu . . ., Xk into the model. A point prediction Fis the expected value of Y given that the explanatory variables take the values X*, . . ., X*:

F=p1A-* + p2X2*+ . . . +¹* .

It is not possible to generate a point prediction of the actual value of Y when the explanatory variables take the values X*, . . ., X* because the actual value of Y is Y + e, where the residual e is the value of the error term at the point of the forecast, and this value is not known. However, it is possible to generate a confidence interval for the true value of Y given that the explanatory variables take the values x* = (X*, . . ., X*). This confidence interval is generated by assuming normality and using the estimated standard deviation of = * + . The forecast error is e = Y- Y - x*(P - p) + e and so

V(e) = x* F(P)x* + F(e). When p = b. the OLS estimators, then F(P) = fj2(XX) 1 and so ( ) = -2( *( ) 1 * + 1).

Thus, assuming normality.

(Y- Y)/(a(x*(XXy]x* + 1)) ~ N(0, 1),

and using the estimated value s2 for rj2 gives

(Y- Y)/(sJ(x*(XXylx* + 1)) ~ tT k. (A.5.1)

This forms the basis of interval predictions for Y. A two-sided 100(1 -a)% confidence interval for the actual value of Y when the explanatory variables take the values x* is

(Y-\, Y+ Q, with \ = ?r W2(V(x*(XX)x* + 1)). (A.5.2)

Confidence intervals are used in scenario analysis, where the explanatory variable values in x* may be anything, and in model validation, to ensure that real data fall into the confidence intervals predicted by the model.

A.5.2 Backtesting

It is very important that the developer should backtest the model with a large number of predictive tests. Normally the historical data are divided into two parts: the in-sample data used to specify the model, and the out-of-sample data used to evaluate model predictions. Often out-of-sample data are taken immediately after the in-sample data, in which case they are usually called post-sample data.

A common form of backtest is to use a rolling window for the in-sample and post-sample data, and each time compute several in-sample and post-sample diagnostic performance measures. A reliable model will show stability in the time series of performance measures that are produced by the backtest. In-sample diagnostics can range from the basic autocorrelation and heteroscedasticity tests that are necessary for the unbiasedness and efficiency of parameter estimates (if obtained by OLS), to more advanced in-sample diagnostics such as the unit root tests described in §11.1.5. Commonly backtests employ a number of post-sample performance measures that can be either purely statistical or operational or both. A number of out-of-sample diagnostic measures are described in §A.5.3. See also §12.5.

There may be no problem churning out the statistics, but the interpretation of backtesting results is not always straightforward. The results will depend on the diagnostics that are chosen and it is unlikely that a given model will perform well according to all possible prediction performance measures.19 The main problem with backtesting is that results will be specific to the trading metric that is chosen for operational evaluation, or the diagnostic statistic that is chosen for statistical evaluation. The most that can be achieved is that a

19The first International Non-linear Financial Prediction Competition used 15 statistical and operational post-sample performance measures (§13.3).