is through a mapping function of the returns: the forecast should fit the mapped returns Yj rather than the real returns ,.

A suitable mapping function makes the mapped returns less leptokurtic thun the original ones. The rest of the regression problem remains unchanged. The desired effects can be obtained with an underproportional mapping function presenting the following properties:

Small returns should be amplified when considered in the regression, in order to establish a sufficient penalty against forecasts of the wrong direction.

Large returns should be reduced when considered in the regression, so the distribution function of mapped returns is no longer leptokurtic.

The mapping effects should decrease with the increasing time horizon size.

The choice of such a mapping function M is arbitrary provided it has the above properties. The one function used in our model is

Yi = M (Yj) = - (9.13)

[Y? +

with the parameters A, B, and a depending on the time horizon At}/. The same parameters are used for all different FX rates. They have been calibrated by trial and error in order to keep the ful 1 sample variance of the mapped returns on the same level as that of the original returns. The parameter a must follow the condition 0 < a < 0.5 because the mapping function must be an underproportional bijection.

9.4 MEASURING FORECAST QUALITY

Two questions are relevant for testing any forecasting model of foreign exchange (FX) rates:

What data should be used for testing?

What is a good measure of forecasting accuracy?

Since the classical paper by Meese and Rogoff (1983), researchers in this field have been aware of the need for out-of-sample tests to truly check the forecast validity. Because of the statistical nature of FX rates, there would be little significance in the forecast accuracy measured on the same data that were used for optimizing the models. The real test comes when the model is run on data that were not used in constructing the model. In our case, our model being run in real-time, we have a continuous out-of-sample test. Besides the question of in and out-of-sample, there is a question as to what constitutes the relevant quantity for measuring the accuracy of forecasting methods. In Makridakis et al. (1983) the main measures are reviewed. We limit ourselves here to presenting the reasons as to why we chose certain types of measures and how we compute the uncertainty of these measures.

9.4.1 Appropriate Measures of Forecast Accuracy

Most standard measures rely on the mean square error (MSE) and the mean absolute error (MAE) for each time horizon. These errors are then compared to the similar ones produced by a naive forecasting model serving as a benchmark. One naive model may be the random walk forecast where expected returns are zero and the best forecast for the future is the current price. These accuracy measures are, however, all parametric in the sense that they rely on the desirable properties of means and variances, which occur when the underlying distributions are normal. The selection of the random walk model to derive the benchmark MSE or MAE is inherently inappropriate. It is in effect comparing the price volatility (MSE or MAE) with the forecasting error. There is no reason to expect that the heteroskedasticity and the leptokurticity of returns would not affect their MSE or MAE for a particular horizon. Thus the significance of their comparison with the forecast MSE or MAE is unclear. It might only reflect the properties of price volatility.

Such considerations have led us to formulate here nonparametric methods of analyzing forecast accuracy. These are generally "distribution free" measures in that they do not assume a normally distributed population and so can be used when this assumption is not valid. One measure that has this desirable property is the percentage of forecasts in the right direction. To a trader, for instance, it is more important to correctly forecast the direction (up or down) of any trend than its magnitude. We term this measure the direction quality, also known as the sign test:

N({xr I (If - xc)(xr - xc) > 0})

D(Atf) = V }----- (9.14)

fJ N({ / I ( / - )( / - xc) 0})

where N is a function that gives the number of elements of a particular set of variables {x}, and xc and Jt/ have the same definition as in Equation 9.8. We give the forecasting horizon in physical time Af/ because the quality must be measured in the time scale in which people look at the forecasts. It should be clarified here that this definition does not test the cases where either the forecast is the same as the current price or when the price at time Dc + A??/ is the same as the current one. To illustrate this problem, let us note that the random walk forecast cannot be measured by this definition. Other definitions could be used, like counting the case when the direction is zero as half right and half wrong. Excluding the cases where one of the two variables is zero would be a problem if this would occur very often. Our results show that it only occurs quite seldom and for the real signal ( / - xc) (few percent of the observations on the very short horizons) and almost never for the forecast signal (Jt/ - xc).

Unlike more conventional forecasts, for instance, a weather forecast, an FX rate forecast is valuable even when its direction quality is slightly above 50% and statistically significant. No trader expects to be right all the time. In practice we assume that a D significantly higher than 50% means that the forecasting model is better than the random walk. The problem lies in defining the word "significant."

As much as one would like to be independent of the random walk assumption, we are still forced to go back to it in one way or the other, as here, when we want to define the significance level of the direction quality. As a first approximation, we define the significance level as the 95% confidence level of the random walk:

1.96

oD * -= (9.15)

2 y/n

where n is the number of tests. The factor 2 comes from the assumption of an equal probability of having positive or negative signals. It is a similar problem to the one of tossing a coin.

Another measure we use in conjunction with the previous one is the signal correlation between the forecasting signal and the real price signal:

c(Atf) S ELt(*/./--rc,-)(*/,--*c,i)- (9 6) VT?li(*/,--*c,,-)2 T!i=i(xfJ -xc,,-)2

where n is the number of possible measures in the full sample, n the number of full forecasting horizons in the full sample, and / is n - n. Here again the forecasting horizon is given in physical time, At/. We estimate the significance of this quantity using 1.96/Vn7.

Both the direction quality and the signal correlation unfortunately have a slight drawback. They do not provide a measure of the forecast effectiveness. Nevertheless, we believe that they are superior to standard measures due to the nonnormality of the return distributions. The direction quality, which for all practical purposes is the most relevant indication of the forecast, and the signal correlation must be highly significant before we accept a model as being "satisfactory."

9.4.2 Empirical Results for the Multi-Horizon Model

Optimization consists of two distinct, but interrelated operations, corresponding to the two main types of parameters in the models. The linear model coefficients cTj and cxj are optimized through least squares (see Section 9.3.5), and under the control of this process, always fulfill the strict out-of-sample condition when applied to a forecast. On the other hand, the nonlinear parameters of the indicators described in Section 9.3.4 must be optimized by trial and error to meet the above criteria: the direction quality and the signal correlation. The data set used in selecting the best combination of indicators is termed the in-sample period where the model parameters are fully optimized.

In Table 9.1 we indicate how our sample is divided to satisfy the different requirements of model initialization, in-sample optimization, and out-of-sample tests. The initialization period is needed for both initializing the different EMAs (see the discussion in Section 3.3.3) and computing the first set of linear coefficients cTj and cxj. The results presented in the next section are computed over two specific periods using our database of intraday market makers quotes. The first