Figure 8.3 Dividing the data set into three groups: in-sample, tuning, and out-of-sample.

You can think of the tuning set as a psychological crutch. If, after having spent considerable intellectual, emotional, and psychological energy developing a trading strategy, you discover that it has a simple flaw when you run it on the out-of-sample data, then your options are limited. If you choose to retool the trading strategy, then your estimates of future performance will be inaccurate because the out-of-sample data set has been tainted. You will have to live with the uncertainty that you may have over-fitted, overoptimized, and "shrink-wrapped" your trading strategy around the out-of-sample data. The benefit of using a tuning set is that it gives you a second chance.

If you decide to use a tuning set in your backtesting, then the standard method is to select a period of time between the in-sample data set and the out-of-sample data set as the tuning set. For example, if you are developing a strategy for the S&P 500 futures, then your in-sample data set might range from 1982 to 1986, your tuning set might range from 1987 to 1991, and your out-of-sample data set might range from 1992 to 1997, as shown in Figure 8.3.

leave-one-out Testing

For the data partitioning strategy previously discussed, the three data sets are chosen in advance and are held fixed through the creation and testing of the trading strategy. In contrast, leave-one-out testing forces the out-of-sample data set to shift throughout the entire data set. With this method, the historical data set is first divided into many groups. One of these groups is temporarily assigned to be the out-of-sample set and the remaining groups become the in-sample set. Next, a trading strategy is partially developed for this particular assignment of the data groups. The cycle repeats as

Table 8.1 Data group allocation in leave-one-out system testing

Data Groups Assigned to Be In-Sample Data Groups Assigned to Be Out-of-Sample

1 2 34 5

1 2 3 5 4

1 24 5 3

1 34 5 2

2 34 5 1

another group is chosen to be the out-of-sample data set, the rest assigned as in-sample, and partial system development continues some more. The process repeats until all the data groups have been used as the out-of-sample data set.

An example serves to illustrate the point. Suppose you divide the historical data into five groups. Then the in-sample and out-of-sample sets are shown in Table 8.1.

The Markowitz/Xu Data Mining Correction Formula

The Markowitz/Xu data mining correction formula is a little known but extremely powerful formula that corrects for overoptimization. You should use the Markowitz/Xu data mining correction formula whenever the out-of-sample data set has been tainted by running more than one trading strategy through it. The Markowitz/Xu formula gives an estimate of the actual return of the top-performing strategy. The intuitive idea behind the formula is that the return of the top-performing strategy should be adjusted in the direction of the average return of all the trading strategies that were run through the data set. The form of this adjustment is given by the following formula:

H = R + B(H-R)

where . . .

H is the new estimate of the return of the best strategy, R is the average return of all of the strategies, H is the return of the best performing strategy, is a number between 0 and 1.

Notice that H will always be between the average return of all of the trading strategies and the return of the top-performing strategy. When = 0 then H = R + 0 (H-R)=R. When B = l then H = R + 1 (H-R) = R + H-R = H. The critical question is: What is the formula for B? Intuitively, should grow smaller as the number of trading strategies grows larger. The smaller is, the closer will move to R, the average return of all of the trading strategies. In addition, should grow smaller as the variance of the returns increases. The formula for is:

- N/D

where

N = C(1 + l/(n- 1)+ 1/(T- 1))-A/(T- 1) D = C(1 + l/(n- 1))

where

A is the average squared difference of the daily returns of each strategy and the

avetage daily tetutn of all sttategies, is the avetaged squared difference of the avetage daily tetutn of each sttategy

and the average daily return of all of strategies, n is the number of trading sttategies,

7* is the number of days for which you have computed daily tetums

Spreadsheet Example

Figute 8.4 shows a spreadsheet that contains an example of the Markowitz/Xu data mining correction formula in action. The first 18 columns of the spreadsheet are organized as follows:

• Column A. Date.

• Column B. The daily returns of Strategy 1. For example, on 10/7/97, the daily return of Sttategy 1 is -.005 oi -.5%.

Figure 8.4 An example spreadsheet for calculating the markowitz/xu correction.