"Will my trading strategy be profitable?" After having gone through the arduous process of crafting a trading strategy, that is the question you must ask yourself. The ability to answer that question is the great promise that backtesting holds out to all traders. A successful backtesting procedure will greatly reduce the probability that you will begin trading with either an unprofitable strategy or one that does not meet your standards.1

By adopting a sound and rigorous backtesting approach, you will:

• Pinpoint which approaches to the market are likely to be successful and which ones are not.

• Generate good estimates of future performance for each trading strategy you test.

• Create a record of your trading strategys historical trading performance.

• Produce data necessary for other components of your trading approach such as your asset allocation strategy.

The ability to backtest is one of the great advantages that computerized traders have over discretionary or intuitive traders. Because computerized trading strategies are clearly and objectively defined, they can be tested on historical data in a way that is all but impossible for a discretionary trading strategy. Computerized traders who do not backtest are surrendering one of the greatest advantages that they have.

In this chapter you will learn the rudiments of backtesting methodology. You will know how to execute simple backtesting protocols on your own data and you will know the benefits and shortcomings of several backtesting strategies. Most important, you will know how to employ the Markowitz/Xu data mining correction formula, which is described for the everyday trader for the first time in this chapter.2

The Basics: Walk-Forward Testing

While there are many different types of backtesting procedures, most of them share a core component: walk-forward testing. The idea behind walk-forward testing is

Backtesting

Michael de la maza

simple: A trading strategy should be tested by going back in time and running the trading strategy as if your historical data were real-time data.

An example helps to illustrate this technique. Suppose that you have developed an intermediate term trading strategy for the S&P 500 futures market and you would like to backtest this system. You have access to data going back to 1983, and your strategy uses the last 10 closing prices to determine whether or not to trade. To perform a walk-forward test of this strategy, you would begin by giving it access to the first 10 closing prices of 1983, and you would run the trading strategy on these 10 data points. You would then record the trade, if any, made by the trading strategy on day 11, along with profit and loss figures and other data you might choose to collect. Then you would replace the first closing price in the 10-day series with day lis closing price and repeat the process. You would sweep a 10-day "window" of closing prices across your historical data and give your strategy access only to the closing prices in this window. This process is identical to the one that you would follow in real-time trading. This 10-day "window" is shown in Figure 8.1. <c

/ There are many commercial programs that automate this walk-forward testingN procedure. Although these packages may be useful for some purposes, in general they are too limited to accommodate sophisticated trading strategies or backtesting protocols. Typically, custom programming is required to implement a walk-forward testing module.

Figure 8.1 Walk-forward testing. At each step, a 10-day "window" of closing prices is available for a trading strategy.

1 2 3 4 5 6 7 8 9 10

3 -

Step 1

Step 2

Step3

Step 4

In-Sample and Out-of-Sample data

With the exception of backtesting protocols that employ the Markowitz/Xu correction (to be discussed) or something similar, all backtesting procedures make a critical distinction between "in-sample" and "out-of-sample" data. In-sample data is the data used to design the trading strategy. Out-of-sample data is the data used to test the trading strategy. A cardinal rule of backtesting is:

In-sample data should be completely separate from the out-of-sample data.

The out-of-sample data should remain "clean" and "pure" throughout the process of creating the strategy. As soon as the out-of-sample data is tainted, it can no longer serve the purpose of providing an accurate estimate of future performance which is one of the most important outcomes of the backtesting process.

One wrinkle on the standard in-sample/out-of-sample data set paradigm is the introduction of a third data set called the "tuning" set. The sole purpose of the tuning set is to provide you with the opportunity to "cheat" without ruining the out-of-sample data set. After creating your trading strategy using the in-sample data set, you then run it on the tuning set. If the results meet your criteria, then you can run the strategy on the out-of-sample data set. If, on the other hand, the results do not meet your criteria, then you can go back to the in-sample set and tune your strategy. The flowchart in Figure 8.2 illustrates this iterative process.

Figure 8.2 The iterative process of creating and testing a trading strategy.