Conclusion

The historical profitability of a system is not enough to justify its trading. Greed and fear affect every trader. This chapter laid groundwork for measuring aspects of a trading system that will likely affect greed and fear. The more that the trader can learn about a system in the evaluation stage, the more confidence the trader will have trading it real time.

No matter how profitable a system appears, if a trader does not have the intestinal fortitude to follow it, he or she should look for another system to trade.

Fear of the unknown always weighs heavier in our mind than that which is known. Preparing for the unknown begins with a thorough system evaluation. Test first-trade second.

During long run-ups and drawdowns, the trading principles behind your system remain constant; it is your perception of the system that changes.

At the end of a systematic evaluation a trader should be able to answer the following questions:

• Do you over- or underestimate your systems true performance?

• How stable are your winning and losing trades?

• What is your systems pessimistic reward/risk ratio?

• How many of your trades are statistical outliers?

• How does your system compare to a Buy-and-Hold strategy?

• What is the average run-up and drawdown for the system?

• Do you really know your trading system well enough to trade it with complete confidence?

Endnotes

1. Schwager, Jack, Fot additional information on underwatet equity curve, see Schwager on Futures: Technical Analysis, John Wiley & Sons, 1995.

2. Jurik, Mark, For further insight into the phenomenon of degrees of freedom (DoF), see "Developing Indicators for Financial Trading," in Virtual Trading, Probus, 1995.

3. Sweeney, John, For a discussion of maximum adverse excursion (MAE), see Campaign Trading John Wiley & Sons, 1995.

4. Ibid.

Part II

SUMMARY

Why Test?

• System performance must be viewed in light of aJl three parameters: profitability, drawdowns, and stress. Before you consider opening up a trading account, are you sure your system is reliably profitable? Will drawdowns wipe you out first? Is it trading in a way you can tolerate? Can you go for long periods of no trading, or too much trading? The only way to know is by subjecting your system to extensive backtesting.

• The more thoroughly you test your system in advance, the less stressful your trading session will be. Does your system perform better than buy and hold or a random buy/sell process? How does it compare with a fixed rate, no-risk investment? Is system performance sensitive to precise values for its parameters? Will a slight change cause a significant decrease in profits? Is system performance on out-of-sample data significantly poorer than on in-sample data?

• Backtesting will help you pinpoint which approaches to the market are likely to be successful and which ones are not. It will also produce a record of your systems historical trading performance, necessary for developing a diversified asset allocation strategy.

• Testing your system requires skill because sloppy testing makes it easy to convince yourself you have a "winning system" despite its flaws. For example, to have a meaningful evaluation, your system should produce a statistically significant number of trades, at least 100. A small number of trades, no matter how perfect, is easy to produce and nearly impossible to reproduce.

General Procedure

• If there is enough data to do so, in-sample data should be completely separate from out-of-sample. Better still, create a third set that can be used in conjunction with the in-sample set to build, test, and reject preliminary systems. This way, the optimization process of building and testing runs without the out-of-sample data. Lastly, any system that passes the preliminary build-and-test phase is then subjected to out-of-sample data for final analysis.

• Whether you should include outlier trades in your test results depends on your strategy. For example, a system designed to suffer lots of small losses while waiting for big, rare, profitable breakouts depends on outlier trades. Removing outliers from that scenario will portray a losing system, which in reality may not be the case at all.

• After you backtest your system on historical data, I advise a round of forward-tests to measure its real time performance. Various brokers are willing to set up

Part II

an account to let you simulate trading.* This will help you answer questions about slippage and timeliness as well as help you master the skill of placing trades.

• When designing a mechanical system, remember that all systems eventually fail, so you will need to periodically update the trading strategy.

Measurements

• System analysis measures overall net profitability. This is important, as no one will trade a system expected to lose money. However, do not let this be your only consideration. Reward/risk ratios point out that sometimes less profitable systems can be more desirable.

• Of all the profit ratio measurements available, I find profit factor (PF) to be the most useful. It tells me how many dollars I gain for each dollar I lose in trading. Dont consider a system whose PF is less than 2.

• The annual return on account (AROA) is another favorite measure of mine. It is the ratio of a years net profits to the total cost of placing those trades in that year (i.e., margin costs + maximum drawdown), averaged across all the years the system was tested.

• Instead of averaging AROA, list the ratio for each year in a table. This sliding window method illuminates any particularly bad years, giving you the opportunity to analyze a system weakness.

• The Markowitz/Xu data mining correction formula adjusts your estimate of a systems average daily return by considering the number of systems that were run through the out-of-sample set.

• The coefficient of variation is a good way to measure, for comparison purposes, the steadiness or consistency of a systems trade-by-trade performance. Look for systems with coefficient of variations of 200 percent or less. Larger numbers indicate instability and should raise your concern.

*TradeComp International (www.tradecomp.com), Larax Software (www.larax.com), and AudiTrack (www.au-ditrack.com) permit simulated trading.