performance on only a few trades. When there is not enough data, there is no way to be sure that the results are robust. The sjstem can only be used if you have complete confidence in the underljing premise, and the sparse results simply confirm that logic.

c. Does the trading "stem produce profits for most combinations of parameters? The best assurance of robustness is when most parameter combinations produce profitable results, even though they varj.

d. Did logic changes improve overall test performance? When you improve the results by adding a special rule, the average of all tests must increase, while the standard deviation remains unchanged. Otherwise, you may have simply overfit part of the data.

e. How did it perform on out-of-sample data?

4. Choosing the specific parameters to trade. Choosing the parameters to be used in trading is another unique and difficult problem. Does the sjstem that showed the peak historic profits give you the best chance of getting the peak future profits, or was it just chance?

a. Did the last test include the most recent data? If you have left out data to perform an out-ofsample test then respedfi the parameters using the most recent data. You must be as current as possible before trading.

b. Did you choose from a broad area of success? Parameters should be chosen from an area that showed consistent profits, but should not be determined by the one that had peak profits. It is best to look for average retums using average parameters.

c. Are profits distributed relatively evenly over the tested histoij? Although there may be attractive total profits, sustained periods of loss would not survive real trading. Retums should be reasonably steadj.

d. Did the historic results show any laige losses due to price shocks? it is not possible to avoid laige price shod in real trading because you never know when they will come. Historic results that do not show laige equity jumps due to price shocks are not realiatic, and result from overfitting. Even worse, those results that only profit from price shocks are benefiting from hindsight. In real frading, 50°o of the price shocks should produce a windfall profit while the other 50°o will give equal losses.

e. Have you risk-adjusted the retums to your acceptable rid; level? The best comparison between tests is to annualize all retums, to avoid comparing unequal time periods, and to risk-adjust the residts. Rid; adjusting changes eadi result so that it has the same unit risk as every other test

5. Trading and performance monitoring. The development of a sjstem is never complete. There is a continuous evolution in the markets, seen in the changing participation, volatility, interrelationships between markets, and the infroduction of new maikets. The only way to see how these changes affect your trading program is to carefully monitor the sjstem signals and the corresponding fills. From this evaluation you can reallocate funds, change the way you execute signals, and even find better rules. Without monitoring your trading, and comparing the results with your expectations, there is no basis for sajing that your sjstem is performing correctly.

a. Are you following the same rules that were tested? It is often convenient to program a simple version of your ideas, but trade a more complex set of rules, if so, you have no basis for comparison.

b. Are you frading the same data that was tested? The use of an artificial, continuous contract for testing futures will not give you the same results as testing eadi contract, beginning at the time it becomes the most active of the deliverj months. Normally, artificial continuous prices are smoother than real prices and sppear to perform better.

c. Are you monitoring the difference between the sjstem and actual enfries and exits? Slippage, the difference between the sjstem price and the actual maiket execution, can change performance from theoretically profitable to a real trading loss. Monitoring this difference will tell you the realistic execution costs, the maximum volume that can be traded, and the best time of dsy to trade.

This chapter has covered a lot of material vital to the successfiil development of a trading program. Because there are so many issues, the following is a brief summary of the most important conc ts:

1. Select a trading strategj with a sound underlying premise; complexity is not necessarj.

2. Reserve out-of-sample data for validation

3. Select the range of parameters to test that is logical, favoring the faster or slower trading approach.

4. Perform the longest reasonable test to include as many market situations as possible.

5. Test as many different markets as possible looking for a common pattem in the results.

6. Standardize the results to facilitate comparisons between sjstems and test periods.

7. Evaluate the robustness of the method rather than the peak performance; avoid fine tuning.

8. Assess the effects of past price sho(ts on profits and rid;.

9. Beta-test (paper trade) the sjstem until you are certain that it is working properly

You might also keep in mind that testing uses historic data and woiks best in real trading when the current market has the same pattems seen during the test data. The premise of optimization is that the markets will continue to behave in the future as they have in the past. This may be why the index markets traditionally have been among the poorest sjstem performers-because they are trading at price levels and volatility that have never been seen before, other areas of interest to readers actively testing and evaluating test performance win be genetic algorithms, a new search technique, discussed in Chapter 20, and most of Chapter 23, especially the section "iVIeasuring Rid;."

22 Practical Considerations

This chapter discusses areas of technical and mathematical analjsis that are not sjstems by themselves, yet essential to the successful development of a sjstem. The first part concems the use of the computer. Technical analjsis and computers have become sj nonymous with eadi other and, with the simple steps provided by development software such as TradeStation, MetaStock, and spreadsheets, it would be rare to trade a new technical sjstem without first running it through a computer to simulate its past performance. Those who do not use a computer directly are strongly influenced by the computer sjstems of others.

The second section, "Price Shocks," is the most underestimated deterrent in finding the right sjstem and generating trading profits. This idea has been discussed in different places during this book, specifically in Chter 21 in the section Anatomy of an Optimization"-, yet it cannot be overstated. Unpredictable events, once they become pari of price history, are often treated as though they could be anticipated. This can inflate performance retums and minimize risk during strategj testing with serious consequences to actual trading. This section will try to explain how to close the gap between testing and trading.

The Theory of Runs is actually an application of gambling techniques, primarily JVIaitingales, to trading. What would seem more reasonable than assuming that the odds are against you in the futures maiket or as a shortterm trader and treating it as a gambling situation? The third section is about combining sjstems together that focus on different maiket characteristics, and the understanding of the trade-offs inherent in eadi type of analytic method. There is also a short analjsis of trading limits to show some of the problems and flaws of moving from a test to an operating environment. The last section reviews the similarity of sjstems and the impact of technical sjstems on futures maikets.

USE AND ABUSE OF THE COMPUTER

Make sure your present report sjstem is reasonably clean and effective before you automate. Otherwise your new computer will just speed up the mess.

Robert Townsend

Computers are not a substitute for thinking. They excel in performing the same tedious tadc over and over again, quickly and accurately-provided, of course, that correct information was entered. But even though technologj has not yet reached the stage depicted in Star Wars, the computer is the only practical tool for evaluating trading ideas This section will consider both good and bad wajs to approadi a computer problem, none of which can be credited to or blamed on the madiine. As a powerful tool, a computer cant be beat many of the sjstems, advancements, and reflnements presented in this book could not have been considered without it.

Computers come in many shapes and sizes, and can be purchased, leased, and borrowed. Their packaging has become second only to the car. During the 1980s, the ai-erage life of a computer generation was about 10 years; however, advances in technologj have

n.rJ.opf,NewY.d 19-C p 3.

reduced that to about 3 years, if you dont mind a year when you feel obsolete. The past few years have seen a remaifcable evolution in the computer industij From an acceptance of the personal computer (PC) to the point at which some households would rather give up their television than their computer, the PC has revolutionized data processing and brought the Internet into tens of millions of homes. Because not everyone knows how to program a computer, software companies have developed turn-key packages, programs that are userfriendly, operated primarily by menu selection. The combination of these two factors, cheap hardware and sophisticated software, has reached far into the realm of individual and institution investors.

One of the first and foremost organizations to provide convenient computer research tools for the trader was the Technical Analjsis Group (TAG), which began in 1975 as a users group. It provided menu-driven capabilities for analjsis of daily price data and programming capability similar to a spreaddieet using its CompuTrac software. It provided daily automatic updates of data at a time when that Iilxuij was the state of the art. The company that breaks ground is not always the one that survives the race, and those analjsts developing sjstems now expect much more Generally, development software falls into two groups:

1. Daily maiket analysis, with price updates at the end of the day