3. They are the same for all products, impljing an attenpt at robustness.

It would not be possible for either the profits or the equity drops to be better in this model than an optimized strategj, such as the Modified Three-Crossover Method; yet, out of 17 commodity maitets simulated during the 1970 through 1976 period, only two lost money. That is a very impressive record. There should be a greater degree of confidence in this performance than in an optimized program.

Importance of Failed Tests

When the period from 1986 to 1996 is tested, as seen in Table 2 I -II, we see much lower profits, large losses in the index maitets, and uniformly high rid;. Net results are worse than other methods tested. This certainly means that this technique is no longer good. Perhaps it means that a rigid set of frends cannot survive over the long term; the lengths of the underljing frends have changed; they are no longer uniform because of changing maitet participation; or there is more noise in the maitet, making it difficult to identify the frend in a timely fashion

These results should make us rethink the conclusions drawn from the previous tests. We have looked at the best dajs and the robustness of the technique. Because we are no longer confident that the best day of the past will be the best day of the future, we need to depend on the robustness of the sjstem even more. If a laige percentage of all tests are profitable, then there is a much greater chance that any parameter selected win be profitable. Our goal is to find the most robust sjstem.

To remove the dependence upon specific fixed parameters that might retum unstable performance over many years, it is necessary to generalize many of the features in a sjstem. For example, instead of a fixed stop-loss at 10 points or $500, it will be necessarj to allow that stop to varj based on maitet volatility Even the calculation period of the frend may change according to volatility, maitet noise, volume, or some other factors. This approach is discussed fiirther in Chter 17 (Adaptive Techniques").

Test Criteria

The method used in these tests is not bad, but real market factors must be included or the results are deceiving. Although the researchers may have been pleased with the out-ofsample profits, they were only paper profits. In real trading, the chances greatly favor losses.

Testing should closely approximate frading. If there is doubt about the costs, make them larger. A sjstem that can profit under testing penalties should make money when actually fraded. Even when using prepackaged computer optimization software, the slippage

TABLE2I-IIResults of a4-9-I8 Crossover (I986-I996),$50 Transaction Costs

fci- JU.,. Trades

can be added by making the commission costs high; both are per frade costs. If a mistake is to be made, do it by being too conservative; however, it is ahvajs best to be realistic.

Gettmg the Big Picture through Testing

As mentioned throughout this chapter, the key to robustness is to test more data, more maitets, more of everjihing, and then to observe the results looking for a common pattem of overall consistency. There is no better way to tell if a maitet has trends than to test it for a variety of trending methods. A maitet that performs well using an interday breakout, but fails with a dual-trend crossover, is not a good candidate for trending profits in the future.

A comprehensive look at 12 trading methods applied to 12 futures maitets gives us a chance to see generalized performance. The annualized percentage retums for a variety of U.S. maitets are shown for 12 well-defined technical methods in Table 21-12. These trading sjstems are:

CHL,Channel breakout, using the closing price penefration of the N-day high and low

PAR Wilders Parabolic Sjstem, in which the stop and reverse gets closer each day

DRMDirectional Movement, a Wilder method that averages the ups and downs separately

RNQ Range Quotient, basing a breakout on a ratio of current change to past change

DRP Directional Parabolic, combining DRM and PAR sjstems

Ml 1 Price Channel, a breakout sjstem using only the oldest and most recent prices

LSO L-S-0 Price Channel, usmg two parameters, including an interval of oldest prices, to decide a channel breakout

REF Reference Deviation, determining a volatility breakout based on a standard deviation of past closing price changes

L.-sPLukacE Wa.leEfTsen.andSc-ttH Innu, "H-ir to testpfbtatility ftecliuical toW sy.-tems Futures i ber 1

TABLE 21-12 Percentage Retums by Sjstem and Maricet, 1978-1984*

CHL MR DRM FNQ DRP LSO REf DMC DRf AIAB ALX Avtn

.JnvichirairniseslinBHJu iO\

DMC Dual Moving Average Crossover, holding a frend position when both moving averages have the same frend

DR] Directional Indicator, a ratio of current price change to total past price change

MAB Moving Average with Price Band, giving a signal when prices pendrate the band

ALX Alexanders Filter Rule, generating signals when prices reverse from a previous swing high or low by a percentage amount

create the results in Table 21-12, each maitet was tested for 3 years and the best parameters used to generate the retums for the nest years, in a step-forward approach. It is easj to see that some maitets are not profitable for any strategj and that some strategies are generally poor. Unless you were creating a strategj for a specific group of markets, you would not want to trade a sjstem that was not profitable in less than 50° of the markets, nor would you want to trade a maitet that failed in most of the trend strategies. For example, cattle lost in every sjstem, and the RNQ, REF, and M AB sjstems were consistently unprofitable; therefore, none of those would be good candidates for trading. The most consistent sjstems were CHL, PAR, MI I, and DMC, each posting 8 of 12 winning maitets; yet, each has a noticeably different technique. T-bills present an interesting choice because the highest profit, 239.8° o. occurred in one of the least reliable sjstems, REF; choosing DRP is likely to be a better choice. Displajing the sjstem results by year for all maitets is another way to look at robustness. In Table 21-13, we see that RNQ and net losses over all maitets in all years, while CHL, Mil, and DMC stand out as very consistent. The combined presentation of results aaoss all maitets and sjstems is a clear way of avoiding overfitting by looking al the big picture.

PRICE SHOCKS

Price shocks are laige changes in price caused by unexpected, unpredictable events. The impact of price shocks on historic tests can change the results from profits to losses, and varj the risk from small to exfremely laige and unfortunately, not enough thought is given to how these moves affect test results and future performance. A discussion of price shocks could easily fill an entire book, but the conc ts that are important can be explained

TABLE 21-13 Percentage Rehims by Sjstem and Year*

briefly. When it comes to actual trading, the difference between your expected results and actual performance (noi attributed to slippage is entirely dependent on the number of price shocks.

By its very nature, a price shock must be unexpected. However, not all events cause shocks. An assassination, abduction, or political coup is likely to be a surprise, while a crop freeze can be anticipated as weather turns unusually cold. Some economic reports, such as a .5°-<, increase in the Producer Price Index or jump in the balance of trade, will come as a shock, but a low carrjover supply of soybeans or a tightening of the money supply after steadj economic growth can be anticipated. When a change is anticipated, the maitet adjusts to the correct level before the news is announced, when it is wrong, prices react in proportion to how poorly it was anticipated.

The problem comes when you test historic data. We know at the time of a price shock that we could not have anticipated the event; at best we have an even chance of being on the right side of the price move. But the computer doesnt know that, and the best results from a laige series of tests often include profiting fran the laigest price sho(ts-a situation that would be impossible in actual trading. For example, if there were 10 price sho(ts in 10 years of data, and your historic tests profited from 8 of those shocks, then you have overestimated your profits. Even worse, you have underestimated your risk by believing that a price shock produced a profit and not a loss

How can you correct this problem? First, you can look at the pattern of profits and losses fran a series of tests