next year. It is better to assume that maiket changes will make it difficult to tell which combination of parameters will be the best in the future. Then you want a sjstem that can generate profits with any parameter set, not just 30 out of 100. Perhaps thats too ambitious; nevertheless, one sjstem is better than the other if the average of all tests is higher than the average of all tests in the other sjstem To be more precise, we could say that the best sjstem is the one in which:

Adjusted Retums = Average of all tests -1 standard deviation of all test retums

is greatest. From the (Kscussion of standard deviations in Chapter 2. we know that a retum greater than the acjusted retums will occur 68°o of the time.

Set aside data for final validation. To give yourself away of knowing whether you have overfitted the data, do not use all the data in your tests. This is needed for out-of-sample verification at the end. If the results of using this data are not good, you will need to reassess the performance. If it is good, you have some assurance that your testing was done correctly, but there are no guarantees other than time.

Stability over time. Be sure that the test data was long enough to have included a substantial change in price, a major bull and bear maiket, and an extended sidewajs period at low volatility. By viewing performance over a number of smaller time intervals, instead of only one average or total figure, it is possible to assess the stability of the strategj.

Elimination of outlier periods. Realistically, a sjstem should not base its profitability on a single major maifcel move over the test interval. In particular, if the move that generated the profit was actually a price shocfc, the sjstem had only a 50°o chance of posting that profit, and has an equal chance of a loss in the future, it may be practical to remove the profits generated on the first day or two of a move that began with a price shock, allowing for the possibility of holding the wrong position in the future

Concentrate on large profits and large losses. The largest profits and losses are critical to the net performance of any sjstem; therefore, you should make a special effort to study those trades. In doing that, you may find that the largest profits were the results of price shocks, as (Kscussed in the preceding point, or that there is an odd piece of data that was not obvious before. If there are a number of large profits and no laige losses, you may have excellent risfc control, but more likely you have overfit the data. There is an unfortunate tendency to look for problems when results are bad, but to accept them when they are good.

Reading between the Lines

It IS easj to show apecific exanples of rules that produce large profits, while the overall strategj is bad. Many books base intuitive proof or justification on specific examples. This can worfc if the general theory is sound, when there is fundamental or economic substantiation, such as expecting a seasonal low in the U.S. crops at harvest. For pattems and many purely technical intKcators, such as those using momentum, this is not as clear. Results are likely to show that the situations that fail greatly outnumber the fewer exceptionally good exanples.

Sjstems That Work in Only One Marfcet

From time to time, all traders receive mail offers for a highly specialized sjstem called Cattle Trader," the "Silver Day-Trading System," or "Easj Profits through Stock Index Trading." It is most likely that each of these sjstems has been finely tuned with rules unique to this one marfcet. Once aware of the optimization process and its results, these offers must be viewed more critically; after all, with optimization techniques able to test combinations of intKcators and rules, you could just as easily create a sjstem that seemed to make as much on a single marfcet. To prove that you have actually found something that worfcs requires an understanding of the rules and the way it was tested, or time to watch the Sjstem operate in the real marfcet. If possible, a comparison of the real trading profile compared with the expected performance based on testing should be available.

Student t-Test

Among tests that help ddermine whether the results are significant, the student t-test is one of the most useful. It tells you whether there is a sjstematic bias in the data by showing whether the mean of the data is significantly different from zero. This is a useful piece of information when you try to decide whether the results of only a few trades represent a good sjstem, or whether a series of losing trades implies that a sjstem has no value. For a single set of trades produced by computer testing or by actual trading, the t-test is:

average trade results ,-------7-

---------X V number of trades

1 standard deviation of trade results

The critical value of t, which can be found in AppentKx A, depends on the number of trades in the sample. For the case of a series of trades produced from the same martlet and same sjstem, the degrees of freedom = number of trades - 1. Using the table in Appendix A- if there were only 10 frades. ttie degrees of freedom would be 9. and the value of t must be greater than 1.833 to reach the 95" confidence level.

Where you want to know if two sjstems are producing results that are significantly efferent from each other, the technique is more complicated. It requires comparing the

-Ben Wam-ick. Event Tra.bg iTnrai. 1996)

mean, variance, and number of trades of the two sjstems; most important is the approximation for the degrees of freedom, which allows you to determine the confidence level of the results:

where Xj = the average of meihod 1 trades X2 = the average of method 2 trades

51 = variance of method 1 trades

52 = variance of method 2 trades n, = number of trades in method 1 «2 = number of trades in method 2

The calculation of degrees of freedom uses Satterthwaites approximation.

(«I - Wj) ( 2 - n,)

POINT-AND-FIGURE TESTING

The point-and-figure sjstem is representative of the group of swing sjstems. Buy and sell signals occur when prices move above or below previous highs and lows, reapectively. The current martlet direction changes when a price reversal exceeds a apecified cumulative price change, designated in points. Because it is a charting method, the reversal is shown in terms of boxes; the reversal criterion fraditionally is expressed as a "3-box reversal- (for a complete explanation, see Chapter 11).

The size of the box and the number of boxes that define a reversal are the two vanables that make up the reversal criteria. If silver were the object of the test, the box size might start as low as 10 points (1/lOth of one cent, the minimum move) and increase in increments of 10 points; the number of boxes in a reversal would he the integer values

1,2,3.....Figure 2 1-1 la shows that the expected pattem of test results is sjmmefric, extending across the (Kagonal

drawn from the top left to the bottom right comers. This example followed the convention of putting the fastest trend change in the upper left (smallest box size and a 1-box reversal) and the slowest in the lower rit comers.

The sjmmetrj is the result of very similar performance for equal reversal values, the product of the box size, and the number of boxes in a reversal. For exanple, a trend change of 40 points for the Swiss franc could have occurred using combinations of 40 points x 1 box, 20 points x 2 boxes, 10 points x 4 boxes, 4 points X 10 boxes, 2 points X 20 boxes, and 1 point x 40 boxes (among others). The difference in profitability occurs because the selection with the smallest box size will satisl the penetration criteria sooner, giving slightly faster signals and more frades. The

average profitability of all combinations of reversal size will appear as in Figure 2 1-1 lb. This representation can be used with the 2dimensional map to find the most consistent set of parameters.

When tested over long periods, most markets have increasingly wide price ranges, as clearly seen in the S&P Testing for fixed combinations of box size and number of reversal

FIGURE21-11 Point-and-figure mapping strategj. (a) Reversal number of boxes, (b) Reversal (box size x number of boxes)

Rfwrnl nunter ol bout

boxes for the past 10 years would find the single parameter set that performed best over fits and

this period; however, the increasing volatility would result in much laiger profits and losses in recent years. This change of risk will force the optimized results to favor parameters that performed best in the last few years and might ignore the profits and losses of the earlier period, which would be relatively small- To put it in perspective, a 10° move in the S&P in 1987 would only be about 25 points, equal to a total of $12,500, while it would be 90 points in 1997, or $45,000. If prices moved Poper day (3.00 points) in 1987, a reasonable reversal combination would be from .45 points for fast frading (. I5-pointboxwith3box reversal) to afull days move of 3.00 (I.OO-pointboxwitha 3box reversal). In 1997, either combination is likely to produce a new trade every few hours, creating avery different performance profile than prior years The problem of volatility was (Kscussed in Chapter 11. The alternative to a fixed box over time was to vary the box size as a percentage of price. Therefore, if the best reversal in 1987 was .45 for the S&P at a value of 3 00, it would be 1.35 in 1997 at 900. However, this may not have increased enough, and many maiket analjsis freat volatility as an exponential function with respect to price. This can be approximated using a square root function or log function, both readily available in a apreaddieet or testing program:

Box size =p X Vprlce Box size =p X In (price)

where p is the percentage that is varied as a parameter in the optimization. This last method would be similar to the technique of using point-and-figure on a log chart.