back start next


[start] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] [29] [30] [31] [32] [33] [34] [35] [36] [37] [38] [39] [40] [41] [42] [43] [44] [45] [46] [47] [48] [49] [50] [51] [52] [53] [54] [55] [56] [57] [58] [59] [60] [61] [62] [63] [ 64 ] [65] [66] [67] [68] [69] [70] [71] [72] [73] [74] [75] [76] [77] [78] [79] [80] [81] [82] [83] [84] [85] [86] [87] [88] [89] [90] [91] [92] [93] [94] [95] [96] [97] [98] [99] [100] [101] [102] [103] [104] [105] [106] [107] [108] [109] [110] [111] [112] [113] [114] [115] [116] [117] [118] [119] [120] [121] [122] [123] [124] [125] [126] [127] [128] [129] [130] [131] [132] [133] [134] [135] [136] [137] [138] [139] [140] [141] [142] [143] [144] [145] [146] [147] [148] [149] [150]


64

Quantifying a Markets Upside and Downside Potential

Timothy w. Hayes

If you are setting out to become a serious trader, you can seek to use a subjective approach that lends itself to trading decisions based on myriad opinions and gut emotions, or you can seek an objective approach that instills discipline into a trading system. The latter is the less risky option, and potentially the more rewarding.

But if handled incorrectly, the objective approach can mislead you into thinking that your system is helping you make money, when in fact it is leading you astray. When the quantification process fails to deliver, instead producing misleading messages, the subjective approach is no worse an alternative-a misguided quantification effort can be worse than none at all. The predicament is how to make your quantification efforts truly pay off in the form of more profitable trading.

A good starting point for testing and optimization is to understand the potential pitfalls in each step of the process. You must also decide the time frame you want to focus on. Are you interested in day trading or only a few trades a year? You must also identify your level of risk tolerance-how much risk are you willing to take, and for how much reward? And you must decide how you will use the indicators that result from your testing. You may decide to combine your indicators into composite models, a "weight-of-the-evidence" method that would reduce the risk of big losses if an individual indicator were to go awry. Or you could use the signals as individual inputs for fine-tuning your model-based trading decisions.

This chapter will start with the issues to keep in mind when embarking on the optimization process, and it will then take a closer look at several methods of quantifying market performance following indicator signals.

Adapted from "The Quantification Predicament," Ned Davis Research. Used with permission. In recognition of this essay, the author was the recipient of the 1996 Charles H. Dow Award. Sponsored by Dow Jones Telerate, the Market Technicians Association, and Barrons, the award recognizes "an outstanding original work or a significant extension of a previously known work that best expounds on the principles of technical analysis." The authors original essay detailed the need to quantify technical indicators and determine their value and role in a truly objective market outlook. He has adapted it here for the trader interested in the basics on testing and optimization.



The Concerns

There are several reasons quantification must be handled with care. The initial concern is the data used in developing the indicators. If it is inaccurate, incomplete, or subject to revision, it can do more harm than good, issuing misleading messages about the market that is under analysis. The data should be clean and contain as much history as possible. When it comes to data, more is better-the greater the data history, the more numerous the like occurrences, and the greater the number of market cycles under study.

This leads to the second quantification concern-sample size. The data may be extensive and clean, and the analysis may yield an indicator that foretold the markets direction with 100 percent accuracy. But if, for example, the record was based on just three trades, the results would lack statistical significance and predictive value. In contrast, there would be few questions regarding the statistical validity of results based on more than 30 observations.

The third consideration is the benchmark, or the standard for comparison. The test of an indicator is not whether it would have produced a profit, but whether the profit would have been any better than a random approach, or no approach at all. Without a benchmark, "random walk" suspicions may haunt the results.1

The fourth general concern is the indicators robustness, or fitness-the consistency of the results of indicators with similar formulas. If, for example, the analysis would lead to an indicator that used a 30-week moving average to produce signals with an excellent hypothetical track record, how different would the results be using moving averages of 28, 29, 31, or 32 weeks? If the answer was "dramatically worse," then the indicators robustness would be thrown into question, raising the possibility that the historical result was an exception to the rule rather than a good example of the rule. An indicator can be considered "fit" if alterations of the formula would produce similar results.

Moreover, the nonrobust indicator may be a symptom of the fifth concern, and that is the optimization process. Much has been written about the dangers of excessive curve fitting and overoptimization, often the result of unharnessed computing power. As analytical programs have become increasingly complex and able to crunch through an ever-expanding multitude of iterations, it has become easy to overoptimize. The risk is that armed with numerous variables to test with minuscule increments, a program may be able to pick out an impressive result that may in fact be attributable to little more than chance. The accuracy rate and gain per annum columns of Figure 11.1 compare results that include an impressive-looking indicator that stands in isolation (top) with indicators that look less impressive but have similar formulas (bottom). One could have far more confidence using an indicator from the latter group even though none of them could match the results using the impressive-looking indicator from the top group.

What follows from these five concerns is the final general concern of whether the indicator will hold up on a real-time basis. One approach is to build the indicator



Figure 11.1 Results from hypothetical indicator tests. The five results in the top half of the table show one with outstanding performance, while all five in the bottom half of the table have similar performance.

SUMMtKl RESULTS FROM HYPOTHETICAL INDICATOR TESTS

These results contain an impressive-looking EXCEPTION to the rule ...

Moving

Number of Average Trades (Periods)

Buy Level

Sell Level

Accuracy Rate (%)

Gain/Annu m

11.2

11.3

65. .

15,1

10.1

These results would

all be good EXAMPLES

of the rule ...

15.6

11.8

15.8

12.0

16.0

12.1

16.2

12.1

16.4

12.0

Buy-Hold Gain/Annum 63

and then let it operate for a period of time as a real-time test. At the end of the test period, its effectiveness would be assessed. To increase the chances that it will hold up on a real-time basis, the alternatives include out-of-sample testing and blind simulation. An out-of-sample approach might, for example, require optimization over the first half of the date range and then a real-time simulation over the second half. The results from the two halves would then be compared. A blind-simulation approach might include optimization over one period followed by several tests of the indicator over different periods.

Whatever the approach, real-time results are likely to be less impressive than the results for the optimization period. The reality of any indicator developed through optimization is that, as history never repeats itself exacdy, it is unlikely that any optimized indicator will do as well in the real-time future. The indicators creator and user must decide how much deterioration can be lived with, which will help determine whether to keep the indicator or go back to the drawing board.



[start] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] [29] [30] [31] [32] [33] [34] [35] [36] [37] [38] [39] [40] [41] [42] [43] [44] [45] [46] [47] [48] [49] [50] [51] [52] [53] [54] [55] [56] [57] [58] [59] [60] [61] [62] [63] [ 64 ] [65] [66] [67] [68] [69] [70] [71] [72] [73] [74] [75] [76] [77] [78] [79] [80] [81] [82] [83] [84] [85] [86] [87] [88] [89] [90] [91] [92] [93] [94] [95] [96] [97] [98] [99] [100] [101] [102] [103] [104] [105] [106] [107] [108] [109] [110] [111] [112] [113] [114] [115] [116] [117] [118] [119] [120] [121] [122] [123] [124] [125] [126] [127] [128] [129] [130] [131] [132] [133] [134] [135] [136] [137] [138] [139] [140] [141] [142] [143] [144] [145] [146] [147] [148] [149] [150]