back start next


[start] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] [29] [30] [31] [32] [33] [34] [35] [36] [37] [38] [39] [40] [41] [42] [43] [44] [45] [46] [47] [48] [49] [50] [51] [52] [53] [54] [55] [56] [57] [58] [59] [60] [61] [62] [63] [64] [65] [66] [67] [68] [69] [70] [71] [72] [73] [74] [75] [76] [77] [78] [79] [80] [81] [82] [83] [84] [85] [86] [87] [88] [89] [90] [91] [92] [93] [94] [95] [96] [97] [98] [99] [100] [101] [102] [103] [104] [105] [106] [107] [108] [109] [110] [111] [112] [113] [114] [115] [116] [117] [118] [119] [120] [121] [122] [123] [124] [125] [126] [127] [128] [129] [130] [131] [132] [133] [134] [135] [136] [137] [138] [139] [140] [141] [142] [143] [144] [145] [146] [147] [148] [149] [150] [151] [152] [153] [154] [155] [156] [157] [158] [159] [160] [ 161 ] [162] [163] [164] [165] [166] [167] [168] [169] [170] [171] [172] [173] [174] [175] [176] [177] [178] [179] [180] [181] [182] [183] [184] [185] [186] [187] [188] [189] [190] [191] [192] [193] [194] [195] [196] [197] [198] [199] [200] [201] [202] [203] [204] [205]


161

TABLE 21-16 Test 3: January 2,1990, through March 28,1991*

Period

NelPffi

/VlnxDD

Trds

-17255

64.0

-26975

-13195

-45.0

29295

4480

26.6

16840

8430

67.6

12475

-»690

-22.6

20725

-1395

-6.4

-21775

28335

2114.6

-1340

-360

-14420

18245

594.3

-3070

8265

246.7

-3350

2485

55.0

-4515

9075

184.8

-4910

7155

100.4

-7130

3815

53.3

-7160

7195

83.9

-8580

-18225

-81.3

-22410

-12945

-70.4

-18390

-4065

40.3

-10080

-8645

56.0

-15430

-3485

-753

-4615

Trade Detdt from Teamg

Trade Dote

May 24.90

23.58

May 25,90

Sell

23.55

Jul 16,90

24.16

-635

-690

Nov 9.90

Sell

40.58

16395

15705

Jan 14.91

3876

1795

17500

Jan 16,91

Sell

39J6

18075

Mar 8.91

29 9

9445

27520

Mar 13,90

Sell

30 9

28495

Mar 19.90

30.75

28610

Mar 28.90

Sell

30.50

-275

2833S

* Test period includes bodi price itiocks. Tesi selects JS-da/ mavipj average. Total prarits are (2.495 without price stiocia

Opdnizicion prarits indude both Intf invasioo (rf Kuwait and At U.S-attack on Iraq.

through historic data to uncover the size of the interest rate increase needed to move the market, or the tjpe of pattem shift fran equities to bonds that could yield a profitable trade.

When we begin looking for a relationship between inventories and price we are reasonably confident that there is a solution, and we can even identify those elements that form the solution. The process is one of careful reasoning and a small amount of testing, or validation. However, when we test a set of trading rules on historic data, we do not actually know that this method will yield a profitable result.

For example, take the simplest case of a trading sjstem that only uses a moving average trendline. A long position is signaled when the trendline tums up, and a short is generated when the trendline turns down. Is this a sound premise? How do you decide that a profitable test result will become profitable real-time trading? The process for understanding this is far fran trivial.



Realistic Assumptions

After you have decided on your underljing strategj, and before you begin testing, there are a number of decisions to make. These must be made without the benefit of hindsight, but according to your own reasoning. They include:

1. The amount of data to be tested

2. The commission cost per unit traded

3. The slippage cost per unit traded

4. The range of each parameter (variable) to be tested

This last item, the range of each parameter, means that you must decide, in advance, which moving average periods will be tested. If this is to be a long-term program trjing to capitalize on the way governments manipulate interest rates

or money supply, then you might choose periods beginning with 50 day5, up to 300 dajs. If you believe that the longterm is too riskj and most trends are medium-term, then a range of 20 to 100 days might be best. The very fast day trader, looking to dowrqalay the trend, would want to test periods from 5 minutes to 1 day. It is important that the periods chosen for testing correspond to how you perceive the price movement. It is not enough to say, "Ill frade anything that makes money." Commissions and slippage are also vital to the results, but more significant for the short-term frader. As a sjstem focuses on shorter holding periods, the profits per frade must become smaller; therefore, the commissions and slippage can become the difference between profits and losses. Because a computer assimies that

orders can be executed perfectly (at the signal price), even during a 5-minute bar where prices jumped 1% with virtually no trading, it takes some effort to decide on a realistic level of slippage. Normally, it is safe to overestimate the cost of entering and exiting the martlet, but for a short-term trader, even a small overestimation could eliminate a potentially profitable frading technique. Accuracy, as near as possible, is the best approach. By making clear assumptions in advance, you gain the advantage of validating your ideas. You thought that medium-speed frends were likely to be best but test results show that only a narrow range of speeds are profitable. You must now rethink why this occurred. Using negative feedback to learn more about the market is a verj healthy process.

Statistics Can Say Whatever You Want

Most computer software that allow historic testing give results that emphasize the highest retums. Given a choice, most users of these programs want to see the highest rehims A single test result, however, should not be considered representative of the worth of the sjstem being tested.

If we test the same 10-year period of historic data for 1,000 different parameter values, spread over a wide range, we are likely to be pleased if 50 of those test results were very attractive. Statistically, we might say that 50 are significant at the 5>o level. But that is not necessarily frue. In a welldistribution, or random, set of results, we can expect 5" to be statistically significant. If the trading sjstem is fruly sound, then we would like to see much more than 5" successes.

Robustness

Practically speaking, robustness franslates into getting the most realistic results by using the fewest parameters and the most data. Robustness is a term used to describe a sjstem or method that wori under many martlet conditions, one in which we have confidence; it is associated with a successful result using an arbifrary set of parameters for a sjstem. This is most likely to occur when only a few parameters are tested over a long time period. For example, if you claim that a long-term moving average sjstem is robust, then someone unfamiliar with the sjstem should be able to choose the moving average period of 130 dajs and have reasonable expectations of a profit. This would be most likely if a laige percentage of all time intervals in the long-term test were profitable.

\Vhen first testing a sjstem, it is best to avoid looking at the laigest profits, but instead, study only the average values of all tests, that is, the average net profit, average number of trades, average drawdown, and average profit per trade, in addition, looking at the standard deviation of profits for all tests will give a very good idea of the consistency of retums over the range of tests.

The following is a checklist along with a brief explanation that might help when trjing to follow a procedure to get robust test results:



1. Deciding what to test. Testing should be used for validation of an existing idea, not for discovery.

a. Is the strategj logical? it should be based on carefiil observation of the maiket, or an awareness of relationships between different factors that drive the maiket, or between different maikets themselves.

b. Can you program all the rules? Programming a sjstem guarantees that all of the contingencies have been thought out. While not everjthing can be programmed, in particular situations requiring crisis intervention, these cases are few.

c. Does the strategj make sense only under certain conditions? A trading program can be limited to a specific situation, and testing should reflect ttiose limitations.

d. Take a guess as to the expected results. It is important to know if your estimate of the sjstem success is correct. Getting a good result for the wrong reason requires just as much rethinking of the problem as getting bad test results.

2. Deciding bow to test. There are a number of steps necessary before you can actually begin the test process.

a. Choose the testing tools and methods. Select a test platform, such as TradeStation or MetaStock, which can speed up the test process and keep it objective.

b. Do you have enough of the rig data? More data is best but if there is less data available, it should have a wide range of market conditions, such as bull and bear markets, price sho(ts, and sustained sidewajs periods.

c. Will you test a full range of parameters? The nature of your trading strategj, for example, longterm or short-term breakouts, may determine the parameters that are tested. Test only the range that makes sense for the strategj. This serves to validate the concept and avoids discovering better techniques.

d. In what order will the parameters be tested? Test the most important parameters first such as the time period. More subtle parameters are often used for refining the results, and may not be helpful at all. You can usually tell the most important parameters because they cause the widest range of results.

e. Are the parameters dishibuted properlj? Testing equal intervals of trend periods can be misleading. The difference between a 3- and 4-dsy moving average is

letailedeiijJanati on testing roburtnesE can be f in Phtv Fjufbian, fti.irter Trading iMci.iwHill, 1995i

0, while the difference between a 99- and loo-dsy average is only Po. When you test periods from I to 100 dsjs, your average results are heavily weighted toward the slow end, which has similar performance.

f Have you defined the evaluation criteria? You must know whether highest profits, highest reliability, or some other combination will be used to evaluate the results. Rid;-adjusted performance can be a very good objective criteria

g. How will the output be presented? Results can look very different when presented in a plain table. Two-and 3-dimensional contour msps can give a better view of the way performance varies when two parameters change.

3. Evaluating the results. Carefiil review of the test results can avoid many hours of unnecessary work. Looking at the detail of the performance, and even the individual trades of a few selected results, will give you insight into the good and bad aspects of your program.

a. Are the calculations correct? just because calculations were done by a computer doesnt mean they are ahvajs correct. A person had to tell the computer what to do, and people make mistakes.

b. Were there enough trades to be significant? We have uncertainty when the results show excellent



[start] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] [29] [30] [31] [32] [33] [34] [35] [36] [37] [38] [39] [40] [41] [42] [43] [44] [45] [46] [47] [48] [49] [50] [51] [52] [53] [54] [55] [56] [57] [58] [59] [60] [61] [62] [63] [64] [65] [66] [67] [68] [69] [70] [71] [72] [73] [74] [75] [76] [77] [78] [79] [80] [81] [82] [83] [84] [85] [86] [87] [88] [89] [90] [91] [92] [93] [94] [95] [96] [97] [98] [99] [100] [101] [102] [103] [104] [105] [106] [107] [108] [109] [110] [111] [112] [113] [114] [115] [116] [117] [118] [119] [120] [121] [122] [123] [124] [125] [126] [127] [128] [129] [130] [131] [132] [133] [134] [135] [136] [137] [138] [139] [140] [141] [142] [143] [144] [145] [146] [147] [148] [149] [150] [151] [152] [153] [154] [155] [156] [157] [158] [159] [160] [ 161 ] [162] [163] [164] [165] [166] [167] [168] [169] [170] [171] [172] [173] [174] [175] [176] [177] [178] [179] [180] [181] [182] [183] [184] [185] [186] [187] [188] [189] [190] [191] [192] [193] [194] [195] [196] [197] [198] [199] [200] [201] [202] [203] [204] [205]