and validation sets. In Run 2 with five inputs, performance was relatively consistent across the three data sets.

In my experience, variable selection consistently ranks as the number one factor in enhancing generalization. If you have more than a handful of variables as input to your model, you have too many. As a side benefit, the performance of a series of networks trained on a reduced data set is often much more consistent. This is comforting in terms of predicting the performance of a deployed system.

Conflicting Data and Expected Values

Local minima are one of the problems often faced in training a neural network, particularly on financial data.* One of the causes of local minima is conflicting data. When two training examples have nearly identical input values, but substantively different output values, this induces a small ripple or local minima in the objective surface. Though most of these are small enough that they do not adversely affect training, they can cause problems.

If the number of inputs have been reduced to a sufficiently small number (5 to 10), it is often possible to eliminate conflicting inputs by computing expected values in local neighborhoods and using those prototypes as training vectors. This process also tends to reduce data distribution effects on network training.

The process is simple. Divide the range of each input into a small number of bins, typically 5 to 10. For each example, construct a "key" that consists of a series of digits. Each digit corresponds to the bin to which the input field belongs. Sort all the data based on this key. If two or more examples have the same key, replace all of them with a new prototype that is the average of all the examples with the same key. Use this reduced set of data for training the network. You are now assured that no two data vectors will conflict.

Eliminating conflicting data results in smaller training sets, reduces the dependence of the model on data distribution, and eliminates the likelihood that a model will get stuck in a local minima.

conclusion

Data preprocessing involves several steps. Some of them are simply mechanical; others require a certain degree of expertise. Use the following seven-step process to prepare raw data for network training:

*A local minima or maxima refers to the parameter settings of a function, so optimized that any small changes made to any of the parameters would degrade the functions performance, as rated by some score. A local minima is produced when the optimal score is at its lowest, and a local maxima is produced when the optimal score is at its highest. (Editor)

1. Identify domain specific transformations that may enhance the signals in the data or make a good solution more obvious. This typically involves technical transformations, financial ratios, correlations, or shape detectors.

2. Convert the problem into a stationary prediction problem. Modify or scale transformations as required.

3. Further modify data by applying distribution-shaping transformations. Continue to maintain transformations from Step 2 as part of the candidate inputs.

4. Extract a subset of data for performing variable selection and training the network that uniformly covers the input space.

5. Select small synergistic sets of variables for network training.

6. Eliminate conflicting data by converting "conflicting" examples to expected values.

You are now ready to train a network or develop any other kind of model-based system. Data preprocessing is a tedious and unglamorous process. Often it is easier to simply use raw data and let the neural network do its "magic." However, as discussed, the time and effort to effectively preprocess data often leads to substantially more profitable models. It is worth it.

Endnotes

1. Press, William H., Teukolsky, Saul A., Vetterling, William ., & Flannery, Brain P., Numerical Recipes in (2nd ed.), Cambridge, UK: Cambridge University Press, 1992.

2. Statman, Meir, (Leavey School of Business, Santa Clara College), "Behavioral Finance" presentation at Chicago Quantitative Alliance annual conference, September 13-14, 1995.

3. Colby, Robert W. & Meyers, Thomas A., Encyclopedia of Technical Market Indicators, Homewood, II: Business One Irwin, 1988.

4. Ibid.

5. Eng, William F, The Technical Analysis of Stocks, Options and Futures, Chicago: Probus Publishing, 1988.

6. Chatterjee, Samprit & Price, Bertram, Regression Analysis by Example (2nd ed.), New York: John Wiley & Sons, 1991, p. 31 f.

7. See Note 1.

Modeling the Markets Using Statistical Network Data Mining

Keith C. Drake and Dale E. Nelson

What is the Market and What Should We model?

When we first begin to think about the markets and think about modeling them, we must ask some salient questions. First, what determines the price of a stock (or the value of an index)? Is it the book value? No. If that were true, stocks would sell at no more than book value. We have all seen stocks selling above or below that value. Is it the amount of profit a company is making? No. There are companies that have been losing money for years and yet their stock prices have increased. On the other hand, thete are very profitable companies whose stock has decteased in value. Why then, do stock prices go up or down? It is simply the perception by investors that the price of the stock will increase or decrease! The investors will make the appropriate choice based on this.

When using the computer to try to influence our investing decisions, what we are really trying to do is develop some sort of model of the market of interest (see Figure 17.1). In engineering, the method often uses first principles of the process at

Figure 17.1 Market modeling paradigm.

INPUT

• • •

MARKET MODEL

OUTPUT

PREDICTION