12.8 IMPLIED f-RATIOS FOR THE VARIOUS CRITERIA 503

Bayes Theorem and Posterior Odds for Model Selection

Bayes theorem is based on the definition of conditional probability. Let £, and Ej be two events. Then by the definition of conditional probability, we have

Hence

P(£, I E,) =

{ I £)P(£,) P{E,)

Now substitute H (hypothesis about the model that generated the data) for E, and D (observed data) for £2- Then we have

Here P{D \ H) is the probability of observing the data given that H is true. This is usually called the likelihood. P(H) is our probability that H is true before observing the data (usually called the prior probability). P(H \ D) is the probability that H is true after observing the data (usually called the posterior probability). P(D) is the unconditional probability of observing the data (whether H is true or not). Often P(£)) is difficult to compute. Hence we write the relation above as

P(H I D) P(D I ) ( )

That is: Posterior probability varies with (or is proportional to) hkelihood times prior probabiHty. If we have two hypothesis H, and H2, then

Hence

( , D) P{D )

I D) I ,) )

(12.16)

The left-hand side is called posterior odds. The first term on the right-hand side is called the likelihood ratio, and the second term on the right-hand side is called the prior odds. Thus we have:

Posterior odds equals likelihood ratio times prior odds.

In the problem of choice of regressors, , and Hj involve some parameters, say P and y. The likelihood ratio is computed by a weighting procedure, the weights being determined by the prior distributions of these parameter sets. Thus we have

, I data) / LiPih) dy )

(12.17)

where L, and are the respective likelihoods and P, and are the respective prior distributions for the parameters in models 1 and 2. The first term on the right-hand side is sometimes called the Bayes factor.

There have been many suggestions for the literature for P, and Pj and thus for the computation of Bayesian posterior odds."" Since our purpose here is to illustrate how the implied F-ratio changes with the sample size according to some Bayesian arguments, we will present one formula suggested by Leamer. He suggests that the posterior probabilities should satisfy the following properties:

1. There must be no arbitrary constants.

2. The posterior probability of a model should be invariant to linear transformations of the data.

3. There should be a degrees of freedom adjustment: of two models that both yield the same residual sum of squares, the one with the fewer explanatory variables should have the higher posterior probability.

Based on these criteria, Leamer suggests prior distributions P, and Pj and computes the posterior odds as prior odds times the Bayes factor given by (in the notation we are using)

„ fRRSS"" =

[uRSs)

,7*2« (12.18)

We say that the evidence favors the restricted model if < 1, Using the F-ratio defined in (12.14), we get the condition as

F< 4- - 1) (12.19)

Compared with the F-ratios presented in Table 12.4, this criterion produces large changes in the F-ratios as the sample size n changes. The F-ratios Leamers criterion implies are presented in Table 12.5.

12.9 Cross-Validation

One of the major criticisms of the different criteria for selection of regressors that we discussed in the preceding section is that a model we choose may be the best for the period used in estimation but may not be the best when we use it for prediction in future periods. To handle this criticism, it is often suggested that we use only part of the data for the purpose of estimation and save the

"An early survey is in K. M. Gaver and M. S. Geisel, "Discriminating Among Alternative Models: Bayesian and Non-Bayesian Methods," in P. Zarembka (ed.). Frontiers of Econometrics (New York: Academic Press, 1974).

PiH, I data) / L,P,0)

Source: E. E. Leamer, Specification Searcties, Wiley, New York, 1978, p. 116.

rest for the purpose of prediction and to check the adequacy of the model chosen. We estimate the different models using the first part and then use the estimated parameters to generate predictions for the second part. The model that minimizes the sum of squared prediction errors is then chosen as the best model.

This procedure of splitting the data into two parts-one for estimation and the other for prediction-is called cross-validation. Actually, we have two sets of prediction errors. The prediction errors for the first part (estimation period) are known as within sample prediction errors. The sum of squares of these prediction errors is the usual residual sum of squares. The prediction errors from the second part are known as the out-of-sample prediction errors. Different criteria in cross-validation depend on different weights given to the sums of squares of these two sets of prediction errors. A criterion often used is to give equal weights to these two sets of prediction errors.

What the cross-validation procedure does is to impose a penalty for parameter instability. If the parameters are not stable between the estimation period and prediction period, the sum-of-squared out-of-sample prediction errors will be large even if the sum of squares of within-sample prediction errors is small. Thus the procedure of model selection by cross-validation impUes choosing the model that minimizes: the usual residual sum of squares plus a penalty for parameter instability.

Instead of splitting the data into two sets and using one for estimation and the other for prediction, we can follow the procedure of using one observation at a time for prediction (and the remaining observations for estimation). That is, in fact, the idea behind "predicted residuals" discussed earlier in Section 12.4. There the sum of squares of the predicted residuals, PRESS, was sug-