A Basic Problem in Testing Nonnested Hypotheses

The specification implied by equations (12.25) and (12.26) does not by itself enable us to make a valid comparison of the two models. In fact, this is the problem with testing nonnested models. Equation (12.25) specifies the conditional distribution of given x. Similarly, equation (12.26) specifies the condition distribution given z. How can we compare these two conditional distributions unless the role of z under and the role of x under , are specified? What we are given are two noncomparable conditional distributions. does not specify anything about the relationship between and z. , does not specify anything about the relationship between and x. Thus to compare the two models given by (12.25) and (12.26), we should be able to derive the conditional distributions of/(y x) and g(y \ z) under both and ,. To do this we have to supplement equations (12.25) and (12.26) by more equations. This is what is done by Mizon and Richard.

Summary

1. Diagnostic tests are tests that "diagnose" specific problems with the model we are estimating. The least squares residuals can be used to diagnose the problems of heteroskedasticity, autocorrelation, omission of relevant variables, and ARCH effects in errors (see Section 12.2). Tests for heteroskedasticity and autocorrelation have been discussed in Chapters 5 and 6, respectively. To test for omitted variables, we first estimate the equation by OLS and use y] and higher powers of y, as additional variables, reestimate the equation, and test that the coefficients of these added variables are zero. A test based on

"Godfrey, "On the Use," p. 76.

-Godfrey, "On the Use," p. 79. Godfrey quotes N. R. Ericksons Ph.D. thesis (1982) at the London School of Economics, which provides a numerical example in which Fand one-degree-of-freedom tests reject only one of the two competing models, but misspecification tests decisively reject , , and the comprehensive model.

jecting the false model. However, this is no consolation because it has also been found that the J-test tends to reject the true model too frequently with the estimated small-sample significance levels being so large that it is often difficult to justify asymptotically vaHd critical values.* Yet another shortcoming of the nonnested tests is that if both ,, and , are false, these tests are inferior to standard diagnostic tests (discussed in earlier sections of this chapter).

What all this suggests is that in testing nonnested hypotheses, one should use the J-test with higher significance levels and supplement it with the F-test on the comprehenesive model and standard diagnostic tests.

SUMMARY 5 ] 9

a regression of , the estimated residual on y] and higher powers of S, is not a valid test. To test for ARCH effects, we estimate the regression

and test 7, = 0.

2. There are two problems with the least squares residuals. They are heteroskedastic and autocorrelated even if the true errors are independent and have a common variance of u- (see Section 12.3). To solve these problems, some alternative residuals have been suggested (see Section 12.4). Two such residuals, which are uncorrelated and have a common variance, are the BLUS residuals and recursive residuals. The BLUS residuals are more difficult to compute than the recursive residuals; moreover, in tests of heteroskedasticity and autocorrelation, their performance is not as good as that of least squares residuals. Hence we consider only recursive residuals. Two other types of residuals are the predicted residuals and studentized residuals. Both of these are, like the least squares residuals, heteroskedastic and autocorrelated. However, some statisticians have found the predicted residuals useful in choosing between regression models and the studentized residuals in the detection of outUers. The predicted residuals, studentized residuals, and recursive residuals can all be computed using the dummy variable method.

3. The usual approach to outliers based on least squares residuals involves looking at the OLS residuals, deleting the observations with large residuals, and reestimating the equation. During recent years an alternative criterion has been suggested. This is based on a criterion called DFFITS which measures the change in the fitted value of that results from dropping a particular observation. Further, we do not drop outliers. We merely see that their influence is minimized. This is what is known as bounded influence estimation (Section 12.5). The SAS regression program computes DFFITS.

4. According to Leamer, the process of model building involves several types of specification searches. These can be classified under the headings: hypothesis-testing search, interpretive search, simplification search, proxy variable search, data selection search, and post-data model construction.

5. Hendry argues that most of empirical econometric work starts with very simplified models and that not enough diagnostic tests are appUed to check whether something is wrong with the maintained model. His suggested strategy is to start with a very general model and then progressively simplify it by applying some data-based simpUfication tests. (The arguments for this are discussed in Section 12.6.)

6. A special problem in model selection that we often encounter is that of selection of regressors. Several criteria have been suggested in the literature: the maximum R criterion and the predictive criteria PC, Cp, and Sp. These criteria are summarized in Section 12.7. There is, however, some difference in these criteria. In the case of the maximum R~ criterion what we are trying to do is pick the "true" model assuming that one of the models considered is the "true" one. In the case of prediction criteria, we are interested in parsimony

and we would like to omit some of the regressors if this improves the MSE of prediction. For this problem the question is whether or not we need to assume the existence of a true model. For the PC and Cp criteria we need to assume the existence of a true model. For the S,, criterion we do not. For this reason the Sp criterion appears to be the most preferable among these criteria.

7. In Section J2.8 we discuss the critical F-ratios implied by the different criteria for selection of regressors. We show that they stand in the relation

FiR) < I < F(PC) < F(Cp) < 2 < F(Sp)

We also present the F-ratios implied by Leamers posterior odds analysis. This method implies that the critical F-ratios used should be higher for higher sample sizes. For most sample sizes frequently encountered, F-ratios are > 2, which is another argument against R criterion and the PC and Cp criteria.

8. Cross-validation methods are often used to choose between different models. These methods depend on splitting the data into two parts: one for estimation and the other for prediction. The model that minimizes the sum-of-squared prediction errors is chosen as the best model. A better procedure than splitting the data into two parts is to leave out one observation at a time for prediction, derive the studentized residuals (the SAS regression program gives these), and use the sum of squares of the studentized residuals as a criterion of model choice.

9. A general test for specification errors that is easy to use is the Hausman test. Let be the hypothesis of no specification error and , the alternative hypothesis that there is a specification error (of a particular type). We compute two estimators Po and p, of the parameter p. Po is consistent and efficient under H„ but is not consistent under ,. p, is consistent under both H„ and H, but is not efficient under - If J = p, - po, then Vid) = V(p,) - V(po). Hausmans test depends on comparing d with V(d). If we are interested in testing for errors in variables or for exogeneity, Hausmans test reduces to a test of omitted variables.

10. Another general specification test (applicable in time-series models) is the PSW differencing test. In this test we compare the estimators from levels and first difference equations. The idea is that if the model is correctly specified, the estimates we get should not be widely apart. This test can also be viewed as an omitted-variable test (Section 12.11).

11. Very often, the comparison of two economic theories implies the testing of two nonnested hypotheses. The earliest test for this is the Cox test, but the alternative (and asymptotically equivalent) /-test is easier to apply. It can also be viewed as an omitted-variable test. There are, however, some conceptual problems with nonnested tests. The two hypotheses and , specify two conditional distributions with different conditioning variables [e.g.,/(y x) and g(y I z)] and it does not really make sense to compare them. A proper way is to derive the conditional distributions /(y x) and g(y \ z) under both and , and compare them, that is, we have to have the same conditioning variables under both the hypotheses. This is the idea behind the encompassing test. What