12.8 Implied f-Ratios for the Various Criteria

We saw earlier (Section 4.10) that maximizing implies deletion of variables with an F-ratio < 1. We can derive similar conditions for the Cp, Sp, and PC criteria discussed in Section 12.7.

Consider the restricted model with (< k) variables, so that ki variables are deleted (k = k, + k,). Define X = RRSS/URSS, where RRSS and URSS are the residual sum of squares from the restricted and unrestricted models, respectively.

Then, as derived in Section 4.8, the F-ratio for testing the hypothesis that the coefficients of the ki excluded variables are zero is

(RRSS - VRSSVki (K - m - k)

URSS/(n - A:) ki

If the restricted model is chosen, then for the R criterion, this means that

RRSS , URSS . n - ki

<-- or X<--f (12.5)

n - ki n - n -

Substituting in (12.4) and noting that = k + ki, we get the condition F < 1. For Amemiyas PC criterion we have

"This criterion is derived in H. Akaike, "Information Theory and an Extension of the Maximum Likelihood Principle," in B. N. Petrov and F. Csaki (eds.), 2nd International Symposium on Information Theory (Budapest: Akademiai Kiado, 1973), and H. Akaike, "On Entropy Maximization Principle," in P. R. Krishniah (ed.). Applications of Statistics (Amsterdam: North-Holland, 1977).

Akaike s Information Criterion

Akaikes information criterion" (AIC) is a more general criterion that can be applied to any model that can be estimated by the method of maximum likelihood. It suggests minimizing

-2 logL 2k n n

where is the number of parameters in L. For the regression models this criterion implies minimizing [RSS exp (2fc/n)], which is the criterion listed in Table 12.3,

We have included AIC in our list because it is the one that is commonly used (at least in nonlinear models) and like the other criteria listed in Table 12.3, it involves RSSj and / , only. Thus it can be computed using the standard regression programs.

12.8 IMPLIED F-RATIOS FOR THE VARIOUS CRITERIA 50 J

+ K, « -

Substituing this in (12.4) and simplifying, we get

F<- (12.7)

For the Cp criterion we have

+ A:,

\ < 1 + (12.8)

n ~

and thus F <2. For the criterion

and thus

For the AIC criterion

( - k){n -k-l)

F<2 + -- (12.10)

« - - 1

X< (12.11)

( ;,/ )

Substituting this in (12.4), we do not get any easy expression as in the other cases, but we can tabulate the values for different values of k/n and k,/n. For n large relative to k, we have

exp() = 1 +

and hence

X< (12.12)

n + k,

Substituting this in (12.4), we get

F< which is <1 (12.13)

n + k.

The F-values for deletion of the variables implied by the different criteria are shown in Table 12.4. They stand in the following relation:

F(AIC) < F(R) < 1 < F(PC) < F(Cp) < 2 < F(5p) (12.14)

If n is large relative to k, note that PC, and all imply a cutoff value of F = 2.

Maximum ¨} F < 1

Mallows Cp F < 2

Amemiyas PC F <

Hockings Sp F<2 +

akaikes AIC (for n large relative to k)

n + ky

kj + 1

n - k- \ n -

Akaikes AIC (for n F<--

n + k.

The choice between the restricted and unrestricted models is based on an F-test for the restrictions. The use of the F-ratios in Table 12.4 implies two things:

1. The significance level that we use for the F-tests, for testing the restrictions, is not the conventional 5% level of significance. In fact, we use a much higher level of significance.

2. The significance level used, in general, decreases with the sample size. This is particularly true for the cases where we use a constant F-ratio like 1 or 2, irrespective of the sample size. It is also true of Amemiyas PC criterion, where the F-ratio is less than 2 for small samples and approaches 2 as « -> CO. However, with Hockings criterion, the cutoff point of F declines toward 2 as the sample size increases. Thus in this case we cannot say unambiguously that the significance level decreases with sample size.

The common procedure of using a constant level of significance in hypothesis testing irrespective of sample size has been often criticized on grounds that with a sufficiently large sample every null hypothesis can be rejected. The procedure increasingly distorts the interpretation of data against a null hypothesis as the sample size grows. The significance level should, consequently, be a decreasing function of sample size." As we have seen, the criteria for choice of regressors imply a decreasing significance level as the sample size increases. However, some Bayesian arguments have led to more substantial changes in the significance levels with sample size than are implied by the criteria in Table 12.4. We will discuss one such criterion, that of Leamer,"" but before that we will explain briefly what the Bayesian approach is.

"This argument was made in D. V. Lindley, "A Statistical Paradox," Biometrika, Vol. 44, 1957, pp. 187-192, and since then in many Bayesian papers on model selection. "Learner, Specification Searches, pp. 114-116.

Table 12.4 F-Values for Different Criteria

Choose the Restricted Model if the F-Ratiofor Testing the Restrictions Criterion Gives: