3.6 Analysis of Variance for the Simple Regression Model

Yet another item that is often presented in connection with the simple linear regression model is the analysis of variance. This is the breakdown of the total sum of squares TSS into the explained sum of squares ESS and the residual sum of squares RSS. The purpose of presenting the table is to test the significance of the explained sum of squares. In this case this amounts to testing the significance of p. Table 3.3 presents the breakdown of the total sum of squares.

Under the assumptions of the model, RSS/cr has a x-distribution with (n -2) degrees of freedom. ESS/a, on the other hand has a x-distribution with 1 degree of freedom only if the true p is equal to zero. Further, these two x distributions are independent. (All these results are proved in the appendix to this chapter.)

Thus under the assumption that = 0, we have the F-statistic (cr cancels) F = (ESS/l)/RSS/(/j - 2), which has an F-distribution with degrees of freedom 1 and (n - 2). This F-statistic can be used to test the hypothesis that p = 0.

For the data in Table 3.2 we are considering, the analysis of variance is presented in Table 3.4. The F-statistic is F = 15.75/1.83 = 8.6. Note that the /-statistic for testing the significance of p is p/SE(P) = 0.75/0.256 = 2.93 and the F-statistic obtained from the analysis of variance Table 3.4 is the square of the /-statistic. Thus in the case of a simple linear regression model, the analysis-of-variance table does not give any more information. We will see in Chapter 4 that in the case of multiple regression this is not the case. What the analysis-of-variance table provides there is a test of significance for all the regression parameters together. Note that

Table 3.3 Analysis of Variance for the Simple Regression Model

Table 3.4 Analysis of Variance for the Data in Table 3.2

t + (n - 2)

We can check in our illustrative example that = 8.6 and (n - 2) = 8. Hence

8.6 8.6 as we obtained earlier. The formula

t + in- 2)

gives the relationship between the /-ratio for testing the hypothesis p = 0 and the r. We will derive a similar formula in Chapter 4.

3.7 Prediction with the Simple Regression Model

The estimated regression equation = d + px is used for predicting the value of for given values of x and the estimated equation x = d + py is used for predicting the values of x for given values of y. We will illustrate the procedures with reference to the prediction of given x.

Let Xo be the given value of x. Then we predict the corresponding value of of by

= d + pxo (3.11)

The true value y is given by

= a + pxo + Mo

where Ug is the error term. Hence the prediction error is

= (a - a} + - p)xo - «0

ESS = 05„ = rS

and

RSS = S„ - pS„ = (1 - ,)S,,

Hence the F-statistic can also be written as

( -2

(I - 1}/{ - 2) 1 -

SSnce F = 1 we get

Since £(a - a) = 0, - ) = 0, and £(t<o) = 0 we have

£(50 - ) = 0

This equation shows that the predictor given by equation (3.11) is unbiased. Note that the predictor is unbiased in the sense that £( ) = £( ) since both So and are random variables. The variance of the prediction error is

a,

Vifo - ) = Via - a) + xVi - p) + 2xo cov(a , /1 x

) + V(mo)

+ cjf-2xocrf + a

1 + - + --

Thus the variance increases the farther away the value of Xq is from x, the mean of the observations on the basis of which a and p have been computed. This is illustrated in Figure 3.5, which shows the confidence bands for y.

If Xo lies within the range of the sample observations on x, we can call it within-sample prediction, and if Xq lies outside the range of the sample observations, we call the prediction out-of-sample prediction. As an illustration, consider a consumption function estimated on the basis of 12 observations. The equation is (this is just a hypothetical example)

Figure 3.5. Confidence bands for prediction of given jc. [Confidence bands shown are ± Z, where Z = tSE(y) and / is the f-value used.]