Also, a and 0 are unbiased estimators. So £(d) = a and = (3. We can show this by noting that

) = 2 ) = Sc,(a + px,) = p

(since cfx = 0 and 5)c,jc, = 1). Also,

£(a) = E{y) - = a + px - Px = a

Thus we get the following results:

1. d has a normal distribution with mean a and variance

2. p has a normal distribution with mean p and variance crVS„.

3. cov(a, p) = (j\-x/S,J.

(3) Distribution of RSS

Note that the derivation of the Mests in the regression model depend on the result that RSS/ct- has a x-distribution with (n - 2) d.f. and that this distribution is independent of the distribution of a and p. The exact proof of this is somewhat complicated.* However, some intuitive arguments can be given as follows. The estimated residual is

, = y, - y,

Note that y, = a + 0 , and y, and are uncorrelated because

= d , + p = 0

by virtue of the normal equations. This proves the independence between , and (a, p). Note that under the assumption of normality zero correlation implies independence.

The next thing to note is that uf/cr has a x-distribution with n d.f. and u; can be partitioned into two components as follows:

S ? = - " - P.) = liy. -y.+y.-a- px.r

= S ? + 2 K« - «) + ( -

(the cross-product term vanishes) = e. + a (say)

"It is easier to prove it with the use of matrix algebra for the multiple regression model. This proof can be found in any textbook of econometrics. See, for instance, Maddala, Econometrics, pp. 456-57.

1 x{x, - x)

log F = S

log(2W) - (y, - a - px,)

= c--loga-2

where = -(w/2)log(2jT) does not involve any parameters Q = liy> - - px,)2

We will maximize log L first with respect to a and p and then with respect to CT. Note that it is only the third term in log L that involves a and p and maximizing this is the same as minimizing Q (since this term has a negative sign). Thus the ML estimators of a and p are the same as the least squares estimators a. and p we considered earlier.

Substituting a for a and p for p we get the likelihood function which is now a function of CT only. We now have

log ) = const. - log CT -

2 * 2o2

= const. - n log CT

2( 2

Qxlu has a x-distribution with (« - 2) d.f. and 0/ has a x-distribution with 2 d.f. Presentation of a detailed proof involves more algebra than is intended here.

(4) The Method of Maximum Likelihood

The method of maximum likelihood (ML) is a very general method of estimation that is applicable in a large variety of problems. In the linear regression model, however, we get the same estimators as those obtained by the method of least squares. The model is

1 = a + Px, + u, ~ IN(0,

This implies that y, are independently and normally distributed with respective means a + px, and a common variance cr-. The joint density of the observations, therefore, is

fiyu . . . , ) = ) exp [- (y, - a - P-)

This function, when considered as a function of the parameters (a, p, cr) is called the likelihood function and is denoted by L{a, p, cr). The maximum likelihood (ML) method of estimation suggests that we choose as our estimates the values of the parameters that maximize this likelihood function. In many cases it is convenient to maximize the logarithm of the likelihood function rather than the likelihood function, and we get the same results since log L and L attain the maximum at the same point. For the linear regression model we have

(5) The Likelihood Ratio Test

The likelihood ratio (LR) test is a general large-sample test based on the ML method. Let be the set of parameters in the model and L(e) the likelihood function. In our example consists of the three parameters a, p, and cr. Hypotheses such asP = Oorp = l,CT = 0 impose restrictions on these parameters. What the likelihood ratio test says is that we first obtain the maximum of ) without any restrictions and with the restrictions imposed by the hypothesis to be tested. We then consider the ratio

max ) under the restrictions

max L(e) without the restrictions

X will necessarily be less than 1 since the restricted maximum will be less than the unrestricted maximum. If the restrictions are not valid, X. will be significantly less than 1. If they are valid, X will be close to 1. The LR test consists

where Q = {y, - a - x,f is nothing but the residual sum of squares RSS. Differentiating this with respect to a and equating the derivative to zero, we get

- - + = 0 This gives the ML estimator for as

Note that this is different from the unbiased estimator for which we have been usmg, namely, RSS/(n - 2). For large n the estimates obtained by the two methods will be very close. The ML method is a large-sample estimation method.

Substituting = Qifi in [og ) we get the maximum value of the log-likelihood as

max log jL = - log - - ,„ 2 n 2

= - - log e + - log - -

This expression is useful for deriving tests for a, , using the ML method. Since we do not change the sample size n, we will just write

max log L = const. - - log j2

maxL = const. {Q)~"" = const. (RSS)-