3.3 The Method of Moments

The assumptions we have made about the error term imply that

£(m) = 0 and cov(x, ) = 0

In the method of moments, we replace these conditions by their sample counterparts.

Let a and p be the estimators for a and p, respectively. The sample counterpart of m, is the estimated error u, (which is also called the residual), defined as

u, = y, - a - px,

The two equations to determine a and p are obtained by replacing population assumptions by their sample counterparts.

Population Assumption Sample Counterpart

In these and the following equations, 1 denotes 1"=i. Thus we get the two equations

2 «, = 0 or liy, - ct - px,) = 0 2 x,w, = 0 or 2 x,(y, - a - px,) = 0

These equations can be written as (noting that 1 a = no)

2 y, = + P Ix, 1 xy, = al X, + 1 x}

Solving these two equations, we get a and p. These equations are also called "normal equations." In Section 3.4, we will show further simplifications of the equations as well as methods of making statistical inferences. First, we consider an illustrative example.

Illustrative Example

Consider the data on advertising expenditures (x) and sales revenue (y) for an athletic sportswear store for 5 months. The observations are as follows:

model we are considering, all three methods give identical estimates. When it comes to generalizations, the methods give different estimates.

S.3 THE METHOD OF MOMENTS

Sales Revenue, Month (thousands of dollars)

I 2 3 4

3 4 2

1 2 3 4

To get a and p, we need to comfHite x, x, xy, and X - We have

The normal equations are (since = 5)

5a + 15p = 23 15a + 55p - 81

These give a = 1.0, = 1-2. Thus the sample regression equation is

= LO + 1.2;r

The sample observations and the estimated regression line are shown in Figure 3.2.

The intercept 1.0 gives the value of when = 0. This says that if advertising expenditures are zero, sales revenue will be $1( 0. The slope coefficient is 1.2. This says that if x is changed by an amount Av, the change in is = 1.2Ajc. For example, tf advertising expenditures are increased by 1 unit ($100), sales revenue increases by 1.2 units ($1200) on the average. Clearly, there is no certainty in this prediction. The estimates 1.0 and 1.2 have some uncertainty attached. We discuss this in Section 3.5, on statistical inference in the linear regression model. For the present, what we have shown is how to obtain estimates of the parameters a and p. One other thing to note is that it is not appropriate to obtain predictions too far from the range of observations. Otherwise, the ov,ner of the store might conclude that by raising the advertising expenditures b\ $1( .( , she can increase sales revenue by $1,200,000.

We have also shown the estimated errors or the residuals, which are given by

, = y, - 1.0 - 1.2x,

These are the errors we would make if we tried to predict the value of on the basis of the estimated regression equation and the values of x. Note that we

y=l.O+l 2x

ilgore 3.2. Sample observations and estimated regression line.

are not trying to predict any future values of y. We are considering only within sample prediction errors. We cannot get , for jc, = 0 or x, = 6 because we do not have the corresponding values of y,. Out of sample prediction is considered in Section 3.6.

Note that 1 , = virtue of the first condition we imposed. The sum of squares of the residuals is

2 ? = (0.8)2 + (0 6)2 + (-2.6)2 + (0.2)2 + (1 0)2 8 8

The method of least squares described in the next section is based on the principle of choosing a and p so that 1 ] is minimum. That is, the sum of squares of prediction errors is the minimum. With this method we get the same estimates of a and p as we have obtained here because we get the same normal equations.

3.4 Method of Least Squares

The method of least squares is the automobile of modern statistical analysis; despite its limitations, occasional accidents, and incidental pollution, it and its numerous variations, extensions and related conveyances carry the bulk of statistical analysis, and are known and valued by all.

Stephen M. Stigler

S. M. Stigler, "Gauss and the Invention of Least Squares, No. 3. 1981, pp. 465-474.

The Annals of Statistics, Vol. 9.