then

A + B: The die shows 1, 3, or 5 AB: The die shows 3

The addition rule of probability states that

( + = P(A) + P(B) - P(AB)

(We can show this by drawing a diagram known as the Venn diagram. This is left as an exercise.) If A and are mutually exclusive, they cannot occur jointly, so ( ) = 0. Thus for mutually exclusive events we have

( + ) = P(A) + ( )

If, in addition, A and are exhaustive, { ) + P{B) = 1.

We denote by A the complement of A. A represents the nonoccurrence of A. Since either A occurs or does not (i.e., A occurs), A and A are mutually exclusive and exhaustive. Hence P(A) + P{A) = 1 or P{A) = 1 - P(A). We can also write P(A) - P(AB) + P(AB) because A can occur jointly with or without B.

Conditional Probability and the Multiplication Rule

Sometimes we restrict our attention to a subset of all possible events. For instance, suppose that when we throw a die, the cases 1, 2, and 3 do not count. Thus the restricted set of events is that the die shows 4, 5, or 6. There are three possible outcomes. Consider the event A that the die shows a 6. The probability of A is now 5 since the total number of outcomes is 3, not 6. Conditional probability is defined as follows: The probability of an event A, given that another event has occurred, is denoted by P(A\B) and is defined by

In the case above, P(AB) = i P(B) = , and hence P(A\B) = i We shall now define independent events. A and are said to be independent if the probability of the occurrence of one does not depend on whether or not the other has occurred. Thus if A and are independent, the conditional and unconditional probabilities are the same, that is, P{A\B) = { ) and P{B\A) = { ). Since P(A\B) = P{AB)IP{B) and P(,A\B) = P(A), we get the multiplication rule, which says that

( ) = f (A) X P{B) if A and are independent

As an example, suppose that we throw a die two times:

A = event that the first throw shows a 6 = event that the second throw shows a 6

Clearly, A and are independent events. Hence Prob(we get a double 6) = P{AB) = P{A)-P{B)

2 2 PROBABILITY \ 5

Bayes Theorem

Bayes theorem is based on conditional probability. We have

Write the second equation as

P(AB) = P(B\A) P(A)

Then we get

P{BA)-P(A)

P{A\B) =

P(B)

This is known as Bayes theorem. It appeared in a text published in 1763 by Reverend Thomas Bayes, a part-time mathematician.

Let , and H2 denote two hypotheses and D denote the observed data. Let us substitute , and H2 in turn for A and substitute Dfor B. Then we get

Hence we get

( , ) PiDH,)

PiHlD) P{D\H2) 2)

The left-hand side of this equation is called posterior odds. The first term on the right-hand side is called the likelihood ratio, and the second term is called the prior odds. We shall make use of this in Chapter 12 for the problem of choice between two models. For the present let us consider the following example.

We have two urns: the first has 1 red ball and 4 white balls, and the second has 2 red balls and 2 white balls. An urn is chosen at random and a ball drawn. The ball is white. What is the probability that it came from the first urn? Let us define:

Hi. The first um was chosen - The second urn was chosen D: Data, that is, the ball is white

We have ( ,) = ) = i Also, (£) ,) = f and \ 2) = i Hence P{H\D)IP{HD) = f or ( ,1)) = A and { = A. The required probability is

Summation and Product Operations

In the examples we considered, we often had only two events. If we have n events, we have to use the summation operator 1 and the product operator .

+ z„, + z„2 + • • • + X„,

Again, where there is no confusion that / runs from 1 to n and j runs from 1 to m, we will just write Yet another notation is SiSj/jy which de-

notes summation over all values of i and j except those for which / = j. For instance, suppose that / goes from 1 to 3 and j goes from 1 to 2. Then

22 ij ~ + 12 + 21 + + 31 + -32

J

22 ~ 12 + 21 + 31 + 32

That is, we have to omit all the terms for which / = j. As yet another example, consider

(Z, + X2 + X,y = + XI + Xl + 2X,X2 + 2X2X3 + 2 , We can write this as

(2 x) = 22

= 2? + 22

Since we shall be using these in other contexts as well, we shall discuss them here.

The summation operator is defined as follows:

2x, = X, + X2 + x„

Some important properties of this operator are: * 2/"= i = nc, where is a constant.

S"=i (c + hx,) = nc + h 5] =1

Where there is no confusion we will just write 2, , or 2 . to denote 5] =1 that is, summation over all the Zs.

Sometimes we use the double-summation operator, which is defined as follows:

n m

2 2 -tf ~ -11 + + + X\„

+ 4- X22 + • • + X2m