Bayes' Theorem

Revising probabilities when new information is obtained is an important aspect of probability analysis.

Theorem: (Bayes' Theorem)

\[ \Pr \left[ A_i | B \right] = \frac{\Pr [A_i ] \,\Pr [B | A_i ]}{\Pr [A_1 ]\,\Pr [B | A_1 ] + \Pr [A_2 ]\,\Pr [B | A_2 ] + \cdots + \Pr [A_n ]\,\Pr [B | A_n ] } . \]
The theorem derives its name from the English statistician Reverend Thomas Bayes (1702--1761), whose 1763 paper (published posthumously by Richard Price) contained the result. The first explicit statement of Bayes' theorem was made by P. Laplace in 1812.

For two events A and B, the above statement is known as Bayes' little theorem:

\[ \Pr \left[ A | B \right] = \frac{\Pr [A ]}{\Pr [B]} \,\Pr [B | A] = \frac{\Pr [A] \, \Pr [B | A]}{\Pr [A]\,\Pr [B | A] + \Pr [A']\,\Pr [B | A']} , \]
where A' is the complementary event. From the inequality \( \left( 1 + x^{-1} \right)^{-1} < x \) for x > 0, it follows
\[ \Pr \left[ A | B \right] < \frac{\Pr [A]\, \Pr [B | A]}{\Pr [A'] \,\Pr[B | A']} . \]
Example: One of the possible practical applications of Bayes' theorem could be found in jurisprudence. Suppose that a test for using a particular drug is 97% sensitive and 98% specific. That is, the test will produce 97% true positive results for drug users and 98% true negative results for non-drug users. Suppose that 0.7% of people are users of the drug. What is the probability that a randomly selected individual with a positive test is a drug user?
\begin{align*} \Pr \left[ \mbox{User} | + \right] &= \frac{\Pr \left[ + | \mbox{User} \right] \Pr [\mbox{User}]}{\Pr [+]} \\ &= \frac{\Pr \left[ + | \mbox{User} \right] \Pr [\mbox{User}]}{\Pr \left[ + | \mbox{User} \right] \Pr [\mbox{User}] + \Pr \left[ + | \mbox{non-User} \right] \Pr [\mbox{non-User}] } \\ &= \frac{0.97 \times 0.007}{0.97 \times 0.007 + 0.03 \times 0.998} \approx \% . \end{align*} \]
Even if an individual tests positive, it is more likely that they do not use the drug than that they do. This is because the number of non-users is large compared to the number of users. The number of false positives outweighs the number of true positives. For example, if 1000 individuals are tested, there are expected to be 995 non-users and 5 users. From the 995 non-users, 0.01 × 995 ≃ 10 false positives are expected. From the 5 users, 0.99 × 5 ≈ 5 true positives are expected. Out of 15 positive results, only 5 are genuine.

The importance of specificity in this example can be seen by calculating that even if sensitivity is raised to 100% and specificity remains at 99% then the probability of the person being a drug user only rises from 33.2% to 33.4%, but if the sensitivity is held at 99% and the specificity is increased to 99.5% then the probability of the person being a drug user rises to about 49.9%.

Example: Suppose you are given the following information regarding a cancer testing scenario:

Put in a table, the probabilities look like this:

bayes table

Now suppose you get a positive test result. What are the chances you have cancer? 80%? 99%? 1%?

The table looks like this:

bayes table computed

And what was the question again? Oh yes: what’s the chance we really have cancer if we get a positive result. The chance of an event is the number of ways it could happen given all possible outcomes:

Probability = desired event / all possibilities

The chance of getting a real, positive result is .008. The chance of getting any type of positive result is the chance of a true positive plus the chance of a false positive (.008 + 0.09504 = .10304).

So, our chance of cancer is .008/.10304 = 0.0776, or about 7.8%.    ■

Example: Each bag in a large box contains 25 tulip bulbs. Three-fourths of the bags are of Type A containing bulbs for 5 red and 20 yellow tulips; one-fourth of the bags are of Type B contain bulbs for 15 red and 10 yellow tulips. A bag is selected at random and one bulb is planted. What is the probability that the bulb will produce a red tulip?

If A denotes the event that a Type A bag is selected, then, because 75% of the bags are of Type A, Pr(A) = 0.75. If B denotes the event that a Type B bag is selected, then, because 25% of the bags are of Type B, Pr(B) = 0.25. Let R denote the event that the selected bulb produces a red tulip. Then:

\begin{align*} \Pr \left[ R \right] &= \Pr \left[ \left( A \cap R \right) \cup \left( B \cap R \right) \right] \\ &= \Pr \left[ R | A \right] \cdot \Pr \left[ A \right] + \Pr \left[ R | B \right] \cdot \Pr \left[ B \right] \\ &= \frac{5}{25}\cdot \frac{3}{4} + \frac{15}{25} \cdot \frac{1}{4} = 0.3 . \end{align*}
What is the probability that the bulb will produce a yellow tulip?
\begin{align*} \Pr \left[ Y \right] &= \Pr \left[ \left( A \cap Y \right) \cup \left( B \cap Y \right) \right] \\ &= \Pr \left[ Y | A \right] \cdot \Pr \left[ A \right] + \Pr \left[ Y | B \right] \cdot \Pr \left[ B \right] \\ &= \frac{20}{25}\cdot \frac{3}{4} + \frac{10}{25} \cdot \frac{1}{4} = 0.7 . \end{align*}
If the tulip is red, what is the probability that a bag having 15 red and 10 yellow tulips was selected?
\[ \Pr \left[ B | R \right] = \frac{\Pr [R | B ] \cdot \Pr [B]}{\Pr [R]} = \dfrac{\frac{15}{25} \cdot \frac{1}{4}}{0.3} = 0.5 . \]