Bayes’ Theorem

 

Bayes’ Theorem – Statement and Formula

Bayes’ Theorem is a way to update our beliefs based on new evidence.

Given events $A$ and $B$, where $P(B) > 0$:

$$\boxed{P(A \mid B) = \frac{P(B \mid A) \cdot P(A)}{P(B)}}$$

More generally , Let $E_1, E_2, \ldots, E_n$ be mutually exclusive and exhaustive events (i.e., they partition the sample space), and let $A$ be any event such that $P(A) > 0$. Then the posterior probability of event $E_i$ given $A$ is:

$$\boxed{ P(E_i \mid A) = \frac{P(E_i) \cdot P(A \mid E_i)}{\sum_{j=1}^n P(E_j) \cdot P(A \mid E_j)} }$$

---

📌 Terminology

Term	Meaning
$P(E_i)$	Prior probability – belief before observing A
$P(A \mid E_i)$	Likelihood – how likely A is given $E_i$
$P(E_i\mid A)$	Posterior probability – updated belief after observing A
Denominator	Total probability(evidence) of A using the law of total probability

✅ Example: Medical Test

Imagine a disease affects 1 in 1000 people. A test for the disease is 99% accurate (i.e., 99% true positive and 99% true negative).

• Let:

  • $D$: person has disease

  • $T$: test is positive

Known:

• $P(D)=0.001$

• $P(T\mid D)=\mathrm{0.99\; (True\; positive)}$

• $P(T \mid \text{no } D) = 0.01$ (False positive)

• $P(\text{no }D)=0.999$

What is the probability the person has the disease given the test is positive?

$$P(D \mid T) = \frac{P(D) \cdot P(T \mid D)}{P(D) \cdot P(T \mid D) + P(\text{no } D) \cdot P(T \mid \text{no } D)}$$ $$P(D \mid T) = \frac{0.001 \cdot 0.99}{0.001 \cdot 0.99 + 0.999 \cdot 0.01} = \frac{0.00099}{0.00099 + 0.00999} \approx 0.09$$

👉 So, even if the test is positive, the person only has about a 9% chance of actually having the disease!

---

📊 Visual Diagram of Bayes’ Theorem

Here’s a graphical interpretation 

🌳 Tree Diagram:

                      Start
                        |
              ---------------------
              |                   |
           Has D             No D
         (0.001)            (0.999)
              |                   |
         Positive             Positive
         (0.99)               (0.01)

      P(D ∩ T) = 0.00099     P(No D ∩ T) = 0.00999

      Total P(T) = 0.00099 + 0.00999 = 0.01098

      Posterior:
      P(D | T) = 0.00099 / 0.01098 ≈ 0.09

---

Example 2: Coin Selection Problem

You have:

• Bag 1: 2 gold coins

• Bag 2: 1 gold + 1 silver coin

• You pick a random bag, then a random coin from it, and it’s gold.
  

  What is the probability it came from Bag 1?

Let:

• $B_1$: chose Bag 1

• $B_2$: chose Bag 2

• $G$: selected gold coin

$$P(B_1 \mid G) = \frac{P(B_1) \cdot P(G \mid B_1)}{P(B_1) \cdot P(G \mid B_1) + P(B_2) \cdot P(G \mid B_2)}$$ $$= \frac{0.5 \cdot 1}{0.5 \cdot 1 + 0.5 \cdot 0.5} = \frac{0.5}{0.5 + 0.25} = \frac{0.5}{0.75} = \frac{2}{3}$$

👉 So the probability the gold coin came from Bag 1 is 2/3. 

Example 3: Spam Filter (Machine Learning)

• 80% of emails are not spam

• "Win money" appears in 10% of spam emails

• Appears in 1% of non-spam emails

What is the probability an email is spam given it contains "win money"?

Let:

• $S$: spam

• $W$: contains "win money"

$$P(S \mid W) = \frac{P(S) \cdot P(W \mid S)}{P(S) \cdot P(W \mid S) + P(\text{Not S}) \cdot P(W \mid \text{Not S})}$$ $$P(S \mid W) = \frac{0.2 \cdot 0.10}{0.2 \cdot 0.10 + 0.8 \cdot 0.01} = \frac{0.02}{0.02 + 0.008} = \frac{0.02}{0.028} ≈ 0.714$$

👉 The email is about 71% likely to be spam.

Example 4: Quality Control

A factory has 3 machines:

• M1 produces 30% of items; 2% are defective.

• M2 produces 50% of items; 1% are defective.

• M3 produces 20% of items; 3% are defective.

If an item is found defective, what is the probability it was produced by M3?

✅ Solution 4:

Let:

• $D$: defective

• $M_1, M_2, M_3$: machines

$$P(M_3 \mid D) = \frac{P(M_3) \cdot P(D \mid M_3)}{\sum_{i=1}^3 P(M_i) \cdot P(D \mid M_i)}$$

Numerator:

$$P(M_3) \cdot P(D \mid M_3) = 0.2 \cdot 0.03 = 0.006$$

Denominator:

$$= 0.3 \cdot 0.02 + 0.5 \cdot 0.01 + 0.2 \cdot 0.03 = 0.006 + 0.005 + 0.006 = 0.017$$
$$P(M_3 \mid D) = \frac{0.006}{0.017} ≈ 0.3529$$

👉 Answer: ~35.3% chance the defective item came from M3

Question 5: Student Performance

Suppose:

• 60% of students are from urban schools, 40% from rural.

• 80% of urban students pass the entrance test.

• 50% of rural students pass.

If a student is selected who passed, what is the probability they are from an urban school?

✅ Solution 5:

Let:

• $U$: urban

• $R$: rural

• $P$: passed

$$P(U \mid P) = \frac{P(U) \cdot P(P \mid U)}{P(U) \cdot P(P \mid U) + P(R) \cdot P(P \mid R)}$$ $$= \frac{0.6 \cdot 0.8}{0.6 \cdot 0.8 + 0.4 \cdot 0.5} = \frac{0.48}{0.48 + 0.2} = \frac{0.48}{0.68} ≈ 0.7059$$

👉 Answer: ~70.6% chance the student is from an urban school.

Q6. Classifier Error (AI/ML context)

An email classifier identifies 90% of spam correctly, but misclassifies 10% of non-spam as spam. If 30% of all emails are spam, what is the probability an email is actually spam given that it was classified as spam?

Solution:

Let:

• $S$: spam → $P(S)=0.3$

• $\overline{S}$: not spam → $P(\overline{S}) = 0.7$
  

• $C$: classified as spam

$$P(S \mid C) = \frac{P(S) \cdot P(C \mid S)}{P(S) \cdot P(C \mid S) + P(\overline{S}) \cdot P(C \mid \overline{S})}$$ $$= \frac{0.3 \cdot 0.9}{0.3 \cdot 0.9 + 0.7 \cdot 0.1} = \frac{0.27}{0.27 + 0.07} = \boxed{0.794}$$

Summary

• Bayes’ Theorem allows us to update probabilities when new information (like test results) is available.

• It’s essential in fields like:

  • Machine learning

  • Medical diagnostics

  • Spam filtering

  • Decision-making under uncertainty