Course contentsShow
← Machine Learning and Deep Learning
Lesson 329 of 3,5387. Instance-Based and Probabilistic ClassifiersFree lesson

Bayes' Theorem and Posterior Probability

Derive Bayes' theorem and understand how it computes posterior probabilities from prior and likelihood.

Bayes' Theorem and Posterior Probability

What you'll learn: How Bayes' theorem lets us update our beliefs about class membership using evidence from data.

The Core Idea

Bayes' theorem is a mathematical formula that tells us how to reverse conditional probabilities. In machine learning, we often want to know: "Given these features, what's the probability this instance belongs to class C?" — but what we can measure from data is: "Given class C, how often do we see these features?"

Bayes' theorem bridges this gap.

The Formula

Bayes' Theorem states:

P(Class | Features) = [P(Features | Class) × P(Class)] / P(Features)

Breaking this down:

  • Posterior Probability P(Class | Features): What we want — the probability of a class given the observed features
  • Likelihood P(Features | Class): How likely these features are if the instance belongs to this class (estimable from training data)
  • Prior Probability P(Class): Our initial belief about the class frequency (before seeing features)
  • Evidence P(Features): The overall probability of observing these features (a normalizing constant)

An Analogy

Think of medical diagnosis. The prior is how common a disease is in the population. The likelihood is how often patients with that disease show certain symptoms. The posterior is the updated probability that you have the disease after seeing your symptoms.

Why It Matters

This theorem forms the foundation of Naive Bayes classifiers (coming soon!). Instead of learning decision boundaries like logistic regression or SVMs, we model the probability distributions of features for each class, then use Bayes' theorem to "invert" them into classification probabilities.

Key Takeaway: Bayes' theorem converts likelihood and prior into posterior probability, letting us predict class membership by reasoning probabilistically from observed features to hidden labels.