Laplace Smoothing for Zero Probabilities
What You'll Learn
You'll learn how to prevent Naive Bayes classifiers from crashing when they encounter feature values they've never seen during training.
The Zero Probability Problem
When using Naive Bayes classifiers, you multiply probabilities of individual features together. But what happens if your training data never showed a particular word in spam emails, and then you encounter that word when classifying a new email?
The probability becomes zero. And when you multiply by zero, your entire posterior probability becomes zero—even if all other features strongly suggest "spam." One unseen feature value destroys your prediction.
How Laplace Smoothing Works
Laplace smoothing (also called additive smoothing) adds a small "pseudocount" to every possible feature-class combination, even those you've never observed.
Instead of counting:
- "Word 'lottery' appeared 10 times in spam emails"
You pretend:
- "Word 'lottery' appeared 10 + α times in spam emails"
The constant α (alpha) is typically 1, giving every feature a minimum count of 1. This ensures no probability is ever exactly zero.
Analogy: Imagine a coin you've never flipped. Without smoothing, you'd say "probability of heads = 0/0 = undefined." With Laplace smoothing (α=1), you assume you've already seen 1 head and 1 tail, so P(heads) = 1/2—a reasonable starting guess.
Why It Matters
Laplace smoothing makes Naive Bayes robust to unseen data. It prevents mathematical errors while keeping probabilities small but non-zero for rare events. The smoothing parameter α controls how much "faith" you place in unseen combinations versus observed data.
Key Takeaway: Laplace smoothing adds pseudocounts (typically α=1) to all feature-class combinations, preventing zero probabilities that would otherwise break Naive Bayes predictions when encountering unseen features.