Softmax: Converting Logits to Probabilities
What you'll learn: How the softmax function transforms raw network outputs into a valid probability distribution for multi-class classification.
The Problem: Raw Scores Aren't Probabilities
When a neural network produces outputs for multi-class classification, the final layer gives you logits—raw, unbounded scores for each class. These numbers can be negative, exceed 1, or fail to sum to 1. You need a way to convert them into proper probabilities.
How Softmax Works
The softmax function transforms logits into probabilities through two steps:
- Exponentiation: Apply
exp()to each logit, making all values positive - Normalization: Divide each exponential by the sum of all exponentials
Formula:
softmax(z_i) = exp(z_i) / Σ exp(z_j)
Where z_i is the logit for class i.
An Analogy
Imagine five students with test scores: [2.0, 1.0, 0.1, 3.0, -1.0]. These are logits—they don't tell you the probability each student is the best. Softmax is like:
- Amplifying differences:
exp()makes larger scores much larger - Creating proportions: dividing by the total gives probabilities that sum to 1.0
The result might be [0.24, 0.09, 0.04, 0.61, 0.01]—the network is 61% confident in class 4.
Key Properties
- Output range: All values between 0 and 1
- Sum constraint: Outputs always sum to exactly 1
- Monotonic: Higher logits → higher probabilities
- Differentiable: Works with backpropagation (gradient flows through softmax)
Key Takeaway: Softmax converts raw network scores (logits) into a valid probability distribution by exponentiating and normalizing, enabling multi-class classification with interpretable confidence scores.