Course contentsShow
← Machine Learning and Deep Learning
Lesson 661 of 3,53816. Activation Functions and Weight InitializationFree lesson

Softmax: Converting Logits to Probabilities

How softmax transforms raw scores into probability distributions for classification.

Softmax: Converting Logits to Probabilities

What you'll learn: How the softmax function transforms raw network outputs into a valid probability distribution for multi-class classification.

The Problem: Raw Scores Aren't Probabilities

When a neural network produces outputs for multi-class classification, the final layer gives you logits—raw, unbounded scores for each class. These numbers can be negative, exceed 1, or fail to sum to 1. You need a way to convert them into proper probabilities.

How Softmax Works

The softmax function transforms logits into probabilities through two steps:

  1. Exponentiation: Apply exp() to each logit, making all values positive
  2. Normalization: Divide each exponential by the sum of all exponentials

Formula:

softmax(z_i) = exp(z_i) / Σ exp(z_j)

Where z_i is the logit for class i.

An Analogy

Imagine five students with test scores: [2.0, 1.0, 0.1, 3.0, -1.0]. These are logits—they don't tell you the probability each student is the best. Softmax is like:

  • Amplifying differences: exp() makes larger scores much larger
  • Creating proportions: dividing by the total gives probabilities that sum to 1.0

The result might be [0.24, 0.09, 0.04, 0.61, 0.01]—the network is 61% confident in class 4.

Key Properties

  • Output range: All values between 0 and 1
  • Sum constraint: Outputs always sum to exactly 1
  • Monotonic: Higher logits → higher probabilities
  • Differentiable: Works with backpropagation (gradient flows through softmax)

Key Takeaway: Softmax converts raw network scores (logits) into a valid probability distribution by exponentiating and normalizing, enabling multi-class classification with interpretable confidence scores.