Applying Softmax to Get Attention Weights
What you'll learn: How softmax transforms raw attention scores into a probability distribution that tells the model exactly how much to focus on each position.
From Scores to Probabilities
You've computed attention scores—numbers that measure how relevant each key is to your query. But these raw scores can be any value: large, small, negative, or positive. To make them useful, you need to convert them into attention weights that sum to 1.0 and represent a clear distribution of focus.
This is where softmax comes in.
The Softmax Transformation
Softmax takes your vector of scaled attention scores and transforms them into a probability distribution:
attention_weights = softmax(scores)
Each weight will be between 0 and 1, and all weights sum to exactly 1.0. This means you're deciding "I'll pay 40% attention to position 1, 35% to position 2, 15% to position 3..." and so on.
Why This Matters
The softmax function does two critical things:
- Exponentiation: Amplifies differences between scores. Slightly higher scores become much more prominent after softmax.
- Normalization: Forces all weights to sum to 1, creating a valid probability distribution.
Think of it like adjusting the volume on different audio channels—softmax ensures your total attention "budget" is always 100%, distributed across all inputs based on relevance.
After softmax, you have clean attention weights ready to compute your weighted sum (context vector) using the values.
Key Takeaway: Softmax converts raw attention scores into normalized probability weights that sum to 1, giving you a proper distribution of attention across all input positions.