ALiBi: Attention with Linear Biases
What you'll learn: An elegant alternative to positional embeddings that modifies attention scores directly based on how far apart tokens are.
The Core Problem
Traditional Transformers use positional embeddings (like RoPE, which you just learned) to help the model understand token order. ALiBi (Attention with Linear Biases) takes a radically different approach: instead of adding position information to token representations, it directly penalizes attention scores based on distance.
How ALiBi Works
Think of attention like asking "how relevant is token j to token i?" ALiBi adds a simple penalty: the farther away token j is from token i, the more we subtract from their attention score.
The formula is beautifully simple:
attention_score = (query · key) - m × distance
Where m is a head-specific slope (each attention head gets a different slope value), and distance is simply |i - j|.
Why Linear Biases?
Imagine you're reading a sentence. Words closer together are usually more relevant to each other. ALiBi bakes this intuition directly into the attention mechanism. A token 3 positions away gets penalized 3m, while a token 10 positions away gets penalized 10m—a straightforward linear relationship.
Key Advantages
- No learned parameters: The biases are fixed based on distance
- Extrapolation: Models trained on short sequences can handle much longer sequences at inference
- Simplicity: No complex sinusoidal functions or rotations needed
- Efficiency: Just a subtraction operation before softmax
Key Takeaway: ALiBi replaces positional embeddings with distance-based penalties on attention scores, enabling better length extrapolation through a simple linear bias that makes nearby tokens naturally attend to each other more strongly.