ALiBi: Attention with Linear Biases
You'll learn how ALiBi provides positional information by adding simple linear penalties directly to attention scores, without any positional embeddings.
The Core Idea
Instead of adding positional encodings to token embeddings (like sinusoidal or learned approaches), ALiBi modifies the attention mechanism itself. It adds a linearly increasing penalty to attention scores based on the distance between tokens.
Think of it like this: when a query token looks at key tokens, ALiBi makes distant keys slightly less attractive by subtracting a small penalty proportional to their distance. If a token at position 5 attends to position 2, the attention score gets penalized by 3 × m (where m is a head-specific slope). Nearby tokens get small penalties, far-away tokens get larger ones.
How It Works
The attention score between positions i and j becomes:
score(i, j) = q_i · k_j - m × |i - j|
The bias term -m × |i - j| is subtracted before applying softmax. Each attention head gets a different slope m, allowing some heads to focus more locally and others more globally.
Key differences from positional embeddings:
- No extra parameters to learn for position
- Works on sequences longer than those seen during training
- The bias is added during attention computation, not to embeddings
Why This Matters
ALiBi elegantly separates positional information from content representation. You don't modify embeddings at all—position is purely an attention phenomenon. This approach has shown strong extrapolation to longer sequences than the model was trained on.
Key Takeaway: ALiBi encodes position by adding distance-based linear penalties directly to attention scores, keeping embeddings position-agnostic and improving length generalization.