ALiBi: Attention with Linear Biases
You'll learn how ALiBi enables transformers to handle sequences of any length by adding simple biases to attention scores instead of using positional embeddings.
The Core Idea
Traditional positional embeddings (like the ones you learned early in transformers) are fixed to a maximum sequence length. If you train on sequences up to 512 tokens, the model struggles with 1000 tokens at inference time.
ALiBi takes a different approach: instead of adding positional information to token embeddings, it adds a linear penalty directly to the attention scores based on the distance between tokens. The further apart two tokens are, the more negative bias is added to their attention score, making distant tokens slightly less likely to attend strongly to each other.
How It Works
Think of ALiBi like a volume knob that gradually turns down based on distance. When token 10 looks at token 5, there's a small penalty. When it looks at token 2, there's a larger penalty. The bias is simply:
bias = -m × |position_query - position_key|
where m is a head-specific slope (different attention heads use different slopes, giving them different "distance sensitivities").
This bias gets added to the raw attention scores before the softmax, subtly discouraging very long-distance attention without completely blocking it.
Why It Matters
Because ALiBi uses relative distances rather than absolute positions, models can extrapolate to much longer sequences than they saw during training. A model trained on 1024 tokens can often handle 4000+ tokens at inference with minimal performance loss—something RoPE and learned embeddings struggle with.
Key Takeaway: ALiBi replaces positional embeddings with distance-based biases added to attention scores, enabling transformers to generalize to arbitrary sequence lengths with improved extrapolation.