Course contentsShow
← Machine Learning and Deep Learning
Lesson 1656 of 3,53835. Modern Large Language Models: ArchitectureFree lesson

ALiBi: Attention with Linear Biases

Adding linear biases to attention scores as an alternative to positional embeddings for arbitrary-length sequences.

ALiBi: Attention with Linear Biases

You'll learn how ALiBi enables transformers to handle sequences of any length by adding simple biases to attention scores instead of using positional embeddings.

The Core Idea

Traditional positional embeddings (like the ones you learned early in transformers) are fixed to a maximum sequence length. If you train on sequences up to 512 tokens, the model struggles with 1000 tokens at inference time.

ALiBi takes a different approach: instead of adding positional information to token embeddings, it adds a linear penalty directly to the attention scores based on the distance between tokens. The further apart two tokens are, the more negative bias is added to their attention score, making distant tokens slightly less likely to attend strongly to each other.

How It Works

Think of ALiBi like a volume knob that gradually turns down based on distance. When token 10 looks at token 5, there's a small penalty. When it looks at token 2, there's a larger penalty. The bias is simply:

bias = -m × |position_query - position_key|

where m is a head-specific slope (different attention heads use different slopes, giving them different "distance sensitivities").

This bias gets added to the raw attention scores before the softmax, subtly discouraging very long-distance attention without completely blocking it.

Why It Matters

Because ALiBi uses relative distances rather than absolute positions, models can extrapolate to much longer sequences than they saw during training. A model trained on 1024 tokens can often handle 4000+ tokens at inference with minimal performance loss—something RoPE and learned embeddings struggle with.

Key Takeaway: ALiBi replaces positional embeddings with distance-based biases added to attention scores, enabling transformers to generalize to arbitrary sequence lengths with improved extrapolation.