Course contentsShow
← Machine Learning and Deep Learning
Lesson 1612 of 3,53835. Modern Large Language Models: ArchitectureFree lesson

ALiBi: Attention with Linear Biases

Learn an alternative positional encoding that biases attention scores based on token distance.

ALiBi: Attention with Linear Biases

What you'll learn: An elegant alternative to positional embeddings that modifies attention scores directly based on how far apart tokens are.

The Core Problem

Traditional Transformers use positional embeddings (like RoPE, which you just learned) to help the model understand token order. ALiBi (Attention with Linear Biases) takes a radically different approach: instead of adding position information to token representations, it directly penalizes attention scores based on distance.

How ALiBi Works

Think of attention like asking "how relevant is token j to token i?" ALiBi adds a simple penalty: the farther away token j is from token i, the more we subtract from their attention score.

The formula is beautifully simple:

attention_score = (query · key) - m × distance

Where m is a head-specific slope (each attention head gets a different slope value), and distance is simply |i - j|.

Why Linear Biases?

Imagine you're reading a sentence. Words closer together are usually more relevant to each other. ALiBi bakes this intuition directly into the attention mechanism. A token 3 positions away gets penalized 3m, while a token 10 positions away gets penalized 10m—a straightforward linear relationship.

Key Advantages

  • No learned parameters: The biases are fixed based on distance
  • Extrapolation: Models trained on short sequences can handle much longer sequences at inference
  • Simplicity: No complex sinusoidal functions or rotations needed
  • Efficiency: Just a subtraction operation before softmax

Key Takeaway: ALiBi replaces positional embeddings with distance-based penalties on attention scores, enabling better length extrapolation through a simple linear bias that makes nearby tokens naturally attend to each other more strongly.