Course contentsShow
← Machine Learning and Deep Learning
Lesson 1054 of 3,53824. The Transformer ArchitectureFree lesson

Scaling the Dot Product: Why Divide by √d_k

Understand why raw dot products are scaled and how this prevents gradient issues.

Scaling the Dot Product: Why Divide by √d_k

You'll learn why attention mechanisms divide dot products by the square root of the key dimension to maintain stable gradients.

The Problem with Raw Dot Products

When you compute attention scores using dot products between queries and keys, larger dimension sizes (d_k) create a problem. As dimensions increase, the magnitude of dot products grows dramatically. For example, if you have 64-dimensional vectors with random values, their dot product will be much larger on average than dot products of 8-dimensional vectors.

Why This Matters

Large dot product values cause two critical issues:

  1. Extreme softmax outputs: When you feed very large numbers into softmax, it produces outputs close to 0 or 1, not smooth distributions
  2. Vanishing gradients: In these extreme regions, softmax gradients become tiny, making learning nearly impossible

Think of it like volume control: if someone whispers (small values) or shouts (large values), you can't hear the nuances. Scaling brings everything to a comfortable "conversation level."

The Solution: Divide by √d_k

The formula becomes:

Attention(Q, K, V) = softmax(QK^T / √d_k) V

Dividing by √d_k normalizes the variance of the dot products. Mathematically, if Q and K have unit variance, their dot product has variance approximately d_k. Dividing by √d_k brings the variance back to roughly 1, keeping values in a range where softmax behaves well and gradients flow properly.

For a 64-dimensional key vector, you'd divide by √64 = 8. This ensures attention scores remain numerically stable regardless of dimension size.

Key Takeaway: Scaling attention scores by √d_k prevents extreme values that would cause gradient problems, enabling stable training across different model dimensions.