Course contentsShow
← Machine Learning and Deep Learning
Lesson 1071 of 3,53824. The Transformer ArchitectureFree lesson

Computing Attention Scores in Parallel

How scaled dot-product attention is applied independently across all heads using batched matrix operations.

Computing Attention Scores in Parallel

What you'll learn: How multi-head attention computes all attention scores simultaneously using efficient batched matrix operations.

The Core Idea

In multi-head attention, we don't compute each head's attention scores one at a time — that would be slow! Instead, we leverage batched matrix operations to compute attention for all heads in parallel using a single set of efficient matrix multiplications.

Think of it like a restaurant with multiple chefs (heads) all preparing different dishes simultaneously, rather than one chef cooking each dish sequentially. The kitchen (GPU) has the capacity to handle all chefs at once.

How Parallelization Works

After creating linear projections for queries, keys, and values (as we learned), we reshape these tensors to include an explicit "head" dimension:

  • Before reshaping: (batch_size, seq_len, d_model)
  • After reshaping: (batch_size, num_heads, seq_len, d_k)

Where d_k = d_model / num_heads is the dimension per head.

Now, when we compute the scaled dot-product attention score matrix QK^T / √d_k, the operation happens across all heads simultaneously. The batch and head dimensions are treated as independent parallel operations.

The softmax normalization and weighted sum of values (attention_weights @ V) also execute in parallel across all heads at once — no loops required!

Why This Matters

This parallelization is what makes multi-head attention practical. Without it, computing 8 or 16 attention heads would be 8–16× slower. GPUs excel at these batched matrix operations, making the overhead of multiple heads minimal compared to the representational benefits.

Key Takeaway: Multi-head attention computes all heads in parallel by reshaping tensors to include a head dimension, then using batched matrix operations — enabling efficient simultaneous computation across all attention heads without sequential loops.