Course contentsShow
← Machine Learning and Deep Learning
Lesson 1670 of 3,53836. LLM Inference OptimizationFree lesson

Multi-Head KV Cache Organization

How KV tensors are structured across attention heads and efficiently accessed during inference.

Multi-Head KV Cache Organization

You'll learn how KV tensors are structured across multiple attention heads and accessed efficiently during LLM inference.

The Multi-Head Structure Challenge

When generating tokens autoregressively, each attention head in your transformer needs its own key-value pairs from previous tokens. The question is: how do you organize these tensors in memory so they're fast to access and update?

Storage Layout Options

Think of your KV cache like a warehouse with multiple shelves (heads). You have two main organizational approaches:

Head-First Layout

Store all keys for head 1, then all keys for head 2, etc.:

[K_head1, K_head2, ..., K_headN, V_head1, V_head2, ..., V_headN]

Sequence-First Layout

Interleave heads within each sequence position, grouping by token:

[Token1_allheads_K, Token2_allheads_K, ..., Token1_allheads_V, ...]

Practical Considerations

Most implementations use head-first organization because:

  • Each attention head computes independently during generation
  • Memory can be pre-allocated as contiguous blocks per head
  • Parallel access patterns align with GPU memory coalescing
  • Grouped-Query Attention naturally shares KV across head groups

The typical shape: [batch_size, num_heads, seq_len, head_dim]

During generation, you append new KV pairs by simply extending the seq_len dimension for each head, avoiding costly reorganization.

Key Takeaway: Multi-head KV caches are typically organized head-first in memory (one contiguous block per attention head) to enable efficient parallel access and incremental updates during autoregressive generation.