Multi-Head KV Cache Organization
You'll learn how KV tensors are structured across multiple attention heads and accessed efficiently during LLM inference.
The Multi-Head Structure Challenge
When generating tokens autoregressively, each attention head in your transformer needs its own key-value pairs from previous tokens. The question is: how do you organize these tensors in memory so they're fast to access and update?
Storage Layout Options
Think of your KV cache like a warehouse with multiple shelves (heads). You have two main organizational approaches:
Head-First Layout
Store all keys for head 1, then all keys for head 2, etc.:
[K_head1, K_head2, ..., K_headN, V_head1, V_head2, ..., V_headN]
Sequence-First Layout
Interleave heads within each sequence position, grouping by token:
[Token1_allheads_K, Token2_allheads_K, ..., Token1_allheads_V, ...]
Practical Considerations
Most implementations use head-first organization because:
- Each attention head computes independently during generation
- Memory can be pre-allocated as contiguous blocks per head
- Parallel access patterns align with GPU memory coalescing
- Grouped-Query Attention naturally shares KV across head groups
The typical shape: [batch_size, num_heads, seq_len, head_dim]
During generation, you append new KV pairs by simply extending the seq_len dimension for each head, avoiding costly reorganization.
Key Takeaway: Multi-head KV caches are typically organized head-first in memory (one contiguous block per attention head) to enable efficient parallel access and incremental updates during autoregressive generation.