Course contentsShow
← Machine Learning and Deep Learning
Lesson 1610 of 3,53835. Modern Large Language Models: ArchitectureFree lesson

Multi-Query and Grouped-Query Attention

Learn memory-efficient attention variants that share key-value heads across query heads for faster inference.

Multi-Query and Grouped-Query Attention

What you'll learn: How modern language models reduce memory and speed up inference by sharing key-value heads across multiple query heads.

The Memory Bottleneck in Standard Attention

In standard multi-head attention (which you've learned through causal self-attention), each attention head maintains its own separate query (Q), key (K), and value (V) projections. During inference, the K and V matrices for every token must be cached to avoid recomputing them—this is the KV cache. As sequences grow longer and models scale to dozens of heads, this cache becomes massive, slowing down generation and limiting batch sizes.

Multi-Query Attention (MQA)

Multi-Query Attention takes a radical approach: use only one shared K and V head for all query heads. Instead of having 32 separate K/V pairs for 32 Q heads, you have 32 Q heads all attending to the same single K and V projection.

Think of it like a library with 32 readers (Q heads) but only one shared catalog system (K) and one shared collection (V). Each reader can ask different questions, but they all look up answers in the same place.

Grouped-Query Attention (GQA)

Grouped-Query Attention is the middle ground: divide query heads into groups, where each group shares one K/V head. For example, with 32 Q heads and 4 groups, you'd have 8 Q heads per group sharing one K/V pair—giving you 4 K/V heads total instead of 32 or 1.

This balances memory efficiency with model quality: fewer KV heads than full multi-head attention (faster inference, smaller cache) but more flexibility than single-head MQA.

Why This Matters

Models like Llama 2 and GPT-4 use GQA to achieve dramatically faster inference speeds—sometimes 2-5x faster—while maintaining competitive quality. The KV cache shrinks proportionally to the reduction in K/V heads.

Key Takeaway: Multi-Query and Grouped-Query Attention reduce memory and accelerate inference by sharing key-value projections across query heads, making large-scale language model deployment practical.