Multi-Query Attention (MQA)
What You'll Learn
You'll understand how Multi-Query Attention dramatically reduces memory consumption by sharing key-value pairs across attention heads.
The Memory Problem with Standard Attention
In standard multi-head attention, each attention head maintains its own separate key (K) and value (V) projections. If you have 32 attention heads, you're storing 32 copies of keys and 32 copies of values in your KV cache. As you learned in the previous lessons, this KV cache memory requirement scales linearly with the number of heads, quickly becoming a bottleneck during inference.
How MQA Solves This
Multi-Query Attention takes a simple but powerful approach: share a single set of key and value projections across all attention heads, while keeping separate query (Q) projections for each head.
Think of it like a library system: instead of every reader (attention head) maintaining their own copy of reference books (keys and values), they all share the same reference section but bring their own unique questions (queries).
Memory Savings
The memory reduction is substantial:
- Standard MHA: Memory = num_heads × 2 × hidden_size
- MQA: Memory = 2 × hidden_size (constant, regardless of head count)
For a model with 32 heads, MQA uses 32× less KV cache memory. This means you can serve longer sequences, larger batch sizes, or fit models on smaller hardware—all critical for production inference optimization.
Trade-offs
MQA slightly reduces model quality compared to full multi-head attention because heads share information. However, the inference speed gains and memory savings often outweigh the minimal accuracy loss, making it a practical optimization for serving large language models.
Key Takeaway: Multi-Query Attention reduces KV cache memory by sharing keys and values across all attention heads, enabling dramatically more efficient inference with minimal quality impact.