Course contentsShow
← Machine Learning and Deep Learning
Lesson 1673 of 3,53836. LLM Inference OptimizationFree lesson

Multi-Query Attention (MQA)

Extreme KV reduction using a single key-value head shared across all query heads.

Multi-Query Attention (MQA)

You'll learn: How Multi-Query Attention dramatically reduces memory by using a single shared key-value head for all query heads.

The Core Idea

In standard multi-head attention, each head has its own query (Q), key (K), and value (V) projections. In Grouped-Query Attention (GQA), you already saw how multiple query heads can share the same K and V heads. Multi-Query Attention takes this to the extreme: all query heads share a single key-value pair.

Think of it like a library with multiple readers (query heads) but only one catalog system (K) and one set of book summaries (V). Every reader must use the same lookup infrastructure, but each can ask different questions.

How It Works

Instead of maintaining separate K and V projections for each attention head:

  • Queries: Still have num_heads separate projections
  • Keys: Only 1 projection shared across all heads
  • Values: Only 1 projection shared across all heads

During generation, your KV cache now stores only one key-value pair per token, regardless of how many query heads you have. If you had 32 query heads before, you've just reduced KV cache memory by 32×.

The Trade-off

This extreme reduction comes at a cost: the model loses some representational capacity since all query heads must attend using the same key-value information. However, research shows that with proper training, MQA models can achieve nearly identical quality to standard multi-head attention while enabling much faster inference.

Key Takeaway: Multi-Query Attention reduces KV cache size by using a single shared key-value head for all queries, trading minimal quality for massive memory savings and faster autoregressive generation.