Course contentsShow
← Machine Learning and Deep Learning
Lesson 1684 of 3,53836. LLM Inference OptimizationFree lesson

Grouped Query Attention Memory Savings

How GQA reduces KV cache size by sharing key-value heads across query heads without major quality loss.

Grouped Query Attention Memory Savings

What you'll learn: How GQA reduces KV cache memory requirements by sharing key-value heads across query heads without significant quality degradation.

The Memory Problem with Multi-Head Attention

In standard multi-head attention, each attention head maintains its own separate key and value projections. If you have 32 query heads, you also store 32 separate key heads and 32 value heads in your KV cache. This creates enormous memory pressure during inference, especially for long sequences.

How GQA Achieves Memory Savings

Grouped Query Attention (GQA) works by sharing key-value head pairs across multiple query heads. Instead of having a 1:1:1 ratio (queries:keys:values), GQA uses a many-to-one relationship.

For example, with 32 query heads and 4 KV groups:

  • You still have 32 separate query heads
  • But only 4 key heads and 4 value heads
  • Each KV pair is shared by 8 query heads (32 ÷ 4 = 8)

The Memory Math

If your KV cache previously stored 32 key-value pairs per token, GQA with 4 groups now stores only 4 pairs—an 8x reduction in KV cache memory! During a typical inference run generating hundreds of tokens, this translates to dramatic memory savings that enable longer sequences or larger batch sizes.

Why It Works

The insight is that queries need fine-grained diversity to capture different patterns, but keys and values can be coarser without losing much expressive power. Think of it like multiple detectives (queries) searching through the same shared evidence database (keys/values)—they each look for different patterns, but don't need separate copies of the evidence.

Key Takeaway: GQA reduces KV cache memory by a factor equal to the number of query heads per group, enabling more efficient inference with minimal quality loss compared to full multi-head attention.