Course contentsShow
← Machine Learning and Deep Learning
Lesson 1672 of 3,53836. LLM Inference OptimizationFree lesson

Grouped-Query Attention (GQA)

Reducing KV cache size by sharing key-value heads across multiple query heads.

Grouped-Query Attention (GQA)

What you'll learn: How Grouped-Query Attention reduces memory usage during inference by sharing key-value heads across multiple query heads.

The Memory Problem with Multi-Head Attention

In standard multi-head attention, every attention head maintains its own separate key (K) and value (V) projections. During autoregressive generation, the KV cache stores these keys and values for all heads across all previous tokens. As you've learned, this cache grows linearly with sequence length and can consume enormous amounts of memory—especially in large models with many heads.

The GQA Solution

Grouped-Query Attention is a hybrid approach between Multi-Head Attention (MHA) and Multi-Query Attention (MQA). Instead of having one KV head shared across all query heads (MQA) or separate KV heads for every query head (MHA), GQA creates groups of query heads that share the same KV projection.

How It Works

Imagine you have 32 query heads. In standard MHA, you'd have 32 separate KV heads—one per query. With GQA, you might organize those 32 query heads into 8 groups of 4. Now you only need 8 KV heads total, with each KV head serving 4 query heads.

The queries still compute attention independently, but they reference the same shared keys and values within their group. This reduces KV cache size by 4× in this example, while maintaining more representational capacity than pure MQA (which would use just 1 KV head).

Memory Savings

If each KV head stores tensors of size [batch, seq_len, head_dim], reducing from 32 to 8 KV heads cuts your cache memory by 75%—without significantly degrading model quality.

Key Takeaway: Grouped-Query Attention reduces KV cache memory requirements by sharing key-value projections across groups of query heads, offering a practical middle ground between full multi-head attention and aggressive multi-query attention.