Course contentsShow
← Machine Learning and Deep Learning
Lesson 2974 of 3,53865. LLM Inference EnginesFree lesson

Copy-on-Write for Shared Prefixes

Efficiently handling beam search and parallel sampling by sharing KV cache pages until sequences diverge.

Copy-on-Write for Shared Prefixes

What you'll learn: How modern LLM inference engines efficiently handle multiple generation paths that share common prompt prefixes by deferring KV cache copying until sequences actually diverge.

The Core Problem

When you perform beam search or parallel sampling from the same prompt, traditional approaches would duplicate the entire KV cache for each beam/sample. If you have a 1000-token prompt and generate 10 beams, you'd store 10 identical copies of those 1000 prompt tokens—wasteful!

Copy-on-Write (CoW) to the Rescue

Copy-on-Write is a memory optimization borrowed from operating systems. The key insight: multiple sequences can share the same physical KV cache pages as long as they're reading identical content. Only when a sequence diverges (generates different tokens) do you need to create a separate copy.

How It Works

  1. Initial state: All beams/samples point to the same physical pages containing the prompt's KV cache
  2. During generation: Each sequence references shared pages via its own page table (from lesson 2973)
  3. On divergence: When beam A generates a different token than beam B, only that page gets copied to a new physical location
  4. Beam A's page table is updated to point to the new page; beam B keeps using the shared one

The Efficiency Win

With CoW, your memory usage is: prompt KV cache + only the diverged tokens per beam. For a 10-beam search on a 1000-token prompt generating 50 tokens each, you store ~1000 + 500 tokens instead of 10,500 tokens—a 10× improvement early in generation!

Key Takeaway: Copy-on-Write eliminates redundant KV cache storage by sharing physical pages across sequences until they diverge, dramatically reducing memory usage for beam search and parallel sampling with minimal computational overhead.