Course contentsShow
← Machine Learning and Deep Learning
Lesson 3001 of 3,53865. LLM Inference EnginesFree lesson

Batching and KV Cache Management

Handling KV cache for speculative sequences, managing rejected tokens, and batching strategies for speculative decoding workloads.

Batching and KV Cache Management

What you'll learn: How speculative decoding complicates batching and KV cache management, requiring specialized strategies to handle tentative tokens that may be rejected.

The Challenge of Speculative Sequences

In standard autoregressive decoding, each request maintains a simple KV cache that grows monotonically—you accept one token, append it to the cache, and move forward. But speculative decoding introduces tentative futures: the draft model proposes multiple tokens at once, which must be verified in parallel. If the verifier rejects a token at position k, you must discard all speculative tokens from k onward.

This creates a KV cache management nightmare. You've already computed key-value pairs for those rejected tokens—do you keep them? Rewind them? How do you batch multiple requests when some accept 5 tokens and others only accept 1?

Managing Rejected Tokens

The key insight: treat speculative tokens as provisional until verification succeeds. Two common strategies:

  1. Copy-on-write checkpointing: Before speculation, snapshot the current KV cache state. If rejection occurs, roll back to the snapshot.
  2. Lazy commit: Store speculative KV pairs in temporary buffers. Only merge them into the main cache after acceptance is confirmed.

Batching Strategies for Speculative Workloads

Continuous batching becomes more complex because requests now have variable-length speculation windows. You need:

  • Per-request acceptance tracking: Determine how many tokens each request accepted before rejoining the batch
  • Aligned vs unaligned batching: Either synchronize all requests to the same speculation depth (wastes capacity) or allow ragged batching with careful memory planning
  • Memory reservation: Pre-allocate KV cache space for the maximum possible speculation depth to avoid mid-batch reallocation

Key Takeaway: Speculative decoding requires provisional KV cache states and sophisticated batching logic to handle variable acceptance rates without fragmenting memory or stalling high-performing requests.