Preemption and Request Priority
What you'll learn: How LLM inference systems handle memory pressure by temporarily pausing or canceling lower-priority requests to serve higher-priority ones.
The Memory Pressure Problem
When running LLM inference with continuous batching, you'll eventually hit a wall: GPU memory is full, KV cache blocks are exhausted, but new high-priority requests keep arriving. Simply rejecting them wastes valuable serving capacity. Preemption solves this by strategically evicting lower-priority work to make room.
How Preemption Works
Think of it like an emergency room triage system. When a critical patient arrives but all beds are full, you might move a stable patient to a waiting area to free up immediate care capacity.
In LLM serving:
- Priority assignment: Requests receive priorities (e.g., paid tier vs free tier, latency-sensitive vs batch jobs)
- Memory monitoring: The system tracks available KV cache blocks
- Preemption trigger: When memory pressure exceeds a threshold and a high-priority request arrives, the scheduler identifies victims
- State preservation: The preempted request's KV cache blocks are either swapped to CPU memory or deallocated (requiring recomputation later)
- Resumption: When resources free up, preempted requests reload their state and continue
Preemption Strategies
- Swap: Move KV cache to CPU/disk (slower but preserves work)
- Recompute: Discard cache entirely and restart from the beginning (simpler but wasteful)
- Partial recompute: Keep shared prefix blocks, only recompute unique portions
The scheduler balances fairness with efficiency—repeatedly preempting the same request can cause starvation.
Key Takeaway: Preemption allows inference engines to dynamically reallocate memory from lower-priority requests to higher-priority ones during memory pressure, maximizing throughput and meeting SLOs without hard rejections.