Course contentsShow
← Machine Learning and Deep Learning
Lesson 2987 of 3,53865. LLM Inference EnginesFree lesson

Preemption and Request Priority

Learn how systems handle memory pressure by preempting lower-priority requests to make room for new ones.

Preemption and Request Priority

What you'll learn: How LLM inference systems handle memory pressure by temporarily pausing or canceling lower-priority requests to serve higher-priority ones.

The Memory Pressure Problem

When running LLM inference with continuous batching, you'll eventually hit a wall: GPU memory is full, KV cache blocks are exhausted, but new high-priority requests keep arriving. Simply rejecting them wastes valuable serving capacity. Preemption solves this by strategically evicting lower-priority work to make room.

How Preemption Works

Think of it like an emergency room triage system. When a critical patient arrives but all beds are full, you might move a stable patient to a waiting area to free up immediate care capacity.

In LLM serving:

  1. Priority assignment: Requests receive priorities (e.g., paid tier vs free tier, latency-sensitive vs batch jobs)
  2. Memory monitoring: The system tracks available KV cache blocks
  3. Preemption trigger: When memory pressure exceeds a threshold and a high-priority request arrives, the scheduler identifies victims
  4. State preservation: The preempted request's KV cache blocks are either swapped to CPU memory or deallocated (requiring recomputation later)
  5. Resumption: When resources free up, preempted requests reload their state and continue

Preemption Strategies

  • Swap: Move KV cache to CPU/disk (slower but preserves work)
  • Recompute: Discard cache entirely and restart from the beginning (simpler but wasteful)
  • Partial recompute: Keep shared prefix blocks, only recompute unique portions

The scheduler balances fairness with efficiency—repeatedly preempting the same request can cause starvation.

Key Takeaway: Preemption allows inference engines to dynamically reallocate memory from lower-priority requests to higher-priority ones during memory pressure, maximizing throughput and meeting SLOs without hard rejections.