Request Scheduling and Admission Control
What you'll learn: How inference schedulers decide whether to accept new incoming requests based on available KV cache memory.
The Core Challenge
When deploying LLM inference engines with continuous batching and paged attention, you face a critical question: Should I accept this new request right now?
Unlike traditional servers that might queue requests indefinitely, LLM inference requires substantial KV cache memory per request. The scheduler must perform admission control — deciding whether accepting a new request would cause existing requests to fail or degrade system performance.
How Admission Control Works
The scheduler maintains a real-time view of available resources:
- Free KV cache blocks (pages) in GPU memory
- Currently executing requests and their memory footprints
- Expected memory needs for new requests (estimated from prompt length)
When a new request arrives, the scheduler:
- Estimates memory requirement based on prompt length and maximum generation length
- Checks available blocks against this estimate
- Decides: admit (start processing), queue (wait for resources), or reject (insufficient capacity)
Scheduling Policies
The scheduler also prioritizes which queued requests to process first:
- FIFO (First-In-First-Out): Fair, simple ordering
- Shortest-Job-First: Minimize average latency by processing quick requests first
- Priority-based: High-value requests get precedence
The scheduler continuously re-evaluates as blocks free up from completed generation steps, enabling the continuous batching paradigm you learned earlier.
Key Takeaway: Request scheduling and admission control protect system stability by only accepting new LLM inference requests when sufficient KV cache memory is available, preventing out-of-memory errors and maintaining quality of service for in-flight requests.