Course contentsShow
← Machine Learning and Deep Learning
Lesson 2984 of 3,53865. LLM Inference EnginesFree lesson

Request Scheduling and Admission Control

Learn how the scheduler decides when to admit new requests based on available KV cache memory.

Request Scheduling and Admission Control

What you'll learn: How inference schedulers decide whether to accept new incoming requests based on available KV cache memory.

The Core Challenge

When deploying LLM inference engines with continuous batching and paged attention, you face a critical question: Should I accept this new request right now?

Unlike traditional servers that might queue requests indefinitely, LLM inference requires substantial KV cache memory per request. The scheduler must perform admission control — deciding whether accepting a new request would cause existing requests to fail or degrade system performance.

How Admission Control Works

The scheduler maintains a real-time view of available resources:

  • Free KV cache blocks (pages) in GPU memory
  • Currently executing requests and their memory footprints
  • Expected memory needs for new requests (estimated from prompt length)

When a new request arrives, the scheduler:

  1. Estimates memory requirement based on prompt length and maximum generation length
  2. Checks available blocks against this estimate
  3. Decides: admit (start processing), queue (wait for resources), or reject (insufficient capacity)

Scheduling Policies

The scheduler also prioritizes which queued requests to process first:

  • FIFO (First-In-First-Out): Fair, simple ordering
  • Shortest-Job-First: Minimize average latency by processing quick requests first
  • Priority-based: High-value requests get precedence

The scheduler continuously re-evaluates as blocks free up from completed generation steps, enabling the continuous batching paradigm you learned earlier.

Key Takeaway: Request scheduling and admission control protect system stability by only accepting new LLM inference requests when sufficient KV cache memory is available, preventing out-of-memory errors and maintaining quality of service for in-flight requests.