Model Warmup and Cold Start Optimization
What you'll learn: How to minimize the dreaded "first request is slow" problem when launching new inference instances.
The Cold Start Problem
When you spin up a fresh inference server—whether for scaling out or replacing a failed instance—the very first requests often take significantly longer than steady-state performance. This "cold start" happens because:
- Model loading from disk into GPU memory isn't complete
- CUDA kernels need just-in-time compilation on first use
- Memory allocators haven't warmed up their buffer pools
- Dynamic batching systems need to calibrate timing
- Caching layers are empty (KV cache blocks, result caches)
Think of it like a car engine on a winter morning: everything works, but it runs rough until warmed up.
Warmup Strategies
Synthetic request injection is the core technique: before marking an instance "ready," send dummy inference requests through the pipeline. This forces:
- Model weights to be fully loaded and optimized
- CUDA kernels to compile and cache
- Memory pools to pre-allocate common buffer sizes
- Dynamic shape handling to process expected input dimensions
For LLMs specifically, you'll want warmup prompts that exercise:
- Common sequence lengths (e.g., 512, 1024, 2048 tokens)
- Batch sizes you expect in production
- Different sampling parameters (greedy, temperature=0.7, etc.)
Readiness probes should only pass after warmup completes—never route production traffic to a cold instance.
Integration Points
This builds directly on your earlier work with health checks and readiness probes (step 2912) and auto-scaling clusters (step 3008). The warmup phase sits between "container started" and "ready to serve."
Key Takeaway: Strategic warmup procedures—sending representative dummy requests before accepting production traffic—eliminate cold start latency penalties and ensure consistent p99 latencies from the moment an instance joins your serving pool.