Course contentsShow
← Machine Learning and Deep Learning
Lesson 3009 of 3,53865. LLM Inference EnginesFree lesson

Model Warmup and Cold Start Optimization

Minimize initialization latency when spinning up new inference instances through strategic warmup procedures.

Model Warmup and Cold Start Optimization

What you'll learn: How to minimize the dreaded "first request is slow" problem when launching new inference instances.

The Cold Start Problem

When you spin up a fresh inference server—whether for scaling out or replacing a failed instance—the very first requests often take significantly longer than steady-state performance. This "cold start" happens because:

  • Model loading from disk into GPU memory isn't complete
  • CUDA kernels need just-in-time compilation on first use
  • Memory allocators haven't warmed up their buffer pools
  • Dynamic batching systems need to calibrate timing
  • Caching layers are empty (KV cache blocks, result caches)

Think of it like a car engine on a winter morning: everything works, but it runs rough until warmed up.

Warmup Strategies

Synthetic request injection is the core technique: before marking an instance "ready," send dummy inference requests through the pipeline. This forces:

  • Model weights to be fully loaded and optimized
  • CUDA kernels to compile and cache
  • Memory pools to pre-allocate common buffer sizes
  • Dynamic shape handling to process expected input dimensions

For LLMs specifically, you'll want warmup prompts that exercise:

  • Common sequence lengths (e.g., 512, 1024, 2048 tokens)
  • Batch sizes you expect in production
  • Different sampling parameters (greedy, temperature=0.7, etc.)

Readiness probes should only pass after warmup completes—never route production traffic to a cold instance.

Integration Points

This builds directly on your earlier work with health checks and readiness probes (step 2912) and auto-scaling clusters (step 3008). The warmup phase sits between "container started" and "ready to serve."

Key Takeaway: Strategic warmup procedures—sending representative dummy requests before accepting production traffic—eliminate cold start latency penalties and ensure consistent p99 latencies from the moment an instance joins your serving pool.