Course contentsShow
← Machine Learning and Deep Learning
Lesson 3008 of 3,53865. LLM Inference EnginesFree lesson

Auto-Scaling LLM Inference Clusters

Configure dynamic scaling policies to add or remove inference capacity based on demand patterns.

Auto-Scaling LLM Inference Clusters

What You'll Learn

You'll learn how to configure dynamic scaling policies that automatically add or remove inference capacity based on real-time demand patterns.

The Core Problem

LLM inference workloads are rarely constant. You might face peak traffic during business hours and minimal requests at night, or sudden spikes when a feature goes viral. Running maximum capacity 24/7 wastes money; under-provisioning creates latency spikes and service degradation.

Dynamic Scaling Policies

Auto-scaling adjusts your cluster size automatically based on predefined triggers:

Metric-Based Triggers

  • Request queue depth: Scale up when pending requests exceed threshold (e.g., >50 queued requests)
  • GPU utilization: Add capacity when average GPU usage stays above 70-80% for sustained periods
  • Latency percentiles: Scale up if P95 latency exceeds your SLO budget
  • Request rate: Monitor requests-per-second and add nodes proactively

Scaling Dimensions

You can scale horizontally (add/remove replica servers) or vertically (larger GPU instances). For LLMs, horizontal scaling with load balancing is typically more practical since model sharding strategies you learned earlier distribute work across multiple GPUs anyway.

Cooldown and Stability

Set cooldown periods to prevent thrashing—rapidly adding and removing nodes wastes startup time and disrupts KV cache warming. A typical policy: scale up aggressively (2-3 minutes), scale down conservatively (10-15 minutes).

Predictive Scaling

Advanced setups use historical patterns: if you know traffic peaks at 9 AM daily, pre-scale capacity at 8:45 AM rather than reacting after requests pile up.

Key Takeaway: Dynamic auto-scaling policies balance cost and performance by continuously matching inference capacity to demand, using metrics like queue depth, GPU utilization, and latency combined with smart cooldown periods to avoid wasteful oscillation.