Inference vs Training Infrastructure
What you'll learn: Why the infrastructure that serves predictions looks radically different from the infrastructure that trains models.
The Tale of Two Workloads
Training and inference are like marathon training versus sprinting on demand. Training is a deliberate, resource-intensive process you do periodically—loading massive datasets, updating billions of parameters over days or weeks, maximizing GPU utilization at all costs. Inference is responding to real-world requests right now—one prediction at a time (or small batches), where milliseconds matter and idle GPUs waste money.
Training Infrastructure Priorities
When training, you optimize for throughput and GPU utilization:
- Saturate GPUs with large batch sizes to amortize overhead
- Use high-end accelerators (A100s, H100s) running 24/7
- Tolerate minutes or hours of processing time
- Accept higher infrastructure costs because training is infrequent
- Focus on data pipeline efficiency to keep GPUs fed
Think of it like a factory assembly line: maximize production volume, minimize downtime.
Inference Infrastructure Priorities
When serving predictions, you optimize for latency and cost efficiency:
- Respond to single requests in milliseconds (p95 < 100ms often)
- Use smaller, cheaper GPUs or even CPUs for cost-effective serving
- Scale horizontally: add/remove instances based on traffic
- Batch requests dynamically to balance latency and throughput
- Pay only for what you use (serverless, autoscaling)
This is like a restaurant kitchen during rush hour: respond quickly to each order while minimizing idle staff.
The Infrastructure Gap
You can't just "use the training cluster" for inference. Training clusters are expensive to keep idle during low traffic. Inference needs edge deployments, CDN integration, and sub-second cold starts—none of which training requires.
Key Takeaway: Training infrastructure maximizes GPU utilization for batch workloads regardless of time; inference infrastructure minimizes latency and operational costs per prediction, requiring fundamentally different architectural choices.