Cloud GPU Options and Spot Instances
What You'll Learn
You'll learn how to evaluate cloud GPU rentals, spot pricing, and serverless inference as cost-effective alternatives to buying and maintaining your own hardware.
Why Cloud GPUs Matter
After understanding your model's hardware requirements (from previous lessons on sizing and quantization), you face a choice: buy GPUs or rent them. For many AI engineering teams, cloud rentals offer flexibility without the upfront capital expense of $10,000+ per high-end GPU.
Cloud GPU Providers
Major providers include AWS (EC2 P/G instances), Google Cloud (A2/G2 instances), Azure (NC/ND series), and specialized platforms like Lambda Labs, Vast.ai, and RunPod. Each offers different GPU types (A100, H100, L4, V100) at varying hourly rates.
Key factors to compare:
- GPU memory (40GB vs 80GB A100s)
- Network bandwidth for multi-GPU training
- Regional availability and latency
- Support for vLLM, TGI, and other serving frameworks
Spot Instances: The Budget Option
Spot instances are unused cloud capacity offered at 60-90% discounts. The catch? They can be reclaimed with 30-120 seconds notice when demand spikes.
When to use spot instances:
- Development and experimentation
- Batch inference jobs with checkpointing
- Training with frequent saves
When to avoid them:
- Production serving requiring 99.9% uptime
- Real-time user-facing APIs
Think of spots like standby airline tickets—great savings, but you need backup plans.
Serverless Inference Platforms
Services like AWS SageMaker Serverless, Modal, and Banana auto-scale and charge per-request, eliminating idle costs. Trade-off: cold-start latency (3-30 seconds) when scaling from zero.
Key Takeaway: Cloud GPUs offer flexible, on-demand access to inference hardware—spot instances provide massive savings for fault-tolerant workloads, while serverless platforms eliminate infrastructure management at the cost of cold-start delays.