Course contentsShow
← AI Engineering
Lesson 1069 of 1,88626. Self-Hosted LLM DeploymentFree lesson

Cloud GPU Options and Spot Instances

Evaluate cloud GPU rentals, spot pricing, and serverless options as alternatives to on-premise hardware.

Cloud GPU Options and Spot Instances

What You'll Learn

You'll learn how to evaluate cloud GPU rentals, spot pricing, and serverless inference as cost-effective alternatives to buying and maintaining your own hardware.

Why Cloud GPUs Matter

After understanding your model's hardware requirements (from previous lessons on sizing and quantization), you face a choice: buy GPUs or rent them. For many AI engineering teams, cloud rentals offer flexibility without the upfront capital expense of $10,000+ per high-end GPU.

Cloud GPU Providers

Major providers include AWS (EC2 P/G instances), Google Cloud (A2/G2 instances), Azure (NC/ND series), and specialized platforms like Lambda Labs, Vast.ai, and RunPod. Each offers different GPU types (A100, H100, L4, V100) at varying hourly rates.

Key factors to compare:

  • GPU memory (40GB vs 80GB A100s)
  • Network bandwidth for multi-GPU training
  • Regional availability and latency
  • Support for vLLM, TGI, and other serving frameworks

Spot Instances: The Budget Option

Spot instances are unused cloud capacity offered at 60-90% discounts. The catch? They can be reclaimed with 30-120 seconds notice when demand spikes.

When to use spot instances:

  • Development and experimentation
  • Batch inference jobs with checkpointing
  • Training with frequent saves

When to avoid them:

  • Production serving requiring 99.9% uptime
  • Real-time user-facing APIs

Think of spots like standby airline tickets—great savings, but you need backup plans.

Serverless Inference Platforms

Services like AWS SageMaker Serverless, Modal, and Banana auto-scale and charge per-request, eliminating idle costs. Trade-off: cold-start latency (3-30 seconds) when scaling from zero.

Key Takeaway: Cloud GPUs offer flexible, on-demand access to inference hardware—spot instances provide massive savings for fault-tolerant workloads, while serverless platforms eliminate infrastructure management at the cost of cold-start delays.