Spot and Preemptible Instances
What You'll Learn
You'll learn how to slash infrastructure costs by using interruptible compute for fault-tolerant AI workloads.
The Core Idea
Spot instances (AWS) and preemptible instances (GCP/Azure) are heavily discounted compute resources—often 60-90% cheaper than on-demand pricing—that cloud providers can reclaim with short notice (typically 30 seconds to 2 minutes). Think of them like standby airline tickets: you get massive savings, but you must be flexible and ready to lose your seat.
For AI engineering, this is transformative for non-critical workloads like:
- Batch inference jobs
- Model training experiments
- Data preprocessing pipelines
- Development/testing environments
The catch? Your instance can disappear at any moment when the cloud provider needs capacity back.
Architecting for Interruption
To use spot instances effectively, you must design fault-tolerant systems:
Checkpointing
Save training state frequently so you can resume from the last checkpoint rather than starting over.
Job Queues
Use queue-based patterns where interrupted tasks automatically re-enter the queue for retry on another instance.
Stateless Workers
Design inference services that don't hold critical state—if one dies, another handles the next request seamlessly.
Diversification
Request multiple instance types across availability zones to reduce simultaneous interruption risk.
When NOT to Use Spot Instances
Avoid spot instances for:
- Real-time user-facing inference APIs
- Critical production services requiring guaranteed uptime
- Long-running jobs without checkpointing capability
Key Takeaway: Spot and preemptible instances can reduce compute costs by 60-90% for AI workloads, but only when paired with fault-tolerant architectures that gracefully handle interruptions through checkpointing, queuing, and stateless design.