GPU Node Autoscaling and Constraints
What you'll learn: How to configure Kubernetes cluster autoscaler to intelligently add GPU nodes only when needed, using taints, tolerations, and node selectors to prevent misplaced workloads.
Why GPU Autoscaling Is Different
GPU nodes are expensive—often 10-50x the cost of CPU nodes. You don't want them sitting idle, but you also can't wait minutes for them to spin up when inference demand spikes. Cluster autoscaling for GPUs requires careful constraint configuration to ensure:
- GPU pods only land on GPU nodes (not CPU nodes where they'd crash)
- CPU workloads never "steal" precious GPU resources
- The cluster adds GPU nodes only when GPU workloads are pending
The Three-Part Strategy
Taints on GPU Nodes
Think of taints as "Do Not Enter" signs. GPU nodes should be tainted so regular pods avoid them:
# Node pool configuration
taints:
- key: nvidia.com/gpu
value: "true"
effect: NoSchedule
Tolerations on GPU Pods
Your AI inference pods need a "hall pass" to tolerate the taint:
tolerations:
- key: nvidia.com/gpu
operator: Equal
value: "true"
effect: NoSchedule
Node Selectors for Precision
Add node selectors to guarantee GPU pods only target GPU-capable nodes:
nodeSelector:
cloud.google.com/gke-accelerator: nvidia-tesla-t4
How Autoscaling Triggers
When a GPU pod is pending (can't be scheduled), the cluster autoscaler sees it needs a node with matching taints, tolerations, and labels. It provisions a new GPU node from your configured node pool. When GPU utilization drops and pods are removed, nodes scale down after a cooldown period.
Key Takeaway: GPU node autoscaling combines taints (repel non-GPU workloads), tolerations (allow GPU workloads), and node selectors (target correct hardware) to optimize cloud costs while ensuring inference availability.