Course contentsShow
← AI Engineering
Lesson 1128 of 1,88627. Cloud Deployment and ScalingFree lesson

GPU Node Autoscaling and Constraints

Configure cluster autoscaler for GPU nodes with proper taints, tolerations, and node selector constraints.

GPU Node Autoscaling and Constraints

What you'll learn: How to configure Kubernetes cluster autoscaler to intelligently add GPU nodes only when needed, using taints, tolerations, and node selectors to prevent misplaced workloads.

Why GPU Autoscaling Is Different

GPU nodes are expensive—often 10-50x the cost of CPU nodes. You don't want them sitting idle, but you also can't wait minutes for them to spin up when inference demand spikes. Cluster autoscaling for GPUs requires careful constraint configuration to ensure:

  1. GPU pods only land on GPU nodes (not CPU nodes where they'd crash)
  2. CPU workloads never "steal" precious GPU resources
  3. The cluster adds GPU nodes only when GPU workloads are pending

The Three-Part Strategy

Taints on GPU Nodes

Think of taints as "Do Not Enter" signs. GPU nodes should be tainted so regular pods avoid them:

# Node pool configuration
taints:
  - key: nvidia.com/gpu
    value: "true"
    effect: NoSchedule

Tolerations on GPU Pods

Your AI inference pods need a "hall pass" to tolerate the taint:

tolerations:
  - key: nvidia.com/gpu
    operator: Equal
    value: "true"
    effect: NoSchedule

Node Selectors for Precision

Add node selectors to guarantee GPU pods only target GPU-capable nodes:

nodeSelector:
  cloud.google.com/gke-accelerator: nvidia-tesla-t4

How Autoscaling Triggers

When a GPU pod is pending (can't be scheduled), the cluster autoscaler sees it needs a node with matching taints, tolerations, and labels. It provisions a new GPU node from your configured node pool. When GPU utilization drops and pods are removed, nodes scale down after a cooldown period.

Key Takeaway: GPU node autoscaling combines taints (repel non-GPU workloads), tolerations (allow GPU workloads), and node selectors (target correct hardware) to optimize cloud costs while ensuring inference availability.