Node Affinity and GPU Node Pools
What you'll learn: How to ensure your GPU-hungry AI workloads land on the right machines in your Kubernetes cluster.
Why Node Affinity Matters for AI
In a Kubernetes cluster, not all nodes are created equal. Some nodes have GPUs, others have massive amounts of RAM, and some are just standard CPU-only machines. When deploying AI models—especially LLMs that require GPUs or high-memory configurations—you need to direct your workloads to the right hardware.
Think of it like assigning specialized jobs to qualified workers. You wouldn't ask someone without a forklift license to operate heavy machinery. Similarly, you don't want to schedule a GPU-intensive inference workload on a node without a GPU.
Understanding Node Pools
In managed Kubernetes services (GKE, EKS, AKS), you create node pools—groups of nodes with identical configurations. For AI workloads, you might have:
- A GPU node pool with NVIDIA A100s or T4s for inference
- A high-memory pool for embedding generation or large batch processing
- A standard pool for lightweight services
Using Node Affinity
Node affinity is Kubernetes' way of matching pods to nodes based on labels. You label your GPU nodes (e.g., gpu-type: a100), then configure your deployment to require or prefer those labels:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: gpu-type
operator: In
values:
- a100
This ensures your vLLM or TGI deployment only runs on nodes with A100 GPUs, preventing expensive pods from landing on incompatible hardware.
Key Takeaway: Node affinity and GPU node pools let you architect heterogeneous clusters where AI workloads automatically find the specialized hardware they need, optimizing both cost and performance.