Course contentsShow
← AI Engineering
Lesson 1109 of 1,88627. Cloud Deployment and ScalingFree lesson

Node Affinity and GPU Node Pools

Directing AI workloads to specific node types with GPUs or high-memory configurations.

Node Affinity and GPU Node Pools

What you'll learn: How to ensure your GPU-hungry AI workloads land on the right machines in your Kubernetes cluster.

Why Node Affinity Matters for AI

In a Kubernetes cluster, not all nodes are created equal. Some nodes have GPUs, others have massive amounts of RAM, and some are just standard CPU-only machines. When deploying AI models—especially LLMs that require GPUs or high-memory configurations—you need to direct your workloads to the right hardware.

Think of it like assigning specialized jobs to qualified workers. You wouldn't ask someone without a forklift license to operate heavy machinery. Similarly, you don't want to schedule a GPU-intensive inference workload on a node without a GPU.

Understanding Node Pools

In managed Kubernetes services (GKE, EKS, AKS), you create node pools—groups of nodes with identical configurations. For AI workloads, you might have:

  • A GPU node pool with NVIDIA A100s or T4s for inference
  • A high-memory pool for embedding generation or large batch processing
  • A standard pool for lightweight services

Using Node Affinity

Node affinity is Kubernetes' way of matching pods to nodes based on labels. You label your GPU nodes (e.g., gpu-type: a100), then configure your deployment to require or prefer those labels:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: gpu-type
          operator: In
          values:
          - a100

This ensures your vLLM or TGI deployment only runs on nodes with A100 GPUs, preventing expensive pods from landing on incompatible hardware.

Key Takeaway: Node affinity and GPU node pools let you architect heterogeneous clusters where AI workloads automatically find the specialized hardware they need, optimizing both cost and performance.