Resource Requests and Limits for GPU Workloads
You'll learn how to specify CPU, memory, and GPU requirements in Kubernetes to ensure your AI inference workloads get the resources they need without starving other applications.
What Are Resource Requests and Limits?
When deploying AI models to Kubernetes, you need to tell the cluster how much compute power your pods require. Requests are what Kubernetes uses to decide which node can host your pod—it's like reserving a hotel room. Limits are the maximum resources your pod can consume—like the fire code capacity of that room.
For GPU workloads, this becomes critical: GPUs are expensive, discrete resources. You can't share half a GPU between two pods (in most cases), and running inference without enough memory will crash your container mid-request.
Why This Matters for AI Inference
Think of resource specifications as a contract with the cluster. If you request 4GB of memory but your LLM actually needs 16GB, your pod will be OOM-killed during inference. If you don't set limits, a runaway process could monopolize a GPU and starve other workloads.
For AI deployments, you typically specify:
- CPU requests/limits: For preprocessing and orchestration logic
- Memory requests/limits: For model weights, KV cache, and batching buffers
- GPU requests: Usually whole numbers (1, 2, 4 GPUs) since fractional GPU allocation requires special tooling
In your Deployment YAML, these go under resources.requests and resources.limits in the container spec. Setting accurate values prevents scheduling failures, enables better cluster utilization, and ensures predictable performance.
Key Takeaway: Resource requests and limits are mandatory for production GPU workloads—they guarantee your AI models get the compute they need while preventing resource contention and enabling Kubernetes to make intelligent scheduling decisions.