Troubleshooting OOM and Imbalance
What You'll Learn
You'll learn how to diagnose and fix out-of-memory (OOM) errors and load imbalance across GPUs when serving large models in production.
Understanding the Problem
When running multi-GPU inference, two critical issues can derail your deployment:
Out-of-Memory (OOM) errors occur when your model or batch demands more GPU memory than available. Think of it like trying to fit too many passengers in an elevator—eventually, the doors won't close.
Load imbalance happens when some GPUs work harder than others, leaving resources idle. Imagine a restaurant where one waiter serves all customers while others stand around—throughput suffers despite having capacity.
Diagnosing OOM Issues
Check these culprits in order:
- Batch size too large for available VRAM per GPU
- Model sharding incomplete (some layers duplicated across devices)
- KV cache growth exceeding allocated memory
- Activation memory during forward pass (hidden states pile up)
Monitor with nvidia-smi or torch.cuda.memory_summary() to pinpoint which GPU exhausts memory first.
Fixing OOM
- Reduce batch size or implement adaptive batching
- Enable gradient checkpointing (if fine-tuning)
- Use more aggressive quantization (INT8/INT4)
- Increase GPU count with tensor or pipeline parallelism
- Configure smaller KV cache limits
Diagnosing and Fixing Imbalance
Imbalance typically stems from:
- Uneven tensor splits in tensor parallelism
- Pipeline bubble time where GPUs wait for previous stages
- Incorrect device mapping concentrating compute on fewer devices
Solutions:
- Profile per-GPU utilization with monitoring tools
- Adjust device mapping strategies for even distribution
- Tune pipeline micro-batch sizes
- Ensure communication bandwidth isn't bottlenecking one device
Key Takeaway: OOM and imbalance require systematic monitoring—watch memory usage and GPU utilization metrics, then adjust batching, model sharding, and resource allocation iteratively until all GPUs run efficiently without exhausting memory.