Course contentsShow
← AI Engineering
Lesson 1080 of 1,88626. Self-Hosted LLM DeploymentFree lesson

Monitoring Multi-GPU Utilization

Tools and techniques to monitor GPU memory, compute utilization, and identify performance issues.

Monitoring Multi-GPU Utilization

What you'll learn: How to track GPU memory, compute usage, and diagnose bottlenecks when running models across multiple GPUs.

Why Monitoring Matters in Multi-GPU Setups

When you're serving a large language model across multiple GPUs (using tensor or pipeline parallelism), understanding how each GPU performs is critical. Without monitoring, you can't tell if one GPU is overloaded while others idle, if communication between GPUs is the bottleneck, or if you're about to run out of memory.

Think of it like monitoring traffic on a multi-lane highway: you need to know which lanes are congested, which are empty, and where accidents (errors) occur.

Core Monitoring Tools and Techniques

nvidia-smi

The foundational command-line tool for NVIDIA GPUs:

nvidia-smi --query-gpu=index,utilization.gpu,memory.used,memory.total --format=csv -l 1

This polls GPU stats every second, showing compute utilization and memory per device.

Python Integration

Libraries like pynvml let you programmatically monitor GPUs:

import pynvml
pynvml.nvmlInit()
handle = pynvml.nvmlDeviceGetHandleByIndex(0)
util = pynvml.nvmlDeviceGetUtilizationRates(handle)
print(f"GPU 0 Utilization: {util.gpu}%")

Key Metrics to Track

  • GPU Utilization (%): Is the compute actually busy?
  • Memory Used/Total: Are you near OOM errors?
  • Temperature/Power: Thermal throttling can slow inference
  • PCIe/NVLink Bandwidth: Communication overhead between GPUs

Identifying Issues

  • Uneven utilization: Suggests poor load balancing across devices
  • Low utilization + high latency: Communication bottleneck or small batch sizes
  • Memory near capacity: Risk of crashes; consider quantization or smaller batches

Key Takeaway: Effective multi-GPU monitoring requires tracking compute, memory, and communication metrics per device to identify imbalances and optimize resource usage.