Setting Up Multi-GPU with Accelerate
What You'll Learn
You'll learn how to use Hugging Face Accelerate to distribute model inference across multiple GPUs with just a few lines of code, no manual CUDA management required.
The Core Idea
After learning about tensor parallelism and pipeline parallelism conceptually, you need a practical way to implement them. Accelerate is Hugging Face's library that abstracts away the complexity of distributed computing. Instead of writing low-level PyTorch distributed code, you wrap your model and data with Accelerate's utilities, and it handles the multi-GPU orchestration automatically.
Think of Accelerate as a universal translator: your code speaks "single GPU," but Accelerate translates it into "multiple GPUs working together" without you changing your core logic.
How It Works
The basic pattern involves three steps:
- Initialize the Accelerator: Create an
Acceleratorobject that detects your hardware setup - Prepare your components: Pass your model, optimizer, and data through
accelerator.prepare() - Run normally: Your training/inference loop runs as if on a single device
from accelerate import Accelerator
accelerator = Accelerator()
# Your model and data
model = YourLLM()
dataloader = YourDataLoader()
# Magic happens here - Accelerate distributes across GPUs
model, dataloader = accelerator.prepare(model, dataloader)
# Rest of your code remains unchanged
for batch in dataloader:
outputs = model(**batch)
Accelerate automatically detects available GPUs, splits batches appropriately, and synchronizes gradients. For inference specifically, it handles tensor placement and gathering results from different devices.
You can launch multi-GPU scripts using accelerate launch script.py instead of python script.py, and Accelerate configures everything based on your environment.
Key Takeaway: Hugging Face Accelerate transforms multi-GPU complexity into a few wrapper calls, letting you scale from one to many GPUs without rewriting your core inference or training logic.