Course contentsShow
← Machine Learning and Deep Learning
Lesson 2803 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

DeepSpeed Configuration and Integration

Writing DeepSpeed configuration files, integrating with PyTorch models, and launching distributed training jobs.

DeepSpeed Configuration and Integration

What you'll learn: How to configure DeepSpeed through JSON files, integrate it with your PyTorch models, and launch distributed training jobs using the DeepSpeed runtime.

The Core Idea

DeepSpeed uses a JSON configuration file to control all aspects of distributed training—from ZeRO stages to mixed precision to gradient accumulation. This declarative approach separates infrastructure concerns from model code, making it easy to experiment with different optimization strategies without rewriting your training loop.

Think of the config file as a control panel for your distributed training engine: you set the dials (parameters), and DeepSpeed handles the complex orchestration behind the scenes.

Key Configuration Elements

A typical ds_config.json includes:

  • train_batch_size: Total effective batch size across all GPUs
  • train_micro_batch_size_per_gpu: Physical batch size per device
  • gradient_accumulation_steps: Automatically calculated or explicitly set
  • zero_optimization: Stage selection (1, 2, or 3) and offload options
  • fp16/bf16: Mixed precision settings and loss scaling
  • gradient_clipping: Max gradient norm

Integration with PyTorch

Instead of wrapping your model with DDP or FSDP manually, you use DeepSpeed's initialization:

import deepspeed

model_engine, optimizer, _, _ = deepspeed.initialize(
    model=model,
    config="ds_config.json"
)

DeepSpeed returns a model_engine that handles forward/backward passes, optimizer steps, and all communication automatically.

Launching Training

Use DeepSpeed's launcher instead of torchrun:

deepspeed --num_gpus=4 train.py --deepspeed_config ds_config.json

For multi-node jobs, specify a hostfile with node addresses and GPU counts.

Key Takeaway: DeepSpeed configuration files provide a unified, declarative interface to control distributed training strategies, while the deepspeed.initialize() API seamlessly integrates these optimizations into your PyTorch training workflow.