Course contentsShow
← Machine Learning and Deep Learning
Lesson 2778 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

Mixed Precision with Distributed Training

Combine mixed precision with DDP or FSDP, handling gradient synchronization and scaling correctly.

Mixed Precision with Distributed Training

You'll learn how to correctly combine mixed precision (FP16/BF16) with distributed training frameworks like DDP and FSDP, ensuring gradient synchronization and loss scaling work harmoniously across multiple GPUs.

The Integration Challenge

When you combine mixed precision training with distributed data parallelism, you're essentially merging two optimization strategies: reduced precision computation (FP16/BF16) and multi-GPU gradient synchronization. The tricky part? Both systems manipulate gradients, and they must coordinate properly.

In DDP, gradients are synchronized across workers using all-reduce operations after the backward pass. With mixed precision, gradients are scaled up before backprop and unscaled before the optimizer step. The critical insight: gradient scaling must happen before synchronization, and unscaling must account for distributed reduction.

How It Works Together

  1. Forward pass: Computation happens in FP16/BF16 across all workers simultaneously
  2. Backward pass: Gradients computed in mixed precision are scaled (to prevent underflow)
  3. Gradient synchronization: DDP/FSDP performs all-reduce on the scaled gradients
  4. Unscaling: The optimizer unscales gradients after they're synchronized
  5. Optimizer step: Updates happen on FP32 master weights

The beauty is that modern frameworks (PyTorch AMP with DDP, FSDP with mixed precision enabled) handle this automatically. The gradient scaler knows about the distributed context and only unscales once all workers have synchronized their scaled gradients.

Practical Considerations

With FSDP, mixed precision is even more nuanced because parameters themselves are sharded. The MixedPrecision policy in FSDP specifies which precision to use for parameters, gradients, and buffers during computation versus storage.

Key Takeaway: Mixed precision and distributed training compose seamlessly when gradient scaling occurs before synchronization—modern frameworks like PyTorch handle this coordination automatically, but understanding the ordering ensures correct custom implementations and debugging.