Course contentsShow
← Machine Learning and Deep Learning
Lesson 2740 of 3,53859. Distributed Training: Data ParallelismFree lesson

FSDP State Dict Management

Saving and loading sharded checkpoints: gathering full state on rank 0 vs distributed sharded checkpoint formats.

FSDP State Dict Management

What You'll Learn

You'll learn the two main approaches for saving FSDP model checkpoints: gathering the full state on rank 0 versus using distributed sharded checkpoint formats.

The Checkpoint Dilemma in FSDP

When you train with FSDP, model parameters are sharded across multiple GPUs—no single device holds the complete model. So how do you save a checkpoint?

You have two strategies:

Strategy 1: Gather Full State on Rank 0

This is the traditional approach. All workers send their parameter shards to rank 0, which reconstructs the complete model state dict—exactly like a non-distributed checkpoint.

Pros:

  • Simple to use: standard PyTorch checkpoint format
  • Easy to load on different hardware configurations
  • Compatible with non-FSDP inference

Cons:

  • Rank 0 must have enough memory for the entire model
  • Communication overhead during gathering
  • Slow for very large models

Think of this like collecting puzzle pieces from everyone to build the complete picture on one table.

Strategy 2: Distributed Sharded Checkpoints

Each rank saves only its own shard to separate files. The checkpoint remains distributed across multiple files.

Pros:

  • No single-node memory bottleneck
  • Faster saving/loading for huge models
  • Scales to models that don't fit on one GPU

Cons:

  • More complex file management (multiple checkpoint files)
  • Loading requires same or compatible sharding configuration
  • Less portable across different cluster setups

This is like leaving each puzzle piece with its current owner—efficient but requires coordination to reassemble.

Choosing Your Approach

Use full state gathering for smaller models or when you need maximum portability. Use sharded checkpoints when your model is so large that rank 0 can't hold it, or when checkpoint I/O time dominates.

Key Takeaway: FSDP state dict management offers a trade-off between simplicity (gathering full state on rank 0) and scalability (distributed sharded format), with your choice depending on model size and deployment requirements.