Sequence Parallelism
What you'll learn: How to partition activations along the sequence dimension to reduce memory usage when training models on long sequences.
The Memory Problem with Long Sequences
When training Transformers on long sequences (say, 8K or 32K tokens), the activation memory grows dramatically with sequence length. Even with tensor parallelism distributing model parameters and computations across GPUs, each GPU still holds the full sequence's activations for operations like LayerNorm and Dropout that are applied independently per token.
This creates a memory bottleneck that limits how long your sequences can be.
The Core Idea
Sequence Parallelism extends tensor parallelism by partitioning activations along the sequence dimension during operations that don't require cross-token communication.
Think of it like this: if you have a sequence of 4096 tokens spread across 4 GPUs, each GPU only stores activations for 1024 tokens during LayerNorm or Dropout. The memory footprint per GPU shrinks proportionally.
How It Works
- During tensor-parallel attention/MLP: Activations remain partitioned as usual (by tensor parallelism)
- Before LayerNorm/Dropout: Use an
all-gatherto collect full activations, then immediately partition them along the sequence dimension - After LayerNorm/Dropout: Use
reduce-scatterto re-partition back to the tensor-parallel format
The key insight: these "element-wise" operations can be computed on sequence chunks independently, so we don't need every GPU holding the full sequence.
The Win
Combined with tensor parallelism, sequence parallelism enables training on much longer contexts without running out of memory, while adding minimal communication overhead (just the all-gather/reduce-scatter pairs around specific layers).
Key Takeaway: Sequence parallelism partitions activations along the sequence dimension for memory-efficient operations, enabling longer context training by distributing activation memory across GPUs.