Parallelization: The Core Advantage
What You'll Learn
You'll understand how self-attention allows Transformers to train massively faster than RNNs by processing all sequence positions simultaneously.
The Sequential Bottleneck in RNNs
In RNNs (including LSTMs and GRUs), processing happens step-by-step. To compute the hidden state at position 5, you must first compute positions 1, 2, 3, and 4. This creates a sequential dependency chain that prevents parallelization during training.
Analogy: Imagine reading a book where you must finish page 1 before starting page 2, page 2 before page 3, and so on. You can't parallelize this task across multiple readers because each page depends on the previous one.
How Self-Attention Breaks the Chain
Self-attention computes attention scores between all positions simultaneously. Each position's representation can be computed independently using the attention mechanism because:
- Queries, keys, and values are computed from all input embeddings at once
- The attention score matrix is calculated in a single matrix operation
- All weighted sums happen in parallel
Analogy: It's like giving the entire book to multiple readers at once, where each reader can examine any page and reference any other page simultaneously. There's no forced sequential order.
The Training Speed Revolution
During training, when you have the full input sequence available, Transformers can:
- Process all tokens in parallel across the sequence length
- Compute all attention weights simultaneously using matrix operations
- Leverage GPU parallelism efficiently with matrix multiplications
RNNs must process one timestep at a time, making them dramatically slower to train on long sequences.
Key Takeaway: Self-attention eliminates RNN's sequential bottleneck by allowing all positions to attend to each other simultaneously, enabling full parallelization during training and making Transformers orders of magnitude faster on modern hardware.