Bidirectional vs Causal Attention
What you'll learn: How encoder and decoder attention mechanisms differ fundamentally in what they can "see."
The Core Difference
In the Transformer architecture, the encoder and decoder use attention differently based on their roles:
-
Encoder uses bidirectional attention: Each token can attend to all other tokens in the input sequence, both before and after its position. This is self-attention without any masking.
-
Decoder uses causal (masked) self-attention: Each token can only attend to previous tokens and itself, never future tokens. This preserves the autoregressive property needed for generation.
Additionally, the decoder uses cross-attention to attend to the encoder's output, looking at the entire encoded source sequence bidirectionally.
Why This Matters
Think of reading comprehension vs. writing:
-
Encoder (bidirectional): Like reading a complete sentence to understand it. You can look at any word in context of all other words—both what came before and after. This builds rich, context-aware representations.
-
Decoder (causal): Like writing a story one word at a time. You can only look back at what you've already written, not peek at future words (because they don't exist yet during generation!).
The Combined Power
The decoder actually uses both types:
- Causal self-attention on its own output so far (can't see the future)
- Cross-attention on the full encoder output (can see everything from the source)
This combination lets the decoder generate sequences autoregressively while still having full access to the input context.
Key Takeaway: Encoders use bidirectional attention to understand complete input sequences, while decoders combine causal self-attention (for autoregressive generation) with cross-attention (to access the encoded source).