Course contentsShow
← Machine Learning and Deep Learning
Lesson 1104 of 3,53824. The Transformer ArchitectureFree lesson

Bidirectional vs Causal Attention

Encoder uses bidirectional attention; decoder combines causal self-attention with cross-attention.

Bidirectional vs Causal Attention

What you'll learn: How encoder and decoder attention mechanisms differ fundamentally in what they can "see."

The Core Difference

In the Transformer architecture, the encoder and decoder use attention differently based on their roles:

  • Encoder uses bidirectional attention: Each token can attend to all other tokens in the input sequence, both before and after its position. This is self-attention without any masking.

  • Decoder uses causal (masked) self-attention: Each token can only attend to previous tokens and itself, never future tokens. This preserves the autoregressive property needed for generation.

Additionally, the decoder uses cross-attention to attend to the encoder's output, looking at the entire encoded source sequence bidirectionally.

Why This Matters

Think of reading comprehension vs. writing:

  • Encoder (bidirectional): Like reading a complete sentence to understand it. You can look at any word in context of all other words—both what came before and after. This builds rich, context-aware representations.

  • Decoder (causal): Like writing a story one word at a time. You can only look back at what you've already written, not peek at future words (because they don't exist yet during generation!).

The Combined Power

The decoder actually uses both types:

  1. Causal self-attention on its own output so far (can't see the future)
  2. Cross-attention on the full encoder output (can see everything from the source)

This combination lets the decoder generate sequences autoregressively while still having full access to the input context.

Key Takeaway: Encoders use bidirectional attention to understand complete input sequences, while decoders combine causal self-attention (for autoregressive generation) with cross-attention (to access the encoded source).