Attention as Explicit Relationship Modeling
What you'll learn: How attention mechanisms explicitly model relationships between tokens, unlike RNNs that encode relationships implicitly in hidden states.
The Core Difference
When an RNN processes a sequence, it compresses all the relationships and dependencies between tokens into a single hidden state vector. This hidden state is like a black box—you know it contains information about previous tokens, but you can't see which tokens are important or how they relate to the current position.
Attention changes this fundamentally.
Explicit vs Implicit Relationships
RNNs (Implicit): The hidden state at position 5 contains some encoded mixture of all previous tokens. The relationships are entangled and hidden inside the vector's numbers. You can't easily inspect which earlier token matters most.
Attention (Explicit): The attention weight matrix gives you a clear, interpretable map. If position 5 has a weight of 0.7 to position 2 and 0.1 to position 3, you can see that token 2 has a strong relationship with token 5.
Why This Matters
Think of it like a conversation. An RNN is like someone who takes mental notes but can't show you their notebook—they just respond based on their overall impression. Attention is like someone who highlights exactly which previous statements they're responding to, showing you the precise connections.
This explicitness means:
- Interpretability: You can visualize what the model attends to
- Precision: Different tokens can form different relationships without interference
- Directness: Information flows directly between related tokens, not through a compressed bottleneck
Key Takeaway: Attention weights explicitly reveal which tokens relate to each other and how strongly, transforming relationship modeling from an implicit hidden state encoding into a transparent, inspectable matrix of connections.