The Encoder Stack
What you'll learn: How the Transformer encoder builds deep representations by stacking multiple identical layers that process input through self-attention and feed-forward networks.
The Core Architecture
Think of the encoder stack like a multi-story building where each floor refines and enriches information from the floor below. In Transformers, the encoder stack consists of N identical layers (typically 6-12) stacked on top of each other.
Each encoder layer contains two main sub-components:
1. Multi-Head Self-Attention
The input sequence attends to itself, capturing relationships between all positions. Each token learns contextual information from every other token in the sequence.
2. Position-wise Feed-Forward Network
After attention, each position independently passes through the same feed-forward network (two linear transformations with a non-linearity between them). This processes the attended information and adds representational capacity.
How Information Flows
The input embeddings (with positional encodings added) enter the first encoder layer. Each layer transforms its input and passes the output to the next layer:
- Layer 1 learns basic patterns and relationships
- Layer 2 builds on those patterns, finding more complex dependencies
- Deeper layers capture increasingly abstract representations
Crucially, each sub-component uses residual connections (adding the input back to the output) and layer normalization to stabilize training and enable deep stacking.
Why Stack Multiple Layers?
Just like deep CNNs learn hierarchical features (edges → textures → objects), stacked encoders learn hierarchical sequence patterns. Early layers might capture local syntax, while deeper layers understand semantic relationships and global context.
Key Takeaway: The encoder stack transforms input sequences through multiple layers of self-attention and feed-forward processing, with each layer building progressively richer representations that capture both local and global dependencies.