Implementing Adapters in Transformer Blocks
What you'll learn: Where to place adapter layers within transformer blocks and how to initialize them for stable, effective fine-tuning.
Core Concept
Adapter layers are small bottleneck modules inserted into existing transformer blocks. But where exactly do they go, and how do you set them up so they don't disrupt the pre-trained model?
Placement locations:
- After the multi-head attention sublayer (before residual addition)
- After the feedforward network sublayer (before residual addition)
Each adapter typically appears twice per transformer block—once in the attention path, once in the feedforward path. They sit within the residual stream, so their output gets added back to the skip connection.
Initialization Strategy
The critical trick: initialize adapters near-zero. If you start with random weights, the first forward pass produces garbage that corrupts the pre-trained representations. Instead:
- Initialize the down-projection layer normally (e.g., Xavier/Kaiming)
- Initialize the up-projection layer to zeros or very small values
This ensures the adapter initially acts as an identity function (adds nearly nothing), preserving the original model behavior. During training, the adapter gradually learns useful task-specific transformations.
Think of it like adding a detour route to a highway: at first, the detour is closed (zero contribution), then it slowly opens as traffic patterns (gradients) show it's useful.
Typical Flow
Input → LayerNorm → Attention → Adapter₁ → Add & Norm → FFN → Adapter₂ → Add & Norm → Output
Each adapter processes the sublayer's output before it merges with the residual connection.
Key Takeaway: Adapter layers are inserted after attention and feedforward sublayers within the residual path, initialized near-zero to preserve pre-trained behavior before training begins.