Adapter Layers: Architecture and Motivation
What you'll learn: How adapter modules enable efficient fine-tuning by inserting small trainable bottleneck layers into frozen pretrained models.
The Core Problem
After covering LoRA and QLoRA, you've seen parameter-efficient methods that modify existing weight matrices. Adapter layers take a different architectural approach: they insert entirely new small modules between frozen pretrained layers rather than modifying weights in-place.
How Adapters Work
Think of adapters like adding expansion joints to a rigid pipeline. The original model remains completely frozen, but you insert tiny trainable "detours" that learn task-specific transformations:
- Down-projection: Compress the layer's output from dimension
dto bottleneck dimensionr(wherer << d) - Non-linearity: Apply an activation function (typically ReLU or GELU)
- Up-projection: Expand back from
rto original dimensiond - Residual connection: Add the adapter output to the original frozen layer output
The bottleneck dimension r might be 64 or 128, while d could be 4096—training only ~1% of parameters.
Why This Architecture Matters
Unlike LoRA's rank decomposition approach, adapters create independent learned pathways through the network. The residual connection ensures the original model's knowledge is preserved (if adapter weights initialize to zero, the model starts unchanged). Each adapter can specialize to your task without touching pretrained weights.
The modular design means you can train multiple adapters for different tasks and swap them at inference time—similar to LoRA's multi-adapter capability but with different architectural constraints.
Key Takeaway: Adapter layers achieve parameter efficiency by inserting small trainable bottleneck modules between frozen layers with residual connections, offering a modular alternative to weight-modification methods like LoRA.