IA³: Infused Adapter by Inhibiting and Amplifying
What You'll Learn
You'll discover how IA³ achieves parameter-efficient fine-tuning with even fewer trainable parameters than LoRA by learning scalar vectors that rescale activation flows.
The Core Idea
While LoRA adds low-rank matrices to weight updates, IA³ (pronounced "I-A-cubed") takes a radically simpler approach: it learns small vectors that multiply (scale) the activations flowing through the network. Think of it as installing volume knobs at strategic points in your model—some turned up to amplify useful signals, others turned down to inhibit less relevant features.
IA³ introduces three learned vectors:
- Key scaling (
l_k): scales attention keys - Value scaling (
l_v): scales attention values - Feedforward scaling (
l_ff): scales feedforward activations
During fine-tuning, these vectors learn which activations to amplify (values > 1) and which to inhibit (values < 1). The base model weights remain completely frozen—only these lightweight scaling vectors update.
Why So Efficient?
IA³ typically requires 10,000× fewer parameters than full fine-tuning and often fewer than LoRA. If LoRA adds matrices of size d × r, IA³ only adds vectors of size d. For a 7B parameter model, IA³ might introduce only 0.01% trainable parameters.
The magic is in where the scaling happens: at activation bottlenecks where information flow is most critical. By rescaling rather than adding, IA³ modulates existing knowledge rather than injecting new patterns.
The Trade-off
IA³ works exceptionally well for tasks that require adapting what the model already knows (like domain adaptation or instruction following). For tasks requiring genuinely new knowledge, LoRA's expressiveness may still win.
Key Takeaway: IA³ achieves extreme parameter efficiency by learning lightweight scaling vectors that inhibit or amplify activations at strategic points, making it ideal when you want minimal overhead with solid task adaptation.