LoRA Scaling Factor Alpha
What you'll learn: How the alpha hyperparameter controls the magnitude of LoRA's contribution to your model's behavior.
The Scaling Problem
When you add low-rank updates to frozen weights, you need to control how much those updates affect the final model. This is where alpha comes in—it's a scaling factor that determines the strength of your LoRA modifications.
How Alpha Works
Recall from earlier lessons that LoRA adds a low-rank update BA to frozen weights W₀. The actual formula includes alpha:
W = W₀ + (alpha/r) * BA
Where:
alphais the scaling hyperparameter you setris the rank you chose for your low-rank matrices- The ratio
alpha/rdetermines the final scaling
Why Scale by Rank?
Think of it like volume control on different speaker systems. A small rank (r=4) means fewer parameters learning the update—you might want to amplify their effect. A large rank (r=64) has more parameters contributing, so the same alpha value automatically scales down per-parameter via the division.
The typical pattern: Set alpha = r or alpha = 2*r. When alpha = r, the scaling factor becomes 1, giving you a "neutral" baseline. When alpha = 2*r, you're amplifying the LoRA contribution by 2x.
Practical Impact
Higher alpha → stronger LoRA influence → faster adaptation but risk of overfitting
Lower alpha → gentler updates → more stable but potentially underfitting
Key Takeaway: Alpha controls how loudly your low-rank updates "speak" relative to the frozen pretrained weights, with the alpha/r ratio providing rank-normalized scaling that keeps training stable across different rank choices.