Course contentsShow
← Machine Learning and Deep Learning
Lesson 1717 of 3,53837. LLM Fine-Tuning: Full and Parameter-EfficientFree lesson

LoRA Scaling Factor Alpha

Understanding the alpha hyperparameter and how it scales the contribution of LoRA updates.

LoRA Scaling Factor Alpha

What you'll learn: How the alpha hyperparameter controls the magnitude of LoRA's contribution to your model's behavior.

The Scaling Problem

When you add low-rank updates to frozen weights, you need to control how much those updates affect the final model. This is where alpha comes in—it's a scaling factor that determines the strength of your LoRA modifications.

How Alpha Works

Recall from earlier lessons that LoRA adds a low-rank update BA to frozen weights W₀. The actual formula includes alpha:

W = W₀ + (alpha/r) * BA

Where:

  • alpha is the scaling hyperparameter you set
  • r is the rank you chose for your low-rank matrices
  • The ratio alpha/r determines the final scaling

Why Scale by Rank?

Think of it like volume control on different speaker systems. A small rank (r=4) means fewer parameters learning the update—you might want to amplify their effect. A large rank (r=64) has more parameters contributing, so the same alpha value automatically scales down per-parameter via the division.

The typical pattern: Set alpha = r or alpha = 2*r. When alpha = r, the scaling factor becomes 1, giving you a "neutral" baseline. When alpha = 2*r, you're amplifying the LoRA contribution by 2x.

Practical Impact

Higher alpha → stronger LoRA influence → faster adaptation but risk of overfitting
Lower alpha → gentler updates → more stable but potentially underfitting

Key Takeaway: Alpha controls how loudly your low-rank updates "speak" relative to the frozen pretrained weights, with the alpha/r ratio providing rank-normalized scaling that keeps training stable across different rank choices.