LoRA Hyperparameters: Rank and Alpha
What you'll learn: How the rank (r) and alpha scaling factor control LoRA adapter capacity and learning behavior.
Core Idea
When you add LoRA adapters to a model, two key hyperparameters determine how they behave:
Rank (r) controls the capacity of your adapter — essentially how many dimensions it has to learn new patterns. Think of it like choosing between a pencil (r=1, minimal capacity) versus a full paintbrush set (r=64, much more expressive).
Alpha is a scaling factor that controls how strongly the adapter's updates influence the base model. It acts like a learning rate multiplier specifically for your LoRA weights.
How They Work
The rank parameter defines the size of the low-rank decomposition matrices. Lower ranks (r=4, r=8) create tiny adapters that learn focused, narrow adjustments — perfect when your task is close to what the base model already knows. Higher ranks (r=32, r=64) allow more complex learning but increase parameters and training time.
The alpha parameter typically ranges from r to 2×r. The actual scaling applied is alpha/r, so if alpha=r, you get a 1:1 scaling. Higher alpha values amplify the adapter's influence, making learning more aggressive. Lower values make changes more conservative.
The Relationship
These parameters work together: you might use low rank with high alpha for strong but simple adjustments, or high rank with moderate alpha for nuanced learning across many dimensions. The balance depends on your data quality, domain shift, and overfitting risk.
Key Takeaway: Rank controls adapter capacity (expressiveness), while alpha controls how strongly those learned patterns influence the base model — together they're your primary dials for trading off learning power against efficiency and stability.