Choosing the Rank r in LoRA
What you'll learn: How the rank r in LoRA controls the balance between model expressiveness and parameter efficiency.
The Core Trade-off
When using LoRA, you decompose weight updates as W_update = A × B where A is d×r and B is r×k. The rank r is the bottleneck dimension connecting these two matrices, and it fundamentally controls two competing forces:
- Expressiveness: Higher
rmeans the low-rank approximation can capture more complex patterns in the weight updates - Parameter count: Higher
rmeans more trainable parameters (since both A and B grow withr)
Understanding the Impact
Think of r like the resolution of a photograph. A low-rank approximation with r=4 is like a highly compressed image—it captures the broad strokes but misses fine details. With r=64, you get much more detail, but the file size (parameter count) increases proportionally.
For a single weight matrix of dimension 4096×4096:
- Full fine-tuning: ~16.7M parameters
- LoRA with
r=8: ~65K parameters (256× reduction) - LoRA with
r=64: ~524K parameters (32× reduction)
Practical Guidelines
Typical values range from r=4 to r=64. Smaller models or simple adaptation tasks work well with lower ranks. Complex domain shifts or larger models may benefit from higher ranks. The key insight: most weight updates during fine-tuning are inherently low-rank, so even small r values (like 8 or 16) often achieve 90%+ of full fine-tuning performance while using <1% of the parameters.
Key Takeaway: Rank r is your tuning knob for the expressiveness-efficiency trade-off—start low (r=8-16) and increase only if task performance demands it.