Course contentsShow
← Machine Learning and Deep Learning
Lesson 1715 of 3,53837. LLM Fine-Tuning: Full and Parameter-EfficientFree lesson

Choosing the Rank r in LoRA

How rank r controls the expressiveness and parameter count trade-off in LoRA adapters.

Choosing the Rank r in LoRA

What you'll learn: How the rank r in LoRA controls the balance between model expressiveness and parameter efficiency.

The Core Trade-off

When using LoRA, you decompose weight updates as W_update = A × B where A is d×r and B is r×k. The rank r is the bottleneck dimension connecting these two matrices, and it fundamentally controls two competing forces:

  1. Expressiveness: Higher r means the low-rank approximation can capture more complex patterns in the weight updates
  2. Parameter count: Higher r means more trainable parameters (since both A and B grow with r)

Understanding the Impact

Think of r like the resolution of a photograph. A low-rank approximation with r=4 is like a highly compressed image—it captures the broad strokes but misses fine details. With r=64, you get much more detail, but the file size (parameter count) increases proportionally.

For a single weight matrix of dimension 4096×4096:

  • Full fine-tuning: ~16.7M parameters
  • LoRA with r=8: ~65K parameters (256× reduction)
  • LoRA with r=64: ~524K parameters (32× reduction)

Practical Guidelines

Typical values range from r=4 to r=64. Smaller models or simple adaptation tasks work well with lower ranks. Complex domain shifts or larger models may benefit from higher ranks. The key insight: most weight updates during fine-tuning are inherently low-rank, so even small r values (like 8 or 16) often achieve 90%+ of full fine-tuning performance while using <1% of the parameters.

Key Takeaway: Rank r is your tuning knob for the expressiveness-efficiency trade-off—start low (r=8-16) and increase only if task performance demands it.