Merging and Deploying QLoRA Adapters
What You'll Learn
You'll understand the three main strategies for serving QLoRA-adapted models in production: keeping adapters separate, merging back to full precision, or deploying quantized merged models.
Three Deployment Paths
After training QLoRA adapters, you face a critical decision: how to deploy your fine-tuned model?
Option 1: Keep Adapters Separate
Load the quantized base model plus adapter weights at inference time. This approach minimizes memory usage (still 4-bit) and allows hot-swapping between multiple task-specific adapters without reloading the base model. Perfect for serving many specialized variants.
Trade-off: Slight computational overhead from applying low-rank updates during forward passes.
Option 2: Merge to Full Precision
Combine your adapter weights back into the base model at full precision (FP16/BF16). Think of it like "baking" the adapter's modifications permanently into the original layers.
# Conceptual merging
merged_model = base_model + (adapter_A @ adapter_B)
Benefits: Fastest inference, no adapter overhead, standard deployment.
Cost: Returns to full memory footprint (~13GB for 7B models).
Option 3: Merge Then Re-Quantize
First merge adapters into full precision, then quantize the entire result back to 4-bit for deployment. You get permanent improvements from fine-tuning while maintaining quantized efficiency.
Sweet spot: Balances memory efficiency with inference speed, though re-quantization adds slight quality loss.
Choosing Your Strategy
- Multiple tasks? → Keep adapters separate
- Single production deployment? → Merge to full precision
- Memory-constrained serving? → Merge and re-quantize
- Fastest inference needed? → Merge to full precision
Key Takeaway: QLoRA adapters offer deployment flexibility—keep them modular for multi-task serving, merge for maximum speed, or merge-and-quantize for memory-efficient production with permanent improvements baked in.