Loading and Switching Between Multiple Adapters
What You'll Learn
You'll learn how to load multiple LoRA adapters onto a single base model and switch between them dynamically at runtime without reloading the entire model.
The Core Idea
Think of LoRA adapters like interchangeable camera lenses—you have one camera body (base model), but you can snap different lenses (adapters) on and off to achieve different effects. Each adapter specializes the model for a specific task or domain, and you can switch between them instantly.
Once you've trained multiple LoRA adapters (say, one for customer support, another for technical documentation, and a third for creative writing), you don't want to maintain three separate full models. Instead, you keep one base model in memory and swap lightweight adapters as needed.
How It Works
With the PEFT library, you can load multiple named adapters onto the same model instance. Each adapter is identified by a unique name. At inference time, you tell the model which adapter to use by calling set_adapter().
This approach saves massive amounts of memory since the base model weights (often billions of parameters) are shared, while only the small adapter weights (typically <1% of model size) differ.
The switching happens nearly instantaneously because you're just changing which small weight matrices are active in the forward pass—no model reloading required.
Real-World Use Case
A chatbot serving multiple clients might load client-specific adapters: set_adapter("client_a") for Client A's queries, set_adapter("client_b") for Client B's, all using the same base model in a single service.
Key Takeaway: Loading multiple LoRA adapters on one base model and switching between them with set_adapter() enables multi-task or multi-tenant deployments with minimal memory overhead and zero reload latency.