Course contentsShow
← AI Engineering
Lesson 1373 of 1,88633. Parameter-Efficient Fine-Tuning (PEFT)Free lesson

Batching Across Adapters

Techniques for batching requests with different adapters while maintaining inference efficiency.

Batching Across Adapters

What you'll learn: How to efficiently batch inference requests that use different LoRA adapters without sacrificing throughput or latency.

The Core Challenge

When serving multiple LoRA adapters (as you learned in multi-adapter serving), naive batching hits a wall: traditional batching assumes all requests use the same model weights. But with adapters, Request A might need customer-support adapter while Request B needs code-generation adapter. Processing them separately wastes GPU cycles and kills throughput.

How Heterogeneous Batching Works

Modern serving systems solve this by batching at the base model level while keeping adapter computations separate:

  1. Shared base computation: All requests in a batch pass through the base model's layers together (same matrix multiplications)
  2. Per-adapter deltas: At each LoRA-enabled layer, compute the low-rank updates separately for each adapter group
  3. Merge results: Add the appropriate adapter deltas back to each request's activations

Analogy: Imagine a restaurant kitchen where the base ingredients (pasta, sauce) are prepared in bulk, but each dish gets its unique topping (adapter) applied individually at the end. You get bulk-cooking efficiency while serving customized meals.

Technical Considerations

  • Memory layout: Pre-load all active adapters into GPU memory
  • Adapter grouping: Cluster requests by adapter when possible to minimize compute branches
  • Padding strategies: Pad sequences within adapter groups, not across the entire batch
  • Scheduling: Balance between waiting for same-adapter requests vs. dispatching heterogeneous batches

Modern frameworks like vLLM and TGI implement this automatically using techniques like "continuous batching" with adapter-aware scheduling.

Key Takeaway: Batching across adapters maintains high throughput by processing the shared base model computation together while applying adapter-specific deltas separately, giving you the best of both worlds: efficiency and customization.