Multi-Model Serving Considerations
What you'll learn: How to plan hardware capacity and strategies when serving multiple LLMs or switching models dynamically based on workload.
The Challenge of Multiple Models
When self-hosting LLMs, you rarely serve just one model forever. You might run different models for different tasks (a small model for classification, a large one for generation), support multiple versions during upgrades, or swap models based on time-of-day demand patterns. Each scenario requires careful hardware planning.
Planning Hardware Capacity
Think of your GPU memory like a shared apartment. If you want roommates (multiple models), you need to know:
Memory footprint: A 7B model at FP16 needs ~14GB. Two such models need 28GB+ (plus KV cache overhead). Check if they fit simultaneously or require swapping.
Loading time cost: Swapping a 13B model from disk to GPU can take 30-60 seconds. Frequent swaps create user-facing latency. Consider keeping hot models resident.
Workload patterns: If 80% of requests hit Model A during business hours and Model B overnight, you might load/unload on schedule rather than keeping both loaded.
Common Strategies
- Dedicated instances: Run each model on separate hardware (simple but expensive)
- Sequential serving: Load one model at a time, swap on demand (cost-effective, slower switching)
- Memory-efficient multi-tenancy: Use quantization to fit multiple smaller models together
- Router pattern: Front-end service routes requests to model-specific backends
Practical Constraints
GPU memory is your limiting factor. A single A100 (40GB) might handle two 7B models or one 13B + several tiny specialized models. Always account for KV cache (typically 20-40% of total memory during serving) and batching overhead.
Key Takeaway: Multi-model serving requires balancing memory capacity, loading latency, and workload patterns—plan hardware by mapping your model portfolio to available GPU memory while accounting for runtime overhead.