Course contentsShow
← AI Engineering
Lesson 1070 of 1,88626. Self-Hosted LLM DeploymentFree lesson

Multi-Model Serving Considerations

Plan hardware capacity for serving multiple models or swapping models based on workload patterns.

Multi-Model Serving Considerations

What you'll learn: How to plan hardware capacity and strategies when serving multiple LLMs or switching models dynamically based on workload.

The Challenge of Multiple Models

When self-hosting LLMs, you rarely serve just one model forever. You might run different models for different tasks (a small model for classification, a large one for generation), support multiple versions during upgrades, or swap models based on time-of-day demand patterns. Each scenario requires careful hardware planning.

Planning Hardware Capacity

Think of your GPU memory like a shared apartment. If you want roommates (multiple models), you need to know:

Memory footprint: A 7B model at FP16 needs ~14GB. Two such models need 28GB+ (plus KV cache overhead). Check if they fit simultaneously or require swapping.

Loading time cost: Swapping a 13B model from disk to GPU can take 30-60 seconds. Frequent swaps create user-facing latency. Consider keeping hot models resident.

Workload patterns: If 80% of requests hit Model A during business hours and Model B overnight, you might load/unload on schedule rather than keeping both loaded.

Common Strategies

  • Dedicated instances: Run each model on separate hardware (simple but expensive)
  • Sequential serving: Load one model at a time, swap on demand (cost-effective, slower switching)
  • Memory-efficient multi-tenancy: Use quantization to fit multiple smaller models together
  • Router pattern: Front-end service routes requests to model-specific backends

Practical Constraints

GPU memory is your limiting factor. A single A100 (40GB) might handle two 7B models or one 13B + several tiny specialized models. Always account for KV cache (typically 20-40% of total memory during serving) and batching overhead.

Key Takeaway: Multi-model serving requires balancing memory capacity, loading latency, and workload patterns—plan hardware by mapping your model portfolio to available GPU memory while accounting for runtime overhead.