Course contentsShow
← AI Engineering
Lesson 1082 of 1,88626. Self-Hosted LLM DeploymentFree lesson

Cost-Performance Trade-offs

Evaluating when to use multi-GPU versus larger single GPUs, considering cost, latency, and throughput.

Cost-Performance Trade-offs

What you'll learn: How to decide between multi-GPU setups and larger single GPUs by balancing cost, latency, and throughput requirements.

The Decision Point

When deploying large language models, you often face a choice: use multiple smaller GPUs (like 2×A10G) or invest in one powerful GPU (like a single A100). Both can work, but the right answer depends on your specific needs and constraints.

Key Factors to Evaluate

Cost

Multiple consumer or mid-tier GPUs may cost less upfront than a single datacenter GPU. However, cloud pricing often favors single larger instances due to reduced networking overhead. Always calculate total cost per 1000 tokens rather than just hardware price.

Latency

Single GPU setups typically offer lower latency because there's no inter-GPU communication overhead. Multi-GPU configurations using tensor or pipeline parallelism add milliseconds of synchronization time between devices—critical if you're serving real-time applications.

Throughput

Multi-GPU shines for high throughput scenarios. By distributing batches across devices, you can serve more concurrent users. If your workload involves continuous batching with many parallel requests, multi-GPU often wins despite higher per-request latency.

The Analogy

Think of it like delivery services: one large truck (single GPU) delivers faster point-to-point, while multiple smaller vans (multi-GPU) handle more total packages per hour across different routes—but each van needs coordination time at the depot.

Making the Choice

  • Low latency, moderate load? → Single larger GPU
  • High concurrent users, batch-friendly? → Multi-GPU
  • Budget-constrained? → Compare cloud spot pricing for both configurations
  • Model won't fit? → Multi-GPU becomes mandatory

Key Takeaway: Choose single GPUs for latency-critical applications and multi-GPU for throughput-heavy workloads, always benchmarking actual cost-per-token in your target environment.