Course contentsShow
← AI Engineering
Lesson 63 of 1,8862. Working with Pre-trained ModelsFree lesson

CPU vs GPU Inference Trade-offs

When to use CPU vs GPU for inference based on model size, batch size, and latency requirements.

CPU vs GPU Inference Trade-offs

You'll learn: When to choose CPU or GPU for running inference based on your model's characteristics and requirements.

The Core Decision

Think of CPUs and GPUs like different vehicles: a CPU is a versatile car that handles various tasks well, while a GPU is a high-speed train designed specifically for parallel computation. For inference, your choice depends on three key factors:

Model Size: Small models (under ~500MB) often run efficiently on CPUs without justifying GPU costs. Large models like multi-billion parameter transformers need GPU acceleration to avoid unacceptably slow response times.

Batch Size: Processing one request at a time? CPUs excel here with lower overhead and faster startup. Processing 32+ requests simultaneously? GPUs shine because they parallelize computations across many inputs at once, amortizing their setup cost.

Latency Requirements: Need responses in under 50ms for real-time applications? CPUs often win for small models due to lower overhead—GPUs add latency transferring data to/from device memory. Can you tolerate 200-500ms? GPUs become viable and cost-effective at scale.

Practical Guidelines

  • Use CPU when: Model is small, handling single/few requests, latency must be minimal, or GPU costs aren't justified by throughput
  • Use GPU when: Model is large (>1GB), processing batches of 8+, total throughput matters more than per-request latency, or doing continuous high-volume inference

Remember: you measured inference performance in the previous lesson—those metrics directly inform this decision. A small distilled model serving one user at a time wastes money on GPU infrastructure.

Key Takeaway: Choose CPU for small models with low-batch, latency-sensitive workloads; choose GPU for large models with high-batch, throughput-focused workloads—measure your specific performance needs to decide.