Course contentsShow
← Machine Learning and Deep Learning
Lesson 2917 of 3,53863. Model Serving and Inference InfrastructureFree lesson

Batch Size Selection and Timeout Configuration

Determining optimal batch sizes and timeout windows based on traffic patterns, SLA requirements, and hardware constraints.

Batch Size Selection and Timeout Configuration

What You'll Learn

You'll learn how to determine the optimal batch size and timeout windows for your inference server by balancing traffic patterns, SLA requirements, and hardware capabilities.

The Core Challenge

When implementing dynamic batching (which you've already learned collects multiple requests into a single inference batch), you face two critical configuration decisions:

Batch Size: How many requests should we accumulate before processing? Timeout: How long should we wait for a batch to fill before processing it anyway?

These parameters directly control your latency-throughput trade-off.

Finding the Sweet Spot

Think of it like a rideshare service: You want to fill the car (maximize throughput), but passengers can't wait forever (meet latency SLAs).

Batch Size Selection

Your maximum batch size should be determined by:

  • Hardware memory limits: GPU memory constrains how many samples fit simultaneously
  • Model architecture: Transformer models scale differently than CNNs
  • Throughput targets: Larger batches maximize GPU utilization
  • SLA requirements: Bigger batches mean some requests wait longer

Start by testing your model's memory footprint: gradually increase batch size until you hit OOM errors, then back off 20-30% for safety.

Timeout Configuration

Your timeout window answers: "How long do we wait for a batch to fill?"

  • Low traffic: Short timeouts prevent requests from waiting unnecessarily
  • High traffic: Longer timeouts allow batches to fill completely
  • P99 latency SLA: Timeout must be significantly less than your SLA budget

Example thinking: If your SLA is 100ms and inference takes 40ms, your maximum safe timeout is ~50ms (leaving margin for networking and postprocessing).

Adaptive Strategies

Advanced systems adjust parameters based on observed traffic patterns—smaller batches and shorter timeouts during quiet periods, larger configurations during peak load.

Key Takeaway: Optimal batch size balances hardware capacity with throughput goals, while timeout configuration ensures you meet latency SLAs even when traffic is insufficient to fill batches naturally. Test empirically under realistic load patterns.