Course contentsShow
← Machine Learning and Deep Learning
Lesson 1883 of 3,53840. Advanced Prompting TechniquesFree lesson

Cost-Performance Trade-offs

Balancing the number of samples against inference cost and latency to determine optimal sampling budget.

Cost-Performance Trade-offs

What you'll learn: How to balance the number of self-consistency samples against inference cost and latency to find the optimal sampling budget.

The Central Challenge

Self-consistency improves reasoning quality by generating multiple reasoning paths and voting on the final answer. But here's the catch: each additional sample costs money (API tokens) and time (inference latency). Your job is to find the sweet spot where accuracy gains justify the extra expense.

Understanding the Trade-off Curve

Think of self-consistency like polling voters before an election. Asking 3 people is cheap but unreliable. Asking 1,000 people is expensive but accurate. The magic happens somewhere in between—often around 5-20 samples for most LLM reasoning tasks.

The pattern:

  • 1→5 samples: Steep accuracy improvements (biggest bang for buck)
  • 5→20 samples: Diminishing returns begin
  • 20+ samples: Minimal gains, costs skyrocket

Calculating Your Budget

Consider these factors:

Cost considerations:

  • Token cost per generation (input + output)
  • Number of queries per day/month
  • Budget constraints

Performance requirements:

  • Acceptable error rate for your application
  • User tolerance for latency
  • Whether accuracy is mission-critical

Example scenario: If moving from 1 to 5 samples improves accuracy from 70% to 85% but costs 5× more, and 10 samples only reaches 87% at 10× cost, the optimal choice is likely 5 samples.

Practical Decision Framework

Test your specific task with different sample counts (1, 3, 5, 10, 20) on a validation set. Plot accuracy vs. cost. Choose the point where the accuracy curve flattens but before costs become prohibitive.

Key Takeaway: Optimal self-consistency sampling typically falls between 5-20 samples, where you capture most accuracy gains before hitting diminishing returns—but always validate on your specific task and budget constraints.