In this lesson, you will learn how to validate that performance improvements are statistically meaningful rather than just noise by performing differential analysis between two code revisions.
Understanding Differential Performance Testing
When you apply optimizations—like loop tiling or cache-aware data layout—the resulting speedup is often small enough that it can be masked by background OS noise, thermal throttling, or CPU frequency scaling. Differential performance testing involves running benchmarks against two distinct revisions (e.g., main vs. feature/optimized) and applying statistical tests to ensure the difference isn't due to random variance.
Beyond Simple Averages
Comparing simple arithmetic means of benchmark runs is dangerous because execution time is rarely normally distributed; it often contains outliers caused by context switches or background interrupts. Instead, we use tools like criterion.rs to compute the probability distribution of execution times for both revisions.
Consider a case where you optimize a function’s memory access pattern:
// Revision A: Baseline
fn process_data(data: &[u64]) { /* ... */ }
// Revision B: Optimized with alignment and prefetching
fn process_data_opt(data: &[u64]) { /* ... */ }
When comparing these, we calculate the Confidence Interval (CI) for the ratio of the two distributions. If the 95% confidence interval for the execution time difference includes zero (or a ratio of 1.0), the performance change is not statistically significant.
Applying Statistical Significance
To perform this correctly, your CI setup should:
- Warm up: Eliminate cold-start cache effects so you are measuring the "hot" execution path.
- Increase Iterations: Run enough samples to narrow the distribution, ensuring the standard deviation is small enough to detect the expected performance delta.
- Analyze the Regression: Look for a "performance regression" (where the CI shifts toward a slower execution time) or "improvement" (where the CI shifts toward a faster one).
If the overlap between the probability density functions of the two versions is significant, you cannot claim a performance gain, even if the "average" appears slightly lower. Differential testing forces you to trust only the data that stands out above the noise floor.
Key Takeaway: Differential performance testing requires comparing the probability distributions of benchmarks between revisions to ensure that measured gains are statistically significant and not artifacts of system noise.