Comparing Performance with BenchmarkDotNet

In this lesson, you will learn how to use BenchmarkDotNet to generate statistically significant performance data to validate your code optimizations.

When you optimize your C# code—perhaps by applying techniques like object pooling or replacing heap allocations with Span<T>—you need to move beyond "gut feeling" measurements like Stopwatch. Relying on a single run is dangerous because background processes, JIT compilation, and CPU thermal throttling create "noise" that masks the truth. BenchmarkDotNet acts as a scientific laboratory for your code.

Setting Up a Benchmarking Class

To create a benchmark, you define a class where each method representing a test scenario is decorated with the [Benchmark] attribute. You must also decorate the class with [MemoryDiagnoser] to track allocations, which is vital given our previous work with the Large Object Heap and garbage collection overhead.

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

[MemoryDiagnoser]
public class StringConcatenationBenchmark
{
    [Benchmark(Baseline = true)]
    public string UsingPlusOperator() => "a" + "b" + "c";

    [Benchmark]
    public string UsingStringBuilder() => 
        new System.Text.StringBuilder().Append("a").Append("b").Append("c").ToString();
}

Understanding Statistical Rigor

When you run your project in Release configuration, BenchmarkDotNet does the heavy lifting:

1. Warmup: It runs the code multiple times to ensure the JIT compiler has fully optimized the methods.
2. Pilot: It determines how many iterations are necessary to get a stable result.
3. Actual Benchmark: It executes the code across multiple iterations, measuring execution time and memory pressure.

The final report provides columns like Mean (the average time taken), Error (the margin of error), and Gen 0/1/2 (how often your code triggers GC collections). A statistically significant optimization is one where the Mean value is consistently lower than the baseline, and the Error range does not overlap with your previous measurements. If the ranges overlap, your "optimization" might just be statistical noise rather than a genuine performance gain.

Key Takeaway: Always use BenchmarkDotNet in Release mode to generate statistically stable metrics, as anecdotal measurements using Stopwatch are prone to interference from JIT compilation and environmental background noise.