Setting Up Alerting on Golden Signals

In this lesson, you will learn how to implement alerting logic for the "Four Golden Signals"—Latency, Traffic, Errors, and Saturation—to ensure your .NET services remain healthy in production.

While monitoring tools aggregate data, your code needs to be instrumented to expose these metrics effectively. Using the System.Diagnostics.Metrics API in .NET, you can export these signals to your observability platform.

The Four Golden Signals

1. Latency: The time it takes to service a request.
2. Traffic: A measure of how much demand is being placed on your system.
3. Errors: The rate of requests that fail, either explicitly (500s) or implicitly (200s with wrong content).
4. Saturation: How "full" your service is (e.g., thread pool usage or memory pressure).

Implementing Instrumentation

You should define your metrics as static objects to ensure they are captured throughout the application lifecycle. Here is how you might track request duration and error counts:

using System.Diagnostics.Metrics;

public class MetricsReporter
{
    private static readonly Meter ServiceMeter = new("MyService", "1.0.0");
    private static readonly Histogram<double> Latency = ServiceMeter.CreateHistogram<double>("http_request_duration");
    private static readonly Counter<int> ErrorCount = ServiceMeter.CreateCounter<int>("http_request_errors");

    public void ReportRequest(double durationMs, bool isError)
    {
        Latency.Record(durationMs);
        if (isError) ErrorCount.Add(1);
    }
}

From Metrics to Alerts

Once these metrics reach your monitoring dashboard, you define "Alerting Rules" based on thresholds rather than just raw values:

• Latency: Alert if the 99th percentile (P99) exceeds 500ms for 3 minutes.
• Traffic: Alert on sudden drops (potential load balancer failure) or spikes (potential DDoS).
• Errors: Alert if the error rate exceeds 1% of total traffic.
• Saturation: Alert if the ThreadPool queue length consistently exceeds your threshold.

By focusing on these four signals, you move away from tracking "vanity metrics" and start monitoring the actual user experience. If a service is up but the P99 latency is spiking, your alerts will trigger before users begin reporting outages.

Key Takeaway: Monitoring the Four Golden Signals provides a comprehensive view of system health by prioritizing user-impacting performance and availability over raw infrastructure utilization.