Post-Mortem Analysis and Root Cause Identification

In this lesson, you will learn how to transition from reactive firefighting to proactive engineering by building a structured, data-driven post-mortem process for your .NET production incidents.

When a critical failure occurs—such as a cascading failure triggered by a circuit breaker or a database connection pool exhaustion—your goal is to identify the "Root Cause," not just the symptom. A professional post-mortem focuses on systemic gaps rather than human error.

The Standard Analysis Procedure

To conduct an effective analysis, follow this four-step standard:

1. Timeline Reconstruction: Use your OpenTelemetry traces and Centralized Log Aggregation data to pinpoint the exact millisecond the incident began.
2. Hypothesis Formulation: Draft a "Why" statement. For example: "The service became unresponsive because the HttpClient factory hit its maximum connection limit under high load."
3. The "5 Whys" Technique: Ask "why" five times to drill down.
   • Why did it crash? Out of memory.
   • Why OOM? Request queue bloat.
   • Why queue bloat? Backend service latency.
   • Why latency? Database lock contention.
   • Why lock contention? Missing indexing on a specific query.
4. Actionable Remediation: Create tasks that prevent the class of the error, not just the specific occurrence.

Data-Driven Verification

When analyzing your .NET code, your telemetry should provide the evidence. If you suspect an issue with your Polly policies, compare your distributed traces against your metrics.

// Example: Correlating a trace ID with a failure log entry
public void LogIncidentContext(string traceId, Exception ex)
{
    // Ensure your logging middleware captures the trace context 
    // to correlate logs with Distributed Tracing spans.
    _logger.LogError(ex, "Incident detected at trace: {TraceId}. Checking circuit breaker state.", traceId);
}

By linking your Distributed Tracing data with your Golden Signals (latency, traffic, errors, and saturation), you can verify if a code change or an infrastructure configuration was the true catalyst. Avoid the trap of blaming "the cloud provider"; instead, identify how your Designing for Failure strategies (like retries or timeouts) failed to contain the blast radius.

Key Takeaway: A successful post-mortem converts production failures into durable architectural safeguards by using observability data to prove the systemic root cause, rather than assigning blame.