Course contentsShow
← System Design
Lesson 1302 of 1,91932. Monitoring and AlertingFree lesson

Triaging and Root Cause Analysis During Incidents

Quickly isolate failing components, form hypotheses, test theories, and focus on mitigation before deep root cause investigation.

Triaging and Root Cause Analysis During Incidents

You'll learn: How to rapidly narrow down the failing component during a live incident, form testable hypotheses, and prioritize mitigation over perfect understanding.

The Burning House Analogy

When your house is on fire, you don't stop to investigate how the fire started—you grab the extinguisher first. Incident triage works the same way: mitigation beats investigation during active outages.

Triage: Isolate the Blast Radius

Your first job is to quickly identify what's broken and what still works. Use your monitoring dashboards (the ones you built earlier) to answer:

  • Which service is failing? (Check error rates, latency spikes)
  • Is this isolated to one region, customer, or global?
  • When did it start? (Correlate with recent deployments or config changes)

Think of this as "divide and conquer"—eliminate suspects fast. If your checkout service is timing out, but the payment gateway metrics look healthy, you've just narrowed your search.

Hypothesis-Driven Debugging

Don't randomly poke at systems. Form quick, testable theories:

  1. "Did we just deploy?" → Check recent changes
  2. "Is a dependency down?" → Verify external service health
  3. "Are we out of resources?" → Check CPU, memory, saturation metrics

Test the most likely hypothesis first (often recent changes). Each test should take seconds to minutes, not hours.

Mitigation First, Root Cause Later

Your goal during the incident is stopping the bleeding, not writing a PhD thesis. Roll back the deployment, restart the service, failover to a backup region—whatever stops customer impact fastest.

Deep root cause analysis happens in the blameless post-mortem, after systems are stable and you can think clearly with full logs and traces.

Key Takeaway: During incidents, rapidly isolate failing components using symptoms from your monitoring stack, test the most probable hypotheses first, and always prioritize customer mitigation over understanding every detail—comprehensive root cause analysis comes after the fire is out.