Course contentsShow
← System Design
Lesson 1303 of 1,91932. Monitoring and AlertingFree lesson

Incident Mitigation vs Fix

Distinguish between immediate mitigation (traffic rerouting, rollback) and permanent fixes; prioritize restoring service first.

Incident Mitigation vs Fix

What you'll learn: How to distinguish between quickly restoring service and permanently solving the underlying problem during an incident.

When an incident strikes, you face two distinct goals: mitigation (stop the bleeding now) and fix (cure the disease permanently). Understanding this difference is critical for effective incident response.

Mitigation: Restore Service First

Mitigation focuses on one thing: getting users back online as quickly as possible, even if the solution is temporary or incomplete. Common mitigation tactics include:

  • Traffic rerouting: Redirect requests away from failing components to healthy ones
  • Rollback: Revert to the last known good deployment
  • Feature flags: Disable problematic features immediately
  • Rate limiting: Reduce load on struggling systems
  • Cache extension: Serve stale data temporarily

Think of mitigation like using a tourniquet on a bleeding wound—it's not healing the injury, but it's keeping the patient alive.

The Fix: Solve It Properly

After service is restored, you investigate the root cause and implement a permanent solution. This might involve:

  • Code changes and thorough testing
  • Infrastructure upgrades
  • Database migrations or schema fixes
  • Third-party vendor discussions

The fix takes time, planning, and often requires changes during lower-traffic periods.

Why Mitigation Comes First

During an incident, every minute of downtime costs money and user trust. A quick mitigation buys you back your error budget and gives you breathing room to implement a proper fix without pressure. Trying to debug and fix perfectly during an incident often extends the outage unnecessarily.

Your incident runbook should always separate these phases clearly: "Step 1: Mitigate (rollback to v2.1). Step 2: After service restored, investigate root cause."

Key Takeaway: Always prioritize rapid mitigation to restore service first, then invest proper time in permanent fixes afterward—your users care about uptime now, not perfect solutions later.