Incident Mitigation vs Fix
What you'll learn: How to distinguish between quickly restoring service and permanently solving the underlying problem during an incident.
When an incident strikes, you face two distinct goals: mitigation (stop the bleeding now) and fix (cure the disease permanently). Understanding this difference is critical for effective incident response.
Mitigation: Restore Service First
Mitigation focuses on one thing: getting users back online as quickly as possible, even if the solution is temporary or incomplete. Common mitigation tactics include:
- Traffic rerouting: Redirect requests away from failing components to healthy ones
- Rollback: Revert to the last known good deployment
- Feature flags: Disable problematic features immediately
- Rate limiting: Reduce load on struggling systems
- Cache extension: Serve stale data temporarily
Think of mitigation like using a tourniquet on a bleeding wound—it's not healing the injury, but it's keeping the patient alive.
The Fix: Solve It Properly
After service is restored, you investigate the root cause and implement a permanent solution. This might involve:
- Code changes and thorough testing
- Infrastructure upgrades
- Database migrations or schema fixes
- Third-party vendor discussions
The fix takes time, planning, and often requires changes during lower-traffic periods.
Why Mitigation Comes First
During an incident, every minute of downtime costs money and user trust. A quick mitigation buys you back your error budget and gives you breathing room to implement a proper fix without pressure. Trying to debug and fix perfectly during an incident often extends the outage unnecessarily.
Your incident runbook should always separate these phases clearly: "Step 1: Mitigate (rollback to v2.1). Step 2: After service restored, investigate root cause."
Key Takeaway: Always prioritize rapid mitigation to restore service first, then invest proper time in permanent fixes afterward—your users care about uptime now, not perfect solutions later.