Blast Radius and Safety Controls
What You'll Learn
You'll learn how to run chaos experiments safely by limiting their potential damage and implementing emergency stop mechanisms.
The Core Idea
When running chaos experiments—intentionally breaking parts of your system to test resilience—you need guardrails. "Blast radius" refers to the scope of damage a chaos experiment can cause. Safety controls are the mechanisms that prevent small, controlled experiments from becoming actual production disasters.
Think of it like a controlled burn in a forest. Firefighters create firebreaks and keep water trucks nearby. They never burn the entire forest at once; they work in small, contained sections with clear boundaries and the ability to extinguish the fire immediately if it spreads.
Limiting Blast Radius
Start small and expand gradually:
- Begin experiments in non-production environments (staging, pre-prod)
- Target single instances before entire zones or regions
- Affect only a percentage of traffic (e.g., 1% of users)
- Run experiments during low-traffic periods initially
- Test one component at a time, not multiple simultaneously
Scoping boundaries:
- Use feature flags to target specific user cohorts
- Apply chaos only to canary deployments
- Isolate experiments to specific availability zones
- Never run experiments on critical dependencies simultaneously
Safety Controls
Kill switches are mandatory:
- Automatic rollback triggers based on SLO violations
- Manual abort buttons accessible to on-call engineers
- Time-based auto-termination (experiments expire after X minutes)
- Real-time monitoring dashboards during experiments
- Pre-defined rollback procedures
Before running any experiment:
- Define success/failure criteria upfront
- Establish monitoring thresholds that trigger automatic stops
- Ensure you can restore normal operations in seconds
- Have incident response team on standby
Key Takeaway: Always limit chaos experiments to the smallest possible scope with clear boundaries, and implement multiple layers of kill switches that can immediately halt the experiment if metrics exceed acceptable thresholds—safety must come before learning.