Course contentsShow
← System Design
Lesson 1346 of 1,91933. Reliability Engineering FoundationsFree lesson

Blast Radius and Safety Controls

Limiting the scope of chaos experiments and implementing kill switches to prevent runaway failures during testing.

Blast Radius and Safety Controls

What You'll Learn

You'll learn how to run chaos experiments safely by limiting their potential damage and implementing emergency stop mechanisms.

The Core Idea

When running chaos experiments—intentionally breaking parts of your system to test resilience—you need guardrails. "Blast radius" refers to the scope of damage a chaos experiment can cause. Safety controls are the mechanisms that prevent small, controlled experiments from becoming actual production disasters.

Think of it like a controlled burn in a forest. Firefighters create firebreaks and keep water trucks nearby. They never burn the entire forest at once; they work in small, contained sections with clear boundaries and the ability to extinguish the fire immediately if it spreads.

Limiting Blast Radius

Start small and expand gradually:

  • Begin experiments in non-production environments (staging, pre-prod)
  • Target single instances before entire zones or regions
  • Affect only a percentage of traffic (e.g., 1% of users)
  • Run experiments during low-traffic periods initially
  • Test one component at a time, not multiple simultaneously

Scoping boundaries:

  • Use feature flags to target specific user cohorts
  • Apply chaos only to canary deployments
  • Isolate experiments to specific availability zones
  • Never run experiments on critical dependencies simultaneously

Safety Controls

Kill switches are mandatory:

  • Automatic rollback triggers based on SLO violations
  • Manual abort buttons accessible to on-call engineers
  • Time-based auto-termination (experiments expire after X minutes)
  • Real-time monitoring dashboards during experiments
  • Pre-defined rollback procedures

Before running any experiment:

  • Define success/failure criteria upfront
  • Establish monitoring thresholds that trigger automatic stops
  • Ensure you can restore normal operations in seconds
  • Have incident response team on standby

Key Takeaway: Always limit chaos experiments to the smallest possible scope with clear boundaries, and implement multiple layers of kill switches that can immediately halt the experiment if metrics exceed acceptable thresholds—safety must come before learning.