Course contentsShow
← System Design
Lesson 858 of 1,91922. Service MeshFree lesson

Fault Injection for Testing

Inject delays or errors into service calls to test resilience and validate circuit breaker behavior.

Fault Injection for Testing

What you'll learn: How to deliberately introduce failures into service calls to validate that your resilience mechanisms actually work before real incidents occur.

The Core Idea

Fault injection is the practice of intentionally causing errors or delays in service communications to test how your system behaves under failure conditions. Instead of waiting for production outages to discover that your circuit breakers, retry policies, and timeouts don't work as expected, you proactively inject faults in controlled environments to verify resilience.

Think of it like a fire drill: you simulate an emergency to ensure everyone knows the evacuation plan works before a real fire happens.

How Service Meshes Enable Fault Injection

Since you've learned about traffic management, retry policies, and circuit breaking in the mesh, fault injection is the testing tool that validates those configurations. The service mesh control plane lets you configure rules that inject:

  • Latency delays: Add 5 seconds to 10% of requests to a specific service
  • Error responses: Return HTTP 503 for 20% of calls to test retry logic
  • Connection failures: Simulate network partitions or complete service unavailability

These rules operate at the data plane (sidecar proxy) level, meaning your application code doesn't change—the mesh transparently corrupts traffic based on your test scenarios.

Testing Real Resilience

Without fault injection, your circuit breaker configuration might look correct but never actually trigger in testing. By injecting 50% error rates, you can observe whether:

  • Circuit breakers open at the right thresholds
  • Retry policies don't amplify failures
  • Timeout settings prevent cascading delays
  • Fallback mechanisms activate properly

Key Takeaway: Fault injection transforms theoretical resilience patterns into validated behavior by deliberately breaking things in controlled ways, ensuring your circuit breakers, retries, and timeouts actually work before production failures test them for real.