Course contentsShow
← System Design
Lesson 1347 of 1,91933. Reliability Engineering FoundationsFree lesson

Common Chaos Experiments

Killing instances, injecting latency, exhausting resources, simulating network partitions, and degrading dependencies.

Common Chaos Experiments

What you'll learn: The specific types of failures you'll inject to test your system's resilience in controlled chaos engineering practice.

Now that you understand chaos engineering principles, safety controls, and blast radius, it's time to explore the actual experiments you'll run. These are the "controlled fires" you set to see if your smoke alarms work.

The Five Core Experiment Types

Killing instances is the simplest experiment. You terminate servers, containers, or processes to verify automatic recovery, health checks, and load balancer behavior. Think of it as unplugging a server to see if traffic reroutes smoothly.

Injecting latency adds artificial delays to network calls or internal operations. You might slow down a database query from 10ms to 3 seconds to see if timeouts trigger correctly and if the system uses graceful degradation instead of cascading failures.

Exhausting resources means consuming CPU, memory, disk I/O, or file descriptors until components struggle. This tests whether your monitoring alerts fire, whether bulkheads prevent neighbor impact, and whether the system fails gracefully under pressure.

Simulating network partitions creates the split-brain scenarios you learned about. You block communication between zones or regions to verify quorum systems work and that active-active or active-passive failover behaves correctly.

Degrading dependencies makes external services slow, unavailable, or return errors. You might make your payment API return 500 errors to confirm circuit breakers open and users see helpful error messages instead of hanging requests.

Each experiment validates assumptions from your redundancy design, fault tolerance patterns, and SLOs. Start small—kill one instance in staging—then gradually expand scope and severity.

Key Takeaway: Common chaos experiments—killing instances, injecting latency, exhausting resources, simulating partitions, and degrading dependencies—systematically test whether your fault tolerance patterns work under real failure conditions.