In this lesson, you will learn how to systematically drill down from surface-level production incidents to the underlying technical or process triggers using the Five Whys technique.
When a Kubernetes cluster experiences an outage, it is tempting to settle for the first observable symptom—like a CrashLoopBackOff on a deployment. However, fixing the symptom often leads to recurring incidents. The Five Whys technique forces you to peel back the layers of your CI/CD pipeline and infrastructure to find the systemic failure point.
The Five Whys in Practice
Imagine your service suddenly spiked in 5xx errors. Instead of simply restarting the pod, ask "Why?" until you reach a root cause:
- Why did the service return 5xx errors? The pods were OOMKilled by Kubernetes.
- Why were the pods OOMKilled? They exceeded the memory limit defined in the
deployment.yaml. - Why did memory usage spike? A recent code change introduced a memory leak during high-concurrency requests.
- Why did the memory leak reach production? The CI/CD pipeline automated tests lacked load testing or performance benchmarks.
- Why were there no performance benchmarks? Our current testing strategy focuses solely on functional unit tests, missing resource-intensive edge cases.
By reaching the fifth "Why," you’ve moved from a transient "restart the pod" fix to a structural improvement: Integrating load testing into the CI/CD pipeline.
Applying the Technique
When investigating an incident, treat each answer as a bridge to the next question. In a distributed system, your answers should alternate between:
- Technical triggers: Configuration drift, resource starvation, or faulty container images.
- Process triggers: Lack of automated validation, insufficient observability, or gaps in the release policy.
Avoid the "blame game." If you stop at "an engineer pushed bad code," you haven't performed a Five Whys analysis; you’ve simply pointed a finger. A true Five Whys analysis explores why your automated safety nets—your CI/CD gates, your canary deployments, or your Prometheus alerts—allowed that "bad code" to reach the cluster in the first place.
Key Takeaway: The Five Whys technique transforms an incident from a frustrating roadblock into a catalyst for hardening your infrastructure and CI/CD processes against future failures.