Automating Infrastructure Rollback Procedures

Create automated recovery paths that revert infrastructure to a previous stable state when deployment health checks fail in production.

In this lesson, you will learn how to trigger automated infrastructure reversion to a previous stable state when post-deployment health checks fail in production.

The Philosophy of Automated Rollbacks

When infrastructure is defined declaratively (as we covered in Lesson 14) and managed through CI/CD pipelines, you gain the ability to treat "rollback" as a programmatic deployment. Instead of manually scrambling to patch a failing system, your pipeline should be designed to detect instability—such as failing liveness/readiness probes or elevated 5xx error rates—and automatically execute a revert to the last known good configuration state.

Implementation Pattern

The most effective way to automate this is by integrating a "watchdog" mechanism into your deployment orchestration. After the infrastructure code is applied, the pipeline enters a verification phase. If this phase fails, the pipeline immediately triggers a re-application of the previous version of your configuration state.

In a Kubernetes-based CI/CD workflow, this is often handled by tracking the revision history of your deployments. If a health check script—running as a step in your pipeline—fails, it triggers a kubectl rollout undo command or re-runs the CI pipeline using the previous git commit SHA as the source of truth.

# Example logic for a pipeline health check step
deploy_infrastructure
if ! run_health_checks --timeout 300; then
  echo "Health checks failed. Triggering automated rollback..."
  git checkout ${PREVIOUS_STABLE_COMMIT}
  apply_infrastructure_state
  exit 1
fi

Ensuring Consistency

For this to be reliable, your infrastructure must strictly follow the Immutable Infrastructure pattern (Lesson 18). If you attempt to roll back infrastructure that has undergone "configuration drift" (Lesson 17), the rollback may fail because the current state no longer matches your stored declarative configuration. By enforcing idempotent provisioning (Lesson 15), you ensure that reverting to a previous state is a predictable, repeatable operation that reliably restores system availability.

Automated rollbacks move the responsibility of stability from the engineer on-call to the CI/CD pipeline, reducing the Mean Time to Recovery (MTTR) by eliminating human latency during a production incident.

Key Takeaway: Automated rollbacks transform infrastructure restoration from a manual emergency response into a standardized, programmatic deployment process triggered by failed health verification.