Rollback and Recovery Patterns
What you'll learn: How to design CI/CD pipelines that automatically undo failed deployments and restore service when health checks or validation tests fail.
The Core Idea
Rollback and Recovery Patterns ensure your pipeline doesn't just push forward—it knows how to step backward safely when something goes wrong. Think of it like an automatic parachute that deploys when your deployment "falls" below acceptable performance thresholds.
When you deploy new code, things can fail after the initial deployment succeeds. A health check might fail, validation tests might catch runtime issues, or monitoring might detect performance degradation. Instead of leaving broken code in production, your pipeline should automatically revert to the last known good version.
How It Works
The pattern integrates validation gates after deployment:
- Deploy the new version
- Run health checks (HTTP endpoints, database connectivity)
- Execute smoke tests or validation tests
- Monitor key metrics (error rates, latency)
- If any check fails → trigger automatic rollback
- If all checks pass → mark deployment as successful
This differs from Manual Approval Gates (which pause before deploying) by adding automated safety checks after deployment.
Common Rollback Mechanisms
- Blue-Green Deployments: Switch traffic back to the previous environment
- Rolling Updates: Stop rollout and redeploy previous version
- Artifact Versioning: Redeploy the last stable artifact from your artifact repository
- Database Migrations: Run reverse migrations if schema changes fail
The key is defining clear failure criteria: What metrics indicate failure? How long should you wait before rolling back? Should rollback be automatic or require confirmation?
Key Takeaway: Design pipelines with automatic rollback triggered by post-deployment health checks and validation—don't assume deployments that "succeed" technically are actually working correctly in production.