In this lesson, you will learn how to identify unauthorized or manual changes to your production environment by automating the comparison between your live state and your Git-based source of truth.
The Challenge of Drift
In a declarative system like Kubernetes, your Git repository acts as the "source of truth." However, manual kubectl edit commands or ad-hoc hotfixes create "drift"—a state where the cluster configuration differs from your version-controlled manifest. Left unchecked, drift leads to "snowflake" environments that are impossible to reproduce and difficult to debug during incidents.
Automating Detection
To maintain reliability, we move away from manual verification and toward automated audit loops. These tools continuously perform a diff between what is defined in Git and what is currently running in your cluster.
Consider a GitOps approach using a tool like Argo CD or Flux. These controllers don’t just deploy code; they observe the live state. If a developer manually changes a replica count or environment variable in production, the controller detects the discrepancy immediately.
The Audit Workflow
An automated audit typically follows these steps:
- Sync: The tool fetches the desired state from your Git repository.
- Observe: It pulls the actual state from the Kubernetes API server.
- Compare: It calculates the delta.
- Alert/Reconcile: It notifies the team via Slack/PagerDuty or automatically overwrites the drift to force the cluster back into the desired state.
Example of a drift detected via CLI:
# Comparing live cluster state against the manifest folder
argocd app diff my-production-service --local ./k8s-manifests
# Output shows drift in replica count:
# - replicas: 3
# + replicas: 5 (manual override detected)
By integrating these audits into your CI/CD pipeline, you ensure that "human-in-the-loop" configuration changes are never the permanent state of your production environment. If a change is required, it must be committed to Git, passing through the same automated testing and PR review process as any other code, preserving the integrity of your infrastructure.
Key Takeaway: Automated audits transform configuration drift from a hidden operational risk into a visible, manageable event by enforcing that the live environment always mirrors the repository’s source of truth.