In this lesson, you will learn how to systematically categorize and interpret configuration drift to maintain the integrity of your GitOps-managed clusters.
In a declarative environment, your Git repository acts as the "source of truth." Drift occurs when the live state of your Kubernetes cluster diverges from the state defined in Git. While many assume drift is purely an error, advanced GitOps practitioners know that drift reports are diagnostic tools that reveal specific operational intent.
Categorizing Drift: Emergency vs. Unintentional
When you view a drift report (e.g., via argocd diff or flux reconcile), you must distinguish between two primary categories:
- Intentional Emergency Patches (Hotfixes): During a production outage, an engineer might manually update a replica count or inject a temporary environment variable via
kubectl edit. This is "intentional" drift. It serves a specific, short-term purpose. To handle this, the drift report will show the live object has fields (likespec.replicas) that don't match Git. Your workflow should mandate that these manual changes be back-ported to Git immediately after the incident to "reconcile" the source of truth. - Unintentional Configuration Changes: These are "silent" drifts often caused by out-of-band automation, side-effects of controllers, or manual mistakes. For example, if a third-party admission controller modifies resource limits or labels automatically, your drift report will show unexpected field values.
Interpreting the Report
A typical diff highlights exactly which fields have diverged. If you see a delta in metadata.resourceVersion or status fields, ignore them; these are expected Kubernetes-internal updates. However, if you see changes in spec fields, you must investigate:
# Drift report example:
- spec:
- replicas: 3
+ replicas: 5 # Manual intervention detected
If the replicas change was a hotfix, you acknowledge the drift and merge the 5 into your Git manifest. If it was unauthorized, you trigger an automated synchronization to overwrite the manual change with your defined state of 3. By training your eyes to filter out systemic Kubernetes metadata and focus on spec divergence, you can instantly tell if your cluster is being managed by intent or by incident.
Key Takeaway: Drift is a signal—differentiate between necessary emergency hotfixes that require Git back-porting and unauthorized manual changes that require immediate automated reconciliation.