Secrets Rotation in Containers
What you'll learn: How to update secrets in running containers without exposing sensitive data or causing service disruption.
The Challenge of Live Secret Updates
Secrets don't last forever—passwords expire, tokens get compromised, and compliance policies demand regular rotation. In containers, this becomes tricky: how do you replace a database password in a running pod without restarting it or briefly exposing the old credentials?
Think of it like changing the locks on your house while you're still living in it. You need a strategy that keeps you secure during the transition.
Rotation Strategies
Pod Restart Approach
The simplest method: update the secret in your secrets management system (Kubernetes Secrets, Vault, etc.), then restart the pods. The new instances fetch fresh credentials on startup. This works well for stateless applications where brief downtime is acceptable.
# After updating the secret externally
kubectl rollout restart deployment my-app
Dynamic Secret Updates
More sophisticated applications can refresh secrets without restarting. External tools like Vault Agent Injector or Secrets Store CSI Driver continuously sync secrets into volumes. Your application watches for file changes and reloads credentials on-the-fly.
For example, with volume-mounted secrets, your app might watch /mnt/secrets/db-password and reconnect to the database when it detects a change—no pod restart needed.
Best Practices
- Version your secrets so you can coordinate rotation across services
- Overlap periods: keep old credentials valid briefly during rotation to avoid connection failures
- Health checks: ensure your app remains healthy after secret updates
- Automation: use tools like External Secrets Operator to sync rotations automatically
Key Takeaway: Secrets rotation in containers requires either orchestrated pod restarts for simplicity or dynamic file-watching mechanisms for zero-downtime updates—choose based on your application's tolerance for brief interruptions.