In this lesson, you will learn how to refresh security credentials without causing your application to drop active connections or go offline.
When you manage sensitive data like API keys or database passwords using Docker Secrets, the files mounted into your containers are immutable. Because the container sees these as read-only files, you cannot simply edit the existing secret file to update the password. If you try to change the source file, the container will remain oblivious, still holding the old, cached value in its memory.
The Strategy: Versioned Secrets and Rolling Updates
To update a secret without downtime, you must treat the secret as a versioned resource. Instead of modifying an existing secret, you create a new one with a unique name (e.g., db_password_v2) and update your service definition to point to the new version.
When you update your service specification (such as a docker-stack.yml or a Kubernetes deployment) to reference the new secret, the orchestrator initiates a Rolling Update.
Here is how the process looks in a docker-compose or Swarm environment:
- Create the new secret:
docker secret create db_password_v2 ./new_password.txt - Update your configuration:
Change your service definition to point to
db_password_v2. - Trigger the update:
docker stack deploy -c docker-compose.yml my_app
Because you have multiple replicas running (which we covered in our lesson on scaling), the orchestrator replaces your containers one by one. It shuts down one old container, spawns a new one that mounts the db_password_v2 file, and ensures the new instance is "Healthy" (using the health checks we discussed previously) before moving on to the next container.
Since your service always maintains a minimum number of running replicas, your traffic continues to be served by the old containers while the new ones spin up, and then by the new ones as the old ones are retired. This ensures that the transition to the new credentials happens seamlessly in the background.
Key Takeaway: To rotate secrets without downtime, always use versioned secret names and leverage your orchestrator’s rolling update mechanism to replace containers gradually with the new credentials.