In this lesson, you will learn how to perform maintenance on a specific node in your Docker Swarm cluster without interrupting the services running on it.
When you need to perform hardware upgrades, kernel patches, or physical server maintenance, you cannot simply shut down a node that is hosting active containers. If you do, those containers will be abruptly terminated, potentially leading to lost requests or inconsistent states. Instead, you must perform a "graceful drain."
The Drain Process
Draining a node is a two-step process handled by the orchestrator:
- Preventing new workloads: You mark the node as
Drain, which tells the Swarm manager to stop scheduling any new service replicas on that specific host. - Evacuating tasks: The orchestrator identifies every container currently running on that node and sends a termination signal to them. Simultaneously, it schedules new replicas on other available nodes in the cluster to maintain your desired service count.
Putting it into Practice
To start the maintenance process, use the docker node update command. By changing the availability of a node to drain, you effectively initiate the migration of all services:
# Mark the node as drain to move tasks away
docker node update --availability drain worker-node-01
# Verify the node status
docker node inspect worker-node-01 --format '{{ .Spec.Availability }}'
Once the command is issued, you can monitor the transition by running docker service ps <service_name>. You will notice the tasks previously assigned to worker-node-01 shifting to a shutdown state while new, healthy tasks are created on other nodes.
Only once the node status shows no active tasks running on it is it safe to power down the machine for your maintenance tasks. After your work is complete, remember to return the node to service so it can resume hosting workloads:
# Return the node to active status
docker node update --availability active worker-node-01
By following this workflow, you ensure that your cluster remains highly available and your users never experience a disruption while you perform critical infrastructure updates.
Key Takeaway: Always set a node to drain availability before maintenance to allow the orchestrator to safely evacuate your tasks to healthy nodes without causing service downtime.