Estimated reading time: 4 minutes
Draining a node in Kubernetes is a standard maintenance operation that gracefully evicts all pods from the node. You would need to do this when performing maintenance on the node, upgrading it, or decommissioning it. Draining ensures that the workloads are safely moved to other nodes in the cluster, minimizing downtime and maintaining application availability. In this article, we’ll explain the steps to drain a node in Kubernetes.
Table of contents
TL;DR: Command to drain a node in Kubernetes
If you are here just for the command to drain a node and want to skip all the theory, look no further:
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data Now, let’s get into the details and understand why and how to execute this command.
Why drain a node?
You might want to drain a node in Kubernetes for several reasons. Let’s quickly look at some of the most common reasons.
Maintenance
Some maintenance activities, such as OS upgrades or hardware maintenance, require you to take the node offline. During these activities, you should ensure no pods run on the node.
Upgrades
When upgrading Kubernetes, you must drain the node and cordon it off so Kubernetes doesn’t schedule any new pods.
Decommissioning a node
Before permanently decommissioning a node from the cluster, evicting all the pods running on it is necessary.
Testing and troubleshooting
If you suspect any issues on a node, you might want to evict all pods running on the cluster to troubleshoot.
Cluster Scaling
When scaling down a cluster, you would need to drain the nodes to be removed and move the pods to other available nodes.
Pre-requisites
The rest of the section assumes that you have the following pre-requisites met:
kubectl: You havekubectlinstalled and configured with correct contexts to run commands against your target cluster.- Access: You have sufficient access and privileges to execute the drain command.
- Minimizing downtime: You have considered various options to ensure the downtime for the applications is minimized when draining a node. You can use PodDisruptionBudgets to ensure your applications remain highly available during the activity.
Steps to drain a node
Now, let’s discuss all the steps in detail.
Note: Remember that you can do a dry run of these commands using the --dry-run client option.
1. Identify the node to drain
First, identify the name of the node you wish to drain. You can list all of the nodes in your cluster with the following command:
kubectl get nodesThis command will list all the nodes in the cluster, along with their status.
2. Drain the Node
Use the kubectl drain command to evict all pods from the node. This command will evict all pods running on the node except those not managed by a controller, such as DaemonSets or static pods.
Example:
kubectl drain <node-name> 3. Ignoring daemon sets
If daemon set-managed pods exist, the drain command will not proceed without --ignore-daemonsets flag. By default, the command will not delete any daemon-set-managed pods because the daemon-set controller will immediately replace them.
Example:
kubectl drain <node-name> --ignore-daemonsets4. Delete pods with emptyDir volumes
If there are pods that use emptyDir volumes, passing the --delete-emptydir-data will force Kubernetes to delete them. Importantly, this option will cause data loss, as the emptyDir volume is deleted during this process.
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data Verifying the node
When you drain a node, Kubernetes marks it unschedulable to prevent new pods from being scheduled. You can verify this by listing the nodes again or by describing them.
Examples:
kubectl get nodeskubectl describe node <node-name> Bonus: How the drain command works
In this section, we’ll explain how the command is executed.
The command first cordons the node we are trying to drain, preventing new pods from arriving. Next, k8s will fetch all running pods on the node. Once all the pods from the node are listed, k8s will filter out mirror pods and daemon-set-managed pods. If the --ignore-daemonsets is not passed, the daemon-set-managed pods will not be evicted. After arriving at the final list of pods to evict/delete, k8s iteratively deletes each pod from the node. The below image from the official Kubernetes documentation explains this perfectly:
References
- kubectl drain
- Safely Drain a Node
- PodDisruptionBudget – Kubernetes official documentation
