Estimated reading time: 4 minutes
Kubernetes can evict pods from a node for many reasons, such as insufficient disk space, memory or CPU constraints, etc. Though these pods are no longer running, they can consume resources and clutter the Kubernetes cluster. Cleaning up and deleting evicted pods from the cluster regularly is essential. Let’s look at how to find the Evicted Pods and how to delete them.
TL;DR: Command to delete evicted pods
If you want to skip the explanations and jump straight to the command, here it is:
Remember to change the namespace name – there are two places in this command where you need to change the namespace.
kubectl get pod -n socketdaddy --no-headers | grep Evicted | \
awk '{print $1}' | xargs -I {} kubectl delete pod -n socketdaddy {} Table of contents
- Why does Kubernetes evict pods?
- Why should we delete evicted pods?
- Identifying Evicted Pods
- Deleting Evicted Pods
- Conclusions
- References
Why does Kubernetes evict pods?
Kubernetes may decide to evict pods for various reasons.
- Resource Constraints: K8s may evict pods if the node on which they are scheduled runs out of resources such as CPU, memory, or disk space. Kubernetes will prioritise these resources on the node for higher-priority pods.
- Draining of nodes: When a planned maintenance, such as an upgrade or hardware maintenance, calls for draining a node, K8s will evict the pods scheduled on the node.
- Taints and Tolerations: Sometimes, a node may have taints to prevent certain pods from getting scheduled. If a pod does not have the right tolerations for a tainted node, Kubernetes will evict it.
- Policy violations: Kubernetes will evict pods that violate policies, such as those defined in resource quotas or PodDistributionBudgets, to ensure that the workloads adhere to the policies.
Why should we delete evicted pods?
Evicted pods sometimes continue to hold on to resources even though they are no longer running. This could include resources such as PVCs, ConfigMaps, IP addresses assigned to those pods, disk space, memory, etc. Over time, as the number of evicted pods increases, they can clutter the etcd key-value store and impact its performance.
In many cases, this can also exhaust all the available IPs in the CIDR. It may also prevent other pods from using PVCs held by the evicted pod.
Lingering entries for evicted pods also impact the K8s Sceduler’s efficiency. The scheduler has to process and manage all the pods, including the evicted ones. This can slow the scheduler’s decision-making and impact the cluster’s overall performance.
It can also clutter your monitoring systems, making it confusing and hard to identify real issues. Keeping the cluster clean is essential so administrators can manage it more efficiently.
Read also: How to Drain a Node in Kubernetes
Identifying Evicted Pods
There are two ways you can identify evicted pods. You can either list all pods and use the grep tool to filter for Evicted pods or use the field selctors in the kubectl command to filter for pods with the status reason Evicted.
Using the grep tool
You can use the below command to list all Evicted pods in the namespace socketdaddy. If you wish to list the evicted pods in all namespaces, replace the -n socketdaddy option with the -n -A option.
kubectl get pod -n socketdaddy --no-headers | grep Evicted Using Field Selectors
Alternatively, you can use the filed selectors and query a jsonpath to filter for evicted pods from the kubectl command.
Again, don’t forget to replace the namespace name.
kubectl get pods -n socketdaddy --field-selector=status.phase=Failed \
-o jsonpath='{.items[?(@.status.reason=="Evicted")].metadata.name}' Once you’ve identified them, we’ll see different ways to delete evicted pods in the next section.
Deleting Evicted Pods
Like the commands above, you can delete the evicted pods using either the grep tool or the field selectors to filter for evicted pods.
Let’s look at both the options.
Delete using the grep tool
You can remove the pod names from the grep command output and pipe it into an xargs command to delete the pods.
kubectl get pod -n socketdaddy --no-headers | grep Evicted | \
awk '{print $1}' | xargs -I {} kubectl delete pod -n socketdaddy {} Delete using Field Selectors
Using the output of the field selectors command we saw above, you can form a piped command in a for loop to iterate through all namespaces and delete evicted pods.
for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
kubectl get pods -n $ns --field-selector=status.phase=Failed \
-o jsonpath='{.items[(@.status.reason=="Evicted")].metadata.name}' | \
xargs -I {} kubectl delete pod -n $ns {}
done; Download the Kubernetes Evicted Pods Cleanup Scripts
You can also download our scripts that save you time when cleaning up evicted pods in Kubernetes
Conclusion
Regularly cleaning up and deleting evicted pods helps you maintain optimal resource utilization of your cluster’s resources and performance. It also helps the cluster’s administration by keeping it clean and clutter-free.
Also Read: 10 Essential Kubernetes Interview Questions – 2023
