When working with Kubernetes, you might encounter a situation where a pod is evicted. Understanding why this happens and how to address it is critical for maintaining a stable cluster. In this guide, we will explore the causes of Kubernetes pod eviction, offer solutions to prevent it, and share best practices for managing resources in the cluster.
TL;DR
Pod eviction in Kubernetes typically occurs due to resource constraints or node issues. Common causes include insufficient CPU or memory, disk pressure, or manual eviction policies. To resolve this, identify the root cause using logs and metrics, adjust resource requests and limits, and consider upgrading your cluster’s resources or node configurations.
What Does “Kubernetes Pod Evicted” Mean?
Pod eviction happens when Kubernetes forces a pod to terminate and removes it from the node it was running on. Unlike a crash, an eviction is a deliberate action initiated by the Kubernetes control plane, often due to resource shortages or policy rules.
Evicted pods will not restart automatically unless specific configurations like deployments or replica sets manage them.
Common Causes of Pod Eviction
- Resource Constraints
- Insufficient memory (OOMKilled) or CPU on the node.
- Disk pressure caused by high I/O or insufficient storage.
- Node Issues
- A node becoming unschedulable due to maintenance.
- Node cordoning or draining for updates or scaling.
- Eviction Policies
- Custom pod disruption budgets (PDBs).
- Preemption by higher-priority pods.
- Cluster-Level Issues
- Overprovisioned cluster with insufficient resources.
- Misconfigured auto-scaling or monitoring systems.
Read also: Kubernetes Pod Eviction: How It Works and Best Practices
Diagnosing Pod Eviction
To address the “Kubernetes pod evicted” issue, start with diagnosis:
1. Check Pod Status
Use kubectl get pods to list pod statuses. Evicted pods typically show the Evicted state.
kubectl get pods 2. Review Events and Logs
Inspect the pod events for details on eviction:
kubectl describe pod <pod-name>This will provide reasons like “MemoryPressure” or “DiskPressure.”
3. Analyze Node Conditions
Check node health to determine resource issues:
kubectl describe node <node-name>Look for conditions such as:
MemoryPressureDiskPressurePIDPressure
Resolving Pod Evictions
1. Adjust Resource Requests and Limits
Ensure your pods request adequate resources without exceeding node capacity. Update your pod specifications as follows:
resources:
requests:
memory: "512Mi"
cpu: "500m"
limits:
memory: "1Gi"
cpu: "1"2. Address Node Resource Issues
- Memory or CPU Pressure: Scale your nodes horizontally by adding more nodes or vertically by increasing instance sizes.
- Disk Pressure: Clear unused images and logs using Kubelet garbage collection settings.
kubectl delete evicted pods --all-namespaces3. Configure Eviction Thresholds
Modify eviction thresholds in the Kubelet configuration to avoid premature evictions. Example:
evictionHard:
memory.available: "100Mi"
nodefs.available: "10%"4. Use Pod Disruption Budgets
Create Pod Disruption Budgets (PDBs) to limit the number of pods evicted simultaneously.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: example-pdb
spec:
minAvailable: 1
selector:
matchLabels:
app: my-appBest Practices to Prevent Pod Evictions
- Monitor Cluster Resources
- Use tools like Prometheus (official documentation), Grafana (official documentation), or Kubernetes Metrics Server (official documentation).
- Implement Auto-scaling
- Enable Horizontal Pod Autoscaler (HPA) (Kubernetes docs) for dynamic scaling.
- Use Cluster Autoscaler (Cluster Autoscaler guide) for automatic node management.
- Optimize Resource Allocation
- Regularly review and adjust resource requests and limits for pods. For guidelines, see Best Practices for Resource Management.
- Perform Regular Node Maintenance
- Clear unused resources and update nodes to prevent evictions due to pressure conditions. Learn more from Node Problem Detector.
Frequently Asked Questions
What happens to evicted pods?
Evicted pods are removed from the node, and Kubernetes will not restart them unless a replica set or deployment is configured.
Can I restart an evicted pod?
No, evicted pods cannot be restarted directly. You need to delete and recreate them.
Conclusion
Encountering a “Kubernetes pod evicted” issue is quite common, but with proper troubelshooting and resource management, you can minimize disruptions and maintain a healthy cluster. Monitoring tools, optimal resource allocation, and proactive scaling are your best defenses against unexpected evictions.
