Kubernetes Pod Eviction: How It Works and Best Practices

In Kubernetes, pod eviction is the process of terminating one or more pods to maintain the stability and health of a cluster. Evictions are triggered by various factors, such as resource constraints, node maintenance, or policy enforcement. Understanding how Kubernetes manages pod evictions is critical for maintaining application availability and ensuring the cluster runs smoothly. This article explores the types of pod evictions, the factors influencing them, and best practices to minimize their impact.

Types of Pod Evictions

Kubernetes supports several types of pod evictions, each with its own purpose and mechanics:

1. Node-Pressure Eviction

Node-pressure eviction occurs when a node is under resource stress, such as low memory, disk space, or inodes. The kubelet continuously monitors these resources and triggers evictions when thresholds are breached to prevent node instability.

Example: If a node runs low on memory, the kubelet identifies pods consuming the most resources or those with lower priority and evicts them to reclaim memory.

2. API-Initiated Eviction

API-initiated evictions allow administrators or controllers to evict pods gracefully using the Kubernetes Eviction API. These evictions respect PodDisruptionBudgets (PDBs), ensuring application availability during planned disruptions like updates or scaling events.

Command Example:

kubectl delete pod <pod-name> --grace-period=<seconds> 

3. Preemption

Preemption ensures that high-priority pods have access to the resources they need by evicting lower-priority pods. This mechanism is particularly useful for workloads with critical requirements.

Example: A high-priority workload may trigger the eviction of lower-priority pods to free up space on a node.

Factors Influencing Pod Evictions

Kubernetes makes eviction decisions based on several criteria:

• Pod Priority: Pods with higher priorities are less likely to be evicted compared to lower-priority ones.
• Quality of Service (QoS) Class: Pods are classified into three QoS classes:
  • Guaranteed: Least likely to be evicted.
  • Burstable: Evicted only when resource requests exceed usage limits.
  • BestEffort: Most likely to be evicted under resource pressure.
• Resource Usage: Pods that exceed their resource requests are more likely to be targeted for eviction.

How Node-Pressure Eviction Works

Node-pressure eviction is managed by the kubelet and involves the following thresholds:

1. Soft Thresholds: Evictions are initiated when resource usage exceeds configured levels over a specified period.
2. Hard Thresholds: Immediate eviction occurs when critical limits are reached.

Threshold Example:

evictionHard:
  memory.available: "500Mi"
  nodefs.available: "10%"
evictionSoft:
  memory.available: "1Gi"
evictionSoftGracePeriod:
  memory.available: "1m"

In this configuration:

• Pods will be evicted if memory availability drops below 500Mi immediately (evictionHard).
• Graceful evictions will occur if memory availability remains under 1Gi for more than 1 minute (evictionSoft).

Best Practices to Minimize Pod Evictions

1. Set Accurate Resource Requests and Limits

Define appropriate resource requests and limits for your pods to avoid overcommitment and unexpected evictions.

resources:
  requests:
    memory: "512Mi"
    cpu: "500m"
  limits:
    memory: "1Gi"
    cpu: "1"

2. Use PodDisruptionBudgets (PDBs)

PDBs ensure that a minimum number of pods remain available during voluntary disruptions.

Example:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-app-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: web-app

3. Monitor Node Resources

Regularly monitor node health to identify potential resource shortages before they lead to evictions. Use tools like Prometheus and Grafana for real-time insights.

4. Prioritize Critical Workloads

Assign higher priorities to critical pods to ensure they are less likely to be evicted.

Example:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000
  globalDefault: false
  description: "High-priority class for critical workloads"

Example Workflow: Avoiding Node-Pressure Evictions

1. Configure evictionHard and evictionSoft thresholds based on your workload needs.
2. Define resource requests and limits for each pod.
3. Set PodDisruptionBudgets to prevent unintended disruptions during voluntary evictions.
4. Monitor node resource usage using Kubernetes metrics or third-party tools.

Summary

Pod eviction is a fundamental mechanism in Kubernetes that ensures resource availability and cluster stability. Understanding the different types of evictions, such as node-pressure, API-initiated, and preemption, helps administrators design resilient systems. By following best practices like setting appropriate resource limits, using PDBs, and monitoring resource usage, you can minimize disruptions and maintain high application availability.

Reference Links

1. Kubernetes Official Documentation: Node-Pressure Eviction
2. Kubernetes Official Documentation: API-Initiated Evictions
3. The New Stack: How Kubernetes Eviction Works
4. Kubernetes Documentation: Pod Priority and Preemption

Read Next…

• How to delete evicted pods in Kuberenetes
• How to Restart Kubernetes Pods with Kubectl
• The difference between kubectl apply and kubectl create
• Kafka Consumers and Partitions: Why Can’t You Have More Consumers Than Partitions?
• Kubernetes Pod Scheduling: Taints, Tolerations, and Node Affinity Explained