Debug StatefulSet in Kubernetes

Kubernetes StatefulSets manage stateful workloads by preserving pod identity and storage. When pods in a StatefulSet misbehave, engineers struggle to trace the root cause. This guide shows you how to debug StatefulSets, list pods, inspect events, exec into containers, port-forward, use ephemeral containers, and dive into storage lifecycles, CSI drivers, access modes, quotas, and security contexts to restore healthy stateful workloads.

TL;DR

• List StatefulSet pods with labels and observe ordering and identity.
• Use kubectl describe to view events and PVC bindings for each pod.
• Exec, port-forward, and attach ephemeral containers for live debugging.
• Understand PV and PVC lifecycle: static vs. dynamic provisioning by CSI drivers.
• Choose correct access modes (RWO, RWX) and set security contexts and resource quotas.
• Refer to the workflow diagram for a step-by-step debug process.

Debug StatefulSet: List Pods

Start by listing all pods managed by your StatefulSet. Pods carry a stable network identity and storage. Use your app label and observe the ordinal suffix:

kubectl get pods -l app.kubernetes.io/name=MyApp
# Output: myapp-0, myapp-1, myapp-2

This confirms pod identity and ordering.

Debug StatefulSet: Inspect Pod Details and Events

For each pod, run kubectl describe to fetch events, volume mounts, and status conditions:

kubectl describe pod myapp-1

Key fields to watch:

• Events: CrashLoopBackOff, FailedMount.
• Volumes: PVC names bound to PVs – shows storage class and capacity.
• Conditions: Ready, ContainersReady.

Debug StatefulSet: Exec and Port-Forward

Exec into a container for real-time inspection of processes, logs, or file system:

kubectl exec -it myapp-1 -- /bin/bash

Use kubectl port-forward to expose container ports locally:

kubectl port-forward myapp-1 8080:80

Combine with curl or grpcurl to test endpoints inside the pod’s network namespace.

Debug StatefulSet: Use Ephemeral Containers

Attach an ephemeral container if original images lack debugging tools:

kubectl debug pod/myapp-1 -it --image=nicolaka/netshoot --target=myapp

Ephemeral containers share the PID namespace and let you inspect running processes.

Storage in StatefulSet: PVC and PV Lifecycle

StatefulSets mount PersistentVolumeClaims (PVCs) per pod. PVCs bind to PersistentVolumes (PVs) according to storage class and access mode. Understand these phases:

• Pending: PVC waiting for PV binding.
• Bound: PVC bound to matching PV by capacity and storage class.
• Released: PVC deleted; PV enters released and then recycled or retained.
• Available: PV ready for new claims if policy allows.

Check PVC status:

kubectl get pvc -l statefulset.kubernetes.io/pod-name=myapp-1

Provisioning and CSI Drivers in StatefulSet

Kubernetes uses Container Storage Interface (CSI) drivers for dynamic provisioning. StorageClass parameters choose the driver:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer

Dynamic provisioning creates PVs at runtime. CSI supports snapshot, clone, and encryption features for stateful workloads.

Access Modes and Security Contexts

Choose access modes based on workload needs:

• ReadWriteOnce (RWO): Single node write access. Default for block storage.
• ReadOnlyMany (ROX): Read-only across multiple nodes.
• ReadWriteMany (RWX): Multiple nodes write access; often requires network file systems.

Apply security contexts to restrict permissions inside pods:

securityContext:
  runAsUser: 1000
  fsGroup: 2000
  allowPrivilegeEscalation: false

Restrict container capabilities and file system group to secure volume mounts against privilege escalation.

Resource Quotas and Limits

Set resource quotas at namespace level to prevent PVC overconsumption:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: storage-quota
spec:
  hard:
    requests.storage: 50Gi
    persistentvolumeclaims: "5"

This ensures teams cannot provision more than allotted storage or PVC count.

Debugging Workflow

References

• Debug a StatefulSet
• Kubernetes Volumes
• Kubermetes Persistent Volumes
• Kubernetes Container Storage Interface (CSI) Documentation
• Kubermetes Persistent Volumes Access Modes
• Configure a Security Context for a Pod or Container
• Resource Quotas in Kubernetes

Suggested Reading

• How I Monitor Commands in Redis Instance with Minimal Effort
• StatefulSet vs Deployment: Key Differences and Use Cases
• Kubernetes 1.33: Volume Populators Graduate to GA | Kubernetes
• Dump a Kubernetes Pod for Analysis
• Install Redis on Ubuntu Manually

PostHashID: f55db1f5a2bc13d4deea8e3ab7869f3e52c5120dbc0be52e724413ea54d1f959