Best practices for Kubernetes Secrets

Kubernetes Secrets carry sensitive data. Mismanagement risks exposure. This article shows robust patterns for secret lifecycle, provisioning, access control and integration. Follow these steps to secure your cluster and applications.

TL;DR

• Store Secrets out of plain YAML; use sealed-secrets or KMS encryption.
• Mount Secrets as files or environment variables; restrict mode and ownership.
• Leverage CSI drivers for dynamic provisioning and rotation.
• Implement RBAC and SecurityContext to limit access to Secrets.
• Use ephemeral volumes or projected volumes for short-lived use-cases.
• Automate secret snapshotting and rotation with event-driven workflows.

Kubernetes Secrets Overview

Kubernetes provides a native Secrets object to store credentials, tokens, certificates. Secrets reside in etcd, encrypted if configured. By default API server handles base64 encoding only, not encryption, so enable EncryptionConfiguration.

Secrets Lifecycle

Secret lifecycle spans creation, update, use, rotation, deletion.

• Creation: kubectl apply or API client.
  kubectl create secret generic db-creds --from-file=./user.pass
• Update: patch or replace. Automated pipelines must avoid plaintext in git.
• Rotation: generate new credentials, update Secret, trigger rollout of dependent workloads.
• Deletion: remove Secret and ensure pods no longer reference it.

Provisioning Kubernetes Secrets

Two provisioning modes:

• Static: define Secret manifest ahead of time.
• Dynamic: use CSI plugin or external controller to fetch from Vault, AWS KMS, Azure Key Vault.

# Static Secret manifest
apiVersion: v1
kind: Secret
metadata:
  name: api-key
type: Opaque
data:
  key: bXktc2VjcmV0LWtleQ==

Volume Types for Kubernetes Secrets

Mount Secrets using two primary volume types:

• secret: Mounts each key as file, mode 420 (default). Can configure defaultMode:

volumes:
- name: creds
  secret:
    secretName: api-key
    defaultMode: 0440

• projected: Combine multiple volume sources (ConfigMap, Secret, downwardAPI). Good for injecting tokens plus metadata.

Using Kubernetes Secrets in Pods

Two access modes:

• Files: Volume mount under /etc/secret.
• Environment Variables: Set envFrom secrets.

containers:
- name: app
  envFrom:
  - secretRef:
      name: api-key
  volumeMounts:
  - name: creds
    mountPath: "/etc/creds"
volumes:
- name: creds
  secret:
    secretName: api-key

CSI and Kubernetes Secrets

CSI (Container Storage Interface) plugins let you fetch secrets on demand. Drivers like secrets-store-csi-driver integrate with external vaults. They support:

• Dynamic mount and refresh.
• Versioned secrets with auto-rotation.
• Access control via ServiceAccount annotations.

Access Controls and Security Contexts

Lock down Secret access with RBAC and SecurityContext:

• Define Role/ClusterRole granting get/list on secrets in namespace.
• Bind to ServiceAccount used by pods.
• Use PodSecurityPolicy or PSP replacement to restrict hostPath mounts.
• Apply securityContext on container to drop unneeded capabilities.

Use Cases with Kubernetes Secrets

Common patterns:

• Database Credentials: Rotated nightly via CronJob, new Secret triggers Deployment rollout.
• TLS Certificates: Use Cert-Manager to auto-renew and inject via projected volume.
• Event-Driven Flows: Knative Eventing reads webhook tokens from Secret, mounts into function container.

Advanced Practices for Kubernetes Secrets

Implement additional controls:

• Enable etcd encryption and audit logging to track get requests.
• Define resource quotas on secrets to avoid oversubscription.
• Use sealed-secrets to store encrypted Secret in git, decrypted by controller.

References

• Encrypting Confidential Data at Rest
• Secrets Store CSI driver for Kubernetes secrets – Integrates secrets stores with Kubernetes via a CSI volume.
• Kubernetes Secrets
• cert-manager
• Auditing in Kubernetes
• Secrets Good Practices

Suggested Reading

• KEDA Kafka Triggers
• Certificates in Kubernetes
• Kubernetes 1.33: Volume Populators Graduate to GA | Kubernetes
• Image Volumes graduate to beta in Kubernetes v1.33!
• Stateful Applications in Kubernetes – StatefulSets

PostHashID: 1ae769a95bc95e537ccf4a47ad85f4ea1a172c1620a0654fe38803061caab723