Kubernetes supports external Key Management Service (KMS) providers to encrypt secret data at rest. You can plug in a gRPC-based KMS v2 plugin for performance and secure key management. This article shows full setup, configuration file format, plugin lifecycle, caching, and real-world use-cases.
TL;DR
- Use KMS v2 for higher throughput and built-in caching.
- Configure
EncryptionConfigurationwithKMSprovider block. - Run a gRPC plugin; set socket and keyID in config.
- Plugin lifecycle includes init, Encrypt/Decrypt calls, cache for 1h by default.
- Implement RBAC and network policies to restrict plugin access.
- Use event handlers or mutating webhooks to trigger re-encryption on secret rotation.
Why KMS provider encryption matters
Kubernetes stores secrets in etcd by default. Etcd may face compromise or misconfiguration risks. Encrypting secrets at rest adds a strong barrier. External KMS providers keep keys outside etcd. KMS v2 offers a simple gRPC API, auto caching, and improved performance.
KMS provider encryption versions
Kubernetes 1.28 deprecates KMS v1 and disables it by 1.29. KMS v1 uses an HTTP plugin with longer latency. KMS v2 uses gRPC. It supports streaming, larger payloads, and efficient marshalling. Performance tests show v2 reduces per-call latency by 60% under load.
Configuring KMS provider encryption
Create an EncryptionConfiguration file and reference it in the API server manifest. Example snippet:
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- kms:
name: my-kms-plugin
endpoint: unix:///var/run/kms-plugin/socket.sock
cachesize: 1000
timeout: 3s
keyID: projects/myproj/locations/global/keyRings/myring/cryptoKeys/mykey
- identity: {}
Place the file under /etc/kubernetes/encryption-config.yaml. Then update the API server flags:
--encryption-provider-config=/etc/kubernetes/encryption-config.yamlKMS provider encryption caching and lifecycle
The API server initializes the plugin at startup. It establishes a gRPC connection to the socket. On an Encrypt or Decrypt call, the server sends a request proto and waits. The plugin returns a response proto. The API server caches DEK (Data Encryption Key) derived from the plugin response. Default cache TTL is 1h. Cached DEKs avoid extra gRPC calls on each read.
Setting up KMS v2 plugin
Most cloud vendors provide KMS v2 plugin binaries. To install:
- Download plugin binary to
/usr/local/bin/kms-plugin. - Create a systemd unit:
[Unit]
Description=KMS v2 Plugin for Kubernetes
After=network.target
[Service]
ExecStart=/usr/local/bin/kms-plugin --endpoint unix:///var/run/kms-plugin/socket.sock
User=root
Restart=on-failure
[Install]
WantedBy=multi-user.target
3. Reload and start:
systemctl daemon-reload
systemctl enable --now kms-plugin
4. Ensure socket permissions allow API server to connect:
chown root:root /var/run/kms-plugin/socket.sock
chmod 600 /var/run/kms-plugin/socket.sock
Encryption configuration file explained
The EncryptionConfiguration schema uses:
resources: list of resource types, usuallysecrets.providers.kms.name: identifier for this provider block.providers.kms.endpoint: socket path withunix://prefix.providers.kms.keyID: identifier in KMS, e.g., Cloud KMS resource name.cachesizeandtimeout: performance tuning.identity: fallback provider returning plaintext (required last).
Security and access controls
Restrict plugin socket and binary. Use SELinux or AppArmor to confine plugin. Apply RBAC to API server only. In networks, isolate plugin host in a dedicated management VLAN. For cloud KMS, restrict IAM roles to decrypt/encrypt only this key ring. Rotate keys via KMS console. Monitor plugin logs for errors and latency.
Use-cases
In event-driven architectures, secrets often rotate on schedule or on application triggers. Use Kubernetes mutating webhooks to re-encrypt secrets when new keys appear.
Example: Integrate with Knative eventing. Trigger a function on Secret change. That function can call API server to fetch and re-create the secret. API server handles encryption automatically via the KMS provider. This pattern ensures zero downtime during key rotation.
References
Suggested Reading
PostHashID: 1ef482e8a71ae808d274ddc27d2ebc3e946b698f8d06b871300859fc6fae99e2
