Kubernetes Secrets Management Best Practices

Secure Kubernetes secrets using external secret managers, encryption at rest, and access controls — because base64 is NOT encryption.

Table of Contents

• The Problem with Default Secrets
• Encryption at Rest
• RBAC for Secrets
• External Secrets Operator
• Azure Key Vault Integration
• Sealed Secrets (GitOps)
• Common Mistakes

The Problem with Default Secrets

Kubernetes Secrets are base64-encoded, not encrypted by default:

kubectl get secret db-password -o jsonpath='{.data.password}' | base64 -d
# Anyone with read access can decode this instantly

Additional risks:

• Secrets stored in etcd in plaintext (unless encryption configured)
• Visible in kubectl describe events if misused
• Often committed to Git by accident
• Accessible to anyone with namespace read access

Encryption at Rest

Enable encryption for secrets in etcd with an EncryptionConfiguration:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}

Managed Clusters

AKS — use Azure Key Vault KMS provider:

az aks update \
  --resource-group my-rg \
  --name my-cluster \
  --enable-azure-keyvault-kms \
  --azure-keyvault-kms-key-id <key-id>

GKE/EKS — use their native KMS integration. Managed clusters encrypt etcd by default at the infrastructure level, but KMS envelope encryption adds a second layer you control.

Rotate Encryption Keys

# Re-encrypt all existing secrets after enabling encryption
kubectl get secrets --all-namespaces -o json | \
  kubectl replace -f -

RBAC for Secrets

Secrets respect namespace boundaries — restrict access tightly:

# role.yaml — read-only access to specific secrets
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: production
  name: secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["db-credentials"]     # only this secret
  verbs: ["get"]                        # no list — prevents enumeration

# Check who can read secrets
kubectl auth can-i get secrets --as=system:serviceaccount:production:my-sa -n production

# Audit access
kubectl get clusterrolebindings -o wide | grep secret

Key rules:

• Use resourceNames to limit to specific secrets
• Avoid granting list/watch on secrets (exposes all secrets in the namespace)
• get on a secret = full decode access — treat it as "read the password"

External Secrets Operator

Store secrets in a real vault, sync to Kubernetes:

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace

Sync from Azure Key Vault

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: azure-kv
  namespace: production
spec:
  provider:
    azurekv:
      vaultUrl: "https://my-keyvault.vault.azure.net"
      authType: WorkloadIdentity
      serviceAccountRef:
        name: external-secrets-sa

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: azure-kv
    kind: SecretStore
  target:
    name: db-credentials        # K8s secret created automatically
  data:
    - secretKey: password
      remoteRef:
        key: db-password        # key name in Key Vault

Benefits: automatic rotation, centralized audit, secrets never in Git.

Azure Key Vault Integration (CSI Driver)

Mount secrets directly into pods — no Kubernetes Secret object needed:

# Enable the add-on on AKS
az aks enable-addons \
  --addons azure-keyvault-secrets-provider \
  --resource-group my-rg \
  --name my-cluster

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: app-secrets
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true"
    userAssignedIdentityID: "<identity-client-id>"
    keyvaultName: "my-keyvault"
    objects: |
      array:
        - |
          objectName: db-password
          objectType: secret
    tenantId: "<tenant-id>"

# Pod mounts secrets as a volume
volumes:
  - name: secrets
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: app-secrets

Sealed Secrets (GitOps)

If you want secrets in Git safely — encrypt them with Sealed Secrets (Bitnami):

# Install controller
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/latest/download/controller.yaml

# Install kubeseal CLI
brew install kubeseal

# Create an encrypted secret safe for Git
kubectl create secret generic db-creds \
  --from-literal=password=s3cret \
  --dry-run=client -o yaml | \
kubeseal -o yaml > sealed-secret.yaml

Commit sealed-secret.yaml to Git — only the cluster's private key can decrypt it.

Common Mistakes

1. Secrets in environment dumps

# Anyone with exec access sees all secrets
kubectl exec -it pod -- env

Mount as files when possible — they don't appear in env output or crash dumps.

2. Secrets in ConfigMaps

# NEVER do this
kind: ConfigMap
data:
  password: "s3cret"

ConfigMaps have no encryption or access restrictions.

3. Secrets in image layers

# WRONG — persists in image history even if deleted
RUN echo $API_KEY > /tmp/key && rm /tmp/key

4. Overly broad RBAC

# DANGEROUS — reads ALL secrets in ALL namespaces
resources: ["secrets"]
verbs: ["get", "list", "watch"]

5. Forgetting audit logging

# audit-policy.yaml
rules:
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets"]

Secret Rotation Checklist

1. Update the secret in your vault / secret store
2. ESO syncs automatically (respecting refreshInterval)
3. Restart pods or use a reloader to pick up new values:

# stakater/reloader annotation
annotations:
  reloader.stakater.com/auto: "true"

Quick Reference

Related Articles

• DevSecOps Pipeline
• Kubernetes RBAC Guide
• Helm Secrets

---

Last Updated: October 2026
Author: CloudOpsGuide Team
Difficulty: Advanced
Estimated Reading Time: 14 minutes