Secrets and Multi-Environment Charts

The default way to configure a chart is a values file. The default way to configure a database password is also a values file, and that is the mistake — because a values file ends up in more places than people expect.

Topic 1: Where a Value Actually Ends Up

Put password: hunter2 in values-prod.yaml and it is now in four places:

1. Git — forever, in history, even after you delete the line.
2. The release Secret — Helm stores the supplied values with every revision.
3. helm get values checkout -n prod — readable by anyone with get secrets in that namespace.
4. CI logs, if anything echoes the command or renders with --debug.

# All three, demonstrated
git log -p -- values-prod.yaml | grep -i password
helm get values checkout -n prod | grep -i password
kubectl get secret sh.helm.release.v1.checkout.v3 -n prod \
  -o jsonpath='{.data.release}' | base64 -d | base64 -d | gunzip | grep -i password

Note the third one in particular: anyone who can read Secrets in the namespace can read every value ever supplied to the release, including values from revisions you have since changed. Rotating the password does not remove the old one from revision history.

Topic 2: The Three Mechanisms

External Secrets Operator — the chart references a Secret by name; a controller creates it from AWS Secrets Manager, Vault, GCP Secret Manager or Azure Key Vault.

# templates/externalsecret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: {{ include "checkout.fullname" . }}
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: {{ .Values.externalSecrets.storeRef }}
    kind: ClusterSecretStore
  target:
    name: {{ include "checkout.fullname" . }}-db
  data:
    - secretKey: password
      remoteRef:
        key: {{ .Values.externalSecrets.path }}
        property: password

# deployment.yaml — the chart only knows the NAME
env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: {{ include "checkout.fullname" . }}-db
        key: password

The chart contains a path, not a password. Rotation happens at the source, and the operator refreshes the Secret without a deploy. This is the strongest option and the one to prefer when the infrastructure exists.

Secrets Store CSI driver — mounts the secret directly from the provider into the pod as a file, optionally syncing a Kubernetes Secret. Nothing is stored in etcd unless you ask for the sync, which some compliance regimes require.

SOPS + helm-secrets — encrypted values files, committed to Git, decrypted at deploy time:

helm plugin install https://github.com/jkroepke/helm-secrets

sops -e -i values/prod.secrets.yaml       # encrypted with age/KMS, safe to commit
helm secrets upgrade --install checkout . -f values/prod.yaml -f values/prod.secrets.yaml

The values are encrypted in Git and decrypted in memory at deploy. They still land in the release Secret in plaintext, so this protects Git rather than the cluster — a meaningful improvement, and not the same guarantee as the first two options.

A fourth option worth naming to reject: generating a password in the chart with randAlphaNum. It changes on every render unless guarded with lookup, does not work under helm template, and produces a secret nobody can rotate or audit. The functions lesson covers the lookup guard; the better answer is a real secret store.

Topic 3: What a Chart Should Do

Design charts so the secret never has to pass through Helm at all:

# values.yaml
database:
  # -- Name of an EXISTING Secret containing the database password.
  # Takes precedence over database.password.
  existingSecret: ""
  existingSecretKey: password
  # -- Plaintext password. Development only; use existingSecret in production.
  password: ""

{{- if .Values.database.existingSecret }}
- name: DB_PASSWORD
  valueFrom:
    secretKeyRef:
      name: {{ .Values.database.existingSecret }}
      key: {{ .Values.database.existingSecretKey }}
{{- else if .Values.database.password }}
- name: DB_PASSWORD
  valueFrom:
    secretKeyRef:
      name: {{ include "checkout.fullname" . }}
      key: password
{{- else }}
{{- fail "set database.existingSecret (production) or database.password (dev only)" }}
{{- end }}

The existingSecret pattern is the convention across the public chart ecosystem, and supporting it is what makes a chart usable by teams with a secrets platform. The fail at the end is what stops a chart from silently deploying with an empty password.

Topic 4: Layering Environments

charts/checkout/values.yaml      safe defaults: 1 replica, small, no ingress
deploy/values/base.yaml          true everywhere: registry, labels, probes
deploy/values/dev.yaml           only the differences
deploy/values/staging.yaml
deploy/values/prod.yaml
deploy/values/prod.secrets.yaml  SOPS-encrypted, if you use that route

helm upgrade --install checkout charts/checkout -n prod \
  -f deploy/values/base.yaml \
  -f deploy/values/prod.yaml \
  --set image.tag="${GIT_SHA}" \
  --atomic --timeout 8m

Two rules that keep environments comparable:

• An environment file contains only what differs. When prod.yaml is 300 lines, the chart’s defaults are wrong and no reviewer can see what production actually changes.
• diff staging.yaml prod.yaml should be short and deliberate. Every line in that diff is a reason staging might not predict production, and each one should be a decision somebody made rather than drift nobody noticed.

Only the image tag belongs in --set. It is the one value that genuinely varies per pipeline run. Everything else in --set is configuration that bypassed review, and helm get values is how you find the ones added during an incident two years ago.

Topic 5: Naming, Namespaces and Blast Radius

helm upgrade --install checkout ./chart -n prod
helm upgrade --install checkout ./chart -n staging

The release name should not encode the environment. checkout in namespace prod, not checkout-prod — the namespace already says it, and encoding it twice means every values file carries a name override and every resource name gets longer (which matters against the 63-character label limit).

One release per component, not one umbrella per environment, unless the components genuinely deploy together. Separate releases give you per-component rollback and stop a database chart’s failure from rolling back your API.

Namespace strategy is a Kubernetes-level decision covered in that module’s multi-tenancy lesson. The Helm-side implication is only this: a release lives in exactly one namespace, and a chart that creates cluster-scoped resources (ClusterRole, ClusterRoleBinding, ValidatingWebhookConfiguration) must include both the release name and the namespace in those names, or two installations collide across namespaces.

name: {{ printf "%s-%s" .Release.Namespace (include "checkout.fullname" .) }}

Topic 6: Auditing What You Have

# Values containing something secret-shaped, across every release
for ns in $(kubectl get ns -o name | cut -d/ -f2); do
  for r in $(helm list -n "$ns" -q 2>/dev/null); do
    helm get values "$r" -n "$ns" 2>/dev/null |
      grep -iE '(password|secret|token|key|credential):' &&
      echo "  ↑ $ns/$r"
  done
done

# And in Git
git log --all -p -- '*values*.yaml' | grep -iE '^\+.*(password|token|secret):'

If either turns something up, the order is the same as any leaked credential: rotate first, then clean up the storage. Removing the value from Git and from the release history is hygiene; the rotation is what actually protects you. The Git side of that — filter-repo, force-push, coordination — is covered in the version control module’s history-rewriting lesson.

Try it yourself: install a chart with a password in a values file, then read it out of the release Secret with kubectl alone. Doing it once makes the “values files are not private” point permanently, and it is the demonstration that convinces a team to adopt External Secrets.

Common mistake: encrypting values with SOPS and concluding the secret is now safe everywhere. SOPS protects Git. The plaintext still reaches the release Secret in etcd, still appears in helm get values, and is still readable by anyone with get secrets in that namespace. It is a genuine improvement over plaintext in Git and it is not the same guarantee as never putting the value in Helm at all.