ConfigMaps, Secrets & the Configuration Boundary

Configuration is the seam where most deployment bugs live. Kubernetes gives you two objects and four ways to consume them, and the differences between those ways are not cosmetic.

Topic 1: ConfigMaps

apiVersion: v1
kind: ConfigMap
metadata:
  name: api-config
data:
  LOG_LEVEL: "info"
  MAX_CONNECTIONS: "100"
  application.yaml: |
    server:
      port: 8080
      timeout: 30s
binaryData:
  cert.der: <base64>

data holds UTF-8 strings; binaryData holds base64 for anything else. Note that values must be strings — MAX_CONNECTIONS: 100 without quotes is a YAML integer and the API rejects it. That error message (cannot unmarshal number into Go value of type string) is common enough to recognise on sight.

kubectl create configmap api-config --from-literal=LOG_LEVEL=info
kubectl create configmap api-config --from-file=application.yaml
kubectl create configmap api-config --from-file=./config-dir/    # every file becomes a key
kubectl create configmap api-config --from-env-file=.env

Size limit: 1 MiB, because the object lives in etcd. Large files belong in a volume, an init container that downloads them, or an image.

Topic 2: The Four Ways to Consume Config — and Why It Matters

1. Individual env vars

env:
  - name: LOG_LEVEL
    valueFrom:
      configMapKeyRef:
        name: api-config
        key: LOG_LEVEL
        optional: false      # pod won't start if missing — usually what you want

2. Every key as env vars

envFrom:
  - configMapRef:
      name: api-config
    prefix: APP_            # optional

Convenient and slightly dangerous: a new key in the ConfigMap silently becomes a new env var, and a key that is not a valid env var name is skipped without complaint.

3. Mounted as a directory

volumeMounts:
  - name: config
    mountPath: /etc/config
    readOnly: true
volumes:
  - name: config
    configMap:
      name: api-config

Each key becomes a file. /etc/config/LOG_LEVEL contains info.

4. Specific keys to specific paths

volumes:
  - name: config
    configMap:
      name: api-config
      items:
        - key: application.yaml
          path: app.yaml       # → /etc/config/app.yaml
      defaultMode: 0400

The difference that matters:

Env vars are injected once, at container start. The value is copied into the process environment and there is no mechanism to change it afterwards — this is a property of Unix processes, not a Kubernetes limitation.

Mounted ConfigMaps are updated in place by the kubelet using a symlink swap, so the change is atomic (you never see a half-written file). But your application must re-read the file; most do not, which is why “the ConfigMap updated and nothing happened” is such a common report.

The checksum annotation — the standard fix:

spec:
  template:
    metadata:
      annotations:
        checksum/config: "{{ include (print $.Template.BasePath \"/configmap.yaml\") . | sha256sum }}"

Changing a ConfigMap does not change the pod template, so nothing rolls out. Hashing the config into a pod annotation does change the template, which triggers a normal rolling update. Every Helm chart worth using does this. Without Helm, kubectl rollout restart deployment/api after a config change achieves the same thing.

Note: a subPath mount never updates, even for a volume. It copies the file once. Use a directory mount if you want live updates.

Topic 3: Secrets Are ConfigMaps With Different Defaults

apiVersion: v1
kind: Secret
metadata:
  name: api-secrets
type: Opaque
stringData:              # you write plaintext here
  DB_PASSWORD: "s3cr3t"
data:                    # ...and the API stores base64 here
  API_TOKEN: dG9rZW4=

Base64 is not encryption. It is an encoding, reversible by anyone:

kubectl get secret api-secrets -o jsonpath='{.data.DB_PASSWORD}' | base64 -d

What Secrets actually give you over a ConfigMap:

• A distinct RBAC surface — you can grant get configmaps without granting get secrets.
• The kubelet stores them in tmpfs (memory), not on the node’s disk.
• Values are omitted from kubectl describe output.
• They are not sent to a node until a pod there needs them.

What they do not give you by default: encryption at rest in etcd. Anyone with etcd access, or a snapshot of it, reads every Secret in plaintext.

Turning on encryption at rest:

# EncryptionConfiguration, passed to the API server
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: ["secrets"]
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <32-byte base64 key>
      - identity: {}     # must be last — allows reading pre-existing plaintext

On managed clusters this is a provider setting (EKS: KMS envelope encryption; GKE: application-layer secrets encryption). Turning it on does not re-encrypt existing Secrets — you must rewrite them:

kubectl get secrets -A -o json | kubectl replace -f -

Secret types:

ServiceAccount tokens changed:

Since 1.24, creating a ServiceAccount no longer creates a permanent token Secret. Pods get a projected, short-lived, audience-bound token that the kubelet rotates automatically. Long-lived tokens still exist if you create the Secret explicitly, and you should avoid doing so — a leaked bound token expires; a leaked legacy token is valid forever.

Topic 4: Where Secrets Should Actually Come From

Committing Secret YAML to Git is the mistake the ecosystem exists to solve. Three approaches, roughly in order of how much you should want them:

External Secrets Operator / Secrets Store CSI Driver — the secret lives in Vault, AWS Secrets Manager, GCP Secret Manager or Azure Key Vault, and a controller syncs it in or mounts it directly. The cluster stores a reference, not the value; rotation happens at the source.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  secretStoreRef: { name: aws-secretsmanager, kind: ClusterSecretStore }
  target: { name: db-credentials }
  data:
    - secretKey: password
      remoteRef: { key: prod/db, property: password }

Sealed Secrets — encrypt with a cluster-held public key so the encrypted form is safe to commit. Only that cluster’s controller can decrypt it. Good for GitOps, but rotation means re-sealing.

SOPS + age/KMS — encrypt values in-file, decrypt at apply time. Widely used with Flux.

Whichever you pick, two rules hold: the plaintext never enters Git, and rotation must not require a human to remember.

Topic 5: Immutable ConfigMaps and Secrets

apiVersion: v1
kind: ConfigMap
metadata:
  name: api-config-v3
immutable: true
data:
  LOG_LEVEL: "info"

Two benefits, one real constraint:

• Performance. The kubelet stops watching immutable objects, which measurably reduces API server load on large clusters — this is the reason the feature exists.
• Safety. It cannot be edited by accident.
• Constraint: to change it, create a new object (api-config-v4) and update the pod template to reference it. That is a rolling update with a clean rollback, which is arguably the correct workflow anyway.

Topic 6: The Configuration Boundary

A rule that removes a lot of argument: an image should be identical across every environment. If your staging image differs from production, you are not testing what you ship.

containers:
  - name: api
    image: registry.example.com/api:1.4.2      # SAME in dev, staging, prod
    envFrom:
      - configMapRef: { name: api-config }     # differs per environment
      - secretRef:    { name: api-secrets }    # differs per environment

Validate config at startup, loudly:

FATAL: DATABASE_URL is not set

An app that starts with missing config and fails on the first request turns a config error into a production incident. An app that refuses to start turns it into a CrashLoopBackOff that never takes traffic — because the readiness probe never passes and the rolling update stalls. That is the system working exactly as designed, and it depends on your app failing fast.

Try it yourself: Mount a ConfigMap as a volume, then kubectl edit it and watch cat /etc/config/LOG_LEVEL inside the pod. Time how long the change takes to appear. Repeat with a subPath mount and confirm it never updates.

Common mistake: Putting a large config file in a ConfigMap and hitting the 1 MiB etcd limit, then splitting it across several ConfigMaps. That works but the real signal is that the file should be built into the image or fetched at startup — config in etcd is for values that differ per environment, not for shipping data.