Helm, Kustomize & GitOps

Every environment needs the same manifests with a few values changed. The wrong answer is copying the YAML per environment; the two right answers solve the problem from opposite directions.

Topic 1: Kustomize — Overlay, Don’t Template

Kustomize is built into kubectl (kubectl apply -k) and takes a deliberate position: manifests stay valid YAML, and environments are expressed as patches.

base/
├── kustomization.yaml
├── deployment.yaml
├── service.yaml
└── configmap.yaml
overlays/
├── dev/
│   ├── kustomization.yaml
│   └── replicas.yaml
└── prod/
    ├── kustomization.yaml
    ├── replicas.yaml
    └── resources.yaml

# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
  - configmap.yaml
commonLabels:
  app.kubernetes.io/name: api

# overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: payments-prod
namePrefix: prod-
resources:
  - ../../base
images:
  - name: registry.example.com/api
    newTag: 1.4.2
replicas:
  - name: api
    count: 6
patches:
  - path: resources.yaml
    target: { kind: Deployment, name: api }
configMapGenerator:
  - name: api-config
    behavior: merge
    literals:
      - LOG_LEVEL=warn

kubectl kustomize overlays/prod          # render to stdout
kubectl apply -k overlays/prod
kustomize build overlays/prod | kubectl diff -f -

What makes it worth using:

• The base is real, applyable YAML. You can kubectl apply -f base/ and it works. Nothing is templated, so your editor’s schema validation and every YAML tool still function.
• configMapGenerator appends a content hash to the name (api-config-8f4kd2m9t6) and rewrites every reference. Changing config therefore changes the pod template, which triggers a rollout automatically — the problem the config lesson solved with a checksum annotation, solved here for free.
• No new language. Patches are strategic-merge or JSON6902.

Where it strains:

Conditional logic. “Include this Ingress only in prod” means a separate overlay resource, not an if. That constraint is intentional, and it is also why complex charts do not translate cleanly.

Topic 2: Helm — Package and Template

Helm is a package manager. Its unit is a chart, and it does have a templating language.

mychart/
├── Chart.yaml
├── values.yaml
├── values-prod.yaml
├── templates/
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── _helpers.tpl
│   └── NOTES.txt
└── charts/                 # vendored dependencies

# templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "mychart.fullname" . }}
  labels: {{- include "mychart.labels" . | nindent 4 }}
spec:
  replicas: {{ .Values.replicaCount }}
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
    spec:
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          resources: {{- toYaml .Values.resources | nindent 12 }}
          {{- if .Values.probes.enabled }}
          readinessProbe: {{- toYaml .Values.probes.readiness | nindent 12 }}
          {{- end }}

helm install api ./mychart -f values-prod.yaml
helm upgrade api ./mychart -f values-prod.yaml --atomic --timeout 5m
helm template api ./mychart -f values-prod.yaml       # render WITHOUT installing
helm upgrade api ./mychart --dry-run --debug
helm diff upgrade api ./mychart -f values-prod.yaml   # plugin — shows exactly what changes
helm history api
helm rollback api 3
helm lint ./mychart

--atomic is the flag that matters in CI. It rolls back automatically if the upgrade fails or times out, which turns a failed deploy into a no-op instead of a half-applied state.

helm diff upgrade (a plugin) is the closest thing to terraform plan and is worth installing everywhere.

Where Helm genuinely wins:

• Distribution. Publishing something for others to install — prometheus, cert-manager, ingress-nginx — is what charts are for. The values file is a documented API for your component.
• Release lifecycle. helm history and helm rollback track revisions as first-class objects.
• Dependencies. A chart can pull in subcharts.

Where it hurts:

• Go templating produces YAML by string concatenation. Indentation errors are common, nindent is easy to get wrong, and a template that renders invalid YAML fails confusingly.
• Debugging means rendering. You are never reading what will be applied; you are reading the thing that produces it. helm template constantly.
• Charts sprawl. A chart with 300 values is a configuration language nobody understands, including its author.

Topic 3: Choosing

They compose: a common pattern is helm template a third-party chart and then post-process it with Kustomize (helmCharts is supported in kustomization.yaml), which gives you the vendor’s chart plus your own patches without forking it.

Topic 4: GitOps

Both tools still leave the question of who runs the command. GitOps answers it: nobody. A controller in the cluster reconciles it against a Git repository — the same reconciliation loop from lesson 1, applied to deployment itself.

# Argo CD
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-prod
  namespace: argocd
spec:
  project: payments
  source:
    repoURL: https://github.com/example/manifests
    targetRevision: main
    path: overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: payments-prod
  syncPolicy:
    automated:
      prune: true            # delete what is removed from Git
      selfHeal: true         # revert manual kubectl changes
    syncOptions:
      - CreateNamespace=true

What this changes in practice:

• Git is the source of truth. The cluster’s state is a consequence of the repo, not of command history.
• selfHeal: true reverts manual changes. Someone’s emergency kubectl edit is undone within minutes — which is either exactly what you want or deeply frustrating at 3am. Know which mode you are in before an incident.
• Rollback is git revert. Auditable, reviewable, and the same mechanism as any other change.
• No cluster credentials in CI. The controller pulls; your pipeline only writes to Git. That removes a large blast radius from your build system.

Argo CD and Flux are the two mature options. Argo has a strong UI and an application-centric model; Flux is more composable and lighter. Either is a significant improvement over kubectl apply from a pipeline.

The drift question:

argocd app diff api-prod
kubectl diff -k overlays/prod      # works without GitOps too

kubectl diff is underused: it shows what an apply would change against the live cluster, and it is available with no additional tooling.

Topic 5: Secrets in a GitOps World

The obvious tension: Git holds everything, and Git must not hold secrets. Options, from the config lesson, applied here:

• External Secrets Operator — Git holds a reference; the operator fetches from Vault/AWS/GCP. Rotation happens at the source, and the cluster never has a stored copy in Git.
• Sealed Secrets — encrypted with a cluster public key, so the ciphertext is safe to commit.
• SOPS — encrypt values in-file; Flux decrypts natively at apply time.

All three are acceptable. What is not acceptable is a base64 Secret in a repo, which is the failure mode the whole category exists to prevent.

Topic 6: Practices That Prevent Pain

Render before you apply, always.

helm template api ./chart -f values-prod.yaml | kubectl apply --dry-run=server -f -
kustomize build overlays/prod | kubectl apply --dry-run=server -f -

--dry-run=server runs the request through admission control without persisting, so it catches policy rejections, webhook failures and schema errors that a client-side check cannot.

Validate in CI:

helm lint ./chart
kubeconform -strict -kubernetes-version 1.36.0 <(kustomize build overlays/prod)
kubectl apply --dry-run=server -k overlays/prod

Keep values files small. If a chart has 200 values, most are never changed. Expose what varies; hardcode what does not.

Pin everything. Chart versions, image tags, and targetRevision in Argo. targetRevision: HEAD means your production state changes when someone merges, which is either the point of GitOps or a surprise, depending on whether you meant it.

One environment per directory or values file, never per branch. Branch-per-environment produces merge conflicts and drift that nobody can reason about; directory-per-environment makes the differences visible in one diff.

Try it yourself: Take an existing Deployment and express it as a Kustomize base with two overlays. Run kustomize build overlays/prod | kubectl diff -f - against a live cluster and read exactly what would change. That command is the closest thing to a plan step Kubernetes offers natively.

Common mistake: Templating a chart so heavily that the values file becomes its own configuration language. If reading the template is harder than reading the YAML it produces, the abstraction is costing more than it saves — Kustomize’s insistence on real YAML exists precisely to avoid that failure mode.