StatefulSets, DaemonSets, Jobs & CronJobs

Deployments assume every pod is interchangeable. Four other controllers exist for the cases where that assumption breaks.

Topic 1: StatefulSet — Identity That Survives

A Deployment’s pods have random names, random start order and shared storage semantics. A StatefulSet gives each pod three guarantees a Deployment cannot:

1. Stable network identity. Pods are api-0, api-1, api-2 — not api-7d9f-x2k4. A deleted api-1 comes back as api-1, with the same DNS name.

2. Stable storage. Each pod gets its own PVC, created from a template, and it is re-attached to the same ordinal on recreation.

3. Ordered operations. Pods start 0, 1, 2 and each waits for the previous to be Ready. Scale-down and rolling updates go in reverse order.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres          # MUST be a headless Service — this is what gives DNS names
  replicas: 3
  podManagementPolicy: OrderedReady    # or Parallel
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0               # only update ordinals >= this — a canary lever
  selector:
    matchLabels: { app: postgres }
  template:
    metadata:
      labels: { app: postgres }
    spec:
      containers:
        - name: postgres
          image: postgres:17
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:          # one PVC PER POD, not shared
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: gp3
        resources:
          requests: { storage: 100Gi }

postgres-0.postgres.default.svc.cluster.local
postgres-1.postgres.default.svc.cluster.local
PVCs: data-postgres-0, data-postgres-1, data-postgres-2

Things that surprise people:

PVCs are never deleted automatically. Scale from 3 to 1 and data-postgres-1 and data-postgres-2 remain, still costing money. This is deliberate — the data might matter. Scale back up and the old data is reattached. persistentVolumeClaimRetentionPolicy (stable in 1.32) lets you opt into deletion on scale-down or delete.

OrderedReady can deadlock. If postgres-0 never becomes Ready, postgres-1 is never created — so a StatefulSet can be stuck at one broken pod forever. podManagementPolicy: Parallel starts them all at once, which is right when the pods do not actually depend on each other’s ordering.

partition is a real canary mechanism. Set partition: 2 on a 3-replica set and only postgres-2 updates. Verify, then lower the partition to roll the rest.

Do you need one? Only if you need per-pod identity or per-pod storage. A stateless app that happens to have a volume does not need a StatefulSet. And running a database on Kubernetes is a decision to make deliberately — a managed RDS/CloudSQL instance removes an entire category of 3am problems. If you do run one, use a mature operator rather than a hand-written StatefulSet.

Topic 2: DaemonSet — One Pod Per Node

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
spec:
  selector:
    matchLabels: { app: node-exporter }
  template:
    metadata:
      labels: { app: node-exporter }
    spec:
      tolerations:
        - operator: Exists          # run on EVERY node, including tainted ones
      hostNetwork: true
      hostPID: true
      containers:
        - name: node-exporter
          image: quay.io/prometheus/node-exporter:v1.8.2
          securityContext:
            runAsNonRoot: true
            readOnlyRootFilesystem: true

A DaemonSet places one pod on every node that matches its nodeSelector/affinity, and automatically adds pods when nodes join. Used for log collectors, metrics agents, CNI plugins, CSI node drivers and security agents.

It bypasses normal scheduling in one important way: the DaemonSet controller sets nodeAffinity for a specific node directly, and it respects taints only through the tolerations you give it. To run on control-plane or tainted nodes you must tolerate those taints explicitly — tolerations: [{operator: Exists}] tolerates everything, which is what monitoring agents typically want and what application workloads never should.

Operational note: a DaemonSet with high resource requests multiplies across every node. A 500Mi request on 200 nodes is 100Gi of cluster capacity reserved before a single application pod is scheduled. DaemonSet resources deserve more scrutiny than most.

updateStrategy: RollingUpdate with maxUnavailable: 1 is the default; on a large cluster that is a slow rollout, and maxUnavailable: 10% is usually the right change.

Topic 3: Job — Run to Completion

apiVersion: batch/v1
kind: Job
metadata:
  name: migrate
spec:
  completions: 1
  parallelism: 1
  backoffLimit: 4                    # retries before marking Failed
  activeDeadlineSeconds: 600         # hard wall-clock cap, overrides backoffLimit
  ttlSecondsAfterFinished: 3600      # auto-delete the Job (and its pods) after an hour
  podFailurePolicy:                  # stable in 1.31
    rules:
      - action: FailJob              # do not waste retries on a config error
        onExitCodes: { operator: In, values: [42] }
      - action: Ignore               # a preemption is not the job's fault
        onPodConditions:
          - type: DisruptionTarget
  template:
    spec:
      restartPolicy: OnFailure       # or Never — NEVER Always for a Job
      containers:
        - name: migrate
          image: myapp:1.4.2
          command: ["/app/migrate"]

Completion modes:

• completions: 1 — run once.
• completions: 10, parallelism: 3 — ten successful runs, three at a time.
• completionMode: Indexed — each pod gets JOB_COMPLETION_INDEX so it can process shard n. This is how you fan out a partitioned batch.

restartPolicy here is not optional detail. Always is rejected for Jobs. OnFailure restarts the container in the same pod; Never creates a new pod per attempt, which is better for debugging because failed pods stick around with their logs.

ttlSecondsAfterFinished should be set on every Job. Without it, completed Jobs and their pods accumulate forever — a cluster with a year of finished CronJob pods is a real and common source of etcd bloat.

podFailurePolicy is the fix for wasting all four retries on an error that will never succeed (a bad config, exit 42) while still retrying on genuine infrastructure failures.

Topic 4: CronJob — Jobs on a Schedule

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-report
spec:
  schedule: "0 2 * * *"
  timeZone: "Europe/London"          # stable in 1.27 — before this, UTC only
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 300
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  suspend: false
  jobTemplate:
    spec:
      backoffLimit: 2
      ttlSecondsAfterFinished: 86400
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: report
              image: reports:1.2.0

concurrencyPolicy is the field that matters most:

The default is Allow, and it is the wrong default for most real jobs. A report that normally takes 20 minutes but occasionally takes 90 will, under Allow, end up with several copies running simultaneously — competing for the same rows, doubling load on the database, and producing duplicated output. Forbid is the safe choice for anything that is not idempotent and parallel-safe.

Skipped runs are skipped, not queued. Under Forbid, a run that is skipped simply never happens. If you need it to catch up, that is application logic (“process everything since the last watermark”), not a scheduler setting.

startingDeadlineSeconds and the 100-miss rule:

If the CronJob controller is down (or the cluster is), missed runs pile up. When it recovers it starts at most one, then checks: if more than 100 schedule times were missed, it stops scheduling entirely and logs an error. Setting startingDeadlineSeconds bounds how far back it will look, and prevents that state. Without it, a controller outage longer than 100 schedule intervals silently disables the CronJob until you notice.

kubectl get cronjob nightly-report
kubectl get jobs --sort-by=.metadata.creationTimestamp
kubectl create job --from=cronjob/nightly-report manual-run-1     # trigger by hand
kubectl patch cronjob nightly-report -p '{"spec":{"suspend":true}}'

kubectl create job --from=cronjob/... is the correct way to test a CronJob — far better than editing the schedule to two minutes from now and forgetting to change it back.

Topic 5: Choosing the Right Controller

A useful decision test: can any pod serve any request? If yes, Deployment. If a client must reach a specific member, or a pod must reattach to its own disk, StatefulSet.

ReplicaSet directly?

Almost never. A Deployment manages ReplicaSets and gives you rollout history and rollbacks for free. Creating a bare ReplicaSet gives up all of that for nothing.

Topic 6: Debugging Each

# StatefulSet stuck — usually PVC binding or pod-0 not Ready
kubectl get statefulset postgres -o wide
kubectl get pvc -l app=postgres
kubectl describe pod postgres-0 | tail -20

# DaemonSet not on every node — almost always taints
kubectl get ds node-exporter
# DESIRED 6  CURRENT 6  READY 4     ← 2 nodes have pods that are not Ready
kubectl get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints'

# Job failing — read the FAILED pod, not the Job
kubectl get pods -l job-name=migrate
kubectl logs -l job-name=migrate --tail=100
kubectl describe job migrate | tail -20

# CronJob not firing
kubectl get cronjob nightly-report -o jsonpath='{.status.lastScheduleTime}{"\n"}{.spec.suspend}{"\n"}'
kubectl get events --field-selector involvedObject.name=nightly-report

The DaemonSet DESIRED vs READY gap is worth calling out: DESIRED counts nodes the controller wants to place on. If DESIRED is lower than your node count, the missing nodes are excluded by taints or nodeSelector — the controller never even tried.

Try it yourself: Create a CronJob running every minute whose job sleeps for 150 seconds, with concurrencyPolicy: Allow. Watch the pods accumulate. Change to Forbid and watch the skips instead.

Common mistake: Using a StatefulSet because the app “has state”. If the state lives in an external database or object store, the pods are interchangeable and a Deployment is simpler in every way. StatefulSets are for pods whose identity is part of the data model — a Kafka broker, a Postgres replica, a Zookeeper ensemble member.