Deployments, ReplicaSets & Rollout Mechanics

A Deployment does not manage pods. It manages ReplicaSets, and each ReplicaSet manages pods. That indirection is the entire mechanism behind rolling updates and rollbacks, and it explains everything you see in kubectl get rs.

Topic 1: The Three-Layer Hierarchy

Deployment          "I want 4 pods of version 1.4.2, updated 1-at-a-time"
   └── ReplicaSet   "I want 4 pods matching pod-template-hash=7d9f"
          └── Pods  the actual containers

Each ReplicaSet owns one immutable pod template. Change the template — a new image, a new env var, a new label — and the Deployment creates a new ReplicaSet and scales the old one down.

kubectl get rs -l app=api
# NAME         DESIRED  CURRENT  READY  AGE
# api-6f4b2c   0        0        0      3d      ← previous version, kept for rollback
# api-7d9f8a   4        4        4      5m      ← current

Old ReplicaSets are kept at zero replicas — they cost nothing and they are your rollback history. revisionHistoryLimit (default 10) controls how many are retained.

The pod-template-hash label:

The Deployment controller hashes the pod template and adds the result as a label to both the ReplicaSet’s selector and its pods. That is how two ReplicaSets with the same app: api label do not fight over each other’s pods — their selectors differ by the hash. You never set this yourself.

Topic 2: The Rolling Update

spec:
  replicas: 4
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1              # may run 5 total during the update
      maxUnavailable: 1        # may drop to 3 available
  progressDeadlineSeconds: 600
  minReadySeconds: 10

Both accept a count or a percentage (25%, the default for each).

The safe production default is maxSurge: 1, maxUnavailable: 0 — capacity never dips below what you asked for, at the cost of needing room for one extra pod. It is also the setting most likely to leave a rollout stuck if the cluster is full, which is a better failure than silently serving with reduced capacity.

minReadySeconds is the underrated one:

A pod counts as available only after it has been ready for this long. Without it, a pod that passes its readiness probe once and then crashes still counts as a successful step, and the rollout marches on, replacing healthy pods with broken ones. Ten to thirty seconds turns “the rollout completed and then everything died” into “the rollout stopped at pod two”.

Topic 3: Driving and Watching a Rollout

kubectl set image deployment/api api=registry.example.com/api:1.4.3
kubectl apply -f deploy.yaml                    # the declarative way

kubectl rollout status deployment/api --timeout=5m     # BLOCKS; non-zero on failure
kubectl rollout history deployment/api
kubectl rollout history deployment/api --revision=3
kubectl rollout pause deployment/api                    # stop mid-rollout
kubectl rollout resume deployment/api
kubectl rollout undo deployment/api                     # back one revision
kubectl rollout undo deployment/api --to-revision=3
kubectl rollout restart deployment/api                  # restart all pods, same image

Two of those deserve emphasis:

kubectl rollout status is the CI gate. It exits non-zero when the rollout fails or times out. A pipeline that runs kubectl apply and stops has verified nothing.

kubectl rollout restart patches an annotation on the pod template with the current timestamp, which changes the hash, which creates a new ReplicaSet — a rolling restart with no image change. It is the correct way to pick up a rotated Secret or a changed ConfigMap, because neither of those triggers a rollout on their own.

Recording what changed:

metadata:
  annotations:
    kubernetes.io/change-cause: "bump api to 1.4.3 (fixes ORD-4471)"

kubectl rollout history shows this column, and a history of <none> entries is useless at 3am. Set it in CI from the commit message.

Pause for a manual canary:

kubectl rollout pause deployment/api
kubectl set image deployment/api api=api:1.5.0
kubectl rollout resume deployment/api        # rollout starts here

Pausing before the change batches multiple edits into one rollout. Pausing during one freezes it partway, leaving both versions serving — a crude canary you can then either resume or undo.

Topic 4: Why Rollouts Stall

A stalled rollout is the most common Deployment problem, and there are only three real causes.

kubectl rollout status deployment/api
# Waiting for deployment "api" rollout to finish: 1 out of 4 new replicas have been updated...
kubectl describe deploy api | tail -15
kubectl get rs -l app=api
kubectl describe pod <new-pod>          # the actual answer is here

1. The new pods never become ready. Bad image, crash on startup, failing readiness probe, missing ConfigMap. With maxUnavailable: 0 the rollout correctly refuses to remove old pods, so you sit at “1 of 4 updated” indefinitely — which is the system protecting you.

2. There is nowhere to put the surge pod. maxSurge: 1 needs room for one more pod. On a full cluster the new pod is Pending with Insufficient cpu, and the rollout waits. This is the failure people misread as “Kubernetes is stuck” when it is actually “you are out of capacity”.

3. A PodDisruptionBudget or terminating pods block the scale-down. Covered in the upgrades lesson.

progressDeadlineSeconds turns a hang into a failure:

spec:
  progressDeadlineSeconds: 600     # default 600; the Deployment gives up after this

kubectl get deploy api -o jsonpath='{.status.conditions[?(@.type=="Progressing")]}' | jq
# {"reason":"ProgressDeadlineExceeded","status":"False", ...}

This does not roll back. It only marks the Deployment as failed and stops kubectl rollout status from blocking forever. Automatic rollback is not a Deployment feature — your pipeline has to do it:

if ! kubectl rollout status deployment/api --timeout=10m; then
    kubectl rollout undo deployment/api
    kubectl rollout status deployment/api --timeout=5m
    exit 1
fi

Topic 5: The Other Strategy, and What Deployments Cannot Do

strategy:
  type: Recreate      # kill everything, THEN start the new version

Recreate accepts downtime in exchange for never running two versions at once. That is the right choice when versions cannot coexist — an incompatible database migration, or a ReadWriteOnce volume that only one pod can mount.

What a Deployment does not give you:

A poor-man’s canary with plain Deployments is two Deployments sharing one Service selector, scaled to 9 and 1 — traffic splits by replica ratio, roughly. It works, and it is the reason progressive-delivery controllers exist.

Topic 6: Zero-Downtime Requires More Than a Rolling Update

The strategy only sequences pod creation. Actually dropping zero requests needs four things, and missing any one of them produces errors that look like a network fault:

1. A readiness probe that means it. Without one, a pod joins the Service the instant the container starts — before the app can serve. The rolling update then confidently removes an old pod.

2. preStop plus a grace period longer than your slowest request.

lifecycle:
  preStop:
    exec:
      command: ["sleep", "5"]
terminationGracePeriodSeconds: 30

The reason is a race worth understanding: when a pod is deleted, two things happen in parallel — the kubelet sends SIGTERM, and the endpoints controller removes the pod from the Service. Endpoint removal has to propagate to every node’s kube-proxy, which takes a moment. Without a preStop delay the app can shut down before the last node has stopped sending it traffic, producing a handful of connection-refused errors on every single deploy.

3. The app handles SIGTERM by draining rather than exiting immediately. An app that dies instantly on SIGTERM drops its in-flight requests.

4. minReadySeconds, so a pod that is briefly ready and then unhealthy does not advance the rollout.

spec:
  minReadySeconds: 10
  strategy:
    rollingUpdate: { maxSurge: 1, maxUnavailable: 0 }
  template:
    spec:
      terminationGracePeriodSeconds: 45
      containers:
        - name: api
          readinessProbe:
            httpGet: { path: /readyz, port: 8080 }
            periodSeconds: 5
          lifecycle:
            preStop:
              exec: { command: ["sleep", "5"] }

Try it yourself: Run a load generator against a Service while you roll out a change, first with no readiness probe and no preStop, then with both. Count the failed requests in each case.

Common mistake: Believing a rolling update is zero-downtime by itself. The default maxUnavailable: 25% explicitly permits a quarter of your capacity to be gone, and with no readiness probe it removes healthy pods to make way for pods that cannot serve. Zero-downtime is a property you configure, not a property you get.