Project 3: Zero-Downtime Cluster Upgrade

The riskiest routine operation there is, because it exercises every disruption path at once. This project turns it into a checklist with a measurable success criterion: requests dropped.

Step 1: The Pre-Flight Script

Everything that can be checked before you touch anything, is. This script is the deliverable — run it in CI weekly, not just at upgrade time.

#!/usr/bin/env bash
# preflight.sh — refuse to upgrade until the cluster is ready for it
set -Eeuo pipefail

TARGET="${1:?usage: preflight.sh <target-version, e.g. 1.36>}"
fail=0
warn() { printf '  \033[33mWARN\033[0m  %s\n' "$*"; }
bad()  { printf '  \033[31mFAIL\033[0m  %s\n' "$*"; fail=1; }
ok()   { printf '  \033[32m OK \033[0m  %s\n' "$*"; }

echo "== 1. version skew =="
CUR=$(kubectl version -o json | jq -r '.serverVersion.minor' | tr -d '+')
TGT=${TARGET#*.}
if (( TGT - CUR > 1 )); then
    bad "cannot jump ${CUR} → ${TGT}; upgrade one minor at a time"
else
    ok "skew acceptable (1.${CUR} → ${TARGET})"
fi

echo "== 2. node readiness =="
NOTREADY=$(kubectl get nodes --no-headers | grep -cv ' Ready' || true)
(( NOTREADY == 0 )) && ok "all nodes Ready" || bad "${NOTREADY} node(s) not Ready"

echo "== 3. deprecated APIs in use =="
if kubectl get --raw /metrics 2>/dev/null \
     | grep -E '^apiserver_requested_deprecated_apis' | grep -v ' 0$' | grep .; then
    bad "deprecated APIs are being requested (see above)"
else
    ok "no deprecated API usage recorded"
fi
command -v pluto >/dev/null && { pluto detect-helm --target-versions "k8s=v${TARGET}.0" || bad "pluto found removed APIs"; }

echo "== 4. PodDisruptionBudgets that would block a drain =="
BLOCKING=$(kubectl get pdb -A -o json | jq -r '
  .items[] | select(.status.disruptionsAllowed == 0) |
  "\(.metadata.namespace)/\(.metadata.name)"')
if [[ -n $BLOCKING ]]; then
    bad "PDBs with 0 allowed disruptions:"; echo "$BLOCKING" | sed 's/^/          /'
else
    ok "every PDB allows at least one disruption"
fi

echo "== 5. single-replica workloads (will have downtime) =="
kubectl get deploy -A -o json | jq -r '
  .items[] | select(.spec.replicas == 1) |
  "\(.metadata.namespace)/\(.metadata.name)"' | while read -r d; do
    warn "single replica: ${d}"
done

echo "== 6. pods that block node removal =="
kubectl get pods -A -o json | jq -r '
  .items[] |
  select((.metadata.ownerReferences // []) | length == 0) |
  select(.metadata.namespace != "kube-system") |
  "\(.metadata.namespace)/\(.metadata.name)"' | while read -r p; do
    warn "bare pod (no controller, will not be recreated): ${p}"
done

echo "== 7. etcd health =="
kubectl get --raw='/readyz?verbose' 2>/dev/null | grep -E '^\[.\]etcd' || warn "cannot read etcd readyz"

echo "== 8. recent backup =="
LATEST=$(ls -t /backup/etcd-*.db 2>/dev/null | head -1 || true)
if [[ -n $LATEST ]] && [[ $(( $(date +%s) - $(stat -c %Y "$LATEST" 2>/dev/null || stat -f %m "$LATEST") )) -lt 86400 ]]; then
    ok "recent etcd snapshot: ${LATEST}"
else
    bad "no etcd snapshot from the last 24h"
fi

exit "$fail"

Items 3 and 4 are the ones that actually stop upgrades. A removed API breaks a controller silently; a PDB with disruptionsAllowed: 0 hangs the drain forever.

Step 2: The Backup Gate

export ETCDCTL_API=3
CERTS="--cacert=/etc/kubernetes/pki/etcd/ca.crt \
       --cert=/etc/kubernetes/pki/etcd/server.crt \
       --key=/etc/kubernetes/pki/etcd/server.key"
SNAP="/backup/etcd-pre-upgrade-$(date +%Y%m%d-%H%M%S).db"

etcdctl --endpoints=https://127.0.0.1:2379 $CERTS snapshot save "$SNAP"
etcdctl snapshot status "$SNAP" --write-out=table     # VERIFY — never skip
aws s3 cp "$SNAP" s3://cluster-backups/prod/ --sse aws:kms

# Application-level, for selective restore
velero backup create pre-upgrade-$(date +%Y%m%d) --wait

Take both. The etcd snapshot restores the cluster; Velero restores a namespace when only one thing went wrong.

Step 3: Control Plane

# On the FIRST control-plane node
sudo apt-mark unhold kubeadm && sudo apt-get update
sudo apt-get install -y kubeadm=1.36.2-1.1 && sudo apt-mark hold kubeadm

sudo kubeadm upgrade plan                    # read the warnings — do not skim them
sudo kubeadm upgrade apply v1.36.2

# On EACH ADDITIONAL control-plane node
sudo kubeadm upgrade node

# Then the kubelet on every control-plane node
sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=1.36.2-1.1 kubectl=1.36.2-1.1
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload && sudo systemctl restart kubelet

kubectl get nodes           # control-plane nodes now on 1.36.2
kubectl get --raw='/readyz?verbose' | grep -v ok

Managed clusters replace this whole step with one API call, and the provider handles it:

aws eks update-cluster-version --name prod --kubernetes-version 1.36

Step 4: Nodes, One at a Time, Under Load

#!/usr/bin/env bash
# upgrade-nodes.sh — drain, upgrade, verify, repeat. Stop on the first failure.
set -Eeuo pipefail
VERSION="${1:?target version}"

for NODE in $(kubectl get nodes -l '!node-role.kubernetes.io/control-plane' -o name); do
    N=${NODE#node/}
    echo "=== ${N} ==="

    kubectl drain "$N" \
        --ignore-daemonsets \
        --delete-emptydir-data \
        --timeout=10m || { echo "DRAIN FAILED on ${N} — check PDBs"; exit 1; }

    ssh "$N" "sudo apt-mark unhold kubeadm kubelet && \
              sudo apt-get install -y kubeadm=${VERSION}-1.1 && \
              sudo kubeadm upgrade node && \
              sudo apt-get install -y kubelet=${VERSION}-1.1 && \
              sudo apt-mark hold kubeadm kubelet && \
              sudo systemctl daemon-reload && sudo systemctl restart kubelet"

    kubectl uncordon "$N"
    kubectl wait --for=condition=Ready "node/${N}" --timeout=5m

    # Let workloads settle and CHECK before touching the next node
    sleep 60
    UNHEALTHY=$(kubectl get pods -A --field-selector status.phase!=Running --no-headers | wc -l)
    (( UNHEALTHY == 0 )) || { echo "STOPPING: ${UNHEALTHY} unhealthy pods after ${N}"; exit 1; }
    echo "=== ${N} done ==="
done

The sleep 60 plus health check between nodes is the part people skip, and it is the part that turns a bad upgrade into a stopped upgrade rather than a cluster-wide one.

Measure while it runs:

kubectl run -it --rm load --image=williamyeh/hey --restart=Never -- \
  -z 30m -c 20 http://api.shop.svc.cluster.local/

Failed requests is the success metric. Zero, or the workload configuration from Project 1 is wrong — not the upgrade.

The safer alternative — blue/green node groups:

eksctl create nodegroup --cluster prod --name ng-136 --node-ami-family AmazonLinux2023
kubectl cordon -l eks.amazonaws.com/nodegroup=ng-134           # stop scheduling on old
kubectl drain -l eks.amazonaws.com/nodegroup=ng-134 --ignore-daemonsets --delete-emptydir-data
eksctl delete nodegroup --cluster prod --name ng-134

Rollback is “uncordon the old group, delete the new one” — genuinely reversible, unlike an in-place kubelet upgrade.

Step 5: Post-Upgrade Verification

#!/usr/bin/env bash
set -Eeuo pipefail

echo "== versions =="
kubectl get nodes -o custom-columns='NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion'

echo "== unhealthy pods =="
kubectl get pods -A --field-selector status.phase!=Running

echo "== not-ready pods (Running but failing readiness) =="
kubectl get pods -A -o json | jq -r '.items[] |
  select(.status.phase=="Running") |
  select([.status.containerStatuses[]?.ready] | index(false)) |
  "\(.metadata.namespace)/\(.metadata.name)"'

echo "== control plane =="
kubectl get --raw='/readyz?verbose' | grep -v ok || echo "all checks ok"

echo "== warnings since the upgrade started =="
kubectl get events -A --field-selector type=Warning --sort-by=.lastTimestamp | tail -20

echo "== NEW deprecations introduced by this version =="
kubectl get --raw /metrics | grep -E '^apiserver_requested_deprecated_apis' | grep -v ' 0$' || echo "none"

echo "== a real request path =="
kubectl run -it --rm smoke --image=nicolaka/netshoot --restart=Never -- \
  curl -sS -m10 -o /dev/null -w 'HTTP %{http_code} in %{time_total}s\n' \
  http://api.shop.svc.cluster.local/healthz

The last check matters: control-plane health does not prove the data path works. Curl something real.

Step 6: Rollback

Be honest about what is and is not reversible:

On a managed cluster the control-plane upgrade is one-way. That is the strongest argument for upgrading a staging cluster of the same shape first — it is the only rehearsal you get.

The Runbook Template

# Cluster Upgrade: 1.35 → 1.36
Date:            Operator:            Approver:

## Pre-flight (T-7 days)
- [ ] Release notes and deprecation guide read for 1.36
- [ ] preflight.sh exits 0
- [ ] Third-party controllers verified compatible (ingress, cert-manager, CSI, operators)
- [ ] Staging cluster of the same shape upgraded successfully
- [ ] Rollback plan agreed; who calls it, and by when

## T-1 hour
- [ ] etcd snapshot taken AND verified AND copied off-cluster
- [ ] Velero backup complete
- [ ] Load generator running, baseline error rate recorded: ______
- [ ] Change window announced

## Execution
- [ ] Control plane upgraded; readyz clean
- [ ] Nodes upgraded one at a time; health checked between each
- [ ] Failed requests during upgrade: ______   (target: 0)

## Post
- [ ] All nodes Ready on the new version
- [ ] No unhealthy or not-ready pods
- [ ] Smoke test passed
- [ ] New deprecations recorded for next time
- [ ] kubectl updated locally and in CI images

## If it goes wrong
Trigger to roll back: ____________________
Command:              ____________________

Extensions worth building: wire preflight.sh into CI on a weekly schedule so drift is caught continuously; add automated smoke tests per critical service; script the blue/green node group flow end to end.

The lesson to take away: the upgrade commands are three lines. Everything else in this project exists because the failure modes — a removed API, a blocking PDB, a single-replica workload, an unverified backup — are all discoverable before you start, and all invisible once you have.