Cluster Upgrades, PDBs & Node Drain

Upgrades are the operation most likely to cause a self-inflicted outage, because they exercise every disruption path at once: node drains, PDBs, API removals and controller compatibility.

Topic 1: The Version Skew Rules

Kubernetes supports a bounded mismatch between components, and the rules dictate the upgrade order:

Two rules follow, and they are not optional:

1. Upgrade the control plane first, nodes second. A kubelet newer than the API server is unsupported and will misbehave.

2. One minor version at a time. 1.34 → 1.35 → 1.36. You cannot jump 1.34 → 1.36; the skew rules do not permit it and the upgrade tooling will refuse.

Support window: roughly one year of patches per minor version. The current supported set is 1.34–1.36. Running 1.33 or older means no security patches, which is a compliance problem as much as a technical one.

Topic 2: Before You Touch Anything — API Deprecations

This is the step that turns a routine upgrade into an incident three weeks later.

Removed APIs do not warn at upgrade time. Your manifests fail on the next apply, and — worse — a controller using a removed API silently stops working while its pods stay Running.

# What is served now
kubectl api-resources
kubectl api-versions

# What is still USING a deprecated version
kubectl get --raw='/metrics' | grep apiserver_requested_deprecated_apis

That last metric is the good one — the API server records every request against a deprecated API, with the version it will be removed in:

apiserver_requested_deprecated_apis{group="policy",version="v1beta1",resource="poddisruptionbudgets",removed_release="1.25"} 1

Tools worth running in CI before every upgrade:

pluto detect-files -d ./manifests           # scan manifests
pluto detect-helm --helm-version=v3         # scan installed charts
kubent                                       # kube-no-trouble: scans the live cluster

Historical removals worth knowing because they still appear in old material: PodSecurityPolicy (removed 1.25), extensions/v1beta1 Ingress (1.22), batch/v1beta1 CronJob (1.25), in-tree cloud volume plugins (1.26–1.31).

Check third-party controllers too. Your ingress controller, cert-manager, CSI drivers and operators each have their own supported Kubernetes range. An operator using a removed API breaks quietly, and the symptom is “resources stopped reconciling” with nothing obviously wrong.

Topic 3: PodDisruptionBudgets

A PDB constrains voluntary disruptions — drains, autoscaler scale-down, Karpenter consolidation. It does not constrain involuntary ones: a node crashing, an OOMKill, or a preemption.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  minAvailable: 2              # or maxUnavailable: 1 — never both
  selector:
    matchLabels: { app: api }
  unhealthyPodEvictionPolicy: AlwaysAllow    # 1.31+ — see below

kubectl get pdb
# NAME  MIN AVAILABLE  MAX UNAVAILABLE  ALLOWED DISRUPTIONS  AGE
# api   2              N/A              1                    5d

ALLOWED DISRUPTIONS is the number to read. If it is 0, every eviction will be refused and any drain touching those pods hangs.

The three ways to block your own drain:

1. minAvailable equal to replicas. replicas: 3, minAvailable: 3 → zero disruptions allowed, forever. The most common mistake by a wide margin.

2. A PDB on a single-replica Deployment. replicas: 1, minAvailable: 1 can never permit an eviction. Single-replica workloads should either have no PDB or accept the disruption.

3. Pods that are already unhealthy. Historically, if pods were not Ready, a PDB could refuse eviction of any pod — so a broken workload blocked node maintenance indefinitely. unhealthyPodEvictionPolicy: AlwaysAllow (stable in 1.31) fixes this by letting unhealthy pods be evicted regardless. Set it.

Sensible defaults:

# For replicas >= 3
spec:
  maxUnavailable: 1        # scales naturally as replicas grow

Preferring maxUnavailable over minAvailable means the budget stays correct when you scale, rather than becoming a blocker at low replica counts.

Topic 4: Cordon, Drain, Uncordon

kubectl cordon ip-10-0-1-42                # mark unschedulable; nothing moves yet
kubectl drain ip-10-0-1-42 \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=60 \
  --timeout=10m
# ... do the maintenance ...
kubectl uncordon ip-10-0-1-42

Drain uses the eviction API, not delete. That is what makes it respect PDBs. kubectl delete pod bypasses the budget entirely, which is why you should never “fix” a stuck drain that way — you are removing the protection you configured on purpose.

Diagnosing a stuck drain:

kubectl get pdb -A
# ALLOWED DISRUPTIONS: 0     ← there it is

kubectl get pods --field-selector spec.nodeName=ip-10-0-1-42 -A
kubectl get events --field-selector reason=EvictionBlocked -A

ALLOWED DISRUPTIONS: 0 on any PDB whose pods live on the draining node is your answer, every time.

Topic 5: Performing the Upgrade

Control plane (kubeadm, self-managed):

kubeadm upgrade plan                         # shows what is available and any warnings
sudo kubeadm upgrade apply v1.36.2           # FIRST control-plane node
sudo kubeadm upgrade node                    # each ADDITIONAL control-plane node

# Then the kubelet on each control-plane node
sudo apt-get install -y kubelet=1.36.2-* kubectl=1.36.2-*
sudo systemctl daemon-reload && sudo systemctl restart kubelet

Back up etcd first. Covered in the next lesson; an upgrade is exactly when you want a known-good snapshot.

Worker nodes, one at a time:

kubectl drain "$NODE" --ignore-daemonsets --delete-emptydir-data --timeout=10m
# upgrade kubeadm, run `kubeadm upgrade node`, upgrade kubelet, restart it
kubectl uncordon "$NODE"
kubectl wait --for=condition=Ready node/"$NODE" --timeout=5m

Managed clusters:

aws eks update-cluster-version --name prod --kubernetes-version 1.36
aws eks update-nodegroup-version --cluster-name prod --nodegroup-name ng-1
gcloud container clusters upgrade prod --master --cluster-version 1.36

The provider upgrades the control plane. Node groups are still yours to sequence, and they still drain — so your PDBs still decide whether it completes.

Surge upgrades (EKS updateConfig.maxUnavailable / maxUnavailablePercentage, GKE surge settings) create replacement nodes before draining old ones, which is both faster and safer than in-place. Prefer them.

Blue/green node groups — the safest pattern:

1. Create a new node group on the new version.
2. Cordon every node in the old group.
3. Drain the old group gradually, watching error rates.
4. Delete the old group once traffic has settled.

Rollback is “uncordon the old group and delete the new one” — genuinely reversible, unlike an in-place upgrade.

Topic 6: A Runbook

Before

• Read the release notes and the deprecation guide for every version you cross
• Run pluto / kubent; check apiserver_requested_deprecated_apis
• Verify every third-party controller supports the target version
• etcdctl snapshot save and verify the snapshot
• Confirm every meaningful workload has a PDB with ALLOWED DISRUPTIONS >= 1
• Confirm minReplicas >= 2 on anything that must stay up
• Upgrade a non-production cluster with the same shape first
• Agree a rollback plan and who calls it

During

• Control plane first, then nodes
• One node at a time, --timeout set
• Watch error rates and latency between nodes, not just at the end
• kubectl get pods -A --field-selector status.phase!=Running after each node

After

• kubectl get nodes — all Ready, all on the new version
• kubectl get pods -A | grep -v Running
• Control-plane component health: kubectl get --raw='/readyz?verbose'
• Re-run the deprecated-API check — new deprecations arrived with this version
• Update your local kubectl

# Fast post-upgrade sanity sweep
kubectl get nodes -o custom-columns='NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion,READY:.status.conditions[-1].type'
kubectl get pods -A --field-selector status.phase!=Running
kubectl get events -A --field-selector type=Warning --sort-by=.lastTimestamp | tail -20

Try it yourself: Set minAvailable equal to your replica count and run a drain with --timeout=60s. Read the eviction error, then fix it by raising replicas rather than by weakening the PDB — that is usually the correct production answer.

Common mistake: Deleting a PDB to unblock a drain. It works, and it removes the protection at the exact moment you are deliberately disrupting the workload. Fix the arithmetic instead: raise replicas, or switch minAvailable: N to maxUnavailable: 1.