The Rollout Freeze

A Deployment Stuck at 2/3 Ready With a Perfectly Healthy Cluster

SEV-1 Kubernetes ~40m

The situation you’re stepping into

A routine rolling update of the checkout-api service in the payments namespace, on the payments-ha-prod EKS cluster (Kubernetes 1.28, ap-south-1). Standard, boring, done a hundred times.

Except this time the rollout freezes at 2/3 Ready. Two replicas go healthy and serve traffic; the third sits in Pending indefinitely and the pipeline stalls.

The environment is textbook-clean:

3 public + 3 private subnets, worker nodes private, ALB Ingress Controller out front; addons CoreDNS, kube-proxy, vpc-cni, metrics-server.
checkout-api: 3 replicas, container nginx:1.25-alpine, simple HTTP probes on /, NodePort → ALB, and a PodDisruptionBudget with minAvailable: 2.

🔥 Stakes

This was declared SEV-1 despite the service being partially alive, because it hit all three red flags of a catastrophic event: (1) a revenue-critical service pinned in a degraded state (2/3 replicas under full load), (2) a completely blocked deploy pipeline — no rollback, no hotfix, no scale — and (3) no visible cause anywhere. This is the “pure war-room” version: no answers in the packet, no hints. You reason it out.

What the team observed

checkout-api-xxxxx stuck in Pending; events show only generic scheduling retries, no explicit error.
After ~10 minutes the deployment controller raised ProgressDeadlineExceeded.
Everything else is green: nodes Ready, ALB healthy, CNI healthy, kube-proxy logs normal, CPU/memory/disk normal, cluster DNS normal, downstream DB/API normal, probes passing on the other pods.
New symptom as it drags on: kubectl drain on a node hangs indefinitely.

🧭 Why 'no crashes' is the hard part

Most engineers debug from logs and dashboards. Here the logs are clean, the dashboards are green, and the pods that do run behave perfectly. A Pending pod has no container running yet, so there is nothing to kubectl logs. The contradiction — “deployment wedged, cluster flawless” — is the whole exercise. You have to interrogate the scheduler’s decision, not the application.

Decision Point 1

A pod is Pending with no node assigned and no IP. What does that state specifically mean versus a pod that scheduled but won't start — and what's the single command that almost always names the reason?

Distinguish a scheduling failure from a startup failure (ImagePullBackOff, CrashLoopBackOff). One of them never gets a node at all.

Commit to your answer, then reveal the responder’s move →

✓What the responder did & why

Pending with no node assigned means the scheduler could not place the pod on any node — a scheduling-level problem. That’s different from a pod that scheduled but is failing to start (image pull error, crash loop, failing probe); those already have a node. The first move is always:

kubectl describe pod -n payments checkout-api-xxxxx    # read the Events section

The Events section normally states the scheduler’s reason verbatim — “0/N nodes are available: N node(s) had untolerated taint …” or “didn’t match pod affinity/anti-affinity” or a resource message. Here, though, events only show generic scheduling retries. So you don’t get a free answer — you have to enumerate the constraints yourself.

📟 Reinforce · Interactive War Room Checkout API Pod Stuck in Pending State → The interactive War Room twin of this incident — drive the describe → events → constraints path live.

Decision Point 2

describe pod gives you only generic retries — no 'insufficient cpu', no named taint. How do you systematically find which placement constraint is unsatisfiable, given they're all evaluated together?

Scheduling constraints are an AND. Enumerate every one and check the cluster against it; a single unsatisfiable constraint wedges the pod even if everything else is fine.

Commit to your answer, then reveal the responder’s move →

✓What the responder did & why

You evaluate all placement inputs as a combined AND condition, because any one unsatisfiable constraint is enough:

kubectl get nodes                         # any SchedulingDisabled (cordoned) nodes?
kubectl describe node <each> | grep -i taint    # taints on nodes
kubectl get deploy checkout-api -n payments -o yaml | \
  yq '.spec.template.spec | {nodeSelector, affinity, tolerations}'   # what the pods demand
kubectl top nodes                         # is it just resource exhaustion?

Walking the checklist — nodeSelector, nodeAffinity, podAffinity / podAntiAffinity, tolerations vs node taints, and resource requests vs allocatable — you find the mismatch: a node carries a taint the checkout-api pods don’t tolerate (equivalently, a required pod-anti-affinity that needs a third distinct, schedulable node that isn’t available, or a node left cordoned). Two replicas fit the two viable nodes; the third has nowhere legal to land.

Decision Point 3

You find a taint (or cordon / anti-affinity change) that was applied AFTER the original pods were scheduled. Why did the two already-running replicas keep working perfectly the whole time?

Taints with NoSchedule and affinity's ...IgnoredDuringExecution semantics are about the future, not the present.

Commit to your answer, then reveal the responder’s move →

✓What the responder did & why

Because these constraints only govern future scheduling decisions, not running pods:

A NoSchedule taint (unlike NoExecute) prevents new pods without a matching toleration from being scheduled onto that node — it does not retroactively evict pods already there.
Pod/node affinity uses ...IgnoredDuringExecution — enforced at schedule time, ignored once the pod is running.

So when the taint/cordon/affinity change landed, the two existing replicas kept humming and the cluster “looked fine.” The landmine only detonated on the next fresh scheduling event — this rollout’s third pod. This timing (change made after pods were scheduled) is the single most overlooked root-cause pattern for a Pending pod, precisely because it doesn’t fail immediately.

📟 Reinforce · Interactive War Room Structured Troubleshooting Framework & RCA Masterclass → The RCA methodology and the exact 'taint added without a matching toleration' lesson, written up as a masterclass.

Decision Point 4

Separately, node drains have started hanging forever. Why — and how is the hanging drain the SAME root cause as the frozen rollout, viewed from another angle?

Count how many replicas are actually Available (2), and recall the PDB: minAvailable 2 of 3.

Commit to your answer, then reveal the responder’s move →

✓What the responder did & why

The two symptoms are one shortage seen through two APIs. The checkout-api PDB is minAvailable: 2, and only 2 of 3 replicas are currently Available (the third is stuck Pending). A kubectl drain must evict a pod, but the eviction API refuses any eviction that would drop the service below its PDB floor — evicting one of the two healthy pods would leave 1 (< 2). So the drain blocks indefinitely, waiting for a third healthy replica that can never schedule. The frozen rollout and the hung drain share the identical root cause: there is no place to put a third checkout-api pod.

📖 Reinforce · Guide NodePool Resource Design & Tool-Selection Framework → How taints, tolerations, and node-pool design interact — the background for getting placement right.

Root cause

A placement constraint made the third replica unschedulable — most commonly a taint added to a node without a matching toleration on the checkout-api deployment (with required pod anti-affinity and/or a cordoned node as equivalent variants). Because such changes only affect future scheduling, the two existing replicas kept running and every dashboard stayed green — hiding the fault until the rollout tried to place a new pod. The minAvailable: 2 PDB then converted the shortage into a second, louder symptom by blocking every node drain, freezing maintenance on top of the frozen deploy.

Resolution and prevention

# Make a third placement legal again (pick the one that matches the real finding):
#  a) add the missing toleration to the deployment
kubectl -n payments patch deploy checkout-api --type merge -p \
  '{"spec":{"template":{"spec":{"tolerations":[{"key":"<taint-key>","operator":"Exists","effect":"NoSchedule"}]}}}}'
#  b) or remove the errant node taint / uncordon the node
kubectl taint nodes <node> <taint-key>-        # or: kubectl uncordon <node>
#  c) or loosen requiredDuringScheduling anti-affinity -> preferred, OR add a correctly-configured node

# Once a third replica schedules -> Deployment reaches 3/3, the PDB floor is satisfied, and drains unblock.

Prevention: treat any taint/cordon/affinity change as a scheduling change — validate proposed node taints against currently-deployed workloads’ tolerations before applying (an admission-control or CI check), document each node pool’s taint purpose, and, for required anti-affinity, guarantee enough correctly-configured node capacity (via the cluster autoscaler) so the strict spread is always satisfiable instead of periodically wedging a rollout.

🎯 Transferable principle

A Pending pod is the scheduler telling you no node satisfies its constraints — evaluate every constraint as an AND, and check whether the cluster changed after the running pods were scheduled. NoSchedule taints and ...IgnoredDuringExecution affinity fail silently on a healthy-looking cluster until the next fresh placement. And when a minAvailable PDB meets a service that’s already below its floor, it will freeze drains and deploys alike — so the frozen rollout and the hung drain are usually the same missing pod.

Telling this story to a recruiter

The 30-second version:

“A routine deploy of our checkout API froze at two of three replicas — the third pod sat Pending forever with no error anywhere: no crashes, green dashboards, healthy nodes. Meanwhile the pipeline was blocked, so we couldn’t roll back or hotfix, and node drains started hanging too. I audited every scheduling constraint as a combined condition and found a node taint that had been added after the existing pods were scheduled, with no matching toleration — invisible until the next rollout tried to place a pod. One toleration fixed the freeze and the hung drains at once, and we added a CI check so taint changes get validated against workload tolerations before they ever land.”

The detailed telling:

Situation. A routine rolling update of a revenue-critical checkout API wedged at 2/3 Ready. The third pod stayed Pending indefinitely; after ten minutes the deployment controller raised ProgressDeadlineExceeded. Every signal an engineer normally leans on was useless: no container crashes, no failing probes, clean logs, nodes Ready, load balancer healthy, resources normal. And this was declared SEV-1 despite the service being “partially up,” because two pods were carrying full production load, the deploy pipeline was frozen — no rollback, no hotfix, no scale — and routine node maintenance had started hanging indefinitely.

Task. Explain why a healthy cluster refused to place one pod, unfreeze the deployment safely, and unblock node maintenance — without guessing, because guessing on the checkout service during production hours isn’t an option.

Action. A Pending pod with no node assignment means one thing: the scheduler evaluated every node and rejected all of them. When describe pod gave me only generic retry events, I enumerated the placement constraints myself, treating them as an AND — node selectors, affinity and anti-affinity, tolerations versus node taints, cordons, and resource requests versus allocatable. The audit surfaced the mismatch: a node carried a taint the deployment didn’t tolerate. The subtle part — and the reason nobody had connected it — was timing: NoSchedule taints don’t evict running pods, and affinity rules are ignored during execution, so the two existing replicas kept serving happily when the taint landed days earlier. The landmine only detonated at the next fresh scheduling decision: our rollout. It also explained the hanging drains perfectly — our PodDisruptionBudget required two available replicas, we only had two healthy ones, so the eviction API correctly refused every drain, waiting for a third replica that could never schedule. Both symptoms were one shortage seen through two APIs. I added the missing toleration, the third pod scheduled immediately, the rollout completed, and drains unblocked on their own.

Result. Deployment pipeline restored and maintenance unblocked with a one-line spec change — no restarts, no rollbacks, no downtime added. Prevention went into the pipeline: any node taint change is now validated against the tolerations of currently-deployed workloads before it applies, and every node pool’s taints are documented with their purpose.

What this story demonstrates. Structured elimination when there are no error messages to lean on, understanding scheduler semantics deeply enough to spot a time-delayed misconfiguration, and connecting two seemingly unrelated symptoms to a single root cause.

Interview deep-dive: the full case study

How the issue happened (the mechanism)

Some time before the incident — days earlier, in routine node-group maintenance — a taint was applied to a node without adding the matching toleration to the checkout-api deployment. Nothing broke at that moment, and that’s the crux of the whole incident: a NoSchedule taint only affects future scheduling decisions. The two checkout-api replicas already running were untouched (NoSchedule doesn’t evict, unlike NoExecute), and affinity rules carry ...IgnoredDuringExecution semantics — enforced at placement time, ignored for running pods. So the cluster carried a time-delayed landmine: every dashboard green, both replicas serving, and a configuration that guaranteed the next fresh scheduling decision would fail.

That next decision arrived with a routine rolling update. The rollout needed to place a third pod; the scheduler evaluated every node against the pod’s combined constraint set — an AND of node selectors, affinity/anti-affinity, taints vs. tolerations, and resource requests — and found no legal placement. The pod sat Pending with no node assigned and no IP, which is the specific signature of a scheduling failure (as opposed to ImagePullBackOff/CrashLoopBackOff, which happen after placement). Events showed only generic retries; after ~10 minutes the deployment controller raised ProgressDeadlineExceeded.

The same shortage then surfaced through a second API. The service’s PodDisruptionBudget (minAvailable: 2) met a deployment that had only 2 healthy replicas — already at its floor. Every kubectl drain calls the eviction API, and the eviction API refuses any eviction that would violate a PDB. Evicting one of the two healthy pods would leave 1 < 2, so every drain hung indefinitely, waiting for a third replica that could never schedule. Frozen rollout and frozen maintenance — one root cause, two symptoms.

Impact

Revenue exposure: checkout ran at 2/3 capacity under full production load — elevated latency, reduced redundancy, and a high blast radius if either surviving node failed.
Operational paralysis: the deploy pipeline was fully blocked — no rollback, no hotfix, no scale-up — which is why this was SEV-1 despite partial availability; the team was “flying without controls.”
Maintenance freeze: node drains hung, blocking patching and node lifecycle work cluster-wide for this service’s nodes.
Diagnostic tax: no crashes, no failing probes, clean logs, green dashboards — every standard signal was useless, which escalated uncertainty and severity.

Steps taken to resolve

Classified the failure mode: Pending + no node + no IP = the scheduler rejected every node; started at kubectl describe pod → Events (normally names the constraint; here, only generic retries).
Enumerated the constraint AND-set manually: kubectl get nodes (cordons), kubectl describe node | grep -i taint (taints), the deployment’s nodeSelector / affinity / tolerations from -o yaml, and kubectl top nodes (ruled out resource exhaustion).
Found the mismatch and its timing: a node taint with no matching toleration on the deployment — applied after the current replicas were scheduled, which is why nothing had failed until this rollout.
Connected the second symptom: hung drains traced to the eviction API correctly enforcing minAvailable: 2 against a service already at 2 — same missing pod.
Fixed with the smallest safe change: added the missing toleration to the deployment spec (one patch); the third pod scheduled immediately, the rollout completed to 3/3, and drains unblocked on their own — no restarts, no rollback, zero added downtime.

Outcomes

Deployment pipeline and node maintenance restored simultaneously with a one-line spec change.
The RCA landed the key mechanism — placement constraints fail silently on running workloads and detonate on the next scheduling event — turning a “mystery freeze” into a teachable, checkable pattern.

What we learned

A Pending pod is the scheduler saying “no node satisfies the AND of every constraint” — the diagnosis is enumeration, not log-reading.
Taints/affinity changes are time bombs, not immediate failures: NoSchedule and IgnoredDuringExecution semantics mean the cluster looks healthy until the next placement.
PDBs convert capacity shortages into frozen operations by design — a hung drain is a symptom worth tracing to why the budget can’t be satisfied, not a nuisance to force.
When no component is erroring, interrogate the control plane’s decisions (scheduler, controllers) rather than the data plane’s health.

Prevention — what we changed so it won’t recur

Pre-apply validation: a CI/admission check that validates any proposed node taint change against the tolerations of currently-deployed workloads — the exact class of drift that caused this can no longer land silently.
Documented node-pool intent: every node pool’s taints and their purpose recorded, so deployment authors know which tolerations their workloads need.
Capacity for strict constraints: where required anti-affinity or strict spreads are used, autoscaling guarantees enough distinct, correctly-configured nodes that the constraint is always satisfiable.
Alerting on scheduling health: alerts on Pending pods older than a threshold and on ProgressDeadlineExceeded, so a wedged rollout pages in minutes instead of being discovered mid-incident.