The Eviction Cascade

The situation you’re stepping into

Incident INC-2026-021, SEV-1, Under Investigation. You’re on a real-time payments platform running on AWS EKS — checkout, wallet debits, order confirmation, settlement, and analytics enrichment.

The environment is small and, importantly, undifferentiated:

• Region ap-south-1, a single cluster payments-prod, 2 managed nodes (t3.small), OIDC enabled.
• Namespace payments, with three workloads all sharing the same node pool:

The architecture note that matters: there is no node isolation between the revenue workload and the batch workloads. They compete for the same CPU and memory on the same two small nodes.

What the pager and dashboards say

At ~14:02 IST the platform starts to wobble:

• Increased latency on checkout requests, intermittent HTTP 5xx, client retries.
• Spikes in pod restarts across the payments namespace.
• checkout-gateway pods restart multiple times; the scheduled reconciliation-job terminates prematurely.

And the “nothing changed” checklist is clean: no deployment in progress, no config change, no manual scaling, no ConfigMap update, no secret rotation, no network-policy change. Infra dashboards show nodes as Ready, and no node reported NotReady.

kubectl get events -n payments --sort-by=.lastTimestamp | tail -40
kubectl describe pod -n payments <checkout-gateway-pod>   # look at Last State / Reason
kubectl describe node <node> | sed -n '/Conditions:/,/Events:/p'

kubectl top pods -n payments --sort-by=memory
kubectl top nodes
kubectl get pods -n payments -o wide          # who is co-scheduled on the pressured node?

Root cause

No resource isolation on a shared, undersized node pool. A memory-heavy batch/reconciliation workload ran on the same two t3.small nodes as the revenue-critical checkout-gateway. When the batch working set grew at peak, the nodes crossed their memory-eviction threshold. Because checkout-gateway wasn’t pinned to a protective QoS (Guaranteed) or priority, the kubelet evicted it to reclaim memory — triggering restarts, a client retry storm, and a namespace-wide instability cascade, all while the nodes still reported Ready.

Containment and durable fix

# Immediate: relieve the pressure and protect revenue
kubectl -n payments scale deploy/analytics-batch --replicas=0     # shed batch load
# (or cordon/drain the pressured node's batch pods; pause the CronJob)
kubectl -n payments patch cronjob reconciliation-job -p '{"spec":{"suspend":true}}'

# Durable prevention:
#  - Set requests == limits on checkout-gateway  -> Guaranteed QoS (evicted last)
#  - Assign a high PriorityClass to revenue workloads
#  - Isolate batch to its own node pool via taints + tolerations / nodeSelector
#  - Add ResourceQuota + LimitRange on the payments namespace
#  - Right-size / autoscale the node pool so peak batch + revenue actually fit

Containment is about decoupling the victim from the cause: stop the batch pressure and protect the revenue pods; recovery follows within a couple of minutes as evictions stop and replicas stabilize.

Telling this story to a recruiter

The 30-second version:

“During peak hours our checkout service started restarting over and over — an 18% payment failure rate — with no deployment, no config change, and every node reporting Ready. I traced it to the kubelet evicting our revenue pods under memory pressure caused by a batch job sharing the same undersized nodes. I stopped the cascade by suspending the batch workload, then fixed the design: Guaranteed QoS and priority for revenue services, and a separate node pool for batch, so a background job could never again take checkout down with it.”

The detailed telling:

Situation. A real-time payments platform on Kubernetes: checkout, wallet debits, settlement. At 14:02 during peak, checkout latency spiked, clients started retrying, and pod restarts rippled across the entire payments namespace. A scheduled reconciliation job died mid-run. The “what changed” checklist was completely clean — no rollout, no scaling event, no config or secret change — and infrastructure dashboards showed every node Ready. Business impact was an 18% checkout failure rate and a 12-minute processing degradation, amplified by a client retry storm.

Task. Find out why healthy-looking infrastructure was shedding revenue-critical pods, stop the cascade, and make sure the pattern couldn’t recur.

Action. My first instinct was to distrust the word “Ready” — it describes kubelet health, not free memory. So instead of reading node status, I read reasons: pod events and describes showed Reason: Evicted with “node was low on resource: memory,” and the nodes carried MemoryPressure conditions. That reframed everything — this wasn’t an app crash-loop, it was the kubelet deliberately reclaiming memory. Next question: what was eating it, with no deploy? A schedule, not a human — the reconciliation CronJob had fired and its working set, plus the analytics batch, was co-located on the same two small nodes as checkout, because the cluster had no workload isolation at all. The cruel twist is that Kubernetes evicts by QoS class, not by blame: checkout had loose resource requests, so it was a first-choice victim while the batch job that caused the pressure kept running. Each eviction triggered restarts, retries piled more load on, and the second node tipped over — the cascade. I suspended the CronJob and scaled the batch deployment to zero to relieve pressure; recovery followed within minutes. Then the real fix: requests equal to limits on checkout for Guaranteed QoS, a high PriorityClass on revenue workloads, batch moved to its own tainted node pool, and namespace quotas.

Result. The cascade stopped the moment the batch pressure lifted; checkout stabilized at full replica count. Post-incident, revenue and batch workloads could no longer compete for the same memory, and eviction ordering worked for us instead of against us. The failure class was retired, not just the incident.

What this story demonstrates. Reading Kubernetes internals precisely instead of trusting dashboard summaries, root-causing to a design flaw (no isolation) rather than a trigger (one hungry job), and shipping prevention that changes the platform’s failure behavior permanently.

Interview deep-dive: the full case study

How the issue happened (the mechanism)

The cluster had a design flaw that stayed invisible until load exposed it: a revenue-critical service and memory-hungry batch workloads shared the same two small nodes (t3.small) with no isolation of any kind — no separate node pools, no taints, no priority classes, and loose (or missing) resource requests on the revenue service.

The trigger wasn’t a human change. The reconciliation-job CronJob fired on schedule during peak, and its working set (plus analytics-batch) drove node memory past the kubelet’s eviction threshold. That’s why the “what changed?” checklist — deploys, configs, secrets, scaling events — came back completely clean: a schedule changed the workload, not a person.

Once a node crosses its memory-eviction threshold, the kubelet must reclaim memory, and it selects victims by QoS class, not by blame:

1. BestEffort pods (no requests/limits) are evicted first,
2. then Burstable pods using more than their memory request, ranked by how far over they are,
3. Guaranteed pods (requests == limits) last.

checkout-gateway had loosely-set requests, so it ranked as an early victim — while the batch job that actually caused the pressure kept running. Each eviction restarted a revenue pod; clients retried, adding load; the rescheduled pods landed on the other node and helped tip it into MemoryPressure; and the cascade rolled across the namespace. Meanwhile every node continued to report Ready, because Ready describes kubelet health, not free memory — the actual signal lived in the node’s separate MemoryPressure condition and in the pods’ Reason: Evicted / OOMKilled records.

Impact

• 18% checkout failure rate during the peak window — direct revenue loss on a payments platform.
• 12-minute degradation of transaction processing, amplified by a client retry storm that added load exactly when capacity was lowest.
• The scheduled reconciliation job terminated prematurely (settlement risk), and support tickets spiked.
• Blast radius stayed namespace-scoped — no control-plane or cluster-wide failure — but every workload in the payments namespace destabilized together.

Steps taken to resolve

1. Distrusted the headline status and read reasons instead: kubectl get events --sort-by=.lastTimestamp, kubectl describe pod (Last State / Reason), kubectl describe node (Conditions) → found Reason: Evicted, “node was low on resource: memory,” OOMKilled, and MemoryPressure: True on Ready nodes.
2. Reframed the question from “why is checkout crashing?” to “what is starving the node?” — eviction is the kubelet acting deliberately, not an app bug.
3. Attributed the memory: kubectl top pods --sort-by=memory, kubectl top nodes, and -o wide co-scheduling showed the reconciliation/batch workloads as the hogs, co-located with checkout replicas.
4. Contained by decoupling victim from cause: suspended the CronJob (kubectl patch cronjob reconciliation-job -p '{"spec":{"suspend":true}}') and scaled analytics-batch to zero — evictions stopped and checkout stabilized within minutes.

Outcomes

• Cascade halted immediately once batch pressure lifted; checkout returned to full replica count and error rates normalized.
• The RCA identified the root cause as an architecture flaw (no workload isolation), with the CronJob only as trigger — which redirected the fix from “make the job smaller” to “make the platform unable to fail this way.”

What we learned

• Ready ≠ healthy. Node pressure conditions and pod eviction reasons are the ground truth; dashboard summaries hide them.
• Kubernetes evicts by QoS class, not by blame — an un-tuned revenue service becomes collateral damage for any noisy neighbor.
• ”Nothing changed” is never literally true. Schedules, autoscaling, and data volume change workloads without a human commit; the timeline must include them.
• Retry storms are part of the blast radius: client behavior under failure adds load precisely when the system can least absorb it.

Prevention — what we changed so it won’t recur

• Guaranteed QoS for revenue workloads: requests == limits on checkout-gateway, so it’s evicted last, not first.
• PriorityClasses: high priority on revenue services so scheduling and preemption decisions favor them explicitly.
• Physical isolation: batch and cron workloads moved to their own tainted node pool (tolerations only on batch), so they can never share memory with checkout again.
• Namespace guardrails: ResourceQuota and LimitRange on the payments namespace, so no workload can be deployed without declared requests/limits.
• Capacity honesty: right-sized/autoscaled node pool so peak batch + revenue actually fit, with alerts on node memory pressure and eviction events — the cascade’s earliest signals.