Kubernetes Memory Eviction Cascade & CoreDNS Outage

1. Scenario Situation

During a normal trading day, the payment processing gateway (checkout-gateway microservice) began experiencing sudden latency spikes, elevating from a baseline of 200ms to over 15 seconds, accompanied by HTTP 500 errors.

As clients retried their payments, a retry storm ensued, overwhelming the remaining nodes. Concurrently, metrics showed a spike in pod restarts across the core-banking namespace. The active infrastructure was fixed on a 2-node cluster of general-purpose instances with no autoscaling configured.

2. Investigation Phase

Step 2.1: Checking Cluster Events

To rule out quick-failure hypotheses (such as code updates or cloud infrastructure crashes), list the current pod status inside the namespace:

$ kubectl get pods -n core-banking

Output:

NAME                                READY   STATUS      RESTARTS   AGE
checkout-gateway-6fcfbc-x92f        1/1     Running     3          12d
checkout-gateway-6fcfbc-p2b9        0/1     Evicted     0          2d
analytics-batch-job-7a2b-81ff       1/1     Running     0          4h

Note that multiple instances of the critical checkout-gateway service are marked as Evicted.

Step 2.2: Identifying Eviction Triggers

Run a describe query on one of the evicted pods to locate the reason provided by the scheduler:

$ kubectl describe pod checkout-gateway-6fcfbc-p2b9 -n core-banking

Output:

Status:       Failed
Reason:       Evicted
Message:      The node was low on resource: memory. Threshold quantity: 100Mi, Memory available: 95Mi.

The output confirms that the node hosting the pod ran out of memory, triggering a kubelet eviction.

Step 2.3: Analyzing QoS Classes

Check the resource specifications of the running pods by checking requests and limits:

$ kubectl get pods -n core-banking -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass

Output:

NAME                                QOS
checkout-gateway-6fcfbc-x92f        BestEffort
analytics-batch-job-7a2b-81ff       Burstable

The critical checkout-gateway is running under the BestEffort QoS class because it has no resource requests or limits declared in its deployment manifest. The background batch job runs as Burstable (defines request constraints but no hard limit).

3. Root Cause Analysis

3.1 The Eviction Mechanism

There is a critical distinction between an OOMKilled event and an Eviction event:

• OOMKilled (Linux Kernel): Initiated by the kernel Out-of-Memory killer when a container exceeds its declared limit. It targets the specific container process, terminating it while the pod remains on the node.
• Evicted (Kubelet): Initiated by the node’s kubelet daemon when the entire host node reaches physical memory exhaustion thresholds (e.g., available memory drops below 100Mi). The kubelet attempts to save the node from freezing by reclaiming resources.

Because checkout-gateway had no resource limits defined, it defaulted to the BestEffort QoS class. When the heavy analytics-batch-job triggered and consumed node memory, it pushed available node resources below the 100Mi threshold.

According to Kubernetes eviction rules, under resource pressure, BestEffort pods are evicted first to safeguard Burstable and Guaranteed workloads, regardless of their business criticality.

Node Memory Pressure Event (memory.available < 100Mi)
  -> Kubelet triggers Eviction Protocol
  -> Sorts active pods by QoS Priority:
     1. BestEffort (checkout-gateway) - Targeted First
     2. Burstable  (analytics-batch-job) - Target Second
     3. Guaranteed - Targeted Last
  -> checkout-gateway is evicted, causing service downtime.

3.2 CoreDNS Downstream Outage

During the eviction, network queries to CoreDNS began failing silently. Because CoreDNS pods shared the same nodes without CPU limits, CPU throttling on the node caused CoreDNS network queries to drop packets. This caused DNS resolution timeouts on API services, manifesting as silent, non-crashing HTTP 500 errors.

4. Mitigation and Prevention

To resolve the incident and prevent future cascading failures:

1. Configure QoS Class: Ensure all critical services declare identical resource requests and limits to run as Guaranteed:

   resources:
     requests:
       memory: "512Mi"
       cpu: "250m"
     limits:
       memory: "512Mi"
       cpu: "250m"
   

2. Separate Workloads: Taint the primary node group to host only web services, and move background/analytical batch processes to a dedicated node group.
3. Enable Autoscaling: Integrate Karpenter to dynamically provision new nodes before memory thresholds are crossed.