War-Room Production Outage Drill: Kubernetes Memory Eviction Cascade & CoreDNS Outage

2. Table of Contents

1. Executive Summary
2. Table of Contents
3. Detailed Structured Notes
   • 3.1 Business & Infrastructure Context
   • 3.2 The Incident: Symptoms and Timeline
   • 3.3 Initial Investigation Hypotheses (Trainee Q&A)
   • 3.4 Diagnostic Commands and Observations
   • 3.5 The CoreDNS “Silent Failure” Discussion
   • 3.6 Core Concept: Allocatable vs. Assigned Memory
   • 3.7 Core Concept: Quality of Service (QoS) Classes
   • 3.8 Core Concept: cgroup Memory Enforcement and OOMKilled
   • 3.9 Core Concept: Eviction Thresholds and the MemoryPressure Condition
   • 3.10 OOMKilled vs. Evicted — The Critical Distinction
   • 3.11 The Full Chain Reaction (Root Cause Narrative)
   • 3.12 Troubleshooting Methodology (Layered Diagnostic Order)
   • 3.13 Live Q&A Clarifications
   • 3.14 The CoreDNS Outage (Second Problem Statement / Assignment)
4. Key Concepts Table
5. Architecture & Workflow Analysis
6. Commands & Configurations
7. Tools & Technologies
8. Real-World Production Usage
9. Interview Preparation
10. Exam & Certification Notes
11. Cheat Sheet
12. Gaps & Assumptions

3. Detailed Structured Notes

3.1 Business & Infrastructure Context

Organization (fictional, for the exercise): “CoreBanking” — a fintech company processing payment transactions (checkout, wallet credit/debit, order confirmation, settlement) with an internal analytics dashboard for spend categorization.

Infrastructure:

• Platform runs on AWS EKS, deployed across two regions: ap-south-1 and us-east-1.
• Two parallel lab environments were built to replicate the same outage in both regions, plus a GCP (GKE) replication of the same scenario for cross-cloud practice.
• Critical service: checkout-gateway — described as the single most business-critical microservice, handling the full lifecycle of a payment from initiation to completion. Its downtime is described directly as revenue downtime (“if checkout gateway is down, the business is down”).
• The platform is expected to serve thousands of requests per minute.

Workload layout (all on a single node group):

• Namespace: core-banking
• Node group name: payment-nng
• Node instance type: t3.small
• Desired node count: 2
• No cluster autoscaler and no Karpenter were in use for this node group (explicitly confirmed by the engineer when asked) — capacity was fixed, not elastic.
• No namespace-level ResourceQuota or LimitRange had been configured (explicitly confirmed).

3.2 The Incident: Symptoms and Timeline

At approximately 2:00 PM IST, the payment platform — previously healthy — began exhibiting:

• Increased latency on the checkout-gateway microservice (baseline was under 200ms; this spiked significantly).
• Intermittent HTTP 500 errors on checkout transactions.
• Clients retried failed transactions, which triggered a retry storm — the retries themselves added further load, worsening latency and error rates (a self-reinforcing feedback loop).
• A spike in pod restarts across the payments/core-banking namespace.
• No deployments, no configuration changes, and no code releases had occurred in the window before the incident.
• At the infrastructure level, everything looked outwardly healthy: nodes were Ready, the cluster was reachable, and dashboards showed no obvious red flags at first glance.

Timeline (relative minute markers as given by the engineer):

Business impact:

• ~18% checkout failure rate during the incident window.
• ~12 minutes of transaction processing degradation.
• Customers raised support tickets disputing failed/uncertain transactions.
• The retry storm from clients directly amplified load on the already-stressed infrastructure.

Blast radius: Limited to the core-banking / payments namespace. Other namespaces were unaffected. The control plane remained healthy throughout — this was a data-plane, resource-level incident, not a control-plane failure.

3.3 Initial Investigation Hypotheses (Trainee Q&A)

Before revealing the root cause, the engineer opened the floor for trainees to propose hypotheses, simulating a real incident call. Notable hypotheses raised and how they were addressed:

• “Is there an issue with a third-party API (rate limits, latency)?” → Ruled out by the engineer (“No, no, no”).
• ”Are there namespace-level resource limits causing pods to fail to scale?” → Ruled out — no ResourceQuota/LimitRange existed on the namespace.
• ”Should we check kubectl get events -n kube-system to rule out control-plane/CoreDNS issues?” (raised by a trainee) → Valid technique, but the engineer used this moment to make an important teaching point (see 3.5 below): a CoreDNS meltdown under CPU throttling does not produce obvious crash-style errors — it just gets slow, and slowness is much harder to spot in logs than a stack trace.
• ”What about kubectl top nodes memory (not just CPU)?” (raised by a trainee — flagged by the engineer as a “good question and good catch”) → This became the pivot point of the whole session: CPU was normal, but memory-pressure signals were present when digging deeper (kubectl describe node showed MemoryPressure: True), which CPU-only checks would have missed entirely.

3.4 Diagnostic Commands and Observations

The engineer walked through the actual sequence of commands run against the cluster (see full table in Section 6). Summary of findings at each step:

Key instructor insight during this phase: Seeing OOMKilled alone tempts engineers to jump straight to “just raise the memory limit.” The engineer explicitly warns this is not the correct troubleshooting path — the real issue was architectural (see 3.6–3.11), and simply raising limits would not have resolved it, because the node itself did not have enough allocatable memory to safely support the workloads as designed.

3.5 The CoreDNS “Silent Failure” Discussion

A trainee (“Nish”) raised a nuanced point mid-investigation: if there is a CoreDNS meltdown, would that show up as obvious errors? The engineer’s answer is an important, generalizable lesson:

• Under CPU throttling or resource pressure, CoreDNS does not crash and does not panic — it just becomes slow.
• ”Slow DNS” effectively presents as timeouts elsewhere in the system, not as a DNS-specific error.
• CoreDNS logs may look completely normal (queries received, forwarded, response codes returned) even while DNS resolution is degraded, unless you are specifically checking latency of DNS responses, not just their presence.
• Testing raw connectivity to a public DNS resolver (e.g., pinging/curling 8.8.8.8, or resolving google.com from inside a pod) confirms external internet/DNS reachability works — but this does not prove that internal service-to-service DNS resolution (e.g., resolving checkout-gateway.core-banking.svc.cluster.local) is equally healthy. A trainee suggested explicitly testing internal service DNS resolution with the dig command from within a pod as a more targeted check; the engineer agreed this was a valid technique he had not yet tested in this specific lab environment but that it should work as expected.

This distinction — “no errors in logs” does not mean “no problem” — becomes the conceptual bridge into the second (CoreDNS) outage assigned later in the session.

3.6 Core Concept: Allocatable vs. Assigned Memory

This is the first and most important root-cause concept in the session.

• A node’s capacity (e.g., “4 GB of memory”) is not the amount of memory actually available to pods.
• The OS, the kubelet, the container runtime, and any system daemons/networking agents (e.g., a CNI agent) all reserve a portion of that memory for themselves before any pod workload can use it.
• Allocatable memory = Capacity − Reserved.
• Example given: a node advertised with 4 GB of memory might have only ~3.2 GB allocatable to pods once reservations are subtracted.
• Analogy used: a phone with 64 GB of storage where the OS/system uses ~10 GB, leaving the user ~54 GB usable — the “assigned” number on the box is not the “usable” number.
• The Kubernetes documentation on node allocatable resources provides the exact formula/mechanism and is referenced as the authoritative source for calculating this per-node.

Root architectural mistake in this incident: The system’s original architect designed the workload placement assuming allocatable memory equaled assigned/capacity memory — i.e., ignored the reserved portion entirely. This assumption was the seed of the entire cascade.

3.7 Core Concept: Quality of Service (QoS) Classes

Kubernetes automatically assigns every pod one of three QoS classes, based purely on how CPU/memory requests and limits are set in the pod spec — this is not something explicitly configured as a separate field; it’s derived:

• Under memory pressure, the kubelet evicts pods in this strict order: BestEffort → Burstable → Guaranteed (Guaranteed is evicted only if the node is still failing after removing everything else).
• Eviction is computed per pod, not per node as a single blanket action — each pod is individually scored, and the kubelet removes pods incrementally until the pressure condition clears; it does not necessarily proceed to the next QoS tier if the system stabilizes after the first round of evictions.
• In this incident: the analytical batch job was BestEffort; checkout-gateway and the settlement CronJob were both Burstable. When pressure continued even after BestEffort pods were evicted, the kubelet proceeded to evict Burstable pods — which is how the business-critical checkout-gateway itself became a casualty, not just the background job.
• A PriorityClass can be added to better protect critical pods, but had not been configured in this environment (explicitly noted by the engineer as a gap in the original design).

3.8 Core Concept: cgroup Memory Enforcement and OOMKilled

• Each container runs inside a Linux cgroup, which enforces the container’s individual memory (and CPU) limits.
• If a container exceeds its own configured memory limit, the Linux kernel — not Kubernetes — kills that container immediately. This produces the OOMKilled status.
• This is explicitly not a Kubernetes-level decision. It is a kernel-level, cgroup-enforced action that happens independently of node-level pressure.
• A pod can therefore be OOMKilled even when the node itself is not under memory pressure — it’s purely about that one container exceeding its own limit.

3.9 Core Concept: Eviction Thresholds and the MemoryPressure Condition

• The kubelet continuously monitors the node’s condition based on available memory, disk pressure, and PID pressure.
• When a configured threshold is breached, the kubelet flips the node’s MemoryPressure condition from False to True.
• Once MemoryPressure: True is set, the kubelet proactively starts evicting pods (in QoS order, see 3.7) to protect the node from going down entirely — this is a preventive, node-level action, distinct from the kernel’s per-container OOM kill.
• Eviction thresholds are configurable and come in (at least) two forms referenced in the session:
  • eviction-hard — e.g. (illustrative example given by the engineer): if available memory drops below a hard threshold, the node condition flips to MemoryPressure: True immediately, and the kubelet starts evicting right away.
  • eviction-soft — a softer, typically grace-period-based threshold.
  • The engineer also referenced a third variant/tier of eviction configuration but did not recall its exact name during the session (see Gaps & Assumptions).
• Practically: if a pod status shows Evicted, that unambiguously means the node was under resource pressure and the kubelet intentionally removed the pod to protect the node — the pod does not restart automatically in this case (unlike a normal container crash/restart cycle).

3.10 OOMKilled vs. Evicted — The Critical Distinction

This is called out explicitly as one of the most commonly confused concepts in interviews and real incidents. The engineer states that most candidates he interviews describe these as “basically the same thing,” which is incorrect.

In this incident: the eviction (node-level, kubelet-driven) was the real, root event. The OOMKilled containers seen alongside it were a side effect of the same underlying node allocatable-memory exhaustion — not two separate unrelated problems. Treating OOMKilled as “the” problem and simply bumping container memory limits would not have fixed the underlying node-level exhaustion.

3.11 The Full Chain Reaction (Root Cause Narrative)

Reconstructed end-to-end causal chain as explained by the engineer:

1. Architect assumes allocatable memory = assigned/capacity memory (ignoring OS/kubelet/runtime/CNI reservations) → node’s real usable headroom is smaller than planned for.
2. All three workloads (checkout-gateway [Burstable], analytical batch job [BestEffort], settlement CronJob [Burstable]) are placed on the same node group, with no namespace-level quota and no separation by criticality.
3. The BestEffort analytical batch job has no request/limit ceiling, so the scheduler keeps placing/allowing it to consume memory freely as it scales.
4. Real memory usage climbs and exceeds the node’s true allocatable capacity (which was smaller than assumed, per step 1).
5. The kubelet detects the breach of its configured eviction threshold → node condition MemoryPressure flips to True.
6. Kubelet begins evicting pods in QoS order: BestEffort pods evicted first (the analytical batch job).
7. Pressure does not subside (the underlying capacity miscalculation is still there) → kubelet proceeds to evict Burstable pods next — which includes checkout-gateway itself, since most production workloads (including this critical service) sit in the Burstable tier.
8. In parallel, individual containers exceeding their own cgroup limits are separately OOMKilled by the kernel — a related but mechanistically distinct side effect of the same underlying capacity shortfall.
9. checkout-gateway destabilizes → latency spikes → HTTP 500 errors appear → clients retry failed transactions → a retry storm adds further load → the feedback loop worsens the incident until it naturally subsides (~12 minutes).

Instructor’s explicit warning: If an engineer investigating this incident sees OOMKilled and concludes “just increase pod memory limits,” they will not solve the problem, because the true constraint is the node’s allocatable capacity and workload placement strategy, not any single container’s limit. This is presented as a classic case of treating a symptom instead of the root cause. Note: if scaling (e.g., HPA) is also enabled and its math is set up incorrectly, this same class of cascading resource issue can be triggered by CPU or scaling misconfigurations too, not only memory.

3.12 Troubleshooting Methodology (Layered Diagnostic Order)

Building on a “Kubernetes failure map” concept referenced from a prior session, the engineer lays out a strict, ordered troubleshooting sequence that should always be followed top-down, never starting from the application layer and working backward:

Initial framing questions (ask these first, always):

1. Is the issue on the control plane or the data plane?
2. Is it a pod-level, node-level, or cluster-level issue?
3. Is it an availability issue, a performance issue, or a correctness issue?
4. Is it a random/intermittent issue or a consistent one?

Then, the fixed diagnostic layer order:

1. Node health
        |
2. Resource pressure (Memory / Disk / PID)
        |
3. Scheduling
        |
4. Pod health
        |
5. Service routing
        |
6. Networking
        |
7. Identity / Access (RBAC, IAM, OIDC)
        |
8. Application

Error-signal-to-category mapping (used to classify an incident quickly):

Why events matter most: kubectl get events (sorted by last timestamp) is highlighted as the single highest-value diagnostic step — it surfaces scheduling failures, eviction reasons, admission denials, and volume issues directly, and can eliminate roughly 50% of wrong hypotheses immediately, before deeper manual digging is needed.

Why order matters: Starting troubleshooting from the application layer (as many engineers instinctively do — “is my app healthy?”) wastes significant time, because if the root cause sits at the node/resource/scheduling layer, application-level investigation will never find it. In this incident, checking node conditions (MemoryPressure: True) and allocatable memory would have immediately pointed to the true root cause, whereas jumping straight to “is the checkout-gateway code broken?” would not have.

(The engineer also references a separate written blog post covering each failure category — scheduling, resource, networking, identity, application — in more diagnostic depth, shared with the team separately from this transcript.)

3.13 Live Q&A Clarifications

• Q: “Is Burstable QoS something we configure directly?” A: No — QoS class is not a field you set directly. It is derived automatically by Kubernetes based on how you define requests and limits in the container spec. You control it indirectly through those values.

• Q: “Is a cgroup memory limit hit an OOM event or an eviction?” A: If a container exceeds its own cgroup-enforced limit, that is OOMKilled — a container-level, kernel-driven event. It is only classified as node-level eviction if the node itself crosses its resource pressure threshold. In this incident’s specific chain, the root cause was node allocatable exhaustion → eviction cascade, and the OOM kills were a side effect, not the primary event.

• Q: “If Priority Classes + Preemption are configured, does that override/interfere with QoS-based eviction?” A: No — these are two separate mechanisms operating at different times:

  • Scheduler preemption (influenced by PriorityClass) happens at pod placement time — deciding which pending pod can bump a lower-priority pod to get scheduled.
  • Kubelet eviction (governed by QoS class) happens at runtime, as a survival mechanism to protect an already-running node under resource pressure. PriorityClass does influence eviction ordering within the same QoS tier, but it does not replace or override the QoS-based eviction logic itself.

3.14 The CoreDNS Outage (Second Problem Statement / Assignment)

As a second, parallel learning exercise, the engineer introduces (but does not fully solve live in this transcript) a CoreDNS-related production outage, explicitly labeled to trainees as a DNS outage (this hint is deliberately given up front). Details provided:

• A dedicated cluster (sre-labs-dns-lab on AWS, replicated on GCP/GKE) was built to simulate this outage, containing 4–5 deployments across 2 node pools.
• Trainees are instructed to independently apply the exact same troubleshooting methodology from Section 3.12 (documentation, architecture diagram, and cluster access were shared separately) to diagnose and resolve it themselves as a hands-on assignment.
• The engineer references an additional standalone theoretical video he plans to upload, walking through “what exactly happens inside CoreDNS” and what symptoms appear on screen when CoreDNS is down or degraded — intended as supplementary theory alongside the hands-on lab.
• Conceptually, this outage reinforces the point made in Section 3.5: DNS degradation under load/throttling is a silent failure mode (no crash, no obvious error), so trainees are expected to apply latency-based and response-code-based diagnosis rather than looking only for crash-style log errors.

4. Key Concepts Table

5. Architecture & Workflow Analysis

5.1 Request Flow / Infrastructure Topology

End Users (Clients)
        |
        v
   Load Balancer (ALB)
        |
        v
     Ingress
        |
        v
  EKS Cluster  (Regions: ap-south-1  AND  us-east-1, replicated independently)
        |
        v
  Node Group: "payment-nng"   (instance type: t3.small, desired size: 2, NO autoscaler / NO Karpenter)
        |
        +-- Namespace: core-banking
              |
              +-- Deployment: checkout-gateway   (2 replicas)  [QoS: Burstable]  <-- BUSINESS-CRITICAL
              +-- Deployment: analytical-batch-job (1 replica) [QoS: BestEffort]
              +-- CronJob:   settlement-job                    [QoS: Burstable]

5.2 Downstream Service Dependencies (from checkout-gateway)

checkout-gateway
     |
     +--> Analytics Dashboard (reporting/analytics data)
     +--> Settlement Job (transaction settlement/reconciliation)
     +--> External Third-Party Payment API (external integration)
     +--> Logging Pipeline (centralized logging)

5.3 The Eviction Cascade — Causal Flow Diagram

[Architect assumption: Allocatable Memory == Assigned Memory]  <-- ROOT MISTAKE
                        |
                        v
  All 3 workloads co-located on same node group, no ResourceQuota/LimitRange
                        |
                        v
  BestEffort analytical job consumes memory with no request/limit ceiling
                        |
                        v
  Real memory usage exceeds node's TRUE allocatable capacity
                        |
                        v
  Kubelet detects eviction threshold breach --> Node condition: MemoryPressure = True
                        |
                        v
        +---------------------------------------------+
        |     Kubelet evicts pods in QoS order:        |
        |   1. BestEffort  (analytical-batch-job)       |
        |   2. Burstable   (checkout-gateway!)  <-- HIT |
        |   3. Guaranteed  (n/a here)                    |
        +---------------------------------------------+
                        |
        (in parallel)   v
  Individual containers over their own cgroup limit --> OOMKilled by kernel (side effect)
                        |
                        v
  checkout-gateway destabilizes --> latency spike --> HTTP 500s
                        |
                        v
  Clients retry failed transactions --> RETRY STORM --> more load
                        |
                        v
  Business impact: ~18% checkout failure rate, ~12 min degradation

5.4 Troubleshooting Decision Flow

START
  |
  v
Q1: Control plane or data plane?
  |
  v
Q2: Pod-level, node-level, or cluster-level?
  |
  v
Q3: Availability, performance, or correctness issue?
  |
  v
Check kubectl get events (sorted by lastTimestamp)  --> narrows ~50% of hypotheses
  |
  v
1. Node health  -->  2. Resource pressure  -->  3. Scheduling  -->  4. Pod health
  -->  5. Service routing  -->  6. Networking  -->  7. Identity/Access  -->  8. Application

Each component’s role:

• Load Balancer / Ingress: entry point for external client traffic into the cluster.
• Node Group (payment-nng): fixed-capacity compute layer; no elasticity, so any capacity miscalculation has nowhere to absorb overflow.
• checkout-gateway: the revenue-critical microservice; its Burstable QoS made it vulnerable once BestEffort eviction failed to relieve pressure.
• Analytical batch job (BestEffort): the lowest-protected workload; correctly evicted first, but its uncapped consumption was a major contributor to the pressure in the first place.
• Kubelet: the node-local agent enforcing eviction policy based on configured thresholds.
• Linux kernel / cgroups: enforces hard per-container memory limits independently of kubelet-level node protection.

6. Commands & Configurations

Note: This walkthrough was conceptual/diagnostic and screen-share-driven; no raw YAML manifests (Deployment/CronJob specs, resource requests/limits blocks, or kubelet eviction-hard config files) were dictated verbatim in the transcript. Commands above are reconstructed from the engineer’s spoken descriptions of what was executed and observed.

7. Tools & Technologies

• Amazon EKS (Elastic Kubernetes Service)

  • Purpose: Managed Kubernetes control plane hosting the payment platform.
  • When to use: Production Kubernetes workloads on AWS needing managed control-plane operations.
  • Advantages: Managed upgrades/HA control plane; deep AWS integration (ALB, IAM, EBS/EFS).
  • Limitations: Node-level resource management (allocatable memory, QoS, eviction) is still the operator’s responsibility — EKS does not abstract this away, as this incident demonstrates.

• Google Kubernetes Engine (GKE)

  • Purpose: Used to replicate the same lab scenarios on GCP for cross-cloud practice.
  • When to use: Comparable managed Kubernetes exercises on GCP.
  • Advantages: Similar managed-control-plane benefits to EKS.
  • Limitations: Not deeply explored in this transcript beyond being a replication target.

• kubelet

  • Purpose: Node-local Kubernetes agent responsible for monitoring node conditions (memory/disk/PID pressure) and enforcing pod eviction.
  • When to use: Always running on every node — not something users invoke directly, but its behavior (thresholds) is configurable.
  • Advantages: Protects node stability proactively before a full node crash.
  • Limitations: Its eviction thresholds must be tuned correctly for a given workload profile, or it can evict critical Burstable workloads (as happened here).

• Linux cgroups (control groups)

  • Purpose: Kernel mechanism enforcing per-container CPU/memory limits.
  • When to use: Underlies every containerized workload automatically.
  • Advantages: Hard, reliable enforcement of container resource limits.
  • Limitations: Operates independently of node-level Kubernetes eviction logic — can produce OOMKilled even when the node itself is healthy, or as a side effect during node-level pressure.

• CoreDNS

  • Purpose: In-cluster DNS resolution for service discovery.
  • When to use: Default DNS add-on in most Kubernetes distributions, including EKS.
  • Advantages: Standard, well-integrated service discovery.
  • Limitations: Under CPU throttling/resource pressure, it degrades silently (slow, not crashing), making it a notoriously hard failure mode to detect via logs alone.

• kubectl

  • Purpose: Primary CLI for inspecting and diagnosing cluster/pod/node state.
  • When to use: Every step of this investigation.
  • Advantages: Direct access to events, resource status, and live metrics (top).
  • Limitations: Requires knowing which subcommands/flags to check and in what order — raw access alone doesn’t guarantee a correct diagnosis without the layered methodology in Section 3.12.

• CNI networking agent / add-ons (e.g., kube-proxy, metrics-server, CoreDNS)

  • Purpose: Cluster add-ons that also consume a portion of a node’s reserved (non-allocatable) memory.
  • When to use: Always present on production clusters.
  • Advantages: Provide essential networking/observability/DNS functionality.
  • Limitations: Their resource consumption must be accounted for when calculating true allocatable memory for workloads — a factor the original architect in this incident overlooked.

8. Real-World Production Usage

• Enterprise use case: This exact failure pattern (mixing critical and non-critical workloads on undersized/shared node groups without QoS-aware design) is common in organizations that scale their Kubernetes footprint faster than they mature their resource-governance practices — especially in fast-growing fintech/payments environments where a single microservice carries outsized business risk.

• Production implementation pattern: Separate business-critical services (like checkout-gateway) onto dedicated node groups or node pools, away from best-effort/batch workloads, so that a runaway background job cannot trigger eviction of a revenue-critical service. Alternatively/additionally, apply ResourceQuota/LimitRange at the namespace level to cap what non-critical workloads can consume.

• DevOps/Cloud best practices highlighted:

  • Always calculate and monitor allocatable, not just capacity, memory per node — reference the official Kubernetes documentation formula rather than assuming a 1:1 mapping.
  • Explicitly set requests and limits to intentionally land critical workloads in the Guaranteed QoS class where the business impact justifies it, rather than leaving them in the more vulnerable Burstable tier by default.
  • Configure a PriorityClass for critical workloads as an additional protection layer, on top of (not instead of) QoS-aware design.
  • Consider enabling cluster autoscaler or Karpenter so that genuine capacity shortfalls trigger new node provisioning instead of pod eviction — this environment had neither enabled, removing that safety valve entirely.
  • Use kubectl get events as a first-class, high-priority diagnostic step in any incident response runbook — not an afterthought.

• Security considerations: Not a focus of this particular session, though the layered troubleshooting order does include an explicit “Identity/Access” layer (RBAC, IAM, OIDC) between networking and application, which is directly relevant to production security posture during incident response.

• Cost optimization considerations: Implicitly relevant — using small (t3.small), fixed-size node groups with no autoscaling is a cost-conscious choice, but this incident illustrates the resiliency trade-off of that approach when workload placement/QoS design isn’t also carefully managed. Right-sizing node capacity and enabling elastic scaling (Karpenter/cluster autoscaler) is a cost-vs-resilience balance every team must make deliberately, not by default/oversight.

• Scalability considerations: The engineer explicitly notes that if Horizontal Pod Autoscaling (HPA) were enabled with incorrect scaling math on top of this same flawed allocatable-memory assumption, the same class of cascading resource issue could be triggered via CPU or scaling-driven pressure as well — not exclusively via memory.

9. Interview Preparation

Beginner Questions

Q1: What is the difference between a pod being OOMKilled and a pod being Evicted? A: OOMKilled happens when a single container exceeds its own memory limit; the Linux kernel, via cgroups, kills it — this is a container-level event. Evicted happens when the node itself is under resource pressure (memory/disk/PID); the kubelet proactively removes pods to protect the node — this is a node-level event. They have different causes and require different fixes.

Q2: What are the three Kubernetes QoS classes, and how are they assigned? A: Guaranteed (requests equal limits for every resource), Burstable (some requests set, but not equal to limits), and BestEffort (no requests or limits set at all). They are automatically derived by Kubernetes from how you define requests/limits — you don’t set the QoS class directly.

Q3: In what order does the kubelet evict pods under memory pressure? A: BestEffort pods are evicted first, followed by Burstable pods if pressure continues, and Guaranteed pods only as a last resort.

Intermediate Questions

Q1: Why is “allocatable memory” different from a node’s advertised memory capacity, and why does that matter? A: A portion of a node’s total memory is always reserved for the OS, the kubelet, the container runtime, and system daemons/agents (e.g., CNI). Allocatable memory = Capacity − Reserved. If workload placement assumes allocatable equals capacity (as happened in this incident), the node will run out of real usable memory sooner than expected, triggering premature MemoryPressure and eviction.

Q2: A production pod shows status Evicted. What should you check first, and why? A: Check kubectl describe node <node> for the MemoryPressure (or DiskPressure/PIDPressure) condition — Evicted is a node-level symptom, so the investigation should start at the node, not the pod’s own configuration. Simply raising that pod’s memory limit will not fix a node-level capacity shortfall.

Q3: What is a “retry storm,” and how did it appear in this incident? A: When clients retry failed requests during an outage, those retries add additional load to an already-degraded system, creating a feedback loop that worsens latency and error rates. In this incident, checkout failures triggered client-side retries, which amplified load on the already-pressured checkout-gateway pods.

Q4: Why might kubectl top nodes show “normal” CPU while the node is actually in trouble? A: top reflects live CPU/memory usage snapshots but doesn’t directly surface node conditions like MemoryPressure. A node can have normal CPU utilization while still crossing a memory-based eviction threshold — you need kubectl describe node (or equivalent condition checks) to see pressure conditions directly.

Advanced Questions

Q1: Explain the full causal chain that turned an architectural memory-allocation assumption into a customer-facing checkout outage. A: (See Section 3.11 in full.) In short: incorrect allocatable-memory assumption → co-located critical/non-critical workloads on one node group with no quota → uncapped BestEffort workload consumes memory → real usage exceeds true allocatable capacity → kubelet flips MemoryPressure: True → BestEffort pods evicted first, then Burstable pods (including checkout-gateway) when pressure persists → in parallel, individual containers get OOMKilled as a side effect of the same underlying shortfall → checkout-gateway destabilizes → latency/500s → client retry storm → amplified business impact.

Q2: How do PriorityClass/preemption and QoS-based eviction interact — are they the same mechanism? A: No. PriorityClass (with preemption) operates at scheduling time, determining which pending pod can preempt a lower-priority pod to get placed on a node. QoS-based eviction operates at runtime, as the kubelet’s mechanism for protecting an already-running node under resource pressure. PriorityClass can influence eviction ordering within the same QoS tier, but it does not replace the QoS eviction hierarchy itself. Conflating these two is a common mistake.

Q3: Why can a CoreDNS-related outage be one of the hardest failure modes to diagnose from logs alone? A: Under CPU throttling or resource pressure, CoreDNS typically doesn’t crash or panic — it just becomes slower. This means logs can look entirely normal (queries received, forwarded, response codes returned) while resolution latency silently increases and manifests elsewhere as generic timeouts. Diagnosing this requires checking DNS latency, not just presence/absence of errors, and testing both external DNS reachability and internal service-to-service DNS resolution separately, since one working does not guarantee the other is healthy.

Q4: A namespace has no ResourceQuota/LimitRange, and a BestEffort batch job is co-located with a Burstable production service on the same node group. What’s the risk, and how would you redesign this? A: The BestEffort job has no cap on consumption, so it can drive the node into memory pressure, at which point the kubelet’s QoS-based eviction can escalate to evicting the Burstable production service once BestEffort pods alone aren’t enough to relieve pressure. Redesign options: separate critical and non-critical workloads onto distinct node groups/pools; apply namespace-level ResourceQuota/LimitRange; set explicit requests/limits on the batch job (moving it out of BestEffort); consider Guaranteed QoS plus a PriorityClass for the critical service; enable cluster autoscaler/Karpenter so genuine capacity shortfalls provision new nodes rather than force eviction.

10. Exam & Certification Notes

Relevant primarily to CKA (Certified Kubernetes Administrator) and CKAD (Certified Kubernetes Application Developer) style exam content, as well as general SRE/DevOps interviews:

• Frequently tested concept: The exact QoS class rules — memorize precisely:
  • Guaranteed: every container has both CPU and memory requests and limits set, and they are equal.
  • Burstable: at least one container has a request or limit set, but the pod does not meet the Guaranteed criteria.
  • BestEffort: no requests or limits set on any container in the pod.
• Frequently tested concept: Allocatable memory formula — Allocatable = Capacity − Reserved (kube-reserved + system-reserved + eviction-threshold). Exams may test your understanding that allocatable is always less than or equal to capacity, never more.
• Potential trick question: “A pod is OOMKilled — is this a Kubernetes scheduling problem?” — No. It’s a kernel/cgroup-level container event, not a scheduler or kubelet-eviction event. Don’t conflate the two.
• Potential trick question: “Does setting a PriorityClass prevent QoS-based eviction?” — No. It influences ordering within a QoS tier but does not override QoS eviction logic.
• Memorization-worthy point: Eviction order is always BestEffort → Burstable → Guaranteed, never a different sequence.
• Memorization-worthy point: Evicted pods do not restart automatically; OOMKilled containers typically do (subject to the pod’s restart policy), since it’s treated as a normal container-level restart event.
• Frequently tested troubleshooting concept: The layered diagnostic order (node → resource → scheduling → pod → routing → networking → identity → application) reflects the general “physical to logical” debugging principle emphasized in Kubernetes operations training and is a common structure for scenario-based exam/interview questions.

11. Cheat Sheet

Core Distinction:

• OOMKilled = kernel/cgroup kills one container for exceeding its own memory limit.
• Evicted = kubelet removes a pod because the node is under resource pressure.

QoS Eviction Order: BestEffort → Burstable → Guaranteed

QoS Assignment Rule:

• All requests == all limits → Guaranteed
• Some but not fully matching → Burstable
• No requests/limits at all → BestEffort

Allocatable Memory: Allocatable = Capacity − (kubelet + OS + runtime + system daemons reserved)

Key Commands:

kubectl get pods -n <ns>
kubectl describe pod/<job> -n <ns>
kubectl get endpoints -n <ns>
kubectl get svc -n <ns>
kubectl get nodes
kubectl top pods / kubectl top nodes
kubectl get replicaset -n <ns>
kubectl exec -it <pod> -- curl <endpoint>
kubectl get events -n <ns> --sort-by=.lastTimestamp
kubectl get events -n kube-system
kubectl describe node <node>   # check MemoryPressure / DiskPressure / PIDPressure

Troubleshooting Order (always top-down):

Node health -> Resource pressure -> Scheduling -> Pod health ->
Service routing -> Networking -> Identity/Access -> Application

Error → Category Quick Map:

• Pending → Scheduling
• OOMKilled / Evicted → Resource
• Timeout / 503 / connection refused → Networking
• 401 / 403 → Identity
• All pods unhealthy → Control plane
• CrashLoopBackOff → Application

CoreDNS silent-failure rule of thumb: No crash ≠ no problem — check latency, not just presence of errors; test both external DNS and internal service DNS resolution independently.

12. Gaps & Assumptions

• Unnamed instructor/participants: The transcript does not explicitly label which named program figure (e.g., lead instructor) is delivering the majority of the technical content. Speaker turns are inferred from context (a platform demo presenter identified as “Sankit”; a co-facilitator addressed as “Ravi”; trainees identified in dialogue as “Nish”/“Nisha” and referenced as having questions include “Kishore”). Assumption: the primary technical instructor’s name was not clearly attributable from the transcript text itself and has therefore been left unspecified in this package rather than guessed.
• Eviction threshold example units: The engineer gave an illustrative example of an eviction-hard memory threshold as “less than 100” without stating the unit explicitly. Assumption: based on standard Kubernetes eviction-threshold conventions (e.g., memory.available<100Mi), this is most likely intended as 100Mi, but this was not confirmed verbatim in the transcript.
• Third eviction rule type: The engineer mentioned there being a third eviction rule category beyond “hard” and “soft” but stated on-air that he could not recall its exact name at that moment. This package therefore documents only eviction-hard and eviction-soft as confirmed; the third variant is not included since it was not reliably named in the source material.
• No raw manifests shown: This walkthrough was diagnostic/conceptual and screen-share-driven (viewing live cluster state), not a code-along. No actual YAML (Deployment specs, resource requests/limits blocks, kubelet config files, PriorityClass definitions) was dictated or displayed as text in the transcript. All commands in Section 6 are reconstructed from the engineer’s spoken descriptions of what was executed and observed, not copied from an on-screen file.
• CoreDNS outage not resolved live: The second problem statement (CoreDNS outage) was introduced and hinted at but explicitly left as a self-guided assignment for trainees, with a promised supplementary video and separate lab cluster access. This package documents the setup and the conceptual framing given (Section 3.5 and 3.14) but does not contain a step-by-step resolution, since one was not delivered in this transcript.
• AWS resource quota issue mentioned at the start: The engineer noted that a real AWS resource quota limitation was currently blocking live lab creation for the team at the time of recording, and that trainees would gain lab access once AWS increased the quota. This is program-logistics content, included here only as brief context since it explains why the session was conceptual/screen-share rather than a fully live hands-on walkthrough.
• Non-technical content excluded per instructions: An opening Platform Knowledge Base demo (lab categories “Foundations Track”/“advanced material”/“Elite,” team war-room features, leaderboards, access timelines) and a closing mention of a following day’s resume/LinkedIn optimization session were excluded from this package as general/non-DevOps program administration, per the stated processing rule to ignore non-technical discussion.