SRE Labs (Advanced Track) — Kubernetes War Room: Checkout API Pod Stuck in Pending State

Live Team-Based Debugging — Pod Anti-Affinity, Node Taints/Tolerations & a Conflicting NodeSelector

2. Table of Contents

1. Executive Summary
2. Table of Contents
3. Detailed Structured Notes
   • 3.1 Infrastructure Overview
   • 3.2 Application Overview
   • 3.3 The Incident — Problem Statement
   • 3.4 Documentation Format (Incident Summary Structure)
   • 3.5 Instructor’s Solo Investigation Pass
   • 3.6 Pre-Hands-On Hypotheses from Participants
   • 3.7 Live Team Debugging — Full Walkthrough
   • 3.8 Root Cause — Final Synthesis
   • 3.9 Required vs. Preferred Affinity — The Core Concept
   • 3.10 Tooling Notes (EKS Access Entries, K8s Lens)
   • 3.11 Session Wrap-Up & Continuation Plan
4. Key Concepts Table
5. Architecture & Workflow Analysis
6. Commands & Configurations
7. Tools & Technologies
8. Real-World Production Usage
9. Interview Preparation (Beginner / Intermediate / Advanced)
10. Exam & Certification Notes
11. Cheat Sheet
12. Gaps & Assumptions

3. Detailed Structured Notes

3.1 Infrastructure Overview

A real, unplanned issue encountered while building this environment (not the designed lesson, but worth noting): the instructor hit actual ENI (Elastic Network Interface) exhaustion in the ap-south-1 region while setting this cluster up — new pods couldn’t get IPs assigned because available ENIs were exhausted, requiring the instructor to manually delete existing unused ENIs before pods could be scheduled with networking. This was explicitly flagged as an infrastructure-build artifact, separate from the actual designed root cause of the exercise.

3.2 Application Overview

• Namespace: payments
• Deployment: checkout-api — framed as the business-critical microservice handling the payment gateway on an e-commerce platform’s checkout page.
• Replicas: 3
• Container image: an Alpine-based nginx image (used as a stand-in application for the exercise)
• Resource requests/limits: requests ~50m CPU / 64Mi RAM; limits ~200m CPU / 256Mi RAM
• Probes: liveness and readiness probes configured
• Service exposure: exposed externally via an AWS Application Load Balancer (ALB); confirmed reachable via curl against the ALB’s DNS name at the start of the session
• Downstream dependency (per the incident documentation, architectural context only): a “risk calculation engine” — a fraud-intelligence microservice — plus logging/monitoring as further downstream consumers.

3.3 The Incident — Problem Statement

• Trigger: during a routine deployment (framed as deploying a hotfix), the team observed that of the 3 checkout-api replicas, 2 entered Running state successfully, and 1 remained stuck in Pending — reportedly for as long as 8 hours in the scenario’s narrative.
• No obvious errors anywhere: no container crashes, no CrashLoopBackOff, no failing liveness/readiness probes, no node-level issues — nodes, CNI, and kube-proxy all appeared healthy from every standard health check.
• Business impact escalation path (as explained): with only 2 of 3 replicas actually serving traffic, load concentrates on the remaining 2 — first manifesting as increased latency, and if unresolved, eventually leading to 503 Service Unavailable errors for end users as the remaining pods become overwhelmed.
• Deployment pipeline blocked: because the rollout couldn’t complete (the new pod never became Ready), the CI/CD pipeline was effectively stuck — meaning no further hotfixes or changes could be deployed until this was resolved, compounding the incident’s severity.
• Declared severity: Severity 1 (highest), based on three red flags: (1) the critical service is in a partially degraded state, (2) the deployment pipeline itself is blocked, and (3) there’s no immediately visible error to point to.
• Stated time pressure: framed as needing resolution within roughly 5–10 minutes in a real e-commerce production scenario, given how quickly latency degradation can cascade into customer-visible failures under real traffic.

3.4 Documentation Format (Incident Summary Structure)

Consistent with the documentation pattern established in earlier war-room sessions in this series, the incident was pre-documented with:

• Incident summary — what happened, in plain language.
• Architecture diagram — showing the request path: client → ALB → NodePort → pod → application, with the fraud-detection engine and logging/monitoring as downstream dependencies.
• Environmental context — cluster name, namespace, workload details, region/subnet layout, networking components, add-ons — all matching the live infrastructure walkthrough in Section 3.1–3.2.
• Blast radius — explicitly separating current impact (internal/pipeline-level) from potential/escalating impact (direct customer-facing failure, financial and reputational loss) if unresolved.
• Upstream / midstream / downstream flow — a deliberate, explicit breakdown of how a request enters the system (upstream: client → ALB → NodePort, with ALB performing health checks against target groups), what happens inside the cluster (midstream: kube-proxy resolving the pod IP via iptables-based routing, e.g. round-robin-style algorithms, then forwarding to the pod/application), and how the response leaves the system (downstream: application → kube-proxy → ALB target group → external client).
• Explicit pedagogical point made about this upstream/downstream framing: understanding exactly how data moves through an infrastructure — in detail, end to end — is presented as a prerequisite for effective troubleshooting; without that mental model, it’s much harder to know where in the chain a gap might exist.

3.5 Instructor’s Solo Investigation Pass

Before handing off to the team, the instructor ran through an initial diagnostic pass live, to model the process:

1. kubectl get pods -n payments -o wide — confirmed 2 pods Running, 1 pod Pending with no IP and no node assigned at all (a specific, important detail — a pod with no node assignment hasn’t even been scheduled yet, distinct from a pod that’s scheduled but failing to start).
2. kubectl describe pod <pending-pod> — reviewed configuration (image, resource requests/limits, probes) but at this stage, no immediately obvious error jumped out in the description.
3. Checked node status — all nodes reported Ready, with healthy-looking resource utilization (CPU/memory both “green”).
4. Checked CNI status/logs — reported healthy via the console UI, though the instructor noted being unable to successfully kubectl exec/grep directly into CNI pod logs during the live session (a minor live friction point, not a finding).
5. Checked kube-proxy logs (last ~20 lines) — showed routine “sync loop” activity and periodic warnings, explicitly characterized as normal background chatter, not errors — iptables rules were confirmed in sync.
6. Checked namespace-level events (kubectl get events -n payments, sorted by timestamp) — surfaced some scheduling-related warning content, setting up the eventual pivot toward the actual root cause area.
7. Reviewed the full deployment manifest (YAML) directly.

Instructor’s mid-investigation synthesis, before handing off: since the application layer showed no errors, the working hypothesis narrowed toward the CNI, kube-proxy, or ENI/networking layer as the likely culprit — explicitly not yet landing on the actual scheduling-constraint root cause at this point, modeling honest, in-progress reasoning rather than jumping straight to the answer.

3.6 Pre-Hands-On Hypotheses from Participants

Before the team took over the keyboard, several participants offered hypotheses based on the symptoms described so far — worth preserving since multiple contributors converged on parts of the actual answer independently:

• Scheduler health — check whether the kube-scheduler component itself is running (acknowledged as rarely an issue on EKS specifically, but still a valid first check).
• Taints, tolerations, and affinity/anti-affinity — explicitly named early by a participant as a likely area, based on the “everything looks healthy but nothing schedules” pattern.
• VPC CNI version compatibility — a participant raised the possibility that the VPC CNI add-on had been updated to a version incompatible with the cluster’s EKS version, which can also silently block pod scheduling/networking.
• Priority classes / preemption policies — raised as another possible scheduling-layer cause.
• A cordoned node — a participant specifically hypothesized that a node might have been marked unschedulable (cordon), which wouldn’t affect already-running pods on that node but would block new pods from landing there — this hypothesis turned out to be directly relevant to the actual root cause.

3.7 Live Team Debugging — Full Walkthrough

The session then handed control to the group, with one participant (“Sai”) volunteering to drive hands-on while the rest of the group collaboratively reasoned through next steps in real time — preserved here in reasonable detail because the process, including its false starts, is the pedagogically valuable part.

Step 1 — Gaining cluster access:

• The driving participant used the EKS “Access Entries” feature (IAM-based cluster access management) to self-grant cluster access, rather than the older aws-auth ConfigMap method.
• A brief, useful side-discussion occurred here: participants explicitly contrasted the two access-management approaches — the aws-auth ConfigMap is the older, more manual method for mapping IAM identities to Kubernetes RBAC permissions; EKS Access Entries is confirmed as the newer, AWS-recommended, more manageable approach (UI/IAM-native), and was used here specifically because it was faster to set up live.

Step 2 — Reading the actual scheduling failure reason:

• kubectl describe pod <pending-pod> on the actual pending pod (not just a general describe) revealed the key events:
  • “One node didn’t match anti-affinity rules."
  • "One node has untolerated taint.”
• This immediately confirmed the affinity/taint hypothesis from Section 3.6.

Step 3 — Checking node state:

• kubectl get nodes showed one node in SchedulingDisabled state (i.e., cordoned) and the other two nodes carrying taints.

Step 4 — Reviewing the deployment YAML for affinity/toleration configuration:

• Found a pod anti-affinity rule using requiredDuringSchedulingIgnoredDuringExecution, keyed on topology.kubernetes.io/hostname (later corrected during live editing — see below) — meaning: no two pods from this deployment may run on the same node, and this rule is a hard requirement, not a preference.
• Critically: no matching tolerations were present in the deployment spec for the taints found on the nodes.
• Root cause hypothesis formed at this point: the taints were very likely added to the nodes after the original 3 pods had already been scheduled and were running — meaning existing pods kept running undisturbed (taints don’t evict already-running pods by default), but this specific rollout’s new pod, needing a fresh scheduling decision, had nowhere valid to go: the cordoned node was unschedulable outright, and the other nodes’ taints blocked placement without a matching toleration.

Step 5 — First fix attempt: loosen anti-affinity from required to preferred (live YAML editing, with genuine friction):

• The team attempted to edit the deployment’s affinity block from requiredDuringSchedulingIgnoredDuringExecution to preferredDuringSchedulingIgnoredDuringExecution live via kubectl edit.
• This produced a realistic sequence of YAML syntax errors that the group worked through together in real time: a missing/misplaced podAffinityTerm wrapper, incorrect indentation levels, a missing required weight field (mandatory specifically for the preferred variant, since it expresses relative priority among multiple preferred rules — the API rejected the edit with “invalid value: 0, must be in range 1–100” until a weight was explicitly added), and a stray/misplaced labelSelector positioning issue.
• After several corrected attempts, the edit was successfully saved.

Step 6 — Confirming the fix wasn’t yet complete:

• Even after correcting the affinity block to “preferred,” kubectl get events still showed a scheduling failure — because the taint/toleration mismatch was still unresolved (loosening affinity alone didn’t address the second contributing cause).
• The team also reasoned through resource availability as a possible contributing factor at this point (i.e., is there actually enough CPU/RAM capacity on the remaining schedulable node), using kubectl top nodes — concluded resources were not the limiting factor.

Step 7 — Uncordoning a node, and an important intermediate observation:

• The team uncordoned the previously-SchedulingDisabled node.
• Important nuance surfaced: this node had no taint (distinct from the other two nodes, which did have taints) — meaning once uncordoned, since it required no toleration, all three pods ended up scheduling onto this single node — technically “fixing” the Pending state, but defeating the actual purpose of the anti-affinity rule (spreading replicas across nodes for resiliency) and not actually validating whether the toleration fix would work on the tainted nodes.
• This was explicitly recognized by the team as an incomplete/unsatisfying fix, prompting further investigation and correction.

Step 8 — Adding tolerations to the deployment:

• The team added a tolerations block to the deployment spec, matching it to the taint present on the batch node group’s node (referenced as a “workload=batch”-style taint, matching the earlier-established batch node group’s Spot-instance role).
• More YAML indentation troubleshooting occurred here as well (the tolerations block initially needed repositioning relative to the container spec).

Step 9 — Discovering and resolving a third contributing misconfiguration — a conflicting nodeSelector:

• Even after the affinity and toleration fixes, a participant noticed the deployment also contained an explicit nodeSelector set to a value like primary — a hard requirement that the pod land specifically on a node carrying that exact label/name.
• This was identified as a genuinely separate, additional misconfiguration: a nodeSelector, pod affinity/anti-affinity rules, and toleration/taint compatibility are all evaluated together with AND logic — every single one must be independently satisfiable for the scheduler to place the pod. A nodeSelector pointing toward a node that doesn’t actually align with the other constraints (or that doesn’t exist/match anything valid, as was effectively the case here) makes the entire combination unsatisfiable, regardless of how correctly the other rules are configured.
• The team commented out (and later confirmed removal of) the conflicting nodeSelector, since hardcoding a specific node name/label is generally poor practice in an autoscaling environment in the first place (node names/instances can change, especially with Spot or autoscaled capacity).

Step 10 — Final validation and rolling restart:

• With all three contributing issues addressed (affinity loosened to preferred for immediate resolution, tolerations added, conflicting nodeSelector removed), the team performed a rolling restart of the deployment (kubectl rollout restart deployment/checkout-api -n payments) to force the still-outdated running pods (which had been scheduled under the old, broken configuration) to be replaced with pods reflecting the corrected spec.
• Confirmed via kubectl get pods and kubectl describe deployment that all 3 replicas were now Running successfully.

3.8 Root Cause — Final Synthesis

The team explicitly enumerated the final, combined root cause as three separate, simultaneously-contributing misconfigurations, not a single cause:

1. Required (hard) pod anti-affinity (requiredDuringSchedulingIgnoredDuringExecution, keyed on node hostname) demanding strict one-pod-per-node placement, in an environment where there weren’t actually enough distinct, untainted, schedulable nodes available to satisfy that strict requirement for all 3 replicas simultaneously.
2. Node taints added to the cluster’s nodes without corresponding tolerations ever being added to the deployment spec — very likely added to the nodes after the original pods were already running (a timing-dependent trap: existing pods were unaffected, but the next fresh scheduling attempt was blocked).
3. A conflicting/incorrect nodeSelector hardcoded to a specific node label/name, which — combined with the other two constraints — made the overall combination of requirements effectively impossible to satisfy.

Discussion of the “correct” permanent fix vs. the fast interim fix: the team explicitly debated two different classes of resolution:

• Fast/interim: keep anti-affinity as preferred rather than required — unblocks scheduling immediately, but weakens the original high-availability guarantee (the scheduler may now colocate replicas on the same node under resource pressure, rather than being forced to spread them).
• Slower/more correct, preserving original intent: add a genuinely new, correctly-configured node to the cluster (with matching taint/label setup) so that required anti-affinity can actually be satisfied by 3 truly distinct nodes — preserving the original resiliency guarantee, at the cost of additional infrastructure and time to provision.
• A related point raised: the cluster did not have a cluster autoscaler configured, meaning capacity issues like this require manual node-group scaling rather than automatic remediation — flagged as a separate, standing gap in the cluster’s resiliency posture, beyond the specific incident being debugged.

3.9 Required vs. Preferred Affinity — The Core Concept

Explained clearly mid-session by a participant, confirmed by the instructor, and worth preserving as the single most important conceptual takeaway from this entire incident:

• preferredDuringSchedulingIgnoredDuringExecution (“preferred”): a soft constraint. The scheduler will try to satisfy the rule, but if it can’t (e.g., no node satisfies the preference), it will still schedule the pod anyway on the best available node rather than leaving it Pending.
• requiredDuringSchedulingIgnoredDuringExecution (“required”): a hard constraint. If no node can satisfy the rule, the pod remains Pending indefinitely — the scheduler will not compromise.
• The ...IgnoredDuringExecution suffix (common to both variants) means the rule is only evaluated at scheduling time — if conditions change after a pod is already running (e.g., a node’s labels change, or a taint is added later, as happened in this exact incident), already-running pods are not affected/evicted because of it. This is precisely the mechanism that explains why the original 3 pods were fine, but a fresh rollout hit the wall.

3.10 Tooling Notes (EKS Access Entries, K8s Lens)

• EKS Access Entries (used live in this session): the current, AWS-recommended, IAM-native way to grant Kubernetes cluster access, superseding the older aws-auth ConfigMap approach. Explicitly discussed and contrasted live, with the group agreeing Access Entries is the better, more modern practice going forward.
• K8s Lens (referenced by the instructor as a recommended tool, in response to a participant question about GUI-based troubleshooting options): a graphical, multi-cluster Kubernetes management interface — supports connecting to and visually browsing multiple clusters (EKS, GKE, AKS, etc.) via kubeconfig, offering a more approachable visualization layer than raw CLI output for exploring workloads, nodes, and cluster state. Recommended especially for participants earlier in their Kubernetes learning curve, or when working across many clusters where a unified visual interface adds real value.

3.11 Session Wrap-Up & Continuation Plan

• The team completed roughly 60–70% of the full incident/exercise within this session; the instructor explicitly offered the group a choice to extend by up to an hour or continue the following day — the group chose to continue the next day.
• Explicit acknowledgment that engagement was uneven: the instructor directly noted that only a small number of participants (three, by name) were actively interactive/hands-on during the live debugging, and checked in with the broader group about whether they were able to follow along — one participant candidly noted limited direct Kubernetes experience made real-time participation difficult without more foundational familiarity first.
• A parallel/independent GCP-based version of the same outage scenario was confirmed as already built by the instructor, for participants wanting to explore and practice the same class of problem independently ahead of the next session.
• Planned for the next session: complete the remaining portion of this incident and its assignment component, cover RCA-writing practice specifically (how to write an effective RCA, with examples), finish the security-hardening track, and begin the ~700-microservice-scale project call — total planned session length roughly 2.5–3 hours.
• Explicit note on Advanced Track difficulty: this specific incident was framed as Foundations Track-level (basic to intermediate); the corresponding Advanced Track version of this class of problem is explicitly framed as intermediate to advanced, and participants were encouraged to make sure they genuinely understood this session’s concepts (not just the specific commands run) before attempting it.

4. Key Concepts Table

5. Architecture & Workflow Analysis

5.1 Request Flow (Upstream → Midstream → Downstream)

UPSTREAM:
  End Customer
       |
       v
  ALB (external endpoint, health-checks target groups)
       |
       v
  NodePort (on a healthy worker node)

MIDSTREAM (inside the cluster):
       |
       v
  kube-proxy (iptables-based routing, e.g. round-robin-style
              algorithm to select a healthy pod IP)
       |
       v
  Pod (checkout-api container, listening on port 80)
       |
       v
  Application generates response

DOWNSTREAM:
       |
       v
  kube-proxy -> Node -> ALB target group
       |
       v
  End Customer receives response

5.2 Scheduling Failure Decision Tree (As Actually Diagnosed)

Pod stuck Pending, no IP, no node assigned
        |
        v
kubectl describe pod -> check Events section
        |
        v
Events show: "anti-affinity rule not matched" +
              "untolerated taint"
        |
        v
kubectl get nodes -> one node SchedulingDisabled (cordoned)
                      other nodes carry TAINTS
        |
        v
Review deployment YAML:
        |
   -----------------------------------------------
   |                    |                        |
REQUIRED anti-     NO tolerations for      nodeSelector
affinity (hard      existing node taints    pointing to a
constraint,                                 conflicting/
one-pod-per-                                non-matching
node, strict)                               node label
   |                    |                        |
   -----------------------------------------------
                        |
                        v
        ALL THREE must be resolved together
        (AND logic -- fixing only one is insufficient)

5.3 Fix Sequence (As Actually Executed, Including False Starts)

1. Attempt: required -> preferred anti-affinity
   (multiple YAML syntax errors fixed live: missing weight,
    misplaced podAffinityTerm, indentation issues)
        |
        v
2. Re-check events: STILL failing
   (toleration gap not yet addressed)
        |
        v
3. Check node resources (kubectl top nodes) -> NOT the bottleneck
        |
        v
4. Uncordon the disabled node
   -> ALL 3 pods land on this ONE node (it has no taint)
   -> "works" but defeats anti-affinity's purpose
        |
        v
5. Add tolerations matching the tainted nodes' taint
        |
        v
6. Discover conflicting nodeSelector ("primary")
   -> remove/comment out
        |
        v
7. Rolling restart deployment
        |
        v
8. Verify: all 3 replicas Running, correctly distributed
        |
        v
9. DISCUSS permanent fix:
   - Keep "preferred" (fast, weaker HA guarantee), OR
   - Add a new correctly-tainted node + revert to "required"
     (slower, preserves original HA intent)

5.4 Constraint Combination Logic (AND, Not OR)

Pod Scheduling Decision
        |
        v
    MUST satisfy ALL of:
        |
   -------------------------------------------
   |               |                          |
nodeSelector    Affinity/Anti-Affinity    Toleration matches
match             rules satisfied          any relevant Taint
   |               |                          |
   -------------------------------------------
                    |
                    v
        If ANY ONE fails -> pod stays Pending
        (assuming required/hard constraints;
         a "preferred" constraint alone won't
         block scheduling, but nodeSelector and
         toleration/taint compatibility ARE
         effectively hard requirements)

6. Commands & Configurations

7. Tools & Technologies

EKS Access Entries

• Purpose: IAM-native mechanism for granting Kubernetes RBAC access to an EKS cluster.
• When to use it: As the current, AWS-recommended replacement for manually editing the aws-auth ConfigMap — simpler to manage, especially for granting/revoking access quickly.
• Contrast: the older aws-auth ConfigMap approach requires directly editing a Kubernetes-native ConfigMap to map IAM ARNs to RBAC roles/groups — more manual and error-prone.

K8s Lens

• Purpose: A graphical, multi-cluster Kubernetes management and visualization tool.
• When to use it: As a more approachable alternative/complement to pure CLI-based troubleshooting, especially useful when managing multiple clusters (EKS, GKE, AKS) from one interface, or for engineers still building CLI fluency.

kubectl (core diagnostic commands used throughout)

• Purpose: The standard Kubernetes CLI, used here for essentially the entire live investigation.
• When to use it: For direct, precise inspection and modification of cluster state — describe, get events, top, edit, and rollout restart were the specific subcommands central to this session’s resolution.

8. Real-World Production Usage

• This session is a genuinely authentic, unpolished demonstration of real Kubernetes incident response, including the actual messiness (YAML syntax errors, iterative correction, false starts like the “uncordon alone” partial fix) that characterizes real live debugging far more accurately than a scripted, error-free walkthrough would — this authenticity is itself valuable, since it models what to actually expect when debugging live under time pressure with a team.
• The “pod stuck Pending with a completely healthy-looking cluster” pattern is extremely common in real production Kubernetes environments, and the diagnostic sequence demonstrated here (check pod events first, then nodes, then the full scheduling-constraint picture: affinity, taints/tolerations, nodeSelector, resources) is a directly reusable, general-purpose troubleshooting sequence for this entire class of incident.
• The “taint added after pods were already running” root-cause pattern is a realistic, easily-overlooked operational trap — teams that add node taints for legitimate reasons (e.g., dedicating nodes to specific workload types) without auditing whether all relevant deployments have matching tolerations can create exactly this kind of delayed, non-obvious incident that only manifests on the next deployment or scale-up event, not immediately.
• The explicit trade-off discussion between “loosen the constraint” and “add real capacity to preserve the original guarantee” reflects a mature, realistic incident-response judgment call that real SREs/platform engineers make constantly — fast mitigation versus fully preserving original architectural intent is a genuine tension, not a simple “right answer,” and this session models discussing that trade-off explicitly rather than picking one silently.
• Using EKS Access Entries over the older aws-auth ConfigMap reflects current, real-world EKS best practice — this is directly relevant, up-to-date operational knowledge for anyone managing real EKS clusters today.
• The three-way combined root cause (anti-affinity + taints/tolerations + nodeSelector) is a realistic illustration of why Kubernetes incidents are so often multi-factor rather than single-cause — and why declaring victory after fixing just the first identified issue (as the team’s own mid-session “uncordon” partial-fix moment demonstrates) can produce an incomplete, symptom-masking resolution rather than a true fix.

9. Interview Preparation

Beginner Questions

Q1: What does it mean when a Kubernetes pod is stuck in Pending state with no node or IP assigned? A: It means the Kubernetes scheduler has not yet been able to place the pod onto any node at all — this is different from a pod that’s been scheduled but is failing to start (e.g., due to an image pull error or crash). A pod with no node assignment indicates a scheduling-level problem: the scheduler couldn’t find any node satisfying all of the pod’s placement constraints (resource requests, node affinity/anti-affinity, taints/tolerations, node selectors).

Q2: What’s the difference between requiredDuringSchedulingIgnoredDuringExecution and preferredDuringSchedulingIgnoredDuringExecution in Kubernetes affinity rules? A: required is a hard constraint — if no node satisfies it, the pod remains Pending indefinitely; the scheduler will not compromise. preferred is a soft constraint — the scheduler tries to satisfy it, but will still place the pod on the best available node even if the preference can’t be met, rather than leaving it unscheduled.

Q3: What’s the first command you’d run to understand why a specific pod is stuck in Pending state? A: kubectl describe pod <pod-name> — specifically, review its Events section, which directly states the scheduler’s reason(s) for failing to place the pod (e.g., “node(s) didn’t match pod affinity/anti-affinity rules,” “node(s) had untolerated taint,” or resource-related messages).

Intermediate Questions

Q4: A node has a new taint added to it, but the pods that were already running on that node before the taint was added are still running fine. Why doesn’t the taint affect them? A: Taints (with the default NoSchedule effect, as opposed to NoExecute) only affect future scheduling decisions — they prevent new pods without a matching toleration from being scheduled onto that node, but they don’t retroactively evict pods that are already running there. This is exactly why a cluster can “look fine” after a taint is added — until the next deployment, rollout, or scale-up event triggers a fresh scheduling decision that then gets blocked.

Q5: You’ve fixed a pod’s anti-affinity rule from required to preferred, but the pod is still stuck in Pending. What would you check next? A: Loosening the anti-affinity constraint only addresses that one specific requirement — if the pod is still failing to schedule, other independent constraints are likely still unsatisfied. Check kubectl get events again for the current failure reason, and specifically review the pod/deployment spec for taints/tolerations mismatches and any nodeSelector that might be pointing to a node that doesn’t actually exist or doesn’t satisfy the other constraints — all of these are evaluated together (AND logic), so fixing one doesn’t guarantee the pod becomes schedulable if others remain unresolved.

Q6: Why might simply uncordoning a previously-cordoned node “fix” a stuck pod, but not actually be a correct or complete fix? A: Uncordoning makes a node schedulable again, which can allow a Pending pod to finally be placed — but if the underlying cause was a combination of issues (e.g., taints on other nodes with no matching tolerations, as in this session), the newly-uncordoned node may become the only viable target, causing all replicas to concentrate there rather than being properly spread according to the deployment’s original anti-affinity intent. This “resolves” the immediate symptom (pod no longer Pending) without addressing the underlying misconfiguration (missing tolerations), and undermines the resiliency goal the anti-affinity rule was meant to enforce in the first place.

Advanced Questions

Q7: Design a systematic diagnostic sequence for a pod stuck in Pending state in an EKS cluster where all standard component health checks (nodes, CNI, kube-proxy) report healthy. A: Start with kubectl describe pod and read the Events section carefully — this almost always directly names the specific scheduling constraint that’s failing, rather than requiring guesswork. Cross-reference with kubectl get nodes to check for cordoned (SchedulingDisabled) nodes and kubectl describe node to check for taints. Review the full pod/deployment spec for nodeSelector, nodeAffinity, podAffinity/podAntiAffinity, and tolerations — and critically, evaluate all of these together as a combined AND condition rather than checking each in isolation, since a single misaligned constraint can make an otherwise-correct configuration unsatisfiable. Check resource requests against actual available node capacity (kubectl top nodes) to rule out simple resource exhaustion. If the cluster recently had any node group, taint, or labeling changes, specifically investigate whether those changes were made after existing pods were already scheduled — a common, easily-overlooked timing-dependent root cause pattern, since ...IgnoredDuringExecution semantics mean already-running pods aren’t retroactively affected by such changes.

Q8: A team wants to enforce strict one-pod-per-node placement for a critical, high-availability service using required pod anti-affinity, but their cluster doesn’t reliably have enough distinct, correctly-configured nodes available to satisfy this at all times (e.g., during a temporary node group scaling event). What are the trade-offs of different approaches to this tension? A: Using required anti-affinity preserves the strongest guarantee — the scheduler will never colocate replicas on the same node, ensuring true node-level fault isolation — but at the cost of pods potentially being stuck Pending (and therefore under-replicated/under capacity) whenever sufficient distinct nodes aren’t available, which is itself an availability risk during exactly the kind of event (node loss, scaling, maintenance) that high availability is meant to protect against. Using preferred anti-affinity instead guarantees pods will always be scheduled (assuming other resources are available), trading away the strict node-spread guarantee for scheduling reliability — meaning under resource pressure, the scheduler might place two replicas on the same node, reducing the fault-isolation benefit exactly when it might matter most. A more robust approach than choosing one or the other outright is ensuring genuinely sufficient, correctly-configured node capacity is reliably available (e.g., via a properly-configured cluster autoscaler with node groups that match the deployment’s actual constraints) so that required anti-affinity’s guarantee can be honored without regularly hitting scheduling failures — treating capacity planning and autoscaling configuration as the actual fix, rather than permanently weakening the availability guarantee to work around insufficient capacity.

Q9: How would you prevent the specific class of incident demonstrated in this session (a taint added to nodes without corresponding tolerations being added to existing deployments) from recurring in a real production environment? A: Establish a change-management practice where any node-group-level change that introduces or modifies taints is explicitly cross-checked against all deployments that might need to schedule onto affected nodes — ideally as an automated check (e.g., a CI/CD or admission-control step that validates a proposed taint change against currently-deployed workloads’ tolerations before it’s applied) rather than a purely manual review, since this exact class of issue is easy to miss precisely because it doesn’t manifest immediately (existing pods keep running fine) and only surfaces on the next fresh scheduling event. Additionally, maintaining clear documentation of the purpose of each node group’s taints (e.g., “this node group is Spot-only, tainted for batch workloads, requires this specific toleration”) makes it much easier for engineers making unrelated deployment changes to recognize when their workload needs a corresponding toleration update. Regular, proactive audits comparing node taints against deployment tolerations across the cluster (rather than waiting for an incident to reveal a mismatch) would catch this class of drift before it causes a production incident.

10. Exam & Certification Notes

(Highly relevant to CKA (Certified Kubernetes Administrator) and CKAD (Certified Kubernetes Application Developer) certifications — this entire incident maps almost directly onto core CKA/CKAD scheduling-domain exam content.)

• Pod affinity/anti-affinity syntax and semantics: Know the exact structure of requiredDuringSchedulingIgnoredDuringExecution vs. preferredDuringSchedulingIgnoredDuringExecution, including that the preferred variant requires a weight field (1–100) and wraps its rule in a podAffinityTerm, while required does not use a weight — a frequently tested syntax distinction, and one this session’s live YAML-editing friction illustrates concretely.
• topologyKey: Understand that this field determines the granularity at which affinity/anti-affinity is evaluated (e.g., per-node via kubernetes.io/hostname, or per-zone via a zone label) — a commonly tested concept for controlling the scope of a spread constraint.
• Taints and tolerations: Know the three taint effects — NoSchedule (blocks new scheduling, doesn’t evict existing pods), PreferNoSchedule (soft version of NoSchedule), and NoExecute (blocks new scheduling AND evicts existing pods lacking a matching toleration, optionally after a grace period via tolerationSeconds) — this session specifically involved NoSchedule-style behavior (existing pods unaffected), a key distinction from NoExecute.
• nodeSelector vs. nodeAffinity: Know that nodeSelector is the older, simpler, exact-match-only mechanism, while nodeAffinity supports more expressive matching (operators like In, NotIn, Exists) and both required/preferred variants — both were present in this incident’s deployment, and both are commonly tested.
• kubectl rollout restart: Know this command’s purpose — forcing a deployment’s pods to be recreated (picking up any spec changes) without requiring an actual image/config version bump — directly relevant to applying corrected scheduling configuration to already-running pods, as done in this session.
• EKS Access Entries vs. aws-auth ConfigMap: Relevant to AWS-specific EKS certification content (not core CKA/CKAD, which are cloud-agnostic) — know Access Entries as the current AWS-recommended IAM-to-RBAC mapping mechanism.

11. Cheat Sheet

Pod Stuck Pending — Diagnostic Order:

1. kubectl describe pod <name> → read Events section first
2. kubectl get nodes → check for SchedulingDisabled (cordoned) nodes
3. kubectl describe node <name> → check taints
4. Review deployment YAML → nodeSelector, nodeAffinity/podAntiAffinity, tolerations
5. kubectl top nodes → rule out resource exhaustion

Required vs. Preferred (memorize):

Constraint Combination Rule: nodeSelector + affinity/anti-affinity + toleration-vs-taint = ALL must pass (AND logic). Fixing one doesn’t help if another is still broken.

Taint Timing Trap: Taints added to a node after pods are already running there do NOT evict those pods (with NoSchedule) — they only block future scheduling attempts without a matching toleration. “It was working before” ≠ “nothing changed.”

Common YAML Gotchas (from this session’s live debugging):

• preferred anti-affinity requires a weight field (1–100)
• podAffinityTerm must correctly wrap the match criteria
• Indentation errors are the most common source of kubectl edit rejections

Fix Trade-off:

• required → preferred = fast, but weakens the HA guarantee
• Add a real, correctly-tainted node = slower, but preserves original intent

Rolling Restart to Apply a Fix:

kubectl rollout restart deployment/<name> -n <namespace>

EKS Cluster Access — Modern Method: EKS Access Entries (IAM-native) > aws-auth ConfigMap (older, more manual)

12. Gaps & Assumptions

• This session ends with the incident only ~60-70% complete per the instructor’s own estimate — the remaining ~30-40% (further assignment components, and presumably a deeper RCA-writing exercise) was explicitly deferred to a follow-up session not captured in this transcript. Treat the resolution documented here as the actual, confirmed fix for the pod-scheduling incident itself, but not necessarily the full scope of the planned exercise.
• Exact YAML content is reconstructed from the live dialogue, not from directly-viewed file content — since this is a text transcript of a live screen-share session, the precise final YAML (indentation, exact field ordering, exact taint key/value/effect strings) is described based on what participants said aloud while editing, not extracted from an actual file. The concepts and sequence of fixes are accurately preserved; treat exact YAML syntax details as illustrative/representative rather than a verbatim copy of the final working manifest.
• The exact taint key/value used (referenced informally as something like “workload=batch”) was not stated with full precision in the transcript — presented here as the best available reconstruction from context (matching the batch node group’s role), not a confirmed exact string.
• Only a subset of participants were actively hands-on during this session, as explicitly acknowledged by the instructor — this document reflects the full technical content of the exercise, but it’s worth noting that real-time engagement was uneven across the full cohort, per the instructor’s own direct check-in.
• The parallel GCP-based version of this same scenario was mentioned as already built but not demonstrated within this transcript — referenced here only as something confirmed to exist for independent practice, not as content covered live in this session.
• This document consolidates a long, genuinely iterative live-debugging session (including real syntax errors and corrections) — content has been reorganized topically for clarity (grouping the debugging steps into a coherent sequence) rather than presented in the exact raw chronological back-and-forth, while preserving the substance and sequence of what was actually tried, in what order, consistent with the approach used for prior packages in this series.