Kubernetes Outage Follow-Up: Structured Troubleshooting and RCA Writing

Full Root Cause Synthesis (Checkout API Deadlock) + Good vs. Bad RCA + Security Wrap-Up

2. Table of Contents

1. Executive Summary
2. Table of Contents
3. Detailed Structured Notes
   • 3.1 Recap: What Was Fixed in the Prior Session
   • 3.2 Filter vs. Score — Why the First Fix Attempt Alone Wasn’t Enough
   • 3.3 Ruling Out CNI and kube-proxy — Expected Symptom Signatures
   • 3.4 The Control Plane vs. Data Plane Framework
   • 3.5 The Four-Step Pod Lifecycle Checklist
   • 3.6 The Scheduler’s Filter Phase — The Five/Six Possible Causes
   • 3.7 When (and When Not) to Use the OSI Model
   • 3.8 Full Root Cause Synthesis — The Three-Factor Policy Paradox
   • 3.9 Solutions Catalog
   • 3.10 RCA Masterclass — What a Good RCA Must Contain
   • 3.11 The “Bad RCA” Case Study — A Real, Anonymized Example
   • 3.12 When to Write an RCA — Qualifying Triggers
   • 3.13 Interview Framing Guidance
   • 3.14 Security Tooling Wrap-Up
   • 3.15 Program Logistics
4. Key Concepts Table
5. Architecture & Workflow Analysis
6. Commands & Configurations
7. Tools & Technologies
8. Real-World Production Usage
9. Interview Preparation (Beginner / Intermediate / Advanced)
10. Exam & Certification Notes
11. Cheat Sheet
12. Gaps & Assumptions

3. Detailed Structured Notes

3.1 Recap: What Was Fixed in the Prior Session

The prior live-debugging session (documented separately in this series) had changed the deployment’s pod anti-affinity rule from requiredDuringSchedulingIgnoredDuringExecution (hard) to preferredDuringSchedulingIgnoredDuringExecution (soft), added matching tolerations, and removed a conflicting nodeSelector. This walkthrough opens by explaining, from first principles, why that fix was necessary and what the underlying scheduler mechanics actually were — filling in the theory that the live session didn’t have time to fully unpack.

3.2 Filter vs. Score — Why the First Fix Attempt Alone Wasn’t Enough

• The Kubernetes scheduler operates in two distinct phases for every pod it tries to place:
  1. Filter phase: removes any node that violates a hard constraint — the node is entirely excluded from consideration.
  2. Score phase: ranks the remaining (filtered-in) nodes by weight/priority, to choose the best among valid candidates.
• required (hard) anti-affinity operates in the filter phase — if a node would violate the rule, it’s removed entirely from consideration. If this removes every candidate node, the pod stays Pending.
• preferred (soft) anti-affinity operates only in the score phase — it never removes a node from consideration; it only adds or subtracts weight/priority among nodes that already passed the filter phase. This means a preferred rule can never by itself cause a pod to stay unschedulable — worst case, it’s simply not honored.
• Why this matters for this incident specifically: converting the anti-affinity rule from required to preferred was a necessary step (it stopped anti-affinity from acting as a hard filter), but — as this walkthrough goes on to explain — it wasn’t sufficient on its own, because other, separate filter-phase-relevant constraints (the node selector, and the PDB’s interaction with the affinity setup) were also contributing to the overall incident.

3.3 Ruling Out CNI and kube-proxy — Expected Symptom Signatures

A key part of this walkthrough is teaching participants the specific, expected symptom signature for each major failure category, so hypotheses can be confirmed or killed quickly based on evidence rather than guesswork:

If this had been a CNI issue, you would expect to see:

• An event specifically named FailedCreatePodSandBox.
• Containers stuck in the ContainerCreating or Init phase (i.e., the pod has been scheduled, but networking setup for it is failing).
• Pods with no IP assigned, but accompanied by ENI-attachment errors or incorrect routing evidence.

If this had been a kube-proxy issue, you would expect to see:

• “Service unreachable” errors.
• NodePort traffic not being forwarded correctly.
• Inconsistent timeouts — some requests succeeding, others failing/timing out — since kube-proxy issues typically manifest as routing problems affecting some traffic, not a total scheduling block.

Why neither applied here: the pod in this incident never progressed past the scheduling step at all — it had no node and no IP assigned from the very start, and CNI/kube-proxy-related failures, by definition, only become relevant after a pod has been successfully scheduled and is at least attempting to start. Since the actual observed symptom (permanently Pending, no node, no IP) doesn’t match either signature above, both hypotheses can be confidently ruled out without needing to inspect their logs at all — a significant time-saver.

Explicitly flagged as a preview: an upcoming exercise in this same cluster environment will deliberately introduce genuine CNI and kube-proxy issues (by adding more microservices), specifically so participants get to practice recognizing those signatures as well, in contrast to this walkthrough’s pure scheduling incident.

3.4 The Control Plane vs. Data Plane Framework

Presented as the foundational mental model for any Kubernetes troubleshooting, regardless of experience level — described explicitly as being about structured clarity, not seniority or years of experience.

Kubernetes has two conceptual “worlds”:

Core discipline being taught: many engineers instinctively mix investigation across both worlds simultaneously, wasting time. The correct approach is to first determine which world the symptom actually belongs to, and then confine investigation entirely to that world until it’s genuinely exhausted. In this incident, the symptom (pod not scheduling) is unambiguously a control-plane problem — so kube-proxy, CNI, networking, and traffic-flow investigation (all data-plane concerns) should be set aside immediately, not explored “just in case.”

3.5 The Four-Step Pod Lifecycle Checklist

The second core framework taught in this walkthrough — a strict, ordered sequence that should never be skipped:

1. Was the pod created?
2. Was the pod scheduled?
3. Is the pod running?
4. Is the pod reachable (by a client)?

The rule: work through this checklist in order, and stop at the first stage where the pod fails. Whatever stage the pod is actually stuck at determines the entire scope of relevant causes — causes belonging to later stages are categorically irrelevant and should not be investigated.

Mapping symptoms to likely cause categories:

• Stuck at container creation → likely CNI or kubelet-related issue.
• Stuck at scheduling (pod created, but never gets a node) → scheduling-layer issue (node selector, taints, affinity, resources, topology).
• Running, but traffic failing → networking-layer issue (kube-proxy, DNS, ALB, service configuration) — this is the scenario where OSI-style layered troubleshooting genuinely applies.
• CrashLoopBackOff → application-layer issue.

Applied to this incident: the pod was confirmed stuck at stage 2 (never scheduled) — meaning stages 3 and 4 (running, reachability) and all their associated causes (kube-proxy, DNS, ALB, application logic) are immediately and confidently ruled out, without needing to investigate them at all. The engineer states this discipline alone — correctly identifying which stage a pod is stuck at, and refusing to investigate later-stage causes — saves approximately 20 minutes of unfocused troubleshooting time in a typical incident.

3.6 The Scheduler’s Filter Phase — The Five/Six Possible Causes

Since this incident was confirmed to be a filter-phase failure (per Section 3.2), the scope of possible causes narrows to a small, specific, enumerable list:

1. Node selector mismatch
2. Taints without matching tolerations
3. Affinity/anti-affinity rules
4. Insufficient resources (requested resources exceed what any candidate node can provide)
5. Topology constraints (e.g., topologySpreadConstraints)

Applied to this incident: the actual root cause involved node selector and affinity (with PDB as an amplifying third factor covered separately in Section 3.8) — resources and topology constraints were specifically confirmed not to be contributing factors in this case, having been checked and ruled out.

3.7 When (and When Not) to Use the OSI Model

• The OSI model is explicitly the wrong tool for this incident, and the engineer uses this as a teaching moment about tool selection generally: OSI-layer troubleshooting is designed for situations where a pod/service is already running but traffic/connectivity to it is failing — it starts from an assumption (a running workload with a networking problem) that simply didn’t apply here, since the pod never got past scheduling at all.
• Cost of misapplying OSI here: since it requires methodically working through up to seven layers, misapplying it to a pure scheduling problem would waste an estimated 20–30 minutes investigating layers that were never relevant to begin with.
• General principle stated: “If you are able to do troubleshooting without OSI, that’s good — only reach for OSI when you genuinely have no other structured path forward.” OSI should be treated as one tool among several structured frameworks (alongside the control-plane/data-plane split and the pod-lifecycle checklist), selected based on which one actually matches the observed symptom, not applied reflexively to every incident.

3.8 Full Root Cause Synthesis — The Three-Factor Policy Paradox

The complete, final explanation of what actually caused the incident (building on and completing the partial diagnosis from the prior live-debugging session):

Cluster topology at the time of the incident:

• 3 total nodes: 2 nodes labeled with role: primary, and 1 node carrying a NoSchedule taint.
• Deployment: checkout-api, 3 desired replicas, with:
  • A nodeSelector requiring role: primary — this alone excludes the third (tainted) node from consideration entirely, regardless of any other configuration.
  • A hard (required) pod anti-affinity rule, keyed on kubernetes.io/hostname — enforcing strictly one pod per node.
  • A Pod Disruption Budget (PDB) configured with minAvailable: 2.

The causal chain, step by step:

1. The nodeSelector immediately narrows the pool of eligible nodes for this deployment from 3 down to 2 (the two role: primary nodes).
2. The hard anti-affinity rule enforces one pod per node — meaning, structurally, at most 2 of the 3 desired replicas can ever be scheduled simultaneously, given only 2 eligible nodes.
3. The third replica is therefore permanently unschedulable under this configuration — not a transient issue, but a structural ceiling.
4. This alone produced the original Pending pod symptom — but the PDB then compounded the problem further, specifically during any drain or eviction attempt (not during normal scheduling — an important distinction the engineer emphasized: PDBs do not block scheduling; they block voluntary disruption/eviction).
5. With minAvailable: 2 set, and the system already structurally capped at exactly 2 schedulable replicas, any attempt to drain or evict either of the 2 running pods (e.g., during a node maintenance operation, or as part of a rollout replacing an old pod with a new one) would cause the “available replica” count to drop from 2 to 1 — directly violating the PDB’s minAvailable: 2 guarantee. Kubernetes’ eviction logic respects the PDB and refuses to proceed, causing the drain/eviction attempt to hang indefinitely — a genuine deadlock.
6. This is explicitly named a “policy paradox”: the PDB is demanding that at least 2 replicas remain available at all times, while the combination of node selector + hard anti-affinity has already structurally capped the system at a maximum of exactly 2 schedulable replicas — leaving zero slack for any disruption, voluntary or otherwise, to ever be safely processed.
7. Downstream consequence on the CI/CD side: because the rollout could never successfully replace the stuck pod, the deployment’s rollout eventually hit Kubernetes’ default progress deadline of 600 seconds, throwing a ProgressDeadlineExceeded error — which is what actually surfaced as a blocked CI/CD pipeline in the original incident.

Critical clarifying point emphasized by the engineer: this was not a single point of failure — it was a chain reaction arising from the combination of three independently-reasonable-looking configuration choices (a node selector, a strict anti-affinity rule, and a conservative PDB) that, together, created an unsatisfiable constraint. Each configuration choice might be defensible in isolation; the incident only emerges from their interaction.

3.9 Solutions Catalog

Presented as a menu of valid options (not a single “correct” answer), each with different trade-offs:

1. Increase the primary-labeled node count from 2 to 3, leaving the hard anti-affinity rule untouched — preserves the original strict one-pod-per-node high-availability guarantee, at the cost of additional infrastructure.
2. Relax anti-affinity from required to preferred (the fix actually applied live in the prior session) — fast, unblocks scheduling immediately, but weakens the strict node-spread guarantee.
3. Adjust the PDB to a more realistic configuration — e.g., using maxUnavailable: 1 instead of a fixed minAvailable: 2, or otherwise setting the PDB’s tolerance relative to the actual achievable replica count given real node constraints, rather than an aspirational number disconnected from what the scheduling configuration can actually sustain.
4. Allow scheduling onto the batch node group, by adding matching tolerations/labels so the third (tainted) node becomes a valid target too — restores the original 3-node eligible pool.
5. Implement cluster autoscaling, so that node capacity can dynamically expand to meet scheduling demand rather than being a fixed, manually-managed ceiling.

3.10 RCA Masterclass — What a Good RCA Must Contain

Framing: an RCA is an engineering artifact, not a casual summary — it must answer three core questions in detail: what exactly broke, what happened (the full detailed sequence), and how do we prevent it from recurring (explicitly connected to the program’s broader chaos-engineering philosophy).

Primary audience consideration: an RCA must serve two very different readers simultaneously — (a) on-call/SRE engineers (the “execution layer”) who need enough precise detail to actually reproduce the troubleshooting steps if the same incident recurs, and (b) non-technical stakeholders (a CEO, CFO, or similar) who need to understand business impact and resolution without needing to parse Kubernetes internals.

Full structure of a professional RCA, as demonstrated:

1. Header/metadata: incident ID, which services were affected, which cluster, start time, end time, total duration (this incident: ~65 minutes).
2. Customer/business impact — stated in concrete numbers, never vague theory: e.g., “33% of pods unavailable,” “rolling update frozen,” “maintenance window extended,” “~66% of intended capacity available, causing latency for end users.” Numbers, not adjectives.
3. Detection method: how was the incident actually discovered? (In this case: honestly documented as manual detection — noticed while draining a node / while a CI/CD deployment got stuck — explicitly not an automated alert, which is itself a finding worth documenting, not something to obscure.)
4. Incident Commander (IC) and Subject Matter Expert (SME) — named explicitly.
5. Summary section (~5 lines): must cover, in brief — what happened, whether customers were impacted, what the root cause was, what the immediate fix was, and what the prevention plan is. Written so that a reader with zero technical background still comes away understanding the full shape of the incident.
6. Detailed timeline: minute-by-minute (or at minimum, clearly timestamped key milestones) — e.g., “7:00 PM: cluster healthy, 3/3 replicas running” → “7:30 PM: incident begins” → “9:10 PM: cluster stabilized by IC and SME.” Explicitly emphasized as critically important, because — when presented to executive stakeholders — every minute of an incident represents real, quantifiable cost to the business, and the timeline is what makes that cost legible.
7. Technical root cause — the full causal chain, not a single symptom: must explain the incident as a sequence (node selector excluded a node → anti-affinity capped scheduling → PDB created a deadlock on any disruption attempt → progress deadline exceeded → pipeline blocked), not collapse it into one sentence.
8. Blast radius: precisely what was and was not affected — e.g., in this case: only the one service was impacted; the control plane, other workloads, and networking were all confirmed healthy throughout. This precision helps readers correctly scope their concern (and confidence) rather than assuming a wider failure than actually occurred.
9. Why wasn’t this caught earlier — a gap analysis: in this case, the honest answer was an observability gap — there was no alerting configured for unschedulable pods, meaning the team only discovered the problem manually rather than being proactively notified.
10. Recovery steps actually taken.
11. Lessons learned: what worked well, what slowed the team down, and what would be done differently next time — explicitly framed as a continuous-improvement mechanism (“improving your infrastructure is a never-ending game”).
12. Action items — must be specific and technical, never generic. (Elaborated further in Section 3.11, since this is where the “bad” example fails most badly.)
13. Stakeholder sign-off: a formal sign-off (e.g., from a CTO, VP, or Director) is required to formally close out the incident — the RCA isn’t considered complete/closed until this sign-off is obtained.

A concrete, official RCA template was confirmed as already uploaded to the shared drive (Foundations Track → weekly modules → “RCA Template” folder) — described as the same template actually used with real this programme clients, including a table of contents, issue-details section, and the stakeholder sign-off section described above.

Internal vs. external RCAs: a participant raised the distinction, confirmed by the engineer — internal RCAs stay within the engineering organization; external RCAs are shared with the affected customer/client when there’s meaningful business or reputational impact, and typically go through an internal review process (e.g., a manager or SME reviewing it) before being sent externally.

3.11 The “Bad RCA” Case Study — A Real, Anonymized Example

The engineer presented a real (anonymized) RCA, explicitly stated to have been written by an engineer with 7 years of DevOps experience — used deliberately to make the point that this failure mode isn’t just a beginner mistake.

Specific, itemized flaws identified:

1. Vague, low-information summary: the entire summary read approximately as “Checkout API went down during maintenance, and pods were not scheduling correctly. The cluster became unstable, and the issue was fixed by adjusting the configuration.” — criticized as giving a reader essentially no actionable detail at all.
2. Root cause stated as a single symptom, not a causal chain: the RCA stated “PDB caused the deployment to fail” as the root cause. Explicitly and emphatically corrected: a PDB can never directly cause a deployment to fail — it can only cause draining/eviction to fail. Conflating “the symptom I observed” with “the actual root cause” is called out as the single most common and most damaging RCA mistake — the correct root cause is the full chain (node selector → anti-affinity → PDB deadlock), not any one link in that chain in isolation.
3. No screenshots or evidence attached anywhere. Direct, memorable framing given: “Without proof… it is your opinion, not an RCA.” A reader (e.g., a CTO) has no way to verify the claims without supporting evidence (describe pod output, describe PDB output, relevant log excerpts).
4. Hedging/weasel language used throughout — specific examples flagged as unacceptable in a professional RCA: “may have,” “probably,” “it seems.” The engineer connected this to a broader, related anti-pattern: RCAs that vaguely blame “the application team” or “the infrastructure team” without concrete evidence — explicitly called out as unprofessional and unhelpful.
5. No timeline at all — no start time, no duration, no indication of who responded or when.
6. No impact section — no statement of who or what was actually affected.
7. Action items were generic, non-actionable platitudes: examples explicitly quoted and criticized — “improve monitoring,” “review configuration,” “avoid similar issues in future.” The engineer’s pointed criticism: everyone already knows they should “avoid incidents” and “improve monitoring” — a real action item must specify exactly what will change, and where (e.g., “add a PrometheusRule alert firing when any pod remains Pending for >5 minutes,” not “improve monitoring”).

3.12 When to Write an RCA — Qualifying Triggers

In response to a direct participant question (“do we write an RCA for every issue, even ones with no customer impact?”), the engineer gave three explicit qualifying triggers — an RCA is required if the incident impacts any one of the following:

1. Internal productivity — engineering or non-technical team productivity is affected.
2. The business — even without direct customer impact (e.g., financial or operational impact internally).
3. Customers — direct customer-facing impact.

If a production incident falls into any of these three categories, an RCA is expected. If an incident occurs but doesn’t meet any of these thresholds, formally skipping an RCA is acceptable (organization-dependent) — but any incident touching production is generally expected to warrant one by default.

3.13 Interview Framing Guidance

An extended, valuable Q&A exchange on how to present this kind of incident experience in a job interview — preserved because it’s genuinely practical career advice tightly coupled to this specific technical incident:

• General guidance: build a coherent “story” version of the incident for interview use — situating it within a plausible, complete project narrative (e.g., “I was working on a payments platform with 4–5 microservices; the checkout service specifically experienced this outage…”) rather than presenting the incident as an isolated, context-free technical puzzle.
• A specific, alternate framing offered for participants from a support/customer-facing SRE background (one participant described their actual role: reviewing customer-provided YAML/deployment configs, and joining live customer outage calls with only partial visibility into the customer’s environment): the engineer validated this as a “powerful SRE narrative” in its own right — reviewing a customer’s deployment configuration, identifying the misconfigured affinity rule, and guiding the customer through a live fix under partial-visibility conditions is a legitimate, compelling interview story distinct from a “I own this infrastructure end-to-end” framing.
• A genuinely important nuance surfaced by a participant, confirmed by the engineer, relevant to a very likely interview follow-up question (“why wasn’t this caught in a lower/staging environment before reaching production?“): in many real organizations, lower environments don’t fully mirror production’s scale (e.g., staging might run with 1 replica instead of 3, for cost reasons) — meaning a purely mathematical/configuration-interaction bug like this one (not an application logic bug, not a config typo) can genuinely and legitimately only manifest once you’re running at production’s actual node count and replica count. This is a real, defensible, common answer to that interview question, not an excuse.
• The engineer committed to preparing reusable, anonymized “story” documentation for participants to use directly in interviews — covering both this Kubernetes incident (with a fintech/payments framing) and, separately, the real GCP cost-optimization work from the FintechPlatform shadowing engagement (logging, monitoring, and related savings) — explicitly because participants found it valuable to have real, well-articulated project narratives to draw on rather than constructing them from scratch under interview pressure.

3.14 Security Tooling Wrap-Up

A shorter, largely recap-oriented closing section, reinforcing material from earlier sessions in this series with some new resource-sharing commitments:

• ScoutSuite reaffirmed as the attacker-perspective tool — reviewed live again briefly, showing its categorized findings (e.g., unencrypted EBS volumes flagged as a “danger”-level finding; overly permissive security groups with 0.0.0.0/0 egress flagged as a security-group-level risk) and its HTML-to-Excel/CSV conversion workflow for non-technical stakeholder presentation.
• Prowler reaffirmed as the compliance-perspective tool — filterable by specific compliance framework, with each finding linking to remediation guidance and references to official best-practice documentation.
• Cloud-specific security tool lists promised for AWS, GCP, and Azure separately (not detailed live in this walkthrough) — one GCP-specific example named live: a bucket/IAM-focused scanning tool from Rhino Security Labs (a recognized security research organization).
• Built-in cloud-native tools reaffirmed as a baseline but shallower option: AWS Trusted Advisor and GCP Active Assist were both mentioned again as useful first-pass, built-in options, but explicitly positioned as offering less depth than dedicated open-source security-scanning tools.
• A new “layer-by-layer” manual security audit methodology introduced, deliberately mirroring the OSI-layer structuring principle used elsewhere in this program: divide a cloud account into Compute, Data, IAM, Logging, and Networking layers, with a dedicated command-reference cheat sheet per layer (e.g., specific commands to audit EBS, EFS, backups, KMS configuration under the “Data” layer) — described as already built and promised to be shared via the drive, intended for engineers who want to perform a fully manual security audit rather than relying solely on automated scanning tools.
• Recommended security-audit cadence: run a full scan-and-fix cycle every 3 to 6 months at minimum — explicitly warned against letting the interval stretch to a year or longer.

4. Key Concepts Table

5. Architecture & Workflow Analysis

5.1 Structured Kubernetes Troubleshooting Decision Flow

Kubernetes incident reported
        |
        v
STEP 1: Control Plane or Data Plane?
        |
   -----------------------------------------
   |                                        |
CONTROL PLANE                          DATA PLANE
(scheduling, controllers,              (kube-proxy, CNI,
 PDB, eviction API)                     networking, traffic)
   |                                        |
   v                                        v
STEP 2: Pod Lifecycle Checklist        Pod IS running --
(created -> scheduled ->                use OSI-layer
 running -> reachable)                  troubleshooting
   |                                    (traffic-focused)
   v
Find FIRST stage pod fails at
   |
   v
STOP HERE. Do not investigate
later-stage causes.
   |
   v
If stuck at "Scheduled":
   -> Scheduler FILTER phase failure
   -> Check: nodeSelector, taints,
      affinity, resources, topology

5.2 The Full Root Cause Chain (Policy Paradox)

nodeSelector: role=primary
        |
        v
Excludes the 3rd (tainted) node entirely
        |
        v
Only 2 nodes eligible for checkout-api
        |
        v
Hard (required) anti-affinity: 1 pod per node
        |
        v
Max 2 of 3 desired replicas can EVER be scheduled
        |
        v
3rd replica: PERMANENTLY Pending (structural ceiling)
        |
        v
[SEPARATELY] PDB: minAvailable = 2
        |
        v
Any drain/eviction attempt on either running pod
would drop available count from 2 -> 1
        |
        v
VIOLATES minAvailable=2 guarantee
        |
        v
Kubernetes REFUSES the eviction -> HANGS INDEFINITELY
        |
        v
Rollout can never complete -> ProgressDeadlineExceeded (600s)
        |
        v
CI/CD pipeline blocked

5.3 Good RCA Structure (Full Template)

1. Header: Incident ID, services, cluster, start/end time, duration
        |
2. Impact (IN NUMBERS): e.g. "33% pods unavailable"
        |
3. Detection method: how was it found? (manual/automated)
        |
4. IC + SME named
        |
5. Summary (~5 lines): what happened, impact, root cause,
   immediate fix, prevention plan -- readable by non-tech audience
        |
6. Detailed timeline: minute-by-minute key milestones
        |
7. Technical root cause: FULL CAUSAL CHAIN, not one symptom
        |
8. Blast radius: precisely what WAS and WAS NOT affected
        |
9. Detection gap analysis: why wasn't this caught earlier?
        |
10. Recovery steps taken
        |
11. Lessons learned: what worked, what slowed us down,
    what we'd do differently
        |
12. Action items: SPECIFIC, technical, verifiable
        |
13. Stakeholder sign-off (CTO/VP/Director) -> formally closes incident

5.4 Good RCA vs. Bad RCA — Side by Side

                    GOOD RCA                    BAD RCA (real example)
                    --------                    -----------------------
Summary             5 lines, full picture       1 vague sentence,
                                                 no real detail

Root Cause          Full causal chain           Single symptom stated
                     (selector -> affinity ->    as root cause
                      PDB -> deadlock)           ("PDB caused failure")

Evidence             Screenshots/command         NONE attached
                     output attached

Language             Precise, factual            Hedging words:
                                                  "may have," "probably,"
                                                  "it seems"

Timeline             Minute-by-minute            MISSING entirely

Impact               Stated in numbers           MISSING entirely
                     (33%, specific %)

Action Items         Specific & technical         Generic platitudes:
                     ("add alert for Pending      "improve monitoring,"
                      pods >5min")                 "avoid similar issues"

6. Commands & Configurations

7. Tools & Technologies

ScoutSuite (recap)

• Purpose: Attacker-perspective, multi-cloud security auditing tool.
• Reinforced in this walkthrough: live re-demonstration of its EC2/security-group findings (unencrypted EBS volumes, overly permissive 0.0.0.0/0 egress rules), plus its HTML-to-Excel/CSV conversion workflow for client-facing reporting.

Prowler (recap)

• Purpose: Compliance-framework-oriented security scanning tool.
• Reinforced in this walkthrough: filterable by specific compliance standard, with remediation guidance and best-practice references linked per finding.

Rhino Security Labs tools (GCP-specific, newly named)

• Purpose: GCP bucket/IAM-focused security scanning tools from a recognized security research organization.
• When to use it: As a GCP-specific complement to broader multi-cloud tools like ScoutSuite/Prowler, for deeper analysis of specific GCP resource types.

AWS Trusted Advisor / GCP Active Assist (recap)

• Purpose: Cloud-native, built-in recommendation tools.
• Positioning reinforced: useful as a baseline first pass, but shallower than dedicated open-source scanning tools for genuinely deep security analysis.

8. Real-World Production Usage

• The control-plane/data-plane split and the four-step pod lifecycle checklist are among the most immediately applicable frameworks in this entire program — they’re fast, require no special tooling, and directly map to how real Kubernetes incidents actually present, making them genuinely usable from day one on a real on-call rotation.
• The “policy paradox” root cause (PDB + affinity + node selector interacting to create a deadlock) is a completely realistic, recurring real-world failure pattern — each individual setting (a node selector for workload placement, a strict anti-affinity rule for resiliency, a conservative PDB for availability) is a defensible best practice in isolation, which is exactly what makes this class of bug dangerous: it emerges only from the interaction of well-intentioned individual decisions, and is easy to miss in code review since no single line looks wrong.
• The RCA masterclass addresses a genuine, widespread real-world skill gap — the engineer’s own framing (a 7-year-experienced engineer producing a genuinely poor RCA) reflects a real, common pattern in the industry: strong debugging skills don’t automatically translate into strong incident documentation skills, and organizations frequently under-invest in explicitly teaching the latter.
• The specific “root cause vs. symptom” distinction, and the concrete PDB-behavior correction (PDBs block eviction, not scheduling), are exactly the kind of precise technical understanding that separates a credible RCA from one that erodes trust with technical stakeholders — a reviewing senior engineer or SRE lead would immediately notice a factually incorrect claim like “PDB caused the deployment to fail.”
• The internal-vs-external RCA distinction, and the requirement for stakeholder sign-off, reflect real incident-management governance practices used at organizations with mature SRE/on-call cultures — this isn’t an academic formality but a genuine accountability and closure mechanism.
• The interview-framing guidance, especially the support/SRE-with-partial-visibility narrative validated for one participant, reflects a realistic breadth of legitimate DevOps/SRE career paths — not everyone owns infrastructure end-to-end, and being able to credibly narrate a support-and-guidance incident-response experience is a real, valuable, and distinct interview asset.

9. Interview Preparation

Beginner Questions

Q1: What’s the difference between the Kubernetes scheduler’s “filter” phase and “score” phase? A: The filter phase removes any node that violates a hard constraint (like a required anti-affinity rule, an unmet node selector, or an unmatched taint) — those nodes are entirely excluded from consideration. The score phase then ranks the remaining, filtered-in nodes by weight or priority to select the best candidate among valid options. A pod stuck in Pending has, by definition, failed the filter phase — every candidate node was excluded.

Q2: Does a Pod Disruption Budget prevent a pod from being scheduled? A: No — this is a common misconception. A PDB only affects voluntary disruptions (like draining a node or evicting a pod) by guaranteeing a minimum number of replicas remain available during such operations. It has no effect on the initial scheduling decision for a new pod.

Q3: A pod is stuck in Pending with no node or IP assigned. Should you check CNI logs first? A: No — a pod with no node assigned hasn’t even passed the scheduling stage yet, and CNI issues (like failed pod sandbox creation or IP assignment) only become relevant after a pod has been scheduled and is attempting to start. Checking CNI logs at this stage would be investigating a stage the pod never reached; the correct first step is to check the pod’s scheduling-related events (kubectl describe pod) instead.

Intermediate Questions

Q4: Explain why converting a pod anti-affinity rule from “required” to “preferred” is a meaningful fix, but why it might not be sufficient on its own. A: required anti-affinity operates in the scheduler’s filter phase — it can entirely exclude nodes from consideration, potentially leaving zero valid candidates and causing a pod to stay Pending indefinitely. preferred anti-affinity operates only in the score phase, meaning it never removes a node from consideration — it can only influence ranking among nodes that already passed filtering, so it can never by itself block scheduling. However, if other filter-phase constraints are also active (like a conflicting node selector, or taints without matching tolerations), those other constraints can still independently cause the pod to remain unschedulable even after the anti-affinity rule is loosened — meaning a full fix requires addressing every contributing filter-phase constraint, not just one.

Q5: What’s the difference between stating a root cause and stating a symptom in an incident report, and why does this distinction matter? A: A symptom is what was directly observed during the incident (e.g., “the PDB caused the deployment to fail”); a root cause is the complete causal chain that actually produced that symptom (e.g., a node selector reduced eligible nodes, a hard anti-affinity rule then capped achievable replica count below the PDB’s guarantee, and the PDB then blocked any disruption from proceeding, ultimately manifesting as a blocked deployment). Stating only the symptom as if it were the root cause produces an incomplete, sometimes factually incorrect account (a PDB, for instance, cannot directly “cause a deployment to fail” — it can only block eviction), and critically, it also produces incomplete or wrong remediation, since fixing only the symptom’s proximate trigger without addressing the full chain risks leaving the underlying structural issue in place.

Q6: What should be included in a Kubernetes incident’s RCA to make it useful to both on-call engineers and non-technical executives? A: A structured RCA should include a short (~5 line), plain-language summary covering what happened, customer impact, root cause, immediate fix, and prevention plan — accessible to a non-technical reader. This should be paired with substantially more technical depth elsewhere in the document: a detailed, timestamped timeline; the full technical root-cause causal chain (not a single symptom); precise blast-radius scoping (what was and wasn’t affected); a gap analysis explaining why the incident wasn’t caught earlier; and specific, technically verifiable action items. Supporting evidence (screenshots, command output) should back every significant claim throughout.

Advanced Questions

Q7: Design a diagnostic approach for a Kubernetes incident where a pod is stuck in Pending, using the frameworks discussed in this walkthrough. A: First, determine whether the symptom belongs to the control plane (scheduling, controllers, PDBs, eviction) or the data plane (kube-proxy, CNI, networking) — a Pending pod with no node/IP assigned is unambiguously a control-plane symptom, so data-plane investigation (CNI logs, kube-proxy logs, networking) should be deferred entirely. Next, apply the pod lifecycle checklist (created → scheduled → running → reachable) to confirm exactly which stage the pod is stuck at — in this case, scheduling. Since scheduling failures originate in the scheduler’s filter phase, narrow investigation to the small, known set of filter-phase causes: node selectors, taints/tolerations, affinity/anti-affinity rules, resource requests vs. available capacity, and topology constraints. Use kubectl describe pod to read the scheduler’s own stated reason for the failure directly from the Events section, rather than guessing, and cross-reference the deployment’s actual spec (nodeSelector, affinity, tolerations) against the cluster’s actual node labels/taints to identify the specific mismatch. Only after confirming the pod has actually progressed past scheduling should OSI-style, traffic-focused troubleshooting be considered — applying it earlier would waste significant time investigating layers that were never relevant to a pure scheduling failure.

Q8: A team’s PDB, anti-affinity rule, and node selector are each individually reasonable, but together they create a scheduling deadlock. How would you design a process to catch this class of compound misconfiguration before it reaches production? A: Since no single setting is individually wrong, code review alone (reviewing each configuration change in isolation) is unlikely to catch this — the risk specifically arises from interaction between settings that may even be defined in separate files or changed at separate times. A more effective approach combines: (1) a pre-deployment or CI validation step that simulates or calculates the maximum achievable replica count given the combination of current node labels/taints, anti-affinity rules, and node selectors for a given deployment, flagging any case where that maximum falls below the deployment’s desired replica count or below any associated PDB’s minAvailable threshold; (2) treating any PDB configuration change as requiring explicit cross-validation against the current scheduling constraints of the deployment it governs, not just against the deployment’s replica count in isolation; and (3) periodic, scheduled audits (not just point-in-time reviews at deployment creation) that re-check this same combination, since node labels/taints can change independently of the deployment spec over time (exactly as happened in this incident, where taints were likely added after the original pods were scheduled) — meaning a configuration that was safe when first deployed can silently become unsafe later without any change to the deployment itself.

Q9: How would you evaluate whether an RCA is “production grade,” using the specific criteria discussed in this walkthrough, if you were reviewing a colleague’s RCA before it goes to a client? A: Check systematically against each of the following: Does the summary give a complete picture in a few lines, understandable without deep technical knowledge? Is there a detailed, timestamped timeline showing exactly when key events occurred? Is the stated root cause a full causal chain, or does it collapse into a single symptom (a strong red flag, as illustrated by the “PDB caused deployment to fail” example)? Is every significant claim backed by attached evidence (screenshots, command output), or does the document rely on unsupported prose? Is impact quantified with actual numbers, or described only in vague terms? Does the document avoid hedging language (“may have,” “probably,” “it seems”) in favor of precise, factual statements? Is there an honest gap analysis explaining why the incident wasn’t caught earlier, rather than glossing over that question? And critically — are the action items specific and technically verifiable (naming an exact change to be made), or are they generic platitudes that don’t commit to anything concrete? An RCA failing on several of these dimensions — even if the underlying incident was actually resolved correctly — should be sent back for revision before being shared externally, since a weak RCA can undermine stakeholder confidence even when the technical response itself was sound.

10. Exam & Certification Notes

(Highly relevant to CKA/CKAD certifications for the technical content; the RCA-writing content is relevant to ITIL-adjacent incident management practices and SRE-focused certifications.)

• Scheduler filter and score phases: Understand this two-phase model as the underlying mechanism behind all Kubernetes scheduling decisions — a frequently tested conceptual foundation for questions involving affinity, taints, or resource-based scheduling failures.
• requiredDuringSchedulingIgnoredDuringExecution vs. preferredDuringSchedulingIgnoredDuringExecution: Reinforced again in this walkthrough as a core, frequently tested CKA/CKAD distinction — know that only the required variant can cause a pod to remain permanently unschedulable.
• PodDisruptionBudget semantics: A frequently tested and frequently misunderstood object — know precisely that a PDB affects voluntary disruptions only (drain, eviction via the Eviction API) and has no effect on initial pod scheduling — a distinction this walkthrough explicitly corrects as a common real-world misconception.
• Deployment progressDeadlineSeconds: Know the default value (600 seconds) and that exceeding it produces a ProgressDeadlineExceeded condition on the Deployment — directly relevant to understanding why a stuck rollout eventually surfaces a distinct, separate error on top of the original scheduling failure.
• Eviction API behavior under PDB constraints: Understand that Kubernetes’ eviction API explicitly checks PDB constraints before allowing an eviction to proceed, and will refuse (rather than force) an eviction that would violate minAvailable/maxUnavailable — this is precisely the mechanism behind the “hanging drain” behavior described in this incident.
• RCA/postmortem structure: While not a Kubernetes-specific exam topic, structured incident postmortem writing is increasingly covered in SRE-oriented certifications and training — the specific components taught here (blast radius, detection-gap analysis, timeline, evidence-backed claims, specific action items) align closely with industry-standard postmortem frameworks (e.g., Google’s SRE postmortem culture).

11. Cheat Sheet

Structured Troubleshooting — Two-Step Framework:

1. Control plane or data plane? (scheduling/PDB/controllers vs. kube-proxy/CNI/networking)
2. Pod lifecycle stage: Created → Scheduled → Running → Reachable (strict order, never skip ahead)

Scheduler Filter-Phase Causes (only 5, when stuck at “Scheduled”):

1. Node selector mismatch
2. Taints without matching tolerations
3. Affinity/anti-affinity rules
4. Insufficient resources
5. Topology constraints

Symptom Signatures (confirm/kill hypotheses fast):

Required vs. Preferred (memorize):

• required → filter phase → CAN block scheduling entirely
• preferred → score phase only → CANNOT block scheduling

PDB Rule to Never Forget: PDBs block eviction/drain, NOT scheduling. A PDB can never be the reason a pod is Pending.

OSI Model Rule: Only use OSI when the pod IS running but traffic is failing. Never use it for a pure scheduling problem — wastes 20-30 min.

Root Cause of This Incident (memorize the chain): nodeSelector excludes 1 node → hard anti-affinity caps scheduling at 2/3 replicas → PDB minAvailable=2 makes any eviction hang → ProgressDeadlineExceeded → CI/CD blocked

Good RCA Checklist:

• 5-line non-technical summary
• Timestamped, minute-level timeline
• Root cause = full causal chain, NOT a single symptom
• Impact stated in NUMBERS
• Screenshots/evidence attached
• No hedging language (“may,” “probably,” “it seems”)
• Blast radius precisely scoped
• Detection-gap analysis included
• Action items SPECIFIC and technical (not “improve monitoring”)
• Stakeholder sign-off obtained

When to Write an RCA: Impacts internal productivity OR business OR customers → RCA required.

12. Gaps & Assumptions

• This walkthrough assumes familiarity with the prior live-debugging session (documented separately in this series) — the exact live YAML editing and step-by-step hands-on sequence is not re-covered here in full detail, since this walkthrough is explicitly the theory follow-up. Refer to the prior package for the full live-debugging blow-by-blow.
• The exact taint key/value and node-role label strings were described narratively rather than shown as literal YAML in this transcript (a whiteboard + verbal explanation session, not primarily a terminal screen-share for this portion) — presented here using the same representative naming conventions established in the prior session’s package for consistency, not as a verbatim citation of exact strings.
• The “master troubleshooting sheet” and the cloud-specific security tool lists (AWS/GCP/Azure) were both committed to as future deliverables but are not part of this transcript’s content — this document describes their intended purpose as stated, not their finished content.
• The layer-by-layer manual security audit command sheet (Compute/Data/IAM/Logging/Networking) was shown briefly on screen but not read aloud in enough detail to reconstruct the specific commands for each layer in this document — flagged as existing and promised for sharing, but its specific contents are outside this transcript’s captured detail.
• Cohort size figures (~50 enrolled, ~35 on onboarding calls, ~29 India-based) were given as approximate, conversational figures in response to a participant’s logistics question — not represented as precisely audited enrollment data.
• The “bad RCA” example is anonymized per the engineer’s explicit choice (“I don’t want to name that guy”) — this document preserves that anonymization; no identifying details about the original author are available or should be inferred.
• This document consolidates a session combining a whiteboard-based theory explanation with an extensive, free-flowing Q&A (particularly the interview-framing discussion) — content has been reorganized topically for clarity rather than presented in strict chronological order, consistent with the approach used for prior packages in this series.