War Room Drill Follow-Up: Solution, Debugging Framework and RCA Writing

Complement to: War Room Drill — Week 3: Kubernetes Production Outage (Pod Stuck in Pending)

What This Session Adds

1. The Compound Root Cause — Formal Explanation

The engineer laid out the full chain cleanly in this walkthrough:

Starting condition:

• 3 replicas (checkout-api)
• 3 nodes total
• 2 primary On-Demand nodes → labelled role=primary
• 1 batch Spot node → tainted workload=batch:NoSchedule

Why the third pod was stuck:

Node A (primary): checkout-api pod-1 → Running ✓
Node B (primary): checkout-api pod-2 → Running ✓
Node C (batch):   Tainted → no toleration in pod spec → ineligible

Anti-affinity rule (required): 1 pod per node
→ Pod-3 checks Node A: already has checkout-api → REJECTED
→ Pod-3 checks Node B: already has checkout-api → REJECTED
→ Pod-3 checks Node C: tainted, no toleration → REJECTED
→ Pod-3: PENDING (indefinitely)

Three simultaneous misconfigs — all acting as AND conditions:

“This is not a single point of failure. This is a chain issue — a configuration deadlock caused by multiple misconfigs acting together.”

2. PDB (PodDisruptionBudget) — Its Role in the Deadlock

PDB does NOT cause scheduling failures. This is the most common misunderstanding, and the session addresses it explicitly.

What PDB actually does: Controls the eviction of pods during voluntary disruptions (drains, rollouts, maintenance). It says: “don’t let the number of available pods fall below minAvailable.”

The PDB configuration in this cluster:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-api-pdb
  namespace: payments
spec:
  minAvailable: 2         # At least 2 checkout-api pods must always be available
  selector:
    matchLabels:
      app: checkout-api

Where PDB entered the picture: When someone tried to drain Node A (to investigate or maintain it):

kubectl drain node-A --ignore-daemonsets --delete-emptydir-data

The drain triggers eviction of pod-1 from Node A. Kubernetes checks the PDB:

• Current available pods: 2 (pod-1 on Node A, pod-2 on Node B)
• If pod-1 is evicted: available pods = 1
• PDB requires minAvailable: 2 → eviction BLOCKED

The scheduler tries to reschedule pod-1 elsewhere first (before the eviction completes). But:

• Node B: anti-affinity says no (pod-2 already there)
• Node C: tainted, no toleration
• Result: rescheduling impossible → PDB blocks eviction → drain hangs indefinitely

The policy paradox:

PDB says:    "Keep at least 2 pods available at all times"
Anti-affinity says: "Each pod must be on a different node"
Available nodes for pod: 2

→ To satisfy anti-affinity: maximum 2 pods can run
→ Draining one node temporarily reduces to 1
→ PDB blocks this (1 < minAvailable:2)
→ Rescheduling would violate anti-affinity
→ DEADLOCK: drain cannot complete, rollout cannot complete

The amplification effect: PDB didn’t cause the pending pod — the scheduling constraints did. But PDB amplified the incident by making the cluster un-drainable (maintenance is blocked) and making the rolling deployment even more stuck (rolling update tries to drain old pods before terminating).

ProgressDeadlineExceeded explanation: The Deployment has a progressDeadlineSeconds (default: 600 seconds = 10 minutes). If the rolling update doesn’t make progress within this window, the Deployment controller marks the rollout as failed and shows:

Deployment checkout-api exceeded its progress deadline.

This is what blocked the CI/CD pipeline. The rolling update couldn’t progress because the new pod was Pending, and couldn’t terminate the old pod because the PDB blocked it.

3. The Policy Paradox (the Real Deadlock Mechanism)

This is the core insight of the session — a concept the engineer named explicitly:

“It is a policy paradox. PDB is demanding two available nodes. The scheduler has the constraint of one pod per node. The cluster has only two eligible nodes. These two policies are mutually unsatisfiable under the current state.”

The mathematical incompatibility:

Desired: 3 replicas
Eligible nodes: 2 (Node A and B; Node C excluded by taint)
Anti-affinity (required): max 1 pod per node
→ Maximum deployable replicas: 2

PDB minAvailable: 2
→ Minimum required to allow drain: at least 2 running pods
→ During drain of Node A: temporarily 1 pod available
→ PDB blocks this

Result:
  Deployment is stuck (can't get to 3 running, can't drain, can't rollout)
  Drain is stuck (PDB blocks eviction)
  CI/CD is stuck (ProgressDeadlineExceeded)

Named pattern: This class of failure is called a scheduling deadlock — not a hardware failure, not an application bug, but a logical contradiction between configured policies.

4. Five Solution Options

The engineer explicitly listed all viable fixes:

Recommended permanent fix (discussed in session): Option 1 (add 3rd primary node) + revert anti-affinity to required. This preserves the original HA intent without compromising the deployment design.

Immediate stabilization (applied in live debug): Option 2 (change to preferred) — fast, no node provisioning needed, but reduces strict HA.

5. The Structured K8s Debugging Framework

This is the session’s most reusable content — the structured debugging approach that eliminates ~70% of hypothesis confusion in a Kubernetes outage.

Step 1: Divide the problem into control plane vs. data plane

KUBERNETES HAS TWO WORLDS:

┌─────────────────────────────────────────────────────┐
│  CONTROL PLANE                                      │
│  Issues here are about LOGIC:                       │
│   - Scheduling (why pods aren't placed)             │
│   - Rollouts (why deployments are stuck)            │
│   - Controller logic (ReplicaSet, Deployment ctrl)  │
│   - PodDisruptionBudget (eviction policies)         │
│   - Eviction API                                    │
│   - RBAC (access control decisions)                 │
└─────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────┐
│  DATA PLANE                                         │
│  Issues here are about NETWORKING / RUNTIME:        │
│   - kube-proxy (service routing, iptables)          │
│   - CNI / VPC CNI (pod IP assignment, ENI)          │
│   - Networking (traffic flow, DNS)                  │
│   - kubelet (container runtime on nodes)            │
│   - Traffic reaching pods                           │
└─────────────────────────────────────────────────────┘

Why this matters: Engineers waste time checking kube-proxy and CNI logs when the pod hasn’t even been scheduled yet. The control plane / data plane split immediately tells you which world to investigate:

• Pod is in Pending → control plane problem (scheduling)
• Pod is in Running but service unreachable → data plane problem (routing, CNI, networking)
• Pod is in CrashLoopBackOff → application or container runtime problem

Step 2: Ask the 4 Pod Lifecycle Questions in order

Question 1: Is the pod CREATED?
  → kubectl get pod <name> -n <namespace>
  → If no pod exists at all: Deployment controller issue, RBAC, namespace issue

Question 2: Is the pod SCHEDULED?
  → STATUS = Pending = NOT scheduled
  → STATUS = Running/ContainerCreating = scheduled
  → If Pending: STOP HERE — don't check kube-proxy, don't check DNS

Question 3: Is the pod RUNNING?
  → STATUS = Running = container started
  → CrashLoopBackOff = container started but keeps dying → application issue
  → ContainerCreating = scheduled but container not started → CNI/kubelet issue

Question 4: Is the pod REACHABLE?
  → Service responding? curl to ClusterIP? ALB health check passing?
  → If running but not reachable → networking issue (kube-proxy, CNI, DNS, SG)

”If you follow this order, it eliminates 70% of your confusion immediately. In our case, pod was Pending — it never passed Question 2. So we never need to ask about kube-proxy, ALB, DNS, or any data-plane component.”

6. Scheduler Filter Phase — All Possible Failure Reasons

Once you’ve established the pod is Pending (failed Question 2), the problem is in the scheduler’s filter phase. The filter phase removes nodes from consideration; if all nodes are removed, the pod stays Pending.

Complete list of filter-phase failure reasons:

”In our case: node selector mismatch + taint/toleration + pod anti-affinity (required) — three filter failures at once. Resources and topology constraints were NOT the issue.”

7. When OSI Applies and When It Does Not

The session explicitly defined OSI’s scope:

Use OSI when:

• Pods are in Running state
• Traffic is failing (service unreachable, timeouts, 503s)
• The problem is in the data plane (networking, routing, DNS)

Do NOT use OSI when:

• Pods are in Pending state (not yet scheduled)
• The problem is in the control plane (scheduling, rollouts, controller logic)

Reason: OSI traverses network layers. If the pod hasn’t started, there is no network path to traverse. Applying OSI to a scheduling problem wastes 20–30 minutes (the full 7-layer traversal) on completely irrelevant checks.

The correct sequencing:

Pod Pending?
  → Use: 4 Pod Lifecycle Questions → Scheduler Filter Analysis
  → Do NOT use: OSI

Pod Running but unreachable?
  → Use: OSI (L1→L7, data plane focus)
  → Relevant layers: L3 (CNI/routing), L4 (kube-proxy/iptables), L7 (app config)

8. Symptom-to-Hypothesis Matching

Instead of guessing (“could be CNI, could be kube-proxy…”), match the symptom to the component:

”If you are matching the symptom to the right component, you kill the wrong hypothesis early and focus on the right one. This saves 15–20 minutes per incident.”

9. Why kube-proxy and CNI Were NOT the Issue (Full Reasoning)

This walkthrough explicitly walked through each component to demonstrate why:

kube-proxy would cause issues if:

• Service is unreachable (iptables out of sync → routing broken)
• NodePort not forwarding
• Intermittent timeouts (some requests succeed, some fail)
• In logs: iptables rules not syncing

In our case: kube-proxy logs showed “sync rules completed,” iptables in sync. Pod was Pending — not Running. kube-proxy operates on running pods’ IPs. A Pending pod has no IP. kube-proxy cannot be the cause.

CNI would cause issues if:

• Pod stuck in ContainerCreating (IP allocation failing)
• Events show: failed to create pod sandbox
• Pod IP never assigned even after node binding
• ENI slot exhaustion (AWS-specific: not enough IPs)

In our case: Pod never got past Pending → never reached the container creation phase → CNI was never invoked. The problem was pre-CNI (scheduler couldn’t even bind the pod to a node).

10. RCA Methodology — What a Good RCA Must Answer

The definition the engineer gave:

“RCA is not story time. RCA is an engineering artifact. It must answer exactly three questions.”

The three core questions every RCA must answer:

The three purposes of an RCA:

1. Knowledge base: If the same issue happens when you’re on leave, a junior engineer can follow the RCA to diagnose and fix it.
2. Accountability: If there was business loss, you explain to the CEO/CFO what happened in terms they understand.
3. Prevention: Documents the specific changes made so the issue cannot recur.

11. The Three Audiences for Every RCA

Understanding who reads the RCA determines what to include:

”If you are writing an RCA, you have to think in these three domains. Not just one. Not just the technical. Not just the business. All three.”

External vs. internal RCA: Some organizations produce two versions:

• Internal RCA: Full technical detail, all layers, action items with owners.
• External RCA: Customer-facing; focuses on impact, resolution, and prevention; technical detail reduced to what customers need.

12. RCA Skeleton — All Required Sections

The engineer’s official template (uploaded to Google Drive, core-ops folder → week modules → RCA template):

═══════════════════════════════════════════════════════
RCA DOCUMENT STRUCTURE
═══════════════════════════════════════════════════════

1. INCIDENT DETAILS (header)
   - Incident ID:         (e.g., INC-7801)
   - Severity:            (SEV-1, P0, etc.)
   - Service affected:    (checkout-api, namespace: payments)
   - Cluster:
   - Start time:
   - End time:
   - Duration:
   - Detected by:         (alert / manual / on-call)
   - IC (Incident Commander):
   - SME:

2. CUSTOMER IMPACT (in numbers, not words)
   - % pods unavailable:  "33% — 1 of 3 replicas"
   - User-facing impact:  "Increased latency; potential 503s if unresolved"
   - Pipeline impact:     "CI/CD blocked — no deployments possible"
   - Revenue at risk:     (if quantifiable)

3. EXECUTIVE SUMMARY (5 lines maximum)
   One line each:
   - What happened
   - Customer impact
   - Root cause (one sentence)
   - Immediate fix applied
   - Prevention plan summary

4. TIMELINE (the most important section)
   Format: Time | Actor | Action/Observation
   T-00:00 | System | Cluster healthy; 3/3 replicas running
   T+00:30 | CI/CD  | Hotfix deployment triggered
   T+00:31 | K8s    | Pod-3 enters Pending state
   T+10:00 | K8s    | ProgressDeadlineExceeded thrown
   T+08:00 | Team   | Incident declared; war room opened
   T+08:30 | IC     | Root cause identified via kubectl describe
   T+09:15 | IC     | Anti-affinity changed to preferred; pods Running
   T+09:15 | System | Cluster stabilized
   [Every action in minutes; every minute of P0 counts]

5. TECHNICAL ROOT CAUSE
   NOT a single line symptom.
   The chain of causation:
   "The node selector restricted eligible nodes to 2 primary nodes.
   The required podAntiAffinity enforced 1 pod per host.
   With 3 replicas and only 2 eligible nodes, pod-3 could never schedule.
   The PDB (minAvailable: 2) amplified the issue by blocking drain,
   preventing the rolling update from making progress.
   The result was a scheduling deadlock — a mathematically unsatisfiable
   set of policies applied to insufficient infrastructure."

6. CONTRIBUTING FACTORS
   - Why wasn't it caught in lower environments?
     (Lower env had fewer replicas; constraints not visible at scale)
   - What monitoring gap allowed this to persist 8 hours?
     (No alert on Pending pods > 5 minutes; no scheduling constraint validation)
   - What process gap prevented early detection?
     (No pre-deployment check: eligible nodes ≥ replicas)

7. BLAST RADIUS
   - What failed:    checkout-api pod-3 (1 of 3 replicas)
   - What was safe:  All other services; cluster control plane; networking
   - Scope:          Single service in payments namespace

8. DETECTION GAP (observability failures)
   - No alert: pod Pending for > 5 minutes
   - No alert: ProgressDeadlineExceeded
   - No validation: replicas > schedulable nodes pre-deploy
   - No runbook: for scheduling constraint deadlock

9. RECOVERY (exact steps taken)
   - Changed podAntiAffinity: required → preferred (deployment edit)
   - Added tolerations for batch node taint
   - Removed conflicting nodeSelector
   - Triggered rolling restart
   - Verified all 3 pods Running

10. LESSONS LEARNED
    - What worked:       (collaborative debugging; kubectl describe events)
    - What slowed us:    (initial guessing at kube-proxy/CNI; no alert)
    - What we'd do differently: (alert on Pending pods immediately; pre-deploy check)

11. ACTION ITEMS (specific and technical — not vague)
    | Action                                              | Owner    | Due    |
    |-----------------------------------------------------|----------|--------|
    | Add alert: pod Pending > 5 min in payments namespace | DevOps  | 1 week |
    | Add pre-deploy check: eligible nodes ≥ replicas     | DevOps   | 1 week |
    | Add 3rd primary node; revert anti-affinity to required| DevOps | ASAP  |
    | Create runbook: scheduling deadlock diagnosis        | SRE      | 2 weeks|
    | Review all deployments: required anti-affinity + PDB | DevOps  | 3 days |
    | Add ProgressDeadlineExceeded alert                   | DevOps  | 1 week |

12. SIGN-OFF
    - IC signature / approval
    - CTO / VP Engineering approval
    - (Signals the incident is formally closed)

13. Good RCA Example (From the Live Session)

The engineer showed the actual RCA created for the checkout-api incident. Key elements highlighted as exemplary:

Incident details header — specific numbers:

Incident ID:    INC-7801
Service:        checkout-api (namespace: payments, cluster: sre-labs-HA-lab)
Start:          20:00 IST, 14 Feb
End:            21:15 IST, 14 Feb
Duration:       65 minutes
Customer impact: 33% pod unavailability; rolling update frozen;
                 maintenance window extended; latency increase for users
Detection:      Manual — observed during CI/CD deployment attempt
IC:             Shai
SME:            Kishor

Executive summary — 5 lines:

What happened:   checkout-api deployment deadlocked due to configuration mismatch;
                 1 of 3 replicas unable to schedule for 65 minutes.
Customer impact: 33% capacity reduction; increased latency; risk of 503 errors.
Root cause:      Required podAntiAffinity + 2 eligible nodes + missing tolerations
                 created a scheduling deadlock; amplified by PDB minAvailable:2.
Immediate fix:   Changed anti-affinity to preferred; added tolerations; removed nodeSelector.
Prevention:      Add Pending pod alert; add pre-deploy eligible-node check; add 3rd node.

Technical root cause — the chain, not a symptom:

1. nodeSelector restricted scheduling to primary-labelled nodes (2 nodes)
2. Required podAntiAffinity enforced 1 checkout-api pod per node
3. 3 replicas required, but only 2 eligible nodes → pod-3 mathematically unschedulable
4. PDB (minAvailable:2) amplified: drain of Node A blocked because eviction would
   drop below 2 available, and rescheduling to Node B violated anti-affinity
5. Result: scheduling deadlock → drain blocked → rollout frozen → ProgressDeadlineExceeded

Evidence / screenshots required:

• kubectl describe pod <pending-pod> → Events section showing scheduler rejections
• kubectl get pdb -n payments → showing minAvailable:2
• kubectl get deployment checkout-api -n payments -o yaml → showing the constraints
• kubectl get nodes → showing 3 nodes, 1 cordoned
• kubectl get pods -n payments → before (1 Pending) and after (3 Running)

14. Bad RCA Example — Annotated

The session showed an actual RCA submitted by a 7-year-experienced DevOps engineer. Annotated here:

Summary: "The checkout API went down during maintenance and pods were not
          scheduling correctly. The cluster became unstable and the issue was
          fixed by adjusting the configuration."

❌ Too vague. “Went down” — what does that mean? “Adjusting the configuration” — which configuration? Why? This tells the reader nothing useful.

Root cause: "PDB caused the deployment to fail."

❌ Wrong and superficial. PDB cannot cause a deployment to fail — it blocks eviction. The actual root cause is the scheduling deadlock. This shows the author didn’t understand the incident. A root cause is a chain, not a symptom.

Impact: "Users may have faced latency."

❌ “May” is unacceptable. You must state what actually happened with evidence. If you’re unsure, say so but quantify: “Based on traffic metrics, 33% of pods were unavailable for 65 minutes. Load balancer metrics showed a 40% increase in P99 latency during this period.”

Action items:
- Improve monitoring
- Review configuration
- Avoid similar issues in future

❌ Completely useless. “Improve monitoring” — which monitors? On what metric? With what threshold? “Avoid similar issues” — everyone knows this. These are not action items, they are wishes. Action items must be specific and technical: “Add Prometheus alert: kube_pod_status_phase{phase="Pending"} > 0 for > 300 seconds in namespace=payments, severity=P1.”

Additional flags from the session:

• No timeline — can’t reconstruct what happened or how long it took
• No impact numbers — “may have affected customers” is not acceptable in a P0 RCA
• No evidence/screenshots — “your opinion, not an RCA”
• No blame language (e.g., “network issue”, “application issue”) — these are symptoms and accusations, not root causes
• Single-page RCA for a 65-minute P0 — grossly insufficient

”RCA is evidence-based engineering. Without screenshots, without events from kubectl, without timeline — you are sharing your opinion. No senior engineer or executive accepts opinion as an RCA.”

15. When to Write an RCA

Threshold: Write an RCA if the incident affected any of these three things:

1. Internal productivity (engineering team blocked or degraded — e.g., CI/CD down)
2. Business (revenue at risk, even without direct customer impact)
3. Customers (any user-facing degradation or failure)

RCA is always required for production incidents touching any of the above. For non-production incidents that are self-contained and quickly resolved, an RCA is optional (org policy dependent).

The RCA as knowledge base:

“If you fix an incident and don’t write an RCA, you have wasted your time. The RCA is the only durable artifact from the incident. Without it, the same incident will happen again — and next time, your colleague will spend the same 65 minutes from scratch.”

16. AWS Security Scanning — ScoutSuite and Prowler

The engineer briefly covered the security scanning piece that had been pending from previous sessions.

ScoutSuite

• Purpose: Provides an attacker’s view of your AWS infrastructure — what can an attacker exploit?
• Type: Open-source; agentless; runs as a CLI tool from any machine with AWS credentials.
• Output: HTML report filterable by: service (EC2, EKS, S3, Lambda, RDS…), severity (danger/warning/good), resource, compliance.
• Key findings shown: EBS volumes not encrypted (danger); unrestricted security group rules (0.0.0.0/0) on inbound ports (warning).
• Download output: Per-finding JSON/CSV export for each resource.

# Install ScoutSuite:
pip install scoutsuite

# Run against AWS account (uses current AWS credentials):
scout aws

# Output: HTML report in ./scoutsuite-report/
# Open: scoutsuite-report/scoutsuite-report.html

Prowler

• Purpose: Compliance-focused AWS security scanner — checks against specific compliance frameworks (CIS, PCI-DSS, SOC2, HIPAA, GDPR, ISO27001, etc.).
• Type: Open-source; cloud-native (also available as a SaaS).
• Key difference from ScoutSuite: ScoutSuite = attacker view (can this be exploited?); Prowler = compliance view (does this meet framework X?).
• Output: Per-control pass/fail/warning; links to AWS documentation for each finding; remediation guidance.

# Install Prowler:
pip install prowler

# Run against AWS (all checks):
prowler aws

# Run against specific compliance framework:
prowler aws --compliance cis_1.5_aws

# Output: HTML, JSON, CSV available

When to use each:

• ScoutSuite → finding attack vectors proactively; security posture for attackers.
• Prowler → audit preparation; compliance certification; auditors will ask about this.
• Use both: they catch different things.

For other clouds:

• GCP: gcloud-security-scanner from Rhino Security; per-resource tools (GCS bucket scanner, IAM scanner, Compute Engine scanner).
• Azure: Prowler (supports Azure); az security CLI commands.
• Cloud-native equivalents: AWS Trusted Advisor (security pillar), GCP Active Assist, Azure Security Center — good starting points but less comprehensive than open-source tools.

17. The 5-Layer Security Audit Framework

The engineer described a layered approach to manual security auditing (a cloud-adapted security equivalent of the OSI troubleshooting model):

Layer 1: COMPUTE
  Resources: EC2, EKS nodes, Lambda, ECS tasks
  Check:     Instance roles (principle of least privilege); IMDSv2 enforced;
             public exposure; unencrypted storage; SSM vs SSH access

Layer 2: DATA
  Resources: EBS, EFS, S3, RDS, DynamoDB, backups, KMS
  Check:     Encryption at rest; public S3 buckets; bucket policies;
             cross-account access; backup retention; KMS key rotation

Layer 3: IAM
  Resources: Users, roles, groups, policies, service accounts
  Check:     MFA on all users; no unused access keys; no overly-permissive
             policies (*:*); cross-account trust; rotation of access keys

Layer 4: LOGGING
  Resources: CloudTrail, VPC Flow Logs, S3 access logs, EKS audit logs
  Check:     CloudTrail enabled in all regions; logs encrypted; not public;
             retention period; no gaps in logging

Layer 5: NETWORKING
  Resources: VPC, security groups, NACLs, NAT GW, load balancers, route tables
  Check:     No 0.0.0.0/0 inbound on sensitive ports (22, 3389, 3306);
             public vs. private subnet architecture; VPC flow logs enabled;
             unnecessary internet gateway exposure

Process:

1. Scan (ScoutSuite / Prowler / manual commands)
2. Fix identified gaps
3. Re-scan to verify
4. Repeat every 3–6 months (not once a year)

“Security hardening is never-ending. Every time your infrastructure changes, new gaps can appear. Quarterly scanning is the minimum. Monthly is better.”

18. Next Project Preview — 500-Microservice MNC Client

The next project, starting from the following Sunday, covers an unnamed Indian MNC with 500+ microservices. Topics to be covered:

1. Architecture — how 500 microservices are structured, grouped, and deployed.
2. CI/CD at scale — GitHub Actions / Jenkins pipelines managing 500 services.
3. Terraform at enterprise scale — how to manage infra-as-code for that footprint without it becoming unmaintainable.
4. Helm at scale — managing 500 Helm charts without chart explosion.
5. Observability — system monitoring + user behaviour monitoring + latency monitoring (three distinct levels).
6. Production outages faced on this client and how they were resolved.

This project is labelled the most complex of the six in the program — it requires the K8s scheduling, Karpenter, and security knowledge from earlier sessions as prerequisites.

19. Interview Framing Guidance

The session addressed how to use this incident in interviews:

The core story structure (for DevOps interview):

Context:    "I was working on a fintech/e-commerce payment platform with
             a Kubernetes-based microservices architecture on EKS."
Situation:  "We had a critical checkout API service with 3 replicas in
             the payments namespace. During a routine hotfix deployment,
             1 of 3 pods stuck in Pending for 8 hours."
Task:       "As the DevOps engineer on call, I needed to diagnose and fix
             the issue within our SLA window."
Action:     "I started by dividing the problem: pod was Pending, not Running,
             so this was a control-plane scheduling issue, not networking.
             I ran kubectl describe pod, which showed scheduler rejections:
             anti-affinity conflicts, missing tolerations, nodeSelector mismatch.
             I identified a scheduling deadlock: 3 replicas, required anti-affinity
             (1 pod per host), but only 2 eligible nodes — mathematically
             unsatisfiable. PDB was amplifying: blocking drain because available
             pods would drop below minAvailable:2.
             Immediate fix: changed anti-affinity to preferred, added tolerations,
             removed conflicting nodeSelector. All 3 pods Running within 5 minutes."
Result:     "Service stabilized. CI/CD unblocked. Post-incident: added alerts for
             Pending pods > 5 minutes, added pre-deploy check for eligible nodes ≥
             replicas, documented the deadlock pattern as a runbook."
Learning:   "Key insight: pod Pending + no error events = scheduling constraint
             problem, not application. The structured K8s debugging order
             (control plane vs. data plane, then filter phase analysis) saved
             20+ minutes of random hypothesis testing."

For support engineers (Prashant’s scenario): Reframe as:

“A customer deployed our Kubernetes-based product with custom configuration. They increased replica count from 2 to 3 and had custom node topology. When they hit production, 1 pod stayed Pending. I joined the outage call, reviewed their deployment YAML, identified the scheduling deadlock (anti-affinity required + insufficient nodes + missing tolerations), guided them through the configuration fix, and the issue was resolved within 30 minutes.”

Interview question this incident answers:

• “Tell me about a production incident you resolved."
• "How do you debug a Kubernetes scheduling issue?"
• "What is podAntiAffinity and when would it cause a problem?"
• "What is a PDB and how can it cause a deadlock?"
• "Walk me through your K8s troubleshooting process.”

20. Key Commands from This Session and Gap-Fills

Validate PDB configuration

# Check if PDB exists and what its constraints are:
kubectl get pdb -n payments
kubectl describe pdb checkout-api-pdb -n payments
# Key fields: minAvailable, maxUnavailable, DisruptionsAllowed, CurrentHealthy

# If minAvailable is too high relative to running pods:
# Temporarily patch PDB to allow drain:
kubectl patch pdb checkout-api-pdb -n payments \
  --type='json' \
  -p='[{"op": "replace", "path": "/spec/minAvailable", "value": 1}]'
# (revert after drain / fix)

Check scheduler filter evidence

# The key command for scheduling failures:
kubectl describe pod <pending-pod-name> -n payments

# In the Events section, look for lines like:
# "0/3 nodes are available:
#  1 node(s) had untolerated taint {workload=batch: NoSchedule},
#  2 node(s) didn't match pod's node affinity/selector,
#  2 node(s) had pod anti-affinity rules rejecting the pod."

# This tells you EXACTLY which filter failed on each node.

Check for ProgressDeadlineExceeded

# Check deployment status:
kubectl describe deployment checkout-api -n payments | grep -A5 Conditions
# Look for: ProgressDeadlineExceeded

# Or:
kubectl rollout status deployment checkout-api -n payments
# Shows: "Waiting for deployment 'checkout-api' rollout to finish: 1 of 3 updated replicas are available..."

# Check the configurable deadline:
kubectl get deployment checkout-api -n payments -o jsonpath='{.spec.progressDeadlineSeconds}'
# Default: 600 (10 minutes)

Pre-deployment validation script (prevention)

#!/bin/bash
# pre-deploy-check.sh
# Validates: eligible nodes ≥ replicas before deploying

NAMESPACE=$1
DEPLOYMENT=$2

REPLICAS=$(kubectl get deployment $DEPLOYMENT -n $NAMESPACE \
  -o jsonpath='{.spec.replicas}')

# Count nodes that satisfy the nodeSelector (simplistic — adjust for affinity):
NODE_SELECTOR=$(kubectl get deployment $DEPLOYMENT -n $NAMESPACE \
  -o jsonpath='{.spec.template.spec.nodeSelector}' 2>/dev/null)

ELIGIBLE_NODES=$(kubectl get nodes --show-labels | grep -v "SchedulingDisabled" | wc -l)

echo "Replicas requested: $REPLICAS"
echo "Eligible nodes (approximate): $ELIGIBLE_NODES"

if [ "$ELIGIBLE_NODES" -lt "$REPLICAS" ]; then
  echo "ERROR: Insufficient eligible nodes for deployment. Aborting."
  exit 1
fi
echo "Pre-deploy check passed."

ScoutSuite quick start

pip install scoutsuite
# Configure AWS credentials (aws configure or assume-role)
scout aws --report-dir ./security-report
# Open: ./security-report/scoutsuite-report.html

Prowler compliance check

pip install prowler
# CIS AWS Foundations Benchmark:
prowler aws --compliance cis_1.5_aws --output-formats html json

# PCI-DSS:
prowler aws --compliance pci_3.2.1_aws

Key Takeaways from This Session

1. PDB doesn’t cause scheduling failures — it amplifies them by blocking drain and rollout when the scheduling deadlock is already present.

2. The policy paradox: When required anti-affinity + minAvailable PDB + insufficient nodes all coexist, neither the scheduler nor the eviction controller can make progress. True deadlock.

3. Control plane vs. data plane is the first split to make in any K8s outage. It immediately eliminates 70% of irrelevant hypotheses.

4. 4 Pod Lifecycle Questions (created? scheduled? running? reachable?) determine which framework to apply. Don’t apply OSI to a scheduling problem.

5. RCA = evidence + chain causation + three audiences. Not a story. Not a symptom. Not vague action items. A specific, numbered chain with screenshots and technical action items.

6. ”May” in an RCA is a red flag. State what happened, with data. If you don’t have data, that itself is a finding (observability gap).

7. Security scanning cadence: Quarterly at minimum; monthly is better. Use ScoutSuite (attacker view) + Prowler (compliance view) together.

8. Immediate fix ≠ complete fix. The live debug applied preferred anti-affinity. The permanent fix (3rd node + revert to required) must be tracked as an action item and completed.