Complete Learning Package (EKS · Pod Scheduling · Taints & Tolerations · Pod Anti-Affinity · NodeSelector · Compound Fault Debugging)

2. Table of Contents

1. Executive Summary
2. Table of Contents
3. Detailed Structured Notes
   • 3.1 Infrastructure Overview (EKS Cluster)
   • 3.2 Application Overview (checkout-api)
   • 3.3 The Incident — Problem Statement
   • 3.4 Blast Radius and Business Impact
   • 3.5 Upstream / Downstream Data Flow
   • 3.6 Initial Observations (Live Console Walk-through)
   • 3.7 Systematic Ruling-Out of Common Causes
   • 3.8 Live Collaborative Debugging — Event Sequence
   • 3.9 The Three Root Causes — Detailed Analysis
   • 3.10 The Fix Applied (Step by Step)
   • 3.11 K8s Scheduling Concepts Surfaced — Deep Dive
   • 3.12 Kubernetes GUI Tooling — K8s Lens
4. Key Concepts Table
5. Architecture & Workflow Analysis
6. Commands Reference (All Commands Executed Live)
7. Deployment YAML — Before and After
8. Tools & Technologies
9. Real-World Production Usage
10. Interview Preparation (Beginner / Intermediate / Advanced)
11. Exam & Certification Notes
12. Cheat Sheet — Pod-Pending Troubleshooting
13. Gaps, Assumptions & Incomplete Areas
14. Gap-Fill — What the Session Left Unfinished, Completed Here

3. Detailed Structured Notes

3.1 Infrastructure Overview (EKS Cluster)

Node Groups:

• Total nodes: 3 (2 primary On-Demand + 1 batch Spot).
• Nodes run in private subnets; ALB is public-facing.
• ASGs back each node group.

Add-ons installed:

• VPC CNI (Amazon VPC CNI plugin — manages pod ENI/IP assignment)
• CoreDNS
• kube-proxy
• Metrics Server

No additional components: No service mesh, no sidecar proxies, no custom observability stack (AWS CloudWatch used).

3.2 Application Overview (checkout-api)

Significance: Checkout API is the most critical microservice in the e-commerce system — it handles all payment processing. Any degradation directly translates to business revenue loss and reputation damage.

3.3 The Incident — Problem Statement

Timeline:

• Routine hotfix deployment triggered via GitHub Actions CI/CD pipeline.
• Deployment targets 3 replicas in the payments namespace.
• 2 pods came up as Running.
• 1 pod stuck in Pending for 8 hours — no IP assigned, no node assigned, no container crash.
• Rolling deployment frozen (cannot proceed to terminate old pod and complete the rollout).
• A ProgressDeadlineExceeded error eventually thrown by the Deployment controller.

What makes this hard:

• No application errors. No container crash. No CrashLoopBackOff. No liveness/readiness probe failure.
• All cluster components appear healthy from outside.
• Nodes all show Ready state.
• ALB health checks passing.
• kube-proxy logs: sync loops (normal warnings), IP tables in sync.
• CNI logs: visible but no explicit errors surfaced.
• CPU and memory on all nodes: well under capacity.

Severity: Declared as Severity 1 / P0 by the organization.

Three immediate red flags identified:

1. Critical service in partially degraded state (2/3 replicas serving all traffic).
2. Deployment pipeline blocked — no further changes can reach production.
3. No visible error — confusing and time-consuming to debug.

3.4 Blast Radius and Business Impact

Current impact (T+0):

• Reduced redundancy: 2 pods handling 3-pod traffic load → latency increases.
• CI/CD pipeline blocked → any hotfix (including a fix for this incident) cannot be deployed.

Projected impact (T+10–15 min if unresolved):

• User-facing 503 errors (Service Unavailable) as remaining pods are overwhelmed.
• Payment failures for customers → direct revenue loss.
• Reputation damage.

Business stakeholders affected:

• External customers (payment processing failures).
• Internal engineering team (pipeline blocked).
• Finance (revenue at risk from payment failures).

Blast radius classification (3 dimensions):

3.5 Upstream / Downstream Data Flow

Understanding the full data path is the first step before debugging — you need to know what “normal” looks like to identify the gap.

UPSTREAM (request entering the system):
External Customer
  → HTTP request
  → ALB (Application Load Balancer)
      performs health checks on NodePorts
      applies listener rules and forwarding rules
      forwards to healthy worker node
  → Worker Node
      → NodePort
      → kube-proxy (looks up IP via iptables — round-robin algorithm)
      → Pod IP (checkout-api pod)
      → Nginx application (listening on port 80)
      → [generates response]

DOWNSTREAM (response leaving the system):
Application (Nginx)
  → kube-proxy
  → Node → ALB target group
  → ALB → External customer

MIDSTREAM (inside the cluster, per new pod creation):
Scheduler: find a suitable node for the pod
  → check node resources (CPU/RAM available?)
  → check node taints vs. pod tolerations
  → check nodeSelector labels
  → check podAntiAffinity / podAffinity rules
  → if all pass: bind pod to node
  → kubelet: pull image, create container, start probes
  → VPC CNI: allocate an ENI slot and assign a pod IP

Key insight from the engineer: Understanding the upstream/downstream flow means you know at which layer to look for the gap. If the pod never gets an IP → the problem is at the scheduler (binding stage) or CNI (IP allocation), not the application. This walkthrough’s problem was at the scheduler binding stage.

3.6 Initial Observations (Live Console Walk-through)

Commands run and what they showed:

# Check pods in the payments namespace:
kubectl get pods -n payments

Output: 2 pods Running, 1 pod Pending. No IP, no node assigned to the Pending pod.

# Describe the pending pod:
kubectl describe pod <pending-pod-name> -n payments

Output: Container spec visible (image, limits, requests, probes). In Events: messages about scheduling failure — “0 nodes available” variants. No container error, no image pull error.

# Check all nodes:
kubectl get nodes

Output: 3 nodes, all Ready. One node: SchedulingDisabled (cordoned).

# Check events in the payments namespace:
kubectl get events -n payments --sort-by='.lastTimestamp'

Output: Warnings about scheduling failures. No application-level errors.

# Check kube-proxy logs:
kubectl logs -n kube-system -l k8s-app=kube-proxy --tail=20

Output: Sync loop warnings (normal). IP tables synced. No errors.

# Check CNI logs:
kubectl logs -n kube-system -l k8s-app=aws-node

Output: No clear errors found. CNI appeared healthy from UI.

# Check deployment manifest:
kubectl get deployment checkout-api -n payments -o yaml

Output: Showed 3 replicas, container spec, limits/requests, liveness/readiness probes — and critically: the scheduling constraints that became the focus of the live debug.

# Check resource utilization:
kubectl top nodes
kubectl top pods -n payments

Output: CPU and memory well under capacity on all nodes. Resources are not the problem.

Summary of initial observations:

• Cluster healthy ✓
• Nodes ready (but one cordoned) ✓/⚠
• Application not crashing ✓
• Resources not exhausted ✓
• CNI and kube-proxy healthy ✓
• Scheduler cannot place the pod ✗ — reason unknown at this point

3.7 Systematic Ruling-Out of Common Causes

The engineer walked through the hypothesis elimination process:

The ENI context from the engineer:

“In AP-South-1, while setting up the cluster, the ENI (Elastic Network Interface) slots were actually full. The network interface couldn’t attach an IP to the pods. I had to delete existing ENIs before the pods could get IPs.” This confirms ENI exhaustion is a real additional risk factor in this environment — though not the root cause here.

3.8 Live Collaborative Debugging — Event Sequence

The engineer asked attendees to debug the live cluster. An attendee (Shai) screen-shared. The debugging sequence that unfolded:

Step 1 — kubectl describe pod <pending-pod> Events revealed:

• "0 nodes didn't match the pod's node affinity/selector" (or similar wording)
• "0 nodes have available space" (related to anti-affinity)
• One node with unschedulable (cordoned)
• Nodes with untolerated taints

Step 2 — kubectl get nodes Confirmed: one node SchedulingDisabled (cordoned). Two active nodes. The batch node group node has a taint (workload=batch:NoSchedule or similar).

Step 3 — kubectl get deployment checkout-api -n payments -o yaml Found:

• podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution with topology key kubernetes.io/hostname (one pod per host, hard requirement)
• A nodeSelector pointing to a label (e.g., nodegroup: primary or disktype: ssd)
• No tolerations in the pod spec

Step 4 — Identify the compound fault:

• 3 pods needed, but only 2 schedulable nodes (one cordoned; batch node has taint; pod has no toleration).
• Anti-affinity required forces 1 pod per host. 2 hosts available → maximum 2 pods schedulable. Third pod stays Pending.
• nodeSelector adds another constraint that conflicts with where the pods can go.

Step 5 — Apply temporary fix (change required → preferred): The attendees tried editing the Deployment YAML live:

• Changed requiredDuringSchedulingIgnoredDuringExecution → preferredDuringSchedulingIgnoredDuringExecution
• Added weight: 100 (required field for preferred that required doesn’t use)
• Syntax issues encountered (unknown field errors when labelSelector structure wasn’t adjusted for preferred format)

Step 6 — Add tolerations: Added a toleration matching the batch node’s taint:

tolerations:
- key: "workload"
  value: "batch"
  effect: "NoSchedule"

Step 7 — Remove / comment out conflicting nodeSelector The nodeSelector was conflicting because it pointed to a label that the available nodes didn’t match (or matched a subset that further restricted placement).

Step 8 — Rolling restart:

kubectl rollout restart deployment checkout-api -n payments

Pods cycled. However, since the existing 2 running pods occupied the 2 available nodes, and the new pod needed a node too, the rolling restart created complexity (can’t create new pod before deleting old one if the node is full).

Final working state: All 3 pods running. Key changes confirmed:

1. Anti-affinity changed to preferred
2. Tolerations added
3. nodeSelector removed / commented out

Debate during session — what’s the “right” permanent fix? The team discussed: the correct long-term fix for a required anti-affinity with 3 replicas is 3 schedulable nodes with no conflicting constraints, not preferred. They identified that changing to preferred compromises the high-availability intent (pods may co-locate on the same node). The architecturally correct fix is to add a third node with the right taint structure. Changing to preferred was the immediate stabilization fix; adding a node is the permanent fix.

3.9 The Three Root Causes — Detailed Analysis

Root Cause 1: podAntiAffinity set to required with insufficient nodes

What it does:

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:   # HARD REQUIREMENT
    - labelSelector:
        matchExpressions:
        - key: app
          operator: In
          values: ["checkout-api"]
      topologyKey: kubernetes.io/hostname              # ONE POD PER HOST

This says: “Do not schedule this pod on any node that already has a pod matching app=checkout-api.”

Why it failed:

• 3 replicas needed.
• Only 2 schedulable nodes (one cordoned, one tainted without toleration).
• Pod 1 → Node A ✓. Pod 2 → Node B ✓. Pod 3 → no available node that doesn’t already have a checkout-api pod → Pending.

requiredDuringSchedulingIgnoredDuringExecution vs preferredDuringSchedulingIgnoredDuringExecution:

• required: hard constraint. If unsatisfiable, pod stays Pending forever.
• preferred: soft constraint. If unsatisfiable, pod still schedules (possibly co-locating). Has a weight field (1–100).

When required is appropriate: When you must ensure no two replicas are on the same node (true HA — losing one node should not lose two replicas). Requires: number of available nodes ≥ number of replicas.

When preferred is appropriate: When you prefer distribution but can tolerate co-location rather than leaving pods unscheduled. More resilient to node loss but less strict.

Root Cause 2: Missing tolerations

What happened:

• The batch-node-group nodes had taints applied (e.g., workload=batch:NoSchedule).
• The checkout-api pod spec had no tolerations section.
• Without a toleration, a pod cannot be scheduled on a tainted node (NoSchedule effect = hard block).
• This left only the primary-node-group nodes available for scheduling.

Taints and tolerations model:

Node taint: key=value:effect
  NoSchedule:      Pods without matching toleration are NOT scheduled here (new pods only)
  PreferNoSchedule: Prefer not to schedule here (soft)
  NoExecute:       Existing pods are evicted if they don't have matching toleration

Pod toleration (must match):
tolerations:
- key: "workload"
  operator: "Equal"
  value: "batch"
  effect: "NoSchedule"

Key insight from session (subtle but important): Taints may have been added to nodes after pods were already running on them. The NoSchedule effect only blocks new scheduling — it does not evict existing pods (NoExecute does that). This is why existing checkout-api pods continued running on tainted nodes even though new pods couldn’t be scheduled there.

Root Cause 3: Conflicting nodeSelector

What happened:

nodeSelector:
  nodegroup: primary   # (or similar label — exact value unclear from transcript)

How nodeSelector works:

• Hard constraint: pod will ONLY be scheduled on nodes with this exact label.
• nodeSelector and podAntiAffinity are AND conditions — both must be satisfied.
• If nodeSelector points to primary nodes AND anti-affinity says “don’t co-locate” AND only 2 primary nodes are available AND one is cordoned → unsatisfiable.

Why it should be removed:

• In an autoscaling environment (ASG/Karpenter), hardcoding a node name or label in nodeSelector is fragile — new nodes may have different names.
• podAntiAffinity already handles spreading. nodeSelector is redundant and actively harmful here.
• If spreading is needed, use nodeAffinity (more expressive) instead of nodeSelector (coarser).

nodeSelector vs nodeAffinity:

3.10 The Fix Applied (Step by Step)

Immediate stabilization fix (applied live):

Step 1: Edit the Deployment

kubectl edit deployment checkout-api -n payments

Changes made in the YAML:

1. Change requiredDuringSchedulingIgnoredDuringExecution → preferredDuringSchedulingIgnoredDuringExecution
2. Add weight: 100 (required for preferred blocks)
3. Add tolerations for the batch node taint
4. Remove (or comment out) the conflicting nodeSelector

Step 2: Trigger rolling restart (to cycle the pods through new scheduling logic)

kubectl rollout restart deployment checkout-api -n payments

Step 3: Verify

kubectl get pods -n payments -w
# All 3 should reach Running within ~60 seconds
kubectl get events -n payments --sort-by='.lastTimestamp'
# No scheduling failure warnings

Permanent architectural fix (discussed but not applied):

Add a third node to the primary node group with the same taint structure as the other primary nodes, then revert anti-affinity to required. This restores true HA (one pod per host, guaranteed) without changing the deployment’s scheduling intent.

# Increase desired capacity on the primary ASG:
aws autoscaling set-desired-capacity \
  --auto-scaling-group-name <primary-asg-name> \
  --desired-capacity 3

# OR update the EKS node group:
aws eks update-nodegroup-config \
  --cluster-name sre-labs-HA-lab \
  --nodegroup-name primary-node-group \
  --scaling-config minSize=2,maxSize=4,desiredSize=3

Then revert preferredDuringScheduling → requiredDuringScheduling in the Deployment.

3.11 Kubernetes Scheduling Concepts — Deep Dive

The Scheduling Decision Tree (what the scheduler checks, in order)

New pod needs to be scheduled
  │
  ├─ 1. Node taints vs. pod tolerations
  │       If node has NoSchedule taint AND pod has no matching toleration → SKIP NODE
  │
  ├─ 2. nodeSelector
  │       If pod has nodeSelector AND node doesn't have all matching labels → SKIP NODE
  │
  ├─ 3. Node Affinity (requiredDuringScheduling)
  │       If pod has required node affinity AND node doesn't match → SKIP NODE
  │
  ├─ 4. Pod Anti-Affinity (requiredDuringScheduling)
  │       If another matching pod already on this node AND anti-affinity is required → SKIP NODE
  │
  ├─ 5. Resource availability
  │       If node doesn't have enough CPU/RAM for pod's requests → SKIP NODE
  │
  └─ 6. Other: Pod Affinity, Priority, Preemption
         → if all previous checks pass, schedule pod here
         → if NO node passes ALL checks → pod stays Pending

If pod stays Pending:
  → Scheduler continuously retries (every ~10s default)
  → Deployment controller tracks progress timeout (progressDeadlineSeconds, default 600s)
  → After deadline: Deployment shows ProgressDeadlineExceeded

Key Scheduling Objects

Taints (on nodes)

# Apply a taint to a node:
kubectl taint nodes <node-name> workload=batch:NoSchedule

# Remove a taint:
kubectl taint nodes <node-name> workload=batch:NoSchedule-

# View taints on a node:
kubectl describe node <node-name> | grep Taints

Tolerations (on pods/deployments)

spec:
  tolerations:
  - key: "workload"           # Must match taint key
    operator: "Equal"         # "Equal" (match key+value) or "Exists" (match key only)
    value: "batch"            # Must match taint value (if operator=Equal)
    effect: "NoSchedule"      # Must match taint effect (or omit to tolerate any effect)
    # tolerationSeconds: 3600  # Only for NoExecute: how long before eviction

nodeSelector

spec:
  nodeSelector:
    nodegroup: primary        # Hard requirement: must have this label
    disktype: ssd             # AND this label (all must match)

Pod Anti-Affinity (required — hard)

spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values: ["checkout-api"]
        topologyKey: kubernetes.io/hostname  # One pod per node

Pod Anti-Affinity (preferred — soft)

spec:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100             # 1-100; higher = stronger preference
        podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values: ["checkout-api"]
          topologyKey: kubernetes.io/hostname

Critical syntax difference: preferred wraps the affinity term in podAffinityTerm and requires a weight. required does not have weight and takes the affinity term directly. This caused the invalid field errors during the live debug.

Node Cordon / Uncordon

# Cordon (mark node unschedulable — no NEW pods, existing pods continue):
kubectl cordon <node-name>

# Uncordon (make node schedulable again):
kubectl uncordon <node-name>

# Drain (evict all pods, then cordon — for node maintenance):
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

Cordon vs. Taint:

Priority and Preemption

Mentioned in the live debug but not a factor here:

• priorityClass — assigns a priority to a pod; higher priority pods can preempt lower-priority ones.
• preemptionPolicy — whether a pod can preempt others.
• If seen in Events: "preemption is not helpful for scheduling" → the scheduler considered preempting other pods to make room but determined it wouldn’t help. This rules out resource starvation as the cause (if preemption would have helped, resources would be the issue).

3.12 Kubernetes GUI Tooling — K8s Lens

Mentioned by the engineer as a useful visual alternative to CLI debugging:

K8s Lens (now: OpenLens):

• A desktop application for Kubernetes cluster management.
• Connect multiple clusters via kubeconfig.
• Visual view of: nodes, pods, deployments, events, resource utilization, logs.
• Can help less CLI-comfortable engineers explore the cluster visually.
• Supports EKS, GKE, AKS, on-prem K8s.

Installation: lens.app (or OpenLens on GitHub)

Workflow with Lens:

1. Add kubeconfig → cluster appears in Lens.
2. Navigate to Workloads → Pods → filter by namespace payments.
3. Click the Pending pod → see Events tab without typing kubectl describe.
4. Navigate to Nodes → see taint/label status visually.

Limitation: Lens shows the same data as kubectl; it does not help with debugging logic. Useful for quick visual scanning but CLI remains essential for production debugging.

4. Key Concepts Table

5. Architecture & Workflow Analysis

5.1 Cluster Topology and the Scheduling Deadlock

EKS Cluster: sre-labs-HA-lab
│
├── primary-node-group (On-Demand)
│   ├── Node A  [Ready]
│   │   ├── checkout-api-pod-1 (Running) ✓
│   │   └── [anti-affinity: no more checkout-api pods here]
│   │
│   └── Node B  [Ready]
│       ├── checkout-api-pod-2 (Running) ✓
│       └── [anti-affinity: no more checkout-api pods here]
│
└── batch-node-group (Spot)
    └── Node C  [SchedulingDisabled] OR [Tainted: workload=batch:NoSchedule]
        └── [checkout-api has no toleration + cordon → cannot schedule here]

checkout-api-pod-3: PENDING
  → Node A: rejected (anti-affinity: already has pod-1)
  → Node B: rejected (anti-affinity: already has pod-2)
  → Node C: rejected (cordoned AND/OR tainted without toleration)
  → Result: 0 eligible nodes → Pending forever

5.2 Data Flow Through the Cluster (Normal vs. Degraded)

NORMAL (3 pods):
External User → ALB → NodePort → kube-proxy
  → round-robin: Pod 1 (Node A) OR Pod 2 (Node B) OR Pod 3 (Node C)
  Response: each pod handles ~33% of load

DEGRADED (2 pods, during incident):
External User → ALB → NodePort → kube-proxy
  → round-robin: Pod 1 (Node A) OR Pod 2 (Node B)  [Pod 3: Pending]
  Response: each active pod handles ~50% of load
  Impact: higher latency → risk of 503 if traffic spikes
  Deployment: FROZEN (rolling update cannot complete)

5.3 Scheduling Decision for the Pending Pod

checkout-api pod-3 scheduling attempt:
  ↓
Check Node A:
  Taint check: no taint → pass ✓
  nodeSelector: "primary" label present? → DEPENDS (may fail)
  Anti-affinity: checkout-api-pod-1 already on Node A → FAIL ✗
  → SKIP Node A

Check Node B:
  Taint check: no taint → pass ✓
  nodeSelector: check
  Anti-affinity: checkout-api-pod-2 already on Node B → FAIL ✗
  → SKIP Node B

Check Node C (batch):
  Taint check: workload=batch:NoSchedule → pod has no toleration → FAIL ✗
  (Also: SchedulingDisabled → FAIL ✗)
  → SKIP Node C

Result: 0 eligible nodes → pod stays Pending

5.4 Pod Anti-Affinity: required vs. preferred

REQUIRED (current broken state):
  3 replicas, 2 eligible nodes
  Pod 1 → Node A ✓  (anti-affinity satisfied)
  Pod 2 → Node B ✓  (anti-affinity satisfied)
  Pod 3 → ??? No node without a checkout-api pod → PENDING ✗

PREFERRED (immediate fix applied):
  3 replicas, 2 eligible nodes (after removing nodeSelector)
  Pod 1 → Node A ✓  (anti-affinity preference checked; Node A is OK)
  Pod 2 → Node B ✓  (anti-affinity preference checked; Node B is OK)
  Pod 3 → Node A or B ✓  (anti-affinity "preferred" — violated but pod scheduled anyway)
  All 3 pods: Running ✓ (but 2 may co-locate on same node — reduced HA)

PERMANENT FIX (required + 3rd node):
  3 replicas, 3 eligible nodes (after adding a node + toleration)
  Pod 1 → Node A ✓  Pod 2 → Node B ✓  Pod 3 → Node C ✓
  All 3 pods: Running ✓  AND  strict HA maintained ✓

6. Commands Reference (All Commands Executed Live)

# ── INITIAL INVESTIGATION ────────────────────────────────────────────

# List pods in the payments namespace:
kubectl get pods -n payments
# Look for: STATUS (should all be Running), READY, IP, NODE assigned

# Describe the pending pod (most important command):
kubectl describe pod <pending-pod-name> -n payments
# Look for Events section at the bottom — shows WHY scheduler rejected nodes

# List all nodes with taint and status info:
kubectl get nodes
kubectl describe node <node-name>   # Look for: Taints, Labels, Conditions

# Check events in the namespace (scheduling failures show here):
kubectl get events -n payments --sort-by='.lastTimestamp' --field-selector type=Warning

# View the deployment YAML (where the scheduling constraints live):
kubectl get deployment checkout-api -n payments -o yaml

# Check resource utilization (rule out resource exhaustion):
kubectl top nodes
kubectl top pods -n payments

# ── COMPONENT HEALTH CHECKS ──────────────────────────────────────────

# Check kube-proxy logs:
kubectl logs -n kube-system -l k8s-app=kube-proxy --tail=20

# Check VPC CNI logs:
kubectl logs -n kube-system -l k8s-app=aws-node --tail=50

# Check CoreDNS:
kubectl get pods -n kube-system | grep coredns

# ── FIX OPERATIONS ───────────────────────────────────────────────────

# Edit deployment live (opens $EDITOR):
kubectl edit deployment checkout-api -n payments

# Apply a patched YAML file:
kubectl apply -f checkout-api-deployment.yaml

# Rolling restart (recycles all pods through new scheduling logic):
kubectl rollout restart deployment checkout-api -n payments

# Check rollout status:
kubectl rollout status deployment checkout-api -n payments

# Rollback if the fix caused issues:
kubectl rollout undo deployment checkout-api -n payments

# ── NODE MANAGEMENT ──────────────────────────────────────────────────

# Cordon a node (mark unschedulable — no new pods):
kubectl cordon <node-name>

# Uncordon a node (make schedulable again):
kubectl uncordon <node-name>

# Add a taint to a node:
kubectl taint nodes <node-name> workload=batch:NoSchedule

# Remove a taint from a node:
kubectl taint nodes <node-name> workload=batch:NoSchedule-

# ── SCALE NODE GROUP ────────────────────────────────────────────────

# Scale EKS node group (permanent fix: add 3rd schedulable node):
aws eks update-nodegroup-config \
  --cluster-name sre-labs-HA-lab \
  --nodegroup-name primary-node-group \
  --scaling-config minSize=2,maxSize=4,desiredSize=3

# ── CONNECT TO EKS CLUSTER ──────────────────────────────────────────

# Update local kubeconfig to use the EKS cluster:
aws eks update-kubeconfig \
  --name sre-labs-HA-lab \
  --region ap-south-1

# Verify connection:
kubectl get nodes
kubectl get pods -n payments

7. Deployment YAML — Before and After

Before (Broken — 3 root causes active)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
  namespace: payments
spec:
  replicas: 3
  selector:
    matchLabels:
      app: checkout-api
  template:
    metadata:
      labels:
        app: checkout-api
    spec:
      # ROOT CAUSE 3: nodeSelector conflicting with anti-affinity
      nodeSelector:
        nodegroup: primary     # ← Points to label not on available/eligible nodes

      # ROOT CAUSE 2: No tolerations (nodes have taints)
      # tolerations: []        # ← Missing entirely

      # ROOT CAUSE 1: Anti-affinity REQUIRED — can't satisfy with only 2 nodes
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:   # ← HARD requirement
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values: ["checkout-api"]
            topologyKey: kubernetes.io/hostname

      containers:
      - name: checkout-api
        image: nginx:alpine
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: "50m"
            memory: "64Mi"
          limits:
            cpu: "200m"
            memory: "256Mi"
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 3
          periodSeconds: 5

After (Immediate Fix — All 3 pods Running)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
  namespace: payments
spec:
  replicas: 3
  selector:
    matchLabels:
      app: checkout-api
  template:
    metadata:
      labels:
        app: checkout-api
    spec:
      # FIX 3: nodeSelector removed (was conflicting)
      # nodeSelector removed

      # FIX 2: Tolerations added (allows scheduling on tainted nodes)
      tolerations:
      - key: "workload"
        operator: "Equal"
        value: "batch"
        effect: "NoSchedule"

      # FIX 1: Anti-affinity changed to PREFERRED (graceful fallback)
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:  # ← SOFT preference
          - weight: 100                                      # ← Required for preferred
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values: ["checkout-api"]
              topologyKey: kubernetes.io/hostname

      containers:
      - name: checkout-api
        image: nginx:alpine
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: "50m"
            memory: "64Mi"
          limits:
            cpu: "200m"
            memory: "256Mi"
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 3
          periodSeconds: 5

Permanent Fix (YAML intent — requires 3rd schedulable node):

# Revert anti-affinity to required, add toleration only for needed nodes:
      tolerations: []   # No tolerations needed if all 3 primary nodes are untainted

      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:  # ← HARD requirement restored
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values: ["checkout-api"]
            topologyKey: kubernetes.io/hostname
# (nodeSelector remains removed)
# Requires: 3 schedulable primary nodes with no blocking taints

8. Tools & Technologies

9. Real-World Production Usage

This is the most common class of Kubernetes incident in real organizations:

• Scaling up a deployment crosses a resource or constraint boundary.
• Adding a node with a new taint without updating pod tolerations.
• Changing anti-affinity from preferred to required without verifying enough nodes exist.
• ENI exhaustion in AWS VPC CNI (pods can’t get IPs even when scheduling succeeds).

Why “no visible error” is so dangerous: The pod shows Pending — not Error, not CrashLoopBackOff. Operators without scheduling knowledge immediately check application logs (empty), CPU/RAM (fine), network (fine), and conclude “it should work.” The bug is in the Deployment YAML, which they may not think to check.

The rolling deployment interaction: Kubernetes rolling deployments (default RollingUpdate strategy) create new pods before terminating old ones (maxSurge: 1, maxUnavailable: 0 by default). If the new pod can’t schedule, the rollout freezes — the old pod never gets terminated. This is why the CI/CD pipeline was also blocked: the Deployment was midway through a rollout with one stuck new pod.

HA design rule: For requiredDuringScheduling podAntiAffinity with topologyKey: hostname:

Number of schedulable nodes ≥ Number of replicas

If you have 3 replicas with required anti-affinity, you need at least 3 nodes that the pod can tolerate and has no blocking constraints against.

ENI exhaustion (AWS-specific): Each EC2 instance type has a maximum number of ENIs, and each ENI can hold a limited number of pod IPs. In small instance types or heavily-used subnets, this cap can be hit silently. kubectl describe pod will show an event like "failed to create pod sandbox: rpc error: ... not enough IPs". The fix: use a larger instance type, use prefix delegation in VPC CNI, or spread pods across more subnets.

10. Interview Preparation

Beginner

Q1. A pod is in Pending state. Where do you look first? A: kubectl describe pod <pod-name> -n <namespace> and read the Events section at the bottom. It will say something like: “0 nodes available: 1 node had taint that the pod didn’t tolerate, 1 node had pod anti-affinity rules rejecting the pod, 1 node was cordoned.” Each reason maps to a specific fix.

Q2. What is a taint and how does it affect pod scheduling? A: A taint is a key=value:effect mark on a node that repels pods without a matching toleration. Effects: NoSchedule (new pods without matching toleration won’t be placed here), PreferNoSchedule (soft — prefer not to place, but will if no other option), NoExecute (evicts existing pods that don’t have a matching toleration). Pods that need to run on a tainted node must include a tolerations spec in their pod spec.

Q3. What is the difference between kubectl cordon and kubectl taint? A: cordon marks a node as SchedulingDisabled (no new pods will be placed there by any pod — total block). taint marks a node with a specific key/value/effect that only repels pods without a matching toleration (other pods without that toleration are blocked, but pods WITH the toleration are fine). Cordon is for maintenance; taint is for workload segregation.

Intermediate

Q4. Explain podAntiAffinity with required vs. preferred and give a real-world scenario where using required would cause a production outage. A: required is a hard constraint — if unsatisfiable, the pod stays Pending forever. preferred is a soft constraint with a weight — if unsatisfiable, the pod schedules anyway. Real-world scenario: a deployment with 3 replicas and required anti-affinity (topologyKey: hostname) runs fine on a 3-node cluster. A new node group with a taint is added but the pod spec has no toleration. A node maintenance event reduces available nodes to 2. The 3rd replica can’t schedule (both remaining nodes already have a replica) → Pending → rolling deployment freezes → CI/CD blocked. Fix: either add a 3rd node, add a toleration for the new node, or change to preferred.

Q5. What is the difference between nodeSelector and nodeAffinity, and why should nodeSelector be avoided in autoscaling environments? A: nodeSelector is a simple key-value map that hard-constrains pod placement to nodes with all matching labels. nodeAffinity supports the same hard constraint (required) but also a soft (preferred) form, and supports complex expressions (In, NotIn, Exists, DoesNotExist, Gt, Lt). In autoscaling environments (ASG, Karpenter), newly-provisioned nodes are assigned dynamic names/IPs; if nodeSelector points to a specific node name or label that new nodes don’t have, pods will never schedule on new nodes. nodeAffinity with preferred is more resilient.

Q6. A checkout-api pod is Pending. kubectl describe pod shows: “0 nodes available: 1 node had taint {workload=batch:NoSchedule}, 1 node was cordoned, 1 didn’t match pod’s node affinity.” What are the three fixes and which should you apply in a P0? A: Fix 1 (immediate/P0): Add a toleration to the Deployment spec for the batch taint, uncordon the cordoned node (if safe), and verify/fix node affinity labels. Fix 2 (medium-term): Add a 3rd on-demand node with the correct labels so anti-affinity can be satisfied with required. Fix 3 (permanent/architectural): Ensure the Deployment’s scheduling constraints are always satisfiable at any scale — document the relationship between replica count, anti-affinity requirements, and minimum node count.

Advanced

Q7. Describe the interaction between nodeSelector, podAntiAffinity, and tolerations as AND conditions in the Kubernetes scheduler. How can these compound to deadlock a pod? A: The scheduler must satisfy ALL constraints simultaneously. If nodeSelector says “only on nodes with label X,” and podAntiAffinity: required says “not on any node that has a pod with label Y,” and tolerations must match all node taints — then any node that satisfies one constraint may fail another. In this walkthrough: nodeSelector filtered to primary nodes → anti-affinity required 1 pod per host → only 2 primary nodes available → 3rd pod can’t satisfy anti-affinity → Pending. All three constraints were AND’d. The deadlock was broken only by removing nodeSelector (eliminated one constraint) and changing anti-affinity to preferred (made another constraint soft).

Q8. You’re in a P0. The checkout-api pod has been Pending for 8 hours. The business wants it fixed NOW, but the engineering lead says “we can’t compromise our HA topology.” How do you resolve the conflict? A: Apply a two-phase fix. Phase 1 (immediate — 5 minutes): Change anti-affinity to preferred, add tolerations, remove conflicting nodeSelector. Pods come up immediately. CI/CD unblocked. Revenue impact stops. Phase 2 (next maintenance window — same day): Add a third schedulable node to the primary node group. Revert anti-affinity to required. Document the constraint: replicas ≤ schedulable nodes. This satisfies both: immediate stabilization AND full HA restoration within hours. The key principle: stabilize first, then restore the intended design.

11. Exam & Certification Notes

CKA (Certified Kubernetes Administrator) — directly testable:

• kubectl taint nodes, kubectl cordon, kubectl uncordon, kubectl drain
• Writing tolerations in a pod spec
• Writing nodeSelector in a pod spec
• Understanding podAntiAffinity syntax (required vs. preferred; topologyKey)
• kubectl rollout restart, kubectl rollout status, kubectl rollout undo
• Debugging a Pending pod via kubectl describe pod → Events

CKAD (Certified Kubernetes Application Developer):

• Scheduling constraints in Deployment YAML: nodeSelector, affinity, tolerations
• resources.requests and resources.limits
• Liveness and readiness probes
• Deployment rolling update strategy

Common exam trick questions:

• “Cordon removes pods from a node” → False. Cordon only prevents NEW pods. Existing pods continue. drain evicts.
• ”Adding a taint to a node immediately evicts pods on it” → Only if the taint effect is NoExecute. NoSchedule does not evict existing pods.
• ”A pod with required anti-affinity will eventually get scheduled if you wait” → False. required means permanently Pending until the constraint can be satisfied. The scheduler retries indefinitely but the constraint doesn’t relax with time.
• ”preferred anti-affinity with weight 100 means the pod won’t schedule if the condition fails” → False. weight 100 means maximum preference, but it’s still soft — the pod WILL schedule even if the condition is violated.
• ”You need nodeSelector to use podAntiAffinity” → False. They are independent. Using both creates an AND condition.

12. Cheat Sheet — Pod-Pending Troubleshooting

POD IS PENDING? Start here:

Step 1: kubectl describe pod <pod-name> -n <namespace>
         → Read EVENTS section at the bottom
         → What does the scheduler say about each node?

Common messages and their fixes:
┌──────────────────────────────────────────────────────────────┬──────────────────────────────────────────┐
│ Event message                                                │ Fix                                      │
├──────────────────────────────────────────────────────────────┼──────────────────────────────────────────┤
│ "N node(s) had taint {X=Y:NoSchedule} that the pod didn't   │ Add tolerations to pod spec              │
│  tolerate"                                                   │                                          │
├──────────────────────────────────────────────────────────────┼──────────────────────────────────────────┤
│ "N node(s) didn't match pod's node affinity/selector"        │ Fix nodeSelector or nodeAffinity labels  │
├──────────────────────────────────────────────────────────────┼──────────────────────────────────────────┤
│ "N node(s) didn't satisfy existing pod anti-affinity rules"  │ Add more nodes OR change to preferred    │
├──────────────────────────────────────────────────────────────┼──────────────────────────────────────────┤
│ "N node(s) were unschedulable" (cordoned)                    │ kubectl uncordon <node>                  │
├──────────────────────────────────────────────────────────────┼──────────────────────────────────────────┤
│ "N node(s) had insufficient cpu/memory"                      │ Right-size pod requests OR add nodes     │
├──────────────────────────────────────────────────────────────┼──────────────────────────────────────────┤
│ "failed to create pod sandbox ... not enough IPs"            │ ENI exhaustion: larger instance OR       │
│                                                              │ VPC CNI prefix delegation                │
└──────────────────────────────────────────────────────────────┴──────────────────────────────────────────┘

Step 2: Check the Deployment YAML for constraint conflicts
  kubectl get deployment <name> -n <namespace> -o yaml
  Look for: nodeSelector / nodeAffinity / podAntiAffinity / tolerations

Step 3: Check node status
  kubectl get nodes
  kubectl describe node <node-name>   # Check: Taints, Labels, Conditions

Step 4: Rule out resources
  kubectl top nodes
  kubectl top pods -n <namespace>

Step 5: Apply fix
  kubectl edit deployment <name> -n <namespace>
  OR kubectl apply -f fixed-deployment.yaml
  THEN kubectl rollout restart deployment <name> -n <namespace>
  THEN kubectl rollout status deployment <name> -n <namespace>

Anti-affinity quick reference:

# Required (hard — pod PENDING if unsatisfied):
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
    matchExpressions:
    - key: app
      operator: In
      values: ["my-app"]
  topologyKey: kubernetes.io/hostname

# Preferred (soft — pod schedules anyway):
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100               # REQUIRED for preferred blocks!
  podAffinityTerm:          # REQUIRED for preferred blocks (wraps the term)!
    labelSelector:
      matchExpressions:
      - key: app
        operator: In
        values: ["my-app"]
    topologyKey: kubernetes.io/hostname

HA design rule: schedulable_nodes ≥ replicas when using required anti-affinity with topologyKey: kubernetes.io/hostname.

13. Gaps, Assumptions & Incomplete Areas

Deferred to the next section:

• The structured Kubernetes V7 debugging framework.
• The second (more advanced) compound outage (the assignment).
• RCA document writing workshop.
• Security hardening project (SecureAsset fintech).
• 500-microservices MNC project kickoff.

Transcription artifacts:

• “Cubernetes / cubinets” = Kubernetes
• ”Cube proxy / Q proxy” = kube-proxy
• ”Cube CDL / KCTL” = kubectl
• ”Enginext / enginext” = Nginx
• ”VK8s / VKS” = EKS
• ”Anti-affffinity” = podAntiAffinity
• ”Ts” (in context of nodes) = Taints
• ”Lin” in context of probes = Liveness probe
• ”Leguling errors” = scheduling errors
• ”No sh” / “no schedule” = NoSchedule (taint effect)
• “Preferred” vs “required” — the session used casual terminology; mapped to K8s API terms above

Session limitations:

• The live debug was collaborative (attendees driving, instructor guiding) — some commands weren’t clearly stated aloud. I reconstructed the logical sequence from context and outcomes.
• The exact taint key/value on the batch node was not stated clearly (“workload=batch” is inferred from context).
• The exact nodeSelector label was not stated clearly (“nodegroup: primary” or similar is inferred).

Assumptions:

• The cordoned node is the batch Spot node. The two available untainted primary nodes are where pods 1 and 2 ran. Pod 3 couldn’t fit.
• The ENI exhaustion mentioned was a setup issue (already resolved), not a contributing factor to this specific outage.
• The “third issue” (priority class / preemption) was mentioned by an attendee and confirmed as not the cause in this case.

14. Gap-Fill — What the Session Left Unfinished, Completed Here

GAP 1 — The Structured Kubernetes Debugging Framework (V7, Deferred from Session)

The engineer mentioned “a structured debugging framework” that would show the speed difference vs. random hypothesis-testing. This is the Kubernetes V7 / Verdict-7 framework (referenced in the SSH war-room session). Applied to a Pending pod:

V7 KUBERNETES PENDING-POD DIAGNOSTIC FLOW

Layer 1: WORKLOAD
  → kubectl get pod <name> -n <ns>: STATUS = Pending?
  → kubectl describe pod <name> -n <ns>: Read Events section

Layer 2: SCHEDULING
  → Events show scheduler reasons for each node rejection
  → Categorize: Taint? Affinity? NodeSelector? Resources? Cordoned?
  → kubectl get deployment <name> -n <ns> -o yaml: check ALL constraints

Layer 3: NODE
  → kubectl get nodes: any NotReady? SchedulingDisabled?
  → kubectl describe node <name>: check Taints, Labels, Capacity, Allocatable

Layer 4: NETWORKING (ENI / CNI)
  → If scheduling passes but IP not assigned: check CNI
  → kubectl logs -n kube-system -l k8s-app=aws-node
  → Look for: "not enough IPs", "ENI limit reached"

Layer 5: RESOURCE
  → kubectl top nodes: is a node near capacity?
  → kubectl describe node: check Allocated resources vs. Capacity
  → Check pod's resources.requests vs. node's Allocatable

Layer 6: CONFIGURATION
  → All scheduling constraints resolved?
  → Image pullable? (ImagePullBackOff would show if the image was the issue)
  → RBAC / ServiceAccount permissions? (less common for Pending)

Layer 7: INFRASTRUCTURE
  → AWS quota limits (vCPU quota per region/AZ)
  → EC2 Spot capacity availability (Spot nodes may not be available in AZ)
  → EKS API server reachable?

Efficiency: V7 takes 5–10 minutes systematically. Random guessing (restart, check logs, check CPU, try a fix…) took 30–40 minutes in the session’s live debug.

GAP 2 — ENI Exhaustion: Full Explanation (Mentioned but Not Explained)

The engineer mentioned ENI exhaustion in ap-south-1 during cluster setup. Here’s what it is and how to detect/fix it:

What is ENI exhaustion? Each EC2 instance type has a maximum number of Network Interfaces (ENIs) it can attach. Each ENI can hold a limited number of pod IPs. The VPC CNI plugin assigns pod IPs by borrowing from the node’s ENIs. When all ENI slots are full, no new pod IPs can be allocated.

# Symptoms:
# kubectl describe pod <pending-pod>:
# Events: "failed to create pod sandbox: ... failed to assign an IP address to the sandbox"
# OR: "networkPlugin cni failed to set up pod ... network: add cmd: failed to assign an IP..."

# Check VPC CNI state:
kubectl get daemonset aws-node -n kube-system
kubectl logs -n kube-system daemonset/aws-node | grep -i "ENI\|IP\|error"

# Check current ENI usage (from EC2 console or CLI):
aws ec2 describe-network-interfaces \
  --filters "Name=attachment.instance-id,Values=<node-instance-id>" \
  --query 'NetworkInterfaces[*].[NetworkInterfaceId,PrivateIpAddressCount]'

Fix options:

1. Use a larger instance type — more ENI slots per instance.
2. Enable VPC CNI prefix delegation — assigns /28 CIDR blocks per ENI slot (16x more IPs per ENI).

   kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true
   

3. Use VPC CNI custom networking — spread pods across additional subnets with more IP space.
4. Delete unused/stale ENIs — what the engineer did manually.

GAP 3 — Rolling Deployment Interaction with Pending Pods

Why did the CI/CD pipeline block? This mechanism wasn’t fully explained:

Deployment strategy: RollingUpdate (default)
  maxSurge: 1           # Can have 1 extra pod above desired replicas during update
  maxUnavailable: 0     # Must maintain all replicas during update

During the hotfix deployment:
  State before rollout: 3 running pods (old version)
  
  Step 1: Create new pod (new version) [pod 4 total, but pod 3 new stuck in Pending]
    → Scheduler cannot place pod 3 (Pending indefinitely)
    → maxUnavailable=0 means can't terminate old pods until new pod is Ready
    → Old pod 3 stays running (good for users), but rollout is frozen

  After progressDeadlineSeconds (default: 600s = 10 min):
    → Deployment controller sets condition: ProgressDeadlineExceeded
    → kubectl rollout status shows: "Waiting for deployment rollout to finish..."
    → kubectl describe deployment shows: "ProgressDeadlineExceeded"

Fix:
  → Resolve the scheduling issue (constraints)
  → kubectl rollout restart deployment <name>   # Forces fresh rollout
  → OR: kubectl rollout undo deployment <name>  # Revert to previous version

GAP 4 — RCA Document Structure (Deferred to Next Session)

The session was going to cover RCA writing. Here is the structure for this specific incident:

# RCA — Checkout API Deployment Failure (P0)
Date: 2026-02-14 | Severity: 1 | Duration: ~8 hours | Presenter: [name]

## 1. Incident Summary
During a routine hotfix deployment of checkout-api (3 replicas) in the payments 
namespace, 1 of 3 pods remained in Pending state for 8 hours, partially degrading 
the payment service and blocking the CI/CD pipeline.

## 2. Timeline
| Time    | Event                                                    |
|---------|----------------------------------------------------------|
| T-0h    | Hotfix deployment initiated via GitHub Actions            |
| T+0m    | Pods 1 and 2 start Running; Pod 3 enters Pending state   |
| T+10m   | progressDeadlineExceeded warning from Deployment controller|
| T+8h    | Incident reported; war-room opened                       |
| T+8h30m | Root causes identified via kubectl describe + YAML review |
| T+9h    | Temporary fix applied; all 3 pods Running                 |
| T+9h    | Incident closed (temporary fix); permanent fix scheduled  |

## 3. Root Cause
Three misconfigurations in the checkout-api Deployment spec acted in combination:
1. podAntiAffinity set to requiredDuringSchedulingIgnoredDuringExecution (1 pod 
   per host, hard), with only 2 schedulable nodes available for 3 replicas.
2. Missing tolerations for the batch node group taint (workload=batch:NoSchedule), 
   reducing available nodes to 2.
3. Conflicting nodeSelector pointing to a label not present on available eligible nodes.

## 4. Contributing Factors
- One primary node was cordoned, reducing schedulable nodes from 3 to 2.
- ENI slot pressure in ap-south-1 was a pre-existing risk (addressed separately).
- No automated validation that scheduling constraints are satisfiable before deploying.

## 5. Impact
- 1/3 replicas unavailable for 8 hours → increased latency risk for payment processing.
- CI/CD pipeline blocked for 8 hours → no further deployments possible.
- No confirmed customer-facing 503 errors during incident window (load stayed below threshold).

## 6. Fix Applied
Immediate: Changed podAntiAffinity to preferred; added tolerations; removed conflicting nodeSelector; ran rolling restart.
Permanent (scheduled): Add 3rd schedulable primary node; revert to required anti-affinity.

## 7. Prevention / Action Items
| Action                                                    | Owner     | Due    |
|-----------------------------------------------------------|-----------|--------|
| Add CI pre-flight check: schedulable nodes ≥ replicas     | DevOps    | 1 week |
| Document scheduling constraint ↔ node count relationship  | DevOps    | 3 days |
| Add alert: any pod Pending > 5 minutes in payments ns     | SRE       | 1 week |
| Review all other deployments for same pattern             | DevOps    | 3 days |
| Add 3rd node to primary-node-group                        | DevOps    | ASAP   |

## 8. Lessons Learned
- Pending pod with no error events = scheduling constraint problem, not application.
- required anti-affinity + # replicas > # schedulable nodes = guaranteed Pending.
- nodeSelector and podAntiAffinity are AND conditions; conflicts cause total block.
- progressDeadlineExceeded is a deployment health signal — add this to monitoring.