Complete Learning Package: Outage Classification, Common Failure Patterns, and Troubleshooting Frameworks

2. Table of Contents

1. Executive Summary
2. Table of Contents
3. Detailed Structured Notes
   • 3.1 The Five Categories of Production Outages
   • 3.2 Common Kubernetes Outage Patterns
   • 3.3 Liveness vs. Readiness Probes — Detailed Q&A
   • 3.4 Sidecar Pattern vs. Multi-Microservice Pods
   • 3.5 Common AWS / Cloud Provider Outage Patterns
   • 3.6 Common CI/CD Outage Patterns
   • 3.7 Common Networking Outage Patterns
   • 3.8 The Knowledge Base Concept
   • 3.9 The Core Teaching Exercise — Debugging an “Invisible” Outage
   • 3.10 Why Structured Debugging Beats Unstructured Debugging
   • 3.11 The Four-Pillar Structured Debugging Framework
   • 3.12 The OSI Troubleshooting Framework (Applied to Outages)
   • 3.13 The VERDICT Model (Kubernetes-Specific Framework)
   • 3.14 RCA Template Structure
4. Key Concepts Table
5. Architecture & Workflow Analysis
6. Commands & Configurations
7. Tools & Technologies
8. Real-World Production Usage
9. Interview Preparation
10. Exam & Certification Notes
11. Cheat Sheet
12. Gaps & Assumptions

3. Detailed Structured Notes

3.1 The Five Categories of Production Outages

Core claim: Every production outage — 100% of them — falls into one of five categories. This is a mental model for triage: when an incident starts, the first question is not “what tool broke” but “which category does this belong to,” because that determines where to look.

Category 1: Capacity Issues

Resource exhaustion at any layer.

• CPU exhaustion
• RAM exhaustion
• Disk exhaustion
• Connection pool / queue exhaustion

Category 2: Configuration Issues

Misconfigured settings, not code bugs.

• Missing or incorrect secrets
• Misconfigured routing
• IAM role misconfiguration
• Configuration drift (actual state ≠ intended state)

Category 3: Dependency Failures

An upstream or downstream system your service depends on fails.

• Database down
• Caching layer issue
• DNS issue
• Load balancer issue
• API gateway issue
• Third-party service failure

Key framing: Understanding dependency failures requires understanding the upstream vs. downstream flow of your system — i.e., what calls your service, and what your service calls.

Category 4: Deployment Issues

Something broke as a direct result of a deployment/release.

• Bad rollout configuration
• Incompatible version between services
• Schema mismatch (e.g., DB schema vs. application code expectation)
• Post-deployment behavioural change (application behaves differently than before)

Category 5: Process Failures

Human/process-driven issues — not a technical bug, but an operational gap.

• Manual changes made outside of IaC (causing state drift)
• Untested changes pushed to production
• Missing alerts for critical systems (you didn’t know because nothing told you)
• Any manual mistake in the deployment/change process

Why this taxonomy matters: When an incident starts, forcing yourself to ask “which of these five buckets does this belong to?” narrows the investigation space immediately, rather than starting with an unbounded “what could be wrong” search.

3.2 Common Kubernetes Outage Patterns

The engineer walked through the most frequent Kubernetes-specific outage symptoms, each requiring different diagnostic starting points:

Presenter’s note: The programme maintains a documentation knowledge base (referenced as containing ~90–100 documented production outage RCAs across categories) with step-by-step troubleshooting guides for each of these patterns, including “ideal” probe configuration guidance.

3.3 Liveness vs. Readiness Probes — Detailed Q&A

This was a live audience question (framed as a real interview question) and the answer is one of the most precise and useful explanations in the session.

The question posed: If a pod runs multiple containers (e.g., 2–3 microservices in one pod — noted later in the session as an anti-pattern, but used here as a hypothetical), and one container’s liveness probe fails while the others are healthy — what happens to the pod?

The answer, generalised to any pod with a failing probe:

Liveness probe

• Purpose: Answers the question “is this container alive?” — detects deadlocks, hung processes, or internal states where the container is running but not functioning.
• On failure: Kubernetes kills the container and restarts it.
• Effect: The pod (or the specific container within it) enters a restart cycle.

Readiness probe

• Purpose: Answers the question “is this container ready to serve traffic?” — used to signal the application is still booting, warming a cache, running a migration, etc.
• On failure: Kubernetes removes the pod from the Service endpoint list. The pod keeps running — it is NOT restarted. Traffic simply stops being routed to it.
• Effect: No restart. Pod stays up but stops receiving traffic until the readiness probe passes again.

Summary table:

Why this distinction matters in interviews and in real debugging: If you see a pod that keeps restarting, that’s a liveness probe (or crash) issue. If you see a pod that’s Running but not receiving traffic, that’s a readiness probe issue — do NOT go looking for a crash; look at what the readiness probe is checking and why it’s failing.

3.4 Sidecar Pattern vs. Multi-Microservice Pods

Audience question: Should you ever run multiple microservices inside a single pod?

Answer: No — this depends on business context, but as a general rule in Kubernetes, you never run multiple independent microservices inside a single pod.

Why:

• A pod is meant to host one primary application.
• Pods have a single lifecycle — they scale together, get scheduled together, and are replaced together.
• Running independent microservices in the same pod means they cannot scale independently, cannot fail independently, and cannot be deployed independently — defeating the purpose of microservices architecture.

What multi-container pods ARE for — the sidecar pattern: A pod can legitimately have multiple containers if the additional containers are sidecars supporting the primary application, not independent services. Examples given:

• A metrics-collection sidecar (e.g., collecting metrics from the primary container)
• A Fluent Bit log collector sidecar (shipping logs from the primary container)
• An authentication sidecar
• A debugging sidecar (e.g., a “busybox” container for troubleshooting the primary container’s network/filesystem)

Rule of thumb: If the additional container exists to serve the primary container (metrics, logs, auth, debug tooling), it’s a legitimate sidecar. If it’s an independently deployable, independently scalable business capability, it belongs in its own pod.

3.5 Common AWS / Cloud Provider Outage Patterns

Documented categories referenced (organised by AWS service, with GCP/Azure equivalents mentioned as also maintained in the knowledge base):

Presenter’s note: The knowledge base is organised into folders by domain: Kubernetes, CI/CD, cloud providers (AWS/GCP/Azure), DB & caching, and networking — with ~90–100 documented outage RCAs total at the time of this walkthrough, actively being expanded via community collaboration (audience members can contribute incidents they’ve experienced).

3.6 Common CI/CD Outage Patterns

Real example referenced: A GitOps sync issue causing an “untouchable release” — the engineer shared a problem statement link in the session chat for this specific scenario (not detailed further in the transcript).

3.7 Common Networking Outage Patterns

Real Incident Shared by an Audience Member (HAProxy + DNS dependency failure)

An attendee shared a real production incident:

• Symptom: Multiple production sites went down one by one, with no obvious cause. Kubernetes side looked completely healthy.
• Investigation: Checked REST controllers and application layer — nothing found. The outage then self-resolved after a few minutes, leaving the team confused about root cause.
• Root cause (found after investigation): The organisation was running a legacy HAProxy load balancer (in the process of being migrated to AWS ALB). HAProxy had Google’s public DNS (8.8.8.8) configured as its resolver. Google DNS experienced a brief outage, breaking name resolution on HAProxy — which meant HAProxy could not resolve backend service hostnames and could not forward any requests to the backend services.
• Fix: Configured failover DNS on HAProxy — so if the primary DNS resolver goes down, a secondary resolver takes over automatically.
• Lesson: This is a textbook example of a Dependency Failure (Category 3) — the internal infrastructure was completely healthy; the failure was in an external dependency (Google DNS) that a legacy component (HAProxy) relied on without redundancy.

3.8 The Knowledge Base Concept

What it is: A structured, searchable documentation repository of production outage RCAs, organised into folders:

• Kubernetes
• CI/CD
• Cloud providers (AWS, GCP, Azure)
• Databases & caching
• Networking

Scale at time of session: ~90–100 documented production outage RCAs, actively growing.

Aspiration stated: Expand to 400–500 documented outages over time.

Collaboration model: Community-contributed — audience members / programme participants can share real incidents they’ve faced; the this programme team helps formalise them into structured RCAs and add them to the shared knowledge base (coordinated via a dedicated Platform Knowledge Base).

Why this matters as a practice: A searchable RCA knowledge base is a real production best practice — it means that when a similar issue recurs (even years later, or faced by a different engineer), the resolution path is already documented rather than being rediscovered from scratch.

3.9 The Core Teaching Exercise — Debugging an “Invisible” Outage

This is the centrepiece of the session — a live Socratic exercise designed to demonstrate the difference between assumption-driven and evidence-driven debugging.

The Scenario (constructed live with the audience)

• A global outage affecting a service deployed across 4 regions — all regions affected simultaneously.
• All logs are clean. No error logs anywhere.
• All metrics look normal/clean.
• Customers are complaining: latency issues and intermittent 500 errors.
• Infrastructure split ~50/50 between AWS and GCP — ruling out a single cloud provider’s outage as the root cause (since both providers show the same symptom simultaneously).
• Not a DNS issue (verified).
• Not an application issue (verified — presenter states this directly as a given for the exercise).
• Minimal third-party dependency — only an authentication dependency, and that authentication service is confirmed working fine.
• Not a password/credential expiration issue.
• The engineer explicitly told the audience: do not throw out more guesses — figure out how to approach this systematically instead.

What Went Wrong in the Audience’s Approach (the actual teaching point)

Multiple audience members immediately began guessing specific root causes:

• “Could it be the load balancer not scaling?"
• "Could it be a caching TTL issue?"
• "Could it be a CDN/Cloudflare issue (referencing a real Atlassian incident where switching load balancers during a deployment caused downtime)?”

The engineer’s explicit correction:

“Let’s try to actually create down a pattern on how exactly we need to debug uh those kind of scenarios… let’s not try to pinpoint any specific issue… let’s try to understand that how, like, how should we start debugging this kind of problem statement in a structured way rather than throwing assumptions.”

The reframe: Rather than guessing at specific components, the engineer pushed the group toward a general-purpose diagnostic sequence that would work regardless of what the actual root cause turns out to be:

1. Curl the API with verbose output — see exactly where the request gets blocked/stalls in the request path.
2. Trace request reachability layer by layer — is it reaching the target at all? If not, check each attached resource (pod, DB, upstream applications) one at a time.
3. Start from the customer side, not the infrastructure side — understand exactly what the customer experiences (latency? blockage? intermittent or constant?) before diving into infrastructure internals.
4. From the client machine outward — DNS resolution → ping response → traceroute/echo response → understand which hop in the network path is degraded.
5. Check underlying cloud provider status — is there a known incident/maintenance window on the cloud provider’s status page (even if it’s not a full outage, partial degradations happen).
6. Check recent deployments/code changes — even in a scenario with “no application issue” as a given, verify whether anything changed recently in the infrastructure or configuration layer (not necessarily application code).

The Real-World Reference Point

The engineer referenced a real 3-hour this programme session where a group of engineers with 15–20 years of combined experience faced a similar unstructured-vs-structured challenge:

• ~2.5 hours were spent on unstructured, assumption-driven guessing (“it could be this, it could be that”).
• The final ~30 minutes, once a structured framework was actually applied, resolved the incident.
• Explicit lesson: Structured debugging isn’t just “faster” — it’s an order of magnitude faster (30 minutes vs. 2.5 hours) because it eliminates wasted investigation into hypotheses that were never grounded in evidence.

3.10 Why Structured Debugging Beats Unstructured Debugging

The core principle stated:

“As a DevOps engineer, you should not be the one to assume things. It’s all about verifying the things in the infrastructure itself.”

The economic argument:

• In a real production incident, you typically have 5–10 minutes to act, not hours.
• Every second spent on an ungrounded assumption is a second the company is losing money/reputation.
• Assumption-driven debugging is fundamentally a search without a map — you might get lucky and land on the right guess quickly, or you might not.
• Structured debugging is a map — even if you don’t yet know the answer, you know exactly which piece of evidence to gather next, and each piece of evidence either confirms or eliminates entire categories of possible causes.

When gut feeling IS acceptable: The engineer clarified: if you have prior direct experience with the exact symptom pattern (“I’ve seen this issue before and it was caused by X”), it’s reasonable to check that specific hypothesis first. The prohibition is against guessing without grounding — pattern-matching from real prior experience is different from speculation.

3.11 The Four-Pillar Structured Debugging Framework

This is the general-purpose (not Kubernetes-specific) framework taught in this walkthrough, built from the audience discussion and formalised by the engineer.

Pillar 1: System Behaviour — “Then vs. Now”

The question to ask: What exactly has changed in my system such that it’s now behaving differently?

Why this matters: You cannot recognise “abnormal” unless you have a clear baseline of “normal.” Before troubleshooting, you must be able to articulate:

• What does my system’s normal latency look like?
• What does my system’s normal error rate look like?
• What does my system’s normal resource utilisation look like?

Worked example given: If your system’s normal latency baseline is 20 milliseconds, and it has suddenly jumped to 400 milliseconds, you now have a concrete, quantified signal: “latency increased 20×.” This gives you a specific hypothesis space (things that could cause a 20× latency increase) rather than a vague “something is slow.”

Pillar 2: Timeline — “Last N Hours”

The question to ask: What changed in my infrastructure in the last 24, 48, or 72 hours (minimum 24 hours, ideally 48 hours per organisational policy)?

What to check across this window:

• Deployments/releases
• Infrastructure changes (e.g., EC2 patching)
• Configuration changes
• Scaling events
• Any manual changes made by any team member

Worked example given: “There was a deployment, and after the deployment, the outage started” OR “there was EC2 patching, and after that, the outage began.” The point of a timeline reconstruction is to find temporal correlation between a change and the onset of the incident — even if the causal mechanism isn’t obvious yet.

Pillar 3: Blast Radius

The question to ask: How deep does this go? Who exactly is affected?

Specific sub-questions:

• Is it one API, or all APIs?
• Is it a single service, or the entire platform?
• Is it internal-only (productivity impact) or customer-facing (revenue/reputation impact)?
• Is it one region, or global?

Why this matters: Blast radius determines severity classification and urgency, and it also narrows the investigation — a blast radius confined to one region points toward regional infrastructure; a global blast radius across all regions and both cloud providers (as in the teaching exercise) rules out single-provider or single-region causes entirely.

Pillar 4: Upstream and Downstream Flow

The question to ask: How does data flow through my system, from the client’s request to my system’s response, and back?

Why this matters: This identifies where the gap is — the point in the data flow where things stop behaving as expected.

Worked example given (for a latency issue):

• Upstream check: networking layer (load balancer, gateway, DNS, network routing)
• Downstream check: database queries, caching layer, any downstream service calls

By mapping the full request path and checking each hop, you localise exactly where the anomaly is introduced rather than treating the system as an unknowable black box.

Summary — the Four Pillars Together

1. System Behaviour (then vs. now)
      → Establishes WHAT changed (quantified baseline vs. current state)
2. Timeline (last 24–72h)
      → Establishes WHEN it started and correlates to recent changes
3. Blast Radius
      → Establishes WHO/WHAT is affected and how severely
4. Upstream/Downstream Flow
      → Establishes WHERE in the request path the anomaly is introduced

Together: WHAT + WHEN + WHO + WHERE
→ This converges on a testable hypothesis far faster than open-ended guessing.

3.12 The OSI Troubleshooting Framework (Applied to Outages)

Recommendation given: For infrastructure with many interconnected components, use the OSI (Open Systems Interconnection) model as a structured troubleshooting framework — divide the system into its seven canonical layers and work through them systematically, typically starting from the lower/base layers and moving upward (or vice versa, depending on the symptom).

The seven layers (as referenced):

1. Application layer
2. Presentation layer
3. Session layer
4. Transport layer
5. Network layer
6. Data link layer
7. Physical layer

Presenter’s guidance: Start troubleshooting from the layer most relevant to the symptom (e.g., for a latency issue, start closer to the network/transport layers) and move methodically through adjacent layers rather than jumping around.

Note: This walkthrough only introduced the OSI framework at a conceptual level and pointed to an external, more detailed video resource (linked in the session chat) for the full layer-by-layer breakdown with specific commands per layer. The full command-level breakdown was not covered in this transcript — see Gaps & Assumptions.

3.13 The VERDICT Model (Kubernetes-Specific Framework)

For Kubernetes-specific outages, the engineer introduced a dedicated framework called VERDICT — each letter represents a diagnostic step:

How it’s meant to be used: When facing a Kubernetes-specific outage with limited time (the engineer frames this as a “10–15 minutes only, no internet support” scenario), work through V → E → R → D → I → C → T in order. Each step narrows the investigation and builds on evidence from the previous step, rather than jumping straight to a hypothesis.

Documentation referenced: The engineer has a detailed VERDICT documentation resource with specific commands for each letter, applied to a real example — a Kubernetes “split-brain” scenario related to etcd (~40 pages of detailed RCA documentation, shared as a link in the session). This specific example was referenced but not walked through live in this transcript.

3.14 RCA Template Structure

Purpose: After resolving any production outage, document it as a formal Root Cause Analysis (RCA) and save it to the knowledge base, so future engineers (or your future self) can resolve the same class of issue faster.

Template sections referenced (our standard RCA template):

Example RCA referenced: A detailed (~40-page) RCA for a Kubernetes “split-brain” scenario tied to an etcd issue — described as a comprehensive worked example of applying the VERDICT model end-to-end. This was shared as a link but not detailed in the transcript content itself.

Best practice emphasised: Every resolved production incident should result in a saved RCA — this compounds the organisation’s collective troubleshooting knowledge over time and is what a searchable knowledge base is built from.

4. Key Concepts Table

5. Architecture & Workflow Analysis

5.1 The Outage Classification Decision Tree

Production Incident Detected
         │
         ▼
   Which category does this fall into?
         │
    ┌────┼────┬────────┬─────────┬─────────┐
    ▼    ▼    ▼         ▼         ▼
Capacity Config Dependency Deployment Process
 Issue   Issue   Failure    Issue    Failure
    │    │        │          │        │
    ▼    ▼        ▼          ▼        ▼
Check    Check    Map        Check    Check
CPU/RAM/ secrets, upstream/   recent   manual
disk/    routing, downstream  release  changes,
conn.    IAM,     dependency  /rollout missing
pool     drift    graph       history  alerts

5.2 Structured Debugging Flow (Four-Pillar Framework)

Incident Reported
      │
      ▼
┌─────────────────────────────┐
│ 1. SYSTEM BEHAVIOUR         │  What is my normal baseline?
│    (then vs. now)           │  What is different right now?
└──────────────┬──────────────┘  e.g., latency 20ms → 400ms
               ▼
┌─────────────────────────────┐
│ 2. TIMELINE                 │  What changed in the last
│    (last 24–72 hours)       │  24–72 hours? Deployments?
└──────────────┬──────────────┘  Patching? Config changes?
               ▼
┌─────────────────────────────┐
│ 3. BLAST RADIUS             │  How deep does this go?
│                              │  One API? All APIs? One
└──────────────┬──────────────┘  region? Global? Internal
               ▼                 only or customer-facing?
┌─────────────────────────────┐
│ 4. UPSTREAM/DOWNSTREAM FLOW │  Trace the request path.
│                              │  Where in the chain does
└──────────────┬──────────────┘  the anomaly appear?
               ▼
     Evidence-backed hypothesis
               │
               ▼
     Test hypothesis → Confirm/Reject
               │
               ▼
          Fix + RCA

5.3 The VERDICT Model Flow (Kubernetes-Specific)

K8s Outage Detected (limited time, no internet help available)
         │
         ▼
   V — Version/Variance:
       What versions are running? What has changed (variance)?
         │
         ▼
   E — Establish blast radius:
       Namespace-scoped? Cluster-wide? Which workloads?
         │
         ▼
   R — Reconstruct timeline:
       Recent deployments, Helm releases, node/cluster changes
         │
         ▼
   D — Deep-dive into signals:
       Metrics, logs, traces — detailed technical investigation
         │
         ▼
   I — Identify dependencies:
       What does this workload depend on? (DB, cache, DNS, APIs)
         │
         ▼
   C — Check configuration drift:
       Actual vs. intended config (Helm values, ConfigMaps, RBAC)
         │
         ▼
   T — hypoThesis:
       Formulate and test specific, evidence-backed root cause
         │
         ▼
   Fix + detailed RCA (example: 40-page etcd split-brain RCA)

5.4 The HAProxy/DNS Dependency Failure (Real Incident, Reconstructed)

Client Request
      │
      ▼
   HAProxy (legacy LB, mid-migration to AWS ALB)
      │
      │  needs to resolve backend hostname
      ▼
   DNS Resolver configured: Google DNS (8.8.8.8)
      │
      │  ← Google DNS experiences a brief outage
      ▼
   Name resolution FAILS
      │
      ▼
   HAProxy CANNOT resolve backend service hostnames
      │
      ▼
   HAProxy cannot forward requests to backend services
      │
      ▼
   Multiple production sites go down "one by one" (as different
   cached DNS TTLs expire at different times across services)
      │
      ▼
   K8s layer looks completely healthy (because the failure is
   entirely upstream, at the HAProxy/DNS layer — outside K8s)

FIX: Configure failover DNS resolver on HAProxy
     (primary fails → automatic failover to secondary resolver)

Why this incident is a good teaching example: It demonstrates a Dependency Failure where every internal system (Kubernetes, application code, databases) was completely healthy — the failure was entirely in an external dependency two layers removed from the actual application. This is exactly the kind of outage that “checking the application logs” would never surface, reinforcing the lesson about starting from a structured, full-system view rather than jumping straight to the application layer.

6. Commands & Configurations

The transcript is primarily conceptual/whiteboard-based; explicit commands given in the session are limited. The following are the commands and diagnostic actions explicitly referenced:

Note: This walkthrough did not include a live terminal walkthrough — it was whiteboard/discussion-based. Full command sequences for each outage pattern are stated to exist in the linked external documentation (knowledge base and OSI-framework video), which is outside the scope of what was captured in this transcript. See Gaps & Assumptions.

7. Tools & Technologies

8. Real-World Production Usage

Enterprise pattern — outage taxonomy as an incident triage tool: Many mature SRE organisations use a similar 4–6 category taxonomy as the first triage question when an incident is declared, precisely because it forces responders to commit to a category early, which shapes which dashboards/logs/runbooks get pulled up first.

Production implementation pattern — knowledge base as institutional memory: The RCA knowledge base concept described here mirrors real practices at mature engineering organisations (e.g., Google’s postmortem culture, Amazon’s COE — Correction of Error process). The key design elements that matter for this to work in practice:

• Searchable/categorised (not just a flat list of documents)
• Actively maintained and expanded (stale knowledge bases lose trust and stop being used)
• Every incident, not just severe ones, gets documented (small incidents often reveal patterns before they become large incidents)

DevOps/SRE best practice — establishing baselines before incidents happen: The “then vs. now” pillar depends entirely on having good baseline observability before an incident occurs. This is a proactive practice, not a reactive one — teams that don’t invest in dashboards/alerting for “normal” behaviour will struggle to even articulate what changed during an incident.

Security consideration — the DNS failover lesson: The HAProxy/Google DNS incident is a specific instance of a broader production hardening principle: any single external dependency without a configured fallback is a single point of failure, even if that dependency (a public DNS resolver) seems “too reliable to fail.” This applies broadly — third-party APIs, external identity providers, public package registries, etc. should all have documented fallback/degradation behaviour.

Cost optimization consideration: Not directly addressed in this walkthrough (this masterclass is about troubleshooting methodology, not cost), but the RCA/knowledge-base practice indirectly reduces cost by reducing MTTR (mean time to resolution), which reduces the duration (and therefore cost) of every future incident of a similar type.

Scalability consideration: The blast-radius pillar of the framework is directly relevant to scalability planning — understanding “is this a regional issue or global” requires your infrastructure to actually be instrumented with region-level and service-level granularity in your monitoring. Without that instrumentation, you cannot answer the blast-radius question quickly during a real incident.

9. Interview Preparation

Beginner Questions

Q1: What is the difference between a liveness probe and a readiness probe in Kubernetes? A: A liveness probe checks whether a container is alive/functioning correctly. If it fails, Kubernetes kills the container and restarts it. A readiness probe checks whether a container is ready to receive traffic. If it fails, Kubernetes removes the pod from the Service’s list of endpoints — the pod keeps running, but stops receiving traffic, and it is NOT restarted. The key distinction: liveness failure = restart; readiness failure = traffic removal without restart.

Q2: Name the five categories that production outages typically fall into. A: Capacity issues (CPU/RAM/disk/connection exhaustion), Configuration issues (misconfigured secrets/routing/IAM/drift), Dependency failures (upstream/downstream systems failing — DB, cache, DNS, third-party services), Deployment issues (bad rollouts, schema mismatches, incompatible versions), and Process failures (manual changes, untested changes, missing alerts).

Q3: Should you run multiple independent microservices inside a single Kubernetes pod? A: No. A pod should host one primary application because pods share a single lifecycle — they scale, get scheduled, and get replaced together. Running independent microservices together defeats the purpose of microservices architecture (independent scaling, independent deployment, independent failure isolation). Multiple containers in a pod are appropriate only for the sidecar pattern — containers that support the primary application (e.g., log shippers, metrics collectors, auth proxies), not independent business logic.

Intermediate Questions

Q4: A pod is showing as “Not Ready” but is not restarting. What’s the most likely cause and how would you investigate? A: This points to a readiness probe failure, not a crash. Since the pod isn’t restarting, the liveness probe (if configured) is passing — the container process itself is alive. Investigation: check the readiness probe’s configured endpoint/command and see why it’s failing — common causes include the application still warming up (cache population, DB migration in progress), a dependency the readiness check calls being unavailable, or an incorrectly configured probe path/port/threshold. Check kubectl describe pod for probe failure events and kubectl logs for what the application itself reports about its readiness state.

Q5: Describe a structured approach to debugging a production outage where logs and metrics show no obvious errors, but customers are reporting latency and intermittent errors. A: Apply the four-pillar framework: (1) Establish system behaviour baseline — quantify what “normal” latency/error rate looks like versus current observed values. (2) Reconstruct the timeline — check the last 24–72 hours for deployments, infrastructure changes, patching, or configuration changes that correlate with the onset. (3) Establish blast radius — is this affecting one API or all APIs, one region or all regions, internal only or customer-facing? This rules in/out categories of cause (e.g., if it spans multiple cloud providers simultaneously, it’s not a single provider’s outage). (4) Trace upstream/downstream flow — map the full request path from client to backend and back, checking each hop (load balancer, gateway, DNS, application, cache, database) to localise exactly where the anomaly is introduced. Rather than guessing at specific components, this sequence converges on evidence-backed hypotheses.

Q6: An organisation’s production outage was traced to a legacy load balancer’s DNS resolver going down. What category of outage is this, and what’s the systemic fix (not just the immediate fix)? A: This is a Dependency Failure — the internal application and infrastructure were healthy; an external dependency (the DNS resolver used by the load balancer) failed. The immediate fix is restoring or bypassing the failed resolver. The systemic fix is configuring DNS failover — a secondary resolver that automatically takes over if the primary fails — so that a single external dependency (however reliable it seems) doesn’t become a single point of failure for the entire system.

Advanced Questions

Q7: Explain why “assumption-driven” debugging is inefficient in production incidents, and what specifically makes “structured” debugging faster — not just theoretically, but in terms of what’s actually different about the process. A: Assumption-driven debugging treats the investigation as an unordered search — each guess (“maybe it’s the load balancer,” “maybe it’s caching”) is tested somewhat independently, often without first gathering evidence that would rule entire categories in or out. This means: (a) guesses can be redundant or contradictory across team members, (b) there’s no accumulation of evidence — a wrong guess doesn’t necessarily narrow the search space for the next guess, and (c) it depends heavily on the guesser’s prior pattern-matching experience, which doesn’t generalise to novel failure modes.

Structured debugging (e.g., the four-pillar framework) is faster because each pillar produces evidence that mechanically eliminates or confirms broad categories of cause before any specific hypothesis is tested. For example, establishing “blast radius spans 2 cloud providers and 4 regions simultaneously” immediately eliminates any single-provider or single-region infrastructure cause — this is a category-level elimination achieved in one step, versus potentially dozens of individual component-level guesses. The real-world example cited (2.5 hours unstructured vs. 30 minutes structured, same incident) demonstrates this isn’t a marginal improvement — it’s roughly a 5x reduction in resolution time, driven by the fact that structured evidence-gathering eliminates entire hypothesis categories per step rather than testing one hypothesis at a time.

Q8: Design a lightweight incident-response process for a mid-size engineering organisation that doesn’t yet have a formal RCA/knowledge-base practice. What are the minimum viable components, based on the principles discussed in this walkthrough? A: Minimum viable components: (1) A shared, categorised location for RCAs (even a simple folder structure by domain — K8s, CI/CD, cloud provider, DB/caching, networking — is sufficient to start; sophistication can come later). (2) A lightweight RCA template mandated for every incident above a minimum severity threshold, covering: when reported, description, impact/blast radius, root cause, immediate fix, and prevention steps — this doesn’t need to be exhaustive, but it needs to be consistent so RCAs are comparable and searchable. (3) A standing baseline-observability practice — dashboards/alerts that define “normal” for key services, so that during an incident, responders can immediately answer the “then vs. now” question rather than debating what normal even looks like. (4) A lightweight structured-debugging checklist (even just the four pillars: system behaviour, timeline, blast radius, upstream/downstream flow) posted somewhere visible/accessible during incidents, to counter the natural tendency toward assumption-driven guessing under pressure. (5) A recurring (even monthly) practice of reviewing recent RCAs as a team, to build shared pattern-recognition and catch recurring root causes across seemingly unrelated incidents.

10. Exam & Certification Notes

Frequently tested concepts from this walkthrough (relevant to CKA/CKAD and general SRE interview prep):

• Liveness vs. Readiness probe behaviour — this is one of the most commonly tested Kubernetes concepts in both certification exams and interviews. Memorise precisely: liveness failure → restart; readiness failure → removed from Service endpoints, no restart.
• Sidecar pattern legitimacy — know that multi-container pods are valid only for the sidecar pattern (supporting containers), not for hosting independent microservices.
• CrashLoopBackOff vs. Pending vs. ImagePullBackOff — know the distinct root-cause categories each symptom points toward (application/config crash vs. scheduling failure vs. registry/credential issue respectively).

Potential trick questions:

• “If a container’s liveness probe fails, does the whole pod restart, or just that container?” — Be precise: the specific container is restarted (in a multi-container pod, other healthy containers are not necessarily restarted, though pod-level behaviour can vary by controller/version — the session’s answer focused on the container being killed and restarted).
• ”Does a readiness probe failure cause a pod restart?” — No. This is the most common point of confusion; readiness failures only affect Service endpoint membership, not the pod’s running state.
• ”Is a DNS outage always a Configuration issue?” — No — as shown in the HAProxy example, a DNS failure that originates from an external provider (not your own DNS config) is classified as a Dependency Failure, not a Configuration issue, because nothing about your own configuration was wrong — the external service you depend on failed.

Memorisation-worthy points:

• The five outage categories: Capacity, Configuration, Dependency, Deployment, Process — useful as an immediate triage question when facing any incident scenario in an interview.
• The four debugging pillars: System Behaviour, Timeline, Blast Radius, Upstream/Downstream Flow.
• The VERDICT acronym: Version/Variance, Establish blast radius, Reconstruct timeline, Deep-dive signals, Identify dependencies, Check config drift, hypoThesis.

11. Cheat Sheet

Five Outage Categories:

CAPACITY       → CPU / RAM / disk / connection pool exhaustion
CONFIGURATION  → secrets / routing / IAM / drift
DEPENDENCY     → DB / cache / DNS / LB / third-party failures
DEPLOYMENT     → bad rollout / schema mismatch / version incompatibility
PROCESS        → manual changes / untested changes / missing alerts

Liveness vs. Readiness (memorise this):

LIVENESS fails  → container KILLED + RESTARTED
READINESS fails → pod REMOVED from Service endpoints, NOT restarted

Common K8s Symptom → Likely Cause:

CrashLoopBackOff    → env vars / secrets missing
Pending              → no schedulable node (resources/taints/labels)
Not Ready            → readiness probe failing
ImagePullBackOff     → registry auth/credentials
HPA not scaling      → no metrics data
OOM                  → memory leak or under-provisioned limit
PVC issues           → storage class / provisioning

Four-Pillar Structured Debugging Framework:

1. SYSTEM BEHAVIOUR   → then vs. now (quantify the baseline)
2. TIMELINE            → last 24–72h — what changed?
3. BLAST RADIUS         → how deep, how wide, who's affected?
4. UPSTREAM/DOWNSTREAM  → trace the request path, find the gap

VERDICT Model (K8s-specific):

V — Version/Variance
E — Establish blast radius
R — Reconstruct timeline
D — Deep-dive into signals
I — Identify dependencies
C — Check configuration drift
T — hypoThesis

RCA Template Sections:

1. Issue reported (when)
2. Description
3. Rate of impact (blast radius, quantified)
4. Root cause
5. Recommendation to fix
6. Prevention strategies

Golden Rule: Never guess. Verify. Structured evidence-gathering beats assumption-driven debugging by an order of magnitude (referenced real example: 2.5 hours → 30 minutes for the same incident).

12. Gaps & Assumptions

Sections referenced but not detailed in this transcript (pointed to external resources):

• The OSI framework applied command-by-command to a specific outage — the engineer referenced an external video with full layer-by-layer commands, but this transcript only covers the conceptual introduction.
• The VERDICT model’s specific commands for each letter (V, E, R, D, I, C, T) — referenced as existing in detailed documentation, but not walked through live in this walkthrough.
• The ~40-page etcd split-brain RCA — referenced as a worked example of the VERDICT model in practice, but its actual content (the specific etcd failure mechanism, specific commands, specific fix) is not part of this transcript.
• The GitOps “untouchable release” incident — mentioned as a real recent example the team faced, with a link shared in chat, but not explained in the transcript itself.
• The Atlassian load balancer incident referenced by an audience member (a real published Atlassian postmortem about a load balancer migration causing downtime) — mentioned briefly, not detailed; presented as external validation of the “structured vs. assumption-driven” thesis, not as new technical content taught in this walkthrough.

Assumption made: Given the instruction to exclude general/non-DevOps discussion, all bootcamp-logistics content (L1/L2 categorisation, weekday class structure, mock interview scheduling, Platform Knowledge Base coordination, enrolment questions) has been excluded from this package, even though it appeared substantially in the second half of the transcript. This is a deliberate scoping decision per your instructions, not an oversight — if bootcamp-logistics content is needed, it would need to be requested separately.

Possible transcription/speech-to-text artifacts corrected via context:

• “Verdict model” — spelled out and clarified as an acronym (V-E-R-D-I-C-T) based on context, since the transcript renders the explanation somewhat garbled in places (e.g., “E will establish any kind of a blast radius… D will uh deep dive into the signals itself…”).
• ”GitHubs” in the CI/CD section is inferred to mean GitOps (based on context: “your GitHubs are not in sync with the production” clearly refers to GitOps reconciliation state, not GitHub itself as a platform).
• ”Cubernetes” / “Cube” spelling variants throughout normalized to “Kubernetes”/“K8s” for clarity.
• ”Vidict” / “Vertic” normalized to “VERDICT” based on the acronym breakdown given.

No pricing, enrolment, or certificate information included — this was present in the original transcript but is explicitly excluded per your instruction to ignore non-DevOps content.