Production Outage War Room: Triage Fundamentals

Real-World Outage Case Studies (Slack, Cloudflare, Facebook, Uber) + Live Simulated Debugging + Unresolved Bastion/SSH Mystery

2. Table of Contents

1. Executive Summary
2. Table of Contents
3. Detailed Structured Notes
   • 3.1 War-Room Call Structure & Format (Foundations Track vs. advanced material)
   • 3.2 Troubleshooting Frameworks Referenced (OSI, VERDICT-7)
   • 3.3 Case Study 1 — Slack Outage (January 2021)
   • 3.4 Case Study 2 — Cloudflare Outage (2019)
   • 3.5 Case Study 3 — Facebook/WhatsApp/Instagram Outage (2021)
   • 3.6 Case Study 4 — Uber Login Outage (2017)
   • 3.7 Live Simulated Foundations Track Environment
   • 3.8 Mid-Session Pedagogical Pivot (Participant Feedback)
   • 3.9 advanced material Scenario — The Intermittent Bastion→Production SSH Timeout Mystery
   • 3.10 Diagnostic Q&A Transcript — What Was Ruled Out, What Was Hinted
   • 3.11 Program Logistics & Access Issues
4. Key Concepts Table
5. Architecture & Workflow Analysis
6. Commands & Configurations
7. Tools & Technologies
8. Real-World Production Usage
9. Interview Preparation (Beginner / Intermediate / Advanced)
10. Exam & Certification Notes
11. Cheat Sheet
12. Gaps & Assumptions

3. Detailed Structured Notes

• Each week’s war-room call replicates a piece of real client-style infrastructure in the cloud playground, seeds it with a simulated bug/failure, and the team works to diagnose and resolve it live, closing with an RCA.
• Two parallel tracks, matched to the Foundations Track / advanced material split established during onboarding:
  • Foundations Track — a smaller, single-instance-style problem, intended to build fundamentals (this week: an sre-labs-admin instance with several small, deliberately-seeded misconfigurations).
  • advanced material — a fuller, more realistic multi-component incident (this week: the bastion→production SSH scenario, described as an actual historical incident the engineer personally worked, not a constructed teaching exercise).
• Recommended sequencing: complete Foundations Track first, then move to advanced material — though participants who are confident in Linux/DevOps fundamentals can skip straight to advanced material.
• Infrastructure lifecycle: the simulated environment stays live in the cloud for 3 days after the session (e.g., a Saturday/Sunday call’s environment stays up until the following Wednesday) before being torn down and replaced by the next week’s environment. Terraform code for recreating each week’s simulated failure is promised to be published to GitHub so participants can rebuild and re-practice independently at any time.
• Team-based troubleshooting is the default expectation — teams are encouraged to coordinate their own call and work the incident together, mirroring how real production incidents are handled as a team sport, not solo. Individual per-user practice environments are also provided for those who want to work independently.
• This week specifically included extra “catch-up” outages beyond the normal two (Foundations Track + advanced material), because a prior week had significant onboarding/access delays — participants were given a full catch-up week to work through all of Week 1’s outages before moving to Week 2.

3.2 Troubleshooting Frameworks Referenced (OSI, VERDICT-7)

Two named frameworks were referenced as pre-reads (uploaded to the drive, referenced but not read aloud in full in this walkthrough):

1. OSI-layer troubleshooting framework — the primary framework used in this walkthrough. The approach: outline the problem statement, then work layer by layer (network reachability → DNS resolution → TCP/connection-level behavior → and upward through the stack), drawing a conclusion at each layer before moving to the next, until the root cause is isolated.
2. The “VERDICT-7” model — referenced as the advanced material-tier framework, intended for a more human/organizational-process-aware approach to troubleshooting (i.e., not just technical layers, but how humans coordinate during an incident). Not explained in technical detail in this transcript — participants were pointed to the drive for the full model.

3.3 Case Study 1 — Slack Outage (January 2021)

• What happened: A major Slack service outage, escalating from network degradation to full service unavailability.
• Root cause: An AWS Transit Gateway used to interconnect Slack’s VPCs became saturated, causing packet loss.
• Cascade mechanism: Packet loss → increased request latency → backend timeouts → blocked worker threads → classic resource exhaustion. As users retried failed actions (e.g., re-uploading files/messages), a retry storm formed, piling additional load onto an already-struggling load balancer — a chain reaction the engineer termed a cascading infrastructure failure.
• Duration: Roughly one hour of downtime.
• Key teaching point: A retry storm is self-generated load, distinct from an external DDoS attack, but can be equally or more damaging because it’s driven by your own legitimate users retrying in good faith.

3.4 Case Study 2 — Cloudflare Outage (2019)

• What happened: A global Cloudflare outage; websites behind Cloudflare began returning HTTP 502 (Bad Gateway) errors worldwide.
• Root cause: A newly deployed WAF (Web Application Firewall) managed rule contained a poorly designed regular expression that caused catastrophic backtracking.
• Cascade mechanism: The bad regex drove CPU utilization to 100% on Cloudflare’s edge servers handling HTTP/HTTPS traffic, saturating worker processes and causing a global outage — a single configuration change (one new rule) took down traffic handling worldwide.
• Key teaching point: Even a single new rule/config change (in this case, a WAF rule) can have blast radius far beyond its apparent scope, because it sits directly in a hot path (HTTP/HTTPS request processing) shared by all traffic.

3.5 Case Study 3 — Facebook/WhatsApp/Instagram Outage (2021)

• What happened: Facebook, WhatsApp, and Instagram became globally unreachable for roughly 3–4 hours; DNS names for these services failed to resolve entirely.
• Root cause: A BGP (Border Gateway Protocol) route withdrawal — someone on the Facebook team removed the “best path” route advertisement — disconnected Facebook’s entire network infrastructure from the rest of the internet.
• Cascade mechanism: Because BGP routing determines how traffic finds its way to a network at all, the withdrawal effectively made Facebook’s DNS servers (and everything else) unreachable from the public internet — DNS resolution failed not because the DNS servers were broken, but because nothing could route to them.
• Key teaching point: Explicitly called out as a core lesson: DNS is the root of most production outages you’ll encounter, and even a junior-to-senior DevOps engineer should reflexively suspect DNS early when facing an ambiguous networking-flavored incident.

3.6 Case Study 4 — Uber Login Outage (2017)

• What happened: Riders and users were unable to log into the Uber platform.
• Root cause: A slow third-party DNS server used by a Node.js-based login service caused DNS resolution calls to take excessively long.
• Cascade mechanism: Slow DNS resolution consumed and exhausted resources within the Node.js login service, eventually taking the entire login system down.
• Key teaching point: Third-party dependencies (in this case, an external DNS provider) can become a resource-exhaustion vector even when your own infrastructure is otherwise healthy — a dependency’s latency, not just its availability, can take you down.
• Note: A fifth outage (attributed to GCP) was mentioned by name in the session’s outage roster but not covered in technical detail within this transcript — see Gaps & Assumptions.

3.7 Live Simulated Foundations Track Environment

The engineer demonstrated a small simulated environment (sre-labs-admin instance) seeded with multiple deliberately-introduced problems, meant to mirror signatures from the case studies above:

• DNS latency spike / intermittent resolution failure: Running dig google.com repeatedly showed inconsistent behavior — sometimes resolving normally (5–20ms, the expected baseline), sometimes taking 5–20 seconds, and sometimes failing outright with a timeout. This was driven by a background script deliberately generating pressure on the DNS resolution path — explicitly built to mirror the Slack incident’s DNS latency curve (100ms → 500ms → 1s → 5s, as documented in Slack’s own postmortem).
• CPU / thread-pool exhaustion: A separate background script gradually loads the CPU over roughly 15–20 minutes, intended to mirror the Cloudflare-style thread-pool/worker exhaustion pattern.
• Selective protocol blocking (ping works, curl doesn’t): The engineer had used iptables to drop traffic on port 80 (the port curl/HTTP uses), while leaving ICMP (used by ping) untouched. This produced the classic diagnostic signal: ping succeeds, curl intermittently fails or times out — because ping and curl use fundamentally different protocols (ICMP vs. TCP) that can be independently firewalled.
• DNS resolv.conf misconfiguration: A modification to the system’s DNS resolver configuration was introduced to simulate elevated retry counts and resolution timeouts, echoing the Facebook BGP/DNS case study.
• Demonstrated OSI-layer walkthrough on this environment:
  1. Network reachability layer — “Can the host reach the outside world at all?” Checked via ping, traceroute, and ss. Conclusion in this environment: ping (ICMP) worked, curl (TCP/port 80) did not — pointing toward a firewall/iptables-level block, not a full network outage.
  2. DNS resolution layer — Checked via repeated dig google.com calls. Conclusion: intermittent, worsening resolution times, consistent with resolver-level pressure or misconfiguration.
  3. Further layers (TCP connection state, application layer, etc.) were outlined as the next steps in the framework but not fully walked in this transcript due to time and the pivot described in Section 3.8.

3.8 Mid-Session Pedagogical Pivot (Participant Feedback)

A substantial, candid exchange occurred mid-session that’s worth preserving as its own item, because it materially changed how the rest of the session (and future sessions) were run:

• Several experienced participants (explicitly identifying themselves as practicing DevOps/SRE professionals) pushed back that walking through documentation and RCA blog links for the case studies, without a corresponding live, hands-on architecture walkthrough or terminal-based debugging demo, wasn’t giving them the depth they needed — one participant specifically asked for a whiteboard-level architecture diagram showing where in a request path (client → load balancer → VPC → subnet → etc.) each incident’s failure actually occurred.
• The consensus request that emerged: rather than covering five or six different outage types shallowly, deep-dive one pattern live (e.g., one DNS outage, fully whiteboarded and terminal-debugged) so participants could generalize the pattern themselves to other similar incidents — then be given a second, similar problem to solve independently as reinforcement, in a “watch one, do one” structure.
• The engineer agreed to this restructuring for future sessions: going forward, the format will be (1) instructor shares screen and works a real problem statement live, showing structured debugging start to finish, then (2) a second, related problem statement is released for teams to solve independently, followed by group RCA discussion.
• This pivot happened live, mid-session — a genuine example of curriculum responsiveness to direct learner feedback, not something planned in advance.

Given the pivot above, the engineer used the remaining session time to present the actual historical incident he’d personally worked (rather than a synthetic case) as a live, Socratic-method group exercise. This scenario was not resolved within the session — see Section 3.10 for the full diagnostic thread and the hints given at the end.

Infrastructure context:

• Two EC2 instances (framed as cloud-agnostic — equally applicable as GCP VMs or Azure VMs): a bastion host and a production server, both running Ubuntu, both in the same VPC and the same private subnet (neither has a public IP; access is entirely internal).
• Bastion: 2 vCPU, 4 GB RAM, 10 GB storage. Sole purpose: entry point/jump host for engineers to reach the production server.
• Production server: 4 vCPU, 8 GB RAM, 15 GB storage. Hosts a business-critical payment/analytics application that pulls transaction data from multiple regions and feeds an external dashboard used by finance and data science teams.
• User base: 30 engineers (a mix of DevOps, SRE, and data science), all physically co-located (“under the same roof”), all authenticating via a shared SSH key pair (single RSA key used by everyone to hop from bastion to production).
• Region: ap-south-1 (Mumbai), per a participant’s clarifying question.
• Business criticality/SLA: Incident occurs on a Friday morning, a peak reporting day; the team has 30 minutes to resolve before the issue escalates to the CFO. If the dashboard’s data feed is delayed, it breaches report and fraud-detection SLAs, causing real business/financial impact — this is explicitly framed as a business-critical, time-pressured incident, not an abstract exercise.

The actual symptom:

• All 30 engineers can SSH into the bastion without any issue.
• Of those 30, when attempting to hop from bastion → production:
  • 5–6 people get a hard SSH connection timeout — cannot connect at all.
  • 7–8 people can connect, but the session is painfully slow — commands take up to 30 minutes to execute after being issued.
  • ~11 people experience no issue at all — everything works normally.
• Critically: this is random and intermittent, not tied to a specific user. The same individual might succeed at one moment, fail an hour later, and succeed again after that — there’s no consistent pattern by user identity.
• Everything else looks healthy: CPU, RAM, disk I/O, and storage on both bastion and production are all normal/green on monitoring. No error logs beyond a generic SSH timeout on the bastion side. No packet loss detected. DNS resolution is working fine. No recent patches or deployments in the prior 72 hours. Application itself (as accessed externally, via the dashboard it feeds) has no errors and is fully functional.
• One important operational clue volunteered by the engineer mid-Q&A: restarting the SSH daemon on the production server temporarily resolves the issue for most users — but the problem recurs roughly 10–15 minutes after the restart. This is a strong signal that whatever is degrading is something that accumulates or resets over time rather than a static misconfiguration.

3.10 Diagnostic Q&A Transcript — What Was Ruled Out, What Was Hinted

This section preserves the actual diagnostic reasoning chain from the group, since the value here is in the process, not a final answer (which was not given in this walkthrough).

Ruled out during the live Q&A (confirmed by the engineer, in response to direct questions):

• Not a subnet/routing issue — bastion and production are in the same private subnet; route tables and IP tables (routing rules) look normal; no packet loss detected between bastion and production.
• Not a firewall/security-group issue in the traditional sense — explicitly reasoned out loud by a participant: if it were a blanket firewall block, it would fail for everyone, not randomly for a subset.
• Not file descriptor (FD) limit misconfiguration — the engineer confirmed the FD limit setting itself looked correctly configured (not the same as FD exhaustion, which remained an open hypothesis — see below).
• Not max SSH sessions per user — explicitly checked and found set to 99 (i.e., not an artificially low cap).
• Not a DNS issue in this specific scenario — separately confirmed working; a participant noted the team was using hostname-based (not raw IP) access, which the engineer acknowledged without ruling in or out DNS as a contributing factor definitively.
• Not user-permission-related — ruled out because the same individual user’s access flips between working and failing over time, which permission settings wouldn’t explain.
• Not resource exhaustion in the traditional monitoring sense — CPU, RAM, and disk all showed as normal/healthy throughout, on both hosts.
• Not multi-region load-balancing/health-check related — clarified that while the application pulls data from multiple regions, the SSH access path itself is not multi-region; all engineers are accessing from the same physical location.
• Not a NAT Gateway issue — the engineer confirmed there’s no NAT Gateway in this specific setup at all (both hosts are simply in the same subnet).
• Not a GCP-specific “guest attribute” metadata issue (a participant raised this as a known GCP SSH quirk related to concurrent session limits) — the engineer confirmed this specific scenario is cloud-agnostic and not tied to that particular GCP mechanism, though he acknowledged the concept as real and worth knowing about on GCP specifically.

Actively investigated, inconclusive within the session:

• strace/estrace was reportedly already run on both bastion and production sides by the engineer prior to the call, with nothing conclusive surfaced from it (mentioned briefly, not detailed).
• SSH verbose mode (ssh -vvv) was suggested by a participant as a next diagnostic step (to inspect handshake/negotiation behavior) but not executed live within this transcript.
• A participant suggested checking whether the production server was under cgroup-level CPU throttling (i.e., normal top-level CPU usage looking fine while a specific service’s cgroup allocation is constrained) — a genuinely sharp hypothesis distinguishing CPU usage from CPU time actually granted to a process. Not confirmed or refuted within the session.
• Time synchronization between bastion and production was checked and confirmed to be in sync (ruling out clock-skew-related auth/certificate issues).
• SSH configuration files themselves were checked and reportedly look correct on both sides.
• A non-standard SSH port (not 22) is in use and reportedly correctly whitelisted — ruled out as a simple port-block issue, though notably this doesn’t fully rule out more subtle port/connection-tracking table exhaustion.

Explicit hints given by the engineer at the end of the session (not a full answer):

• The issue is described as a “chain reaction” — a combination of multiple small factors accumulating over time, not one single static misconfiguration.
• Specifically named as contributing factors: something related to SSH itself, something related to ndots (a DNS resolver configuration parameter controlling how many dots must appear in a hostname before it’s treated as fully-qualified, otherwise triggering additional search-domain lookup attempts), and “too many files open” (i.e., a file descriptor exhaustion condition) — with the engineer noting there are “two more factors” involved beyond what had been named by that point in the call, which were not disclosed.
• The engineer’s suggested next diagnostic move: running SSH with verbose output, and thinking carefully about exactly how SSH establishes a connection at a protocol level (TCP three-way handshake, then key exchange) as a way of narrowing down at which specific step in that sequence things are actually breaking.
• A promised ~3-hour recorded deep-dive video (already created, to be shared) walks through this exact incident’s structured debugging process in full, along with a discussion of how SSH works at a lower protocol level — not included in this transcript, and is the actual source of the full resolution.

Working theory, based on the hints (explicitly a synthesis by this document, not the engineer’s stated final answer — see Gaps & Assumptions): The combination of clues (SSH daemon restart provides only temporary relief; “too many files open”; ndots misconfiguration; a single shared SSH key/user account used by all 30 engineers) is consistent with a scenario where lingering/orphaned SSH sessions or connections are slowly exhausting a file descriptor limit or connection-tracking table over time, compounded by a DNS ndots misconfiguration causing extra, slow DNS lookups on every single connection attempt (because hostnames aren’t being recognized as fully-qualified, triggering repeated search-domain resolution attempts) — which would explain both the intermittent, non-user-specific nature of the failures (a shared, degrading resource pool) and why restarting the SSH daemon provides only temporary relief (it clears the daemon’s immediate state but doesn’t fix the underlying resource leak or DNS misconfiguration, which reaccumulates within ~10–15 minutes).

4. Key Concepts Table

5. Architecture & Workflow Analysis

5.1 Cascading Failure Pattern (Generalized, from the Case Studies)

Small Initial Trigger
  (saturated gateway / bad regex / withdrawn route / slow DNS)
        |
        v
Localized Degradation
  (packet loss / CPU spike / unreachable route / slow resolution)
        |
        v
Downstream Symptom
  (increased latency / timeouts)
        |
        v
User/Client Retry Behavior
        |
        v
Retry Storm (self-generated load spike)
        |
        v
Resource Exhaustion
  (thread pools / connection queues / FDs)
        |
        v
Full Cascading Infrastructure Failure

5.2 OSI-Layer Troubleshooting Sequence (as demonstrated)

1. Network Reachability Layer
   "Can this host reach the outside world at all?"
   Tools: ping, traceroute, ss
        |
        v
2. DNS Resolution Layer
   "Is DNS resolving correctly and quickly?"
   Tools: dig, nslookup
        |
        v
3. TCP / Connection Layer
   "Are TCP connections establishing? What's the handshake/wait state?"
   Tools: ss, netstat, tcpdump
        |
        v
4-7. Higher Layers
   (application-specific, service-specific checks)
        |
        v
Conclusion: Root cause isolated at the layer where behavior first deviates from expected

30 Engineers (same physical location)
   |  (local machine, private SSH key)
   v
Bastion Host (EC2, Ubuntu, 2 vCPU / 4GB RAM)
   - Public-facing entry point (no external IP in this specific config;
     accessed via internal network path)
   - Sole purpose: jump host
   |  (shared SSH key, all 30 users)
   v
Production Server (EC2, Ubuntu, 4 vCPU / 8GB RAM)
   - No public IP; reachable ONLY via bastion
   - Hosts business-critical payment/analytics application
   - Pulls transaction data from multiple regions
   - Feeds external dashboard (finance + data science consumers)

Both hosts: same VPC, same private subnet, ap-south-1 (Mumbai)

SYMPTOM (intermittent, not user-specific):
   ~11/30 users: no issue
   ~7-8/30 users: connects but painfully slow (up to 30 min per command)
   ~5-6/30 users: hard SSH timeout, no connection at all

ALL of CPU / RAM / disk / packet loss / DNS / app health: NORMAL

KEY CLUE: SSH daemon restart on production -> temporary fix -> recurs ~10-15 min later

                    Suspected Cause
                          |
        --------------------------------------------
        |            |            |            |
   Routing/       Firewall/    FD Limit      Max SSH
   Subnet         Security      (config)     Sessions
   RULED OUT      Group         RULED OUT    RULED OUT
                  RULED OUT     (limit set    (set to 99,
   (same subnet,  (would fail    correctly,    not the
    no packet      for ALL       but EXHAUSTION  bottleneck)
    loss)          users, not    still open)
                   a subset)
        |
        --------------------------------------------
        |            |            |
   User Perms    Resource      Multi-region/
   RULED OUT     Exhaustion    NAT Gateway
   (same user     (traditional  RULED OUT
    flips pass/   monitoring:   (no NAT gateway;
    fail over      all normal)  SSH path is
    time)                       single-region)
        |
        v
   REMAINING OPEN HYPOTHESES (hinted, not confirmed in-session):
   - FD exhaustion (accumulating "too many open files")
   - ndots DNS resolver misconfiguration (extra lookups per connection)
   - cgroup-level CPU throttling (usage looks normal, allocation may not be)
   - "Two more factors" explicitly not disclosed by instructor

6. Commands & Configurations

7. Tools & Technologies

dig

• Purpose: DNS lookup utility, used to query and troubleshoot DNS resolution.
• When to use it: First-line tool for testing DNS resolution speed and success/failure, especially when an incident has networking-flavored symptoms.

ping / ICMP

• Purpose: Basic network reachability testing using the ICMP protocol.
• When to use it: As a first-pass reachability check — but remember it tests a different protocol than most application traffic (TCP), so a successful ping does not guarantee application-level connectivity.

curl

• Purpose: HTTP/TCP-level connectivity and request testing.
• When to use it: To test actual application/service-level reachability over TCP, as a complement to (not a replacement for) ping.

traceroute

• Purpose: Maps the network path (hop by hop) to a destination.
• When to use it: When basic reachability tests fail or are inconsistent, to identify at which network hop a problem might be occurring.

ss

• Purpose: Modern Linux socket statistics tool (successor to netstat).
• When to use it: Inspecting active TCP connections, their states (e.g., ESTABLISHED, TIME_WAIT), and listening ports — useful for both the network-reachability and TCP-connection-layer diagnostic steps.

strace

• Purpose: Traces system calls made by a running process.
• When to use it: For deep, low-level diagnosis when higher-level tools don’t reveal the issue — e.g., to see exactly what a hanging SSH process is doing (or blocked on) at the syscall level.

iptables

• Purpose: Linux kernel firewall rule management.
• When to use it: Both as a diagnostic target (checking whether rules are unexpectedly blocking traffic) and, in this walkthrough, as the tool used to intentionally construct the “ping works, curl doesn’t” teaching scenario.

Terraform (referenced, code to be published)

• Purpose: Infrastructure-as-Code tool used to build and tear down each week’s simulated war-room environment.
• When to use it: The engineer’s stated plan is to publish the Terraform modules for each week’s simulated outage to GitHub, so participants can recreate the exact same failure scenario in their own environment for independent practice.

8. Real-World Production Usage

• Studying real, published postmortems (Slack, Cloudflare, Facebook, Uber) is a legitimate, high-value practice for building incident-response pattern recognition — these are genuinely useful references precisely because the root causes span the most common outage categories (network saturation, bad config/rule changes, routing failures, slow third-party dependencies) that any production engineer will eventually encounter in some form.
• The retry-storm pattern is one of the most underappreciated real-world failure amplifiers — many engineers focus on the initial trigger (a saturated gateway, a slow DNS server) without recognizing that the system’s own retry behavior is often what turns a contained degradation into a full outage. Designing sane backoff/retry/circuit-breaker behavior into client code is a direct, practical mitigation.
• ping vs. curl divergence as a fast diagnostic heuristic is genuinely used in real on-call troubleshooting — it’s a low-cost, high-signal first check that can quickly narrow a problem from “network is down” to “something specific is blocking this port/protocol.”
• The unresolved bastion/SSH scenario is an authentic illustration of what real production incidents actually feel like — ambiguous, intermittent, with multiple plausible-sounding-but-wrong hypotheses along the way, and a real business SLA clock running. The fact that this walkthrough ended without a clean resolution (deferred to a dedicated follow-up with full context) is arguably more realistic and more valuable than a tidy, fully-resolved teaching example — real incidents often aren’t solved in one sitting, and knowing how to keep narrowing the hypothesis space under time pressure is the actual skill being trained.
• Shared/single-credential access models (one SSH key for 30 engineers) are a real, common anti-pattern in fast-moving organizations, and this walkthrough’s scenario incidentally illustrates why that’s problematic even beyond the security angle — it makes incident diagnosis by user-identity impossible, since you can’t distinguish “this specific user’s session” from “this specific connection attempt” in logs tied only to a shared account.
• The engineer’s live incorporation of direct participant feedback (shifting from documentation-review format to live demo + independent-practice format) is itself a good real-world lesson in coaching/mentoring — adjusting teaching method based on what the specific audience (in this case, already-experienced practitioners) actually needs, rather than a fixed curriculum delivery.

9. Interview Preparation

Beginner Questions

Q1: What’s the difference between what ping and curl actually test, and why might one succeed while the other fails? A: ping tests basic network reachability using the ICMP protocol. curl tests application/service-level connectivity, typically over TCP (e.g., HTTP on port 80/443). Because these are different protocols, a firewall or routing rule can block one without affecting the other — for example, an iptables rule dropping TCP port 80 traffic would cause curl to fail while ping (ICMP) continues to succeed, which is a useful diagnostic signal pointing toward a port/protocol-specific block rather than a full network outage.

Q2: What is a “retry storm,” and why can it be worse than an external attack? A: A retry storm happens when a system’s own legitimate users or clients, encountering failures or timeouts, automatically or manually retry their requests — often many times, often simultaneously. This adds load on top of an already-degraded system, which can push it from a partial degradation into a full outage. It’s sometimes described as worse than an external DDoS attack because it’s driven by real, well-intentioned traffic that’s hard to simply block, and it can catch a team off guard because it looks like organic (if elevated) demand rather than an obvious attack.

Q3: Why is DNS often described as “the root of most production outages”? A: Because DNS sits at the very front of nearly every network interaction — before an application can even attempt a connection, it typically needs to resolve a hostname to an IP address. This means DNS failures (whether from the DNS servers themselves, routing to reach them, or misconfiguration) can silently break connectivity for everything downstream, even when the actual destination service is perfectly healthy. This is why experienced engineers are trained to check DNS early when facing an ambiguous networking-flavored incident.

Intermediate Questions

Q4: Walk through the OSI-layer approach to troubleshooting a suspected network issue. A: Start at the most fundamental layer and work upward, drawing a conclusion at each step before moving on: first confirm basic network reachability (can the host reach the outside world at all — using tools like ping, traceroute, ss), then check DNS resolution (is it working, and is it fast — using dig), then examine TCP/connection-level behavior (are connections establishing, what states are they in), and continue upward through higher layers as needed. This structured approach prevents “directionless debugging” — jumping between unrelated hypotheses — which wastes critical time during a live incident where every minute has real business cost.

Q5: Explain the cascading failure pattern seen in the Slack outage, step by step. A: An AWS Transit Gateway interconnecting Slack’s VPCs became saturated, which caused packet loss. That packet loss increased request latency and caused backend timeouts, which in turn blocked worker threads (a form of resource exhaustion). As users experienced failures, they retried their actions (re-uploading files, resending messages) — often repeatedly — which created a retry storm that piled additional load onto an already-struggling load balancer. This retry-driven overload compounded the original problem into a full, roughly hour-long service outage — illustrating how a single infrastructure component’s saturation can cascade into total unavailability through retry dynamics alone.

Q6: A production incident shows CPU, RAM, and disk all reporting as “healthy” in monitoring, yet a specific service is clearly struggling. What’s a diagnostic angle that basic resource monitoring might miss? A: Top-level system resource monitoring can miss per-process or per-cgroup constraints — a specific service could be CPU-throttled at the cgroup level even while overall system CPU usage looks normal, because cgroup limits control the CPU time actually granted to a process, not just overall utilization. This is a genuinely advanced but important distinction: “CPU usage looks fine” and “this process is getting the CPU time it needs” are not the same claim, and checking cgroup-level allocation (not just top-level top/htop output) can reveal a bottleneck that basic monitoring dashboards wouldn’t surface.

Advanced Questions

Q7: You’re debugging an intermittent SSH access issue affecting a random subset of users (not tied to specific identities), where restarting the SSH daemon provides only temporary relief before the issue recurs. What does this pattern suggest about the nature of the underlying problem, and what would you check next? A: The fact that a service restart provides only temporary relief strongly suggests the underlying cause is something that accumulates or degrades over time rather than a static, one-time misconfiguration — for example, a resource that gets consumed and not properly released (file descriptors, connection-tracking table entries, or similar), which the restart temporarily clears but which then re-accumulates. The randomness across users (rather than being tied to specific accounts) suggests the exhausted resource is likely shared/pooled rather than per-user — consistent with something like a systemwide file descriptor limit, a connection-tracking table, or a shared credential/session pool. Next steps: check file descriptor usage over time (not just at a single snapshot) via ulimit//proc-based inspection, examine whether SSH sessions are being properly closed/reaped or are lingering as zombies, and check DNS resolver configuration (specifically ndots) for anything that might be silently multiplying DNS lookups per connection attempt, since DNS lookups performed as part of SSH’s reverse-DNS or hostname-based authentication flow can itself be a slow, resource-consuming step under the right misconfiguration.

Q8: How would you structure a live war-room troubleshooting session to be maximally useful for a cohort of already-experienced engineers, based on the feedback given in this walkthrough? A: Based on the direct feedback in this walkthrough, the most effective structure is: (1) don’t just review documentation/RCA links passively — walk through a real architecture diagram live, showing exactly where in the request/access path the failure occurred; (2) fully deep-dive one representative incident live, with actual terminal-based debugging shown step by step, rather than shallowly covering many incidents; (3) explicitly connect the specific commands/tools used at each diagnostic step back to the structured framework being taught (e.g., OSI layers), so participants can generalize the pattern, not just memorize the one specific fix; and (4) follow the live demo with a second, related-but-distinct problem statement for the group to solve independently, reinforcing the pattern through active practice rather than passive observation. This “watch one, do one” structure respects that experienced practitioners often already know individual commands/tools — what they need is the structured decision process for applying them under pressure.

Q9: In the unresolved bastion/SSH scenario, several plausible-sounding hypotheses were ruled out one by one (firewall rules, FD limit misconfiguration, max session caps, user permissions, DNS, multi-region routing, NAT gateway). What does this progressive elimination process illustrate about real incident response, and why might it be valuable that the session didn’t resolve the incident? A: This illustrates that real incident response is fundamentally a process of narrowing a hypothesis space under uncertainty, not a lookup-table exercise — many of the ruled-out hypotheses were entirely reasonable given the symptoms and had to be actively tested/confirmed-absent, not just intuited. The value of leaving the scenario unresolved in this teaching context (rather than being handed a clean answer) is that it forces genuine diagnostic reasoning practice: readers/participants have to sit with real ambiguity, weigh which of the remaining open hypotheses (FD exhaustion, ndots misconfiguration, cgroup throttling, and undisclosed additional factors) are most consistent with the specific pattern of clues (intermittent, not user-specific, temporarily fixed by a restart, recurring after ~10-15 minutes) — which is a much closer simulation of an actual on-call incident than a scenario with an immediately revealed answer.

10. Exam & Certification Notes

(Relevant to Linux/networking-focused certifications — e.g., CompTIA Linux+, RHCSA, LPIC, and networking fundamentals sections of cloud certifications.)

• ICMP vs. TCP protocol distinction: A commonly tested networking fundamental — know that ping uses ICMP while most application traffic (HTTP, SSH, etc.) uses TCP, and that these can be independently firewalled. This exact scenario (ping works, application traffic doesn’t) is a classic exam and real-world diagnostic pattern.
• File descriptor limits (ulimit): Know the distinction between the soft limit and hard limit for open file descriptors, and that exhausting available file descriptors causes new connection/file-open attempts to fail — a frequently tested Linux resource-management concept, directly relevant to the “too many files open” hint in this walkthrough.
• resolv.conf and ndots: Understand that the ndots parameter in resolv.conf controls when a hostname is treated as fully qualified vs. triggering search-domain suffix appending — an increasingly relevant exam topic given how often it causes subtle, hard-to-diagnose DNS latency issues in containerized/Kubernetes environments specifically (worth connecting to broader ndots-in-Kubernetes discussions, since Kubernetes’ default DNS search-domain configuration is a well-known source of exactly this kind of issue).
• BGP route withdrawal and its effect on reachability: Understand that BGP determines how networks are reachable across the internet, and that a route withdrawal can make a fully healthy set of servers completely unreachable — a good example for distinguishing “DNS is broken” from “the network can’t route to reach the DNS servers at all,” a subtle but testable distinction.
• Catastrophic regex backtracking: A general software/security concept (not cloud-specific) — know that certain regex patterns can have exponential (rather than linear) time complexity on certain inputs, and that this is a real, recurring cause of CPU-exhaustion incidents in production systems, not just a theoretical computer science curiosity.
• cgroups vs. system-wide resource monitoring: Understand that Linux control groups (cgroups) can constrain a specific process/service’s resource allocation independently of overall system resource usage — a nuance increasingly relevant given how commonly cgroups underpin container resource limits (Docker, Kubernetes).

11. Cheat Sheet

Cascading Failure Pattern (memorize the shape, not just the examples): Small trigger → localized degradation → downstream timeouts → user retries → retry storm → resource exhaustion → full outage

Fast First-Pass Diagnostic Check: ping succeeds + curl fails → suspect port/protocol-specific firewall block (check iptables/security groups), not a full network outage

OSI-Layer Troubleshooting Order:

1. Network reachability (ping, traceroute, ss)
2. DNS resolution (dig)
3. TCP/connection state (ss, netstat, tcpdump) 4–7. Higher layers (application-specific)

Famous Outage Root-Cause Quick Reference:

Signs a Problem “Accumulates Over Time” (not a static config issue):

• A service restart provides only temporary relief
• The issue recurs on a roughly consistent time interval after restart
• → Suspect a resource that’s consumed but not properly released (FDs, connections, sessions)

Diagnostic Questions to Always Ask in an Ambiguous Incident:

• Does this affect everyone or a subset? (Rules firewall/blanket-block issues in/out)
• Is it consistent per-user or random/intermittent? (Points toward shared/pooled resource vs. per-user config)
• Did anything change in the last 24–72 hours (deploys, patches)? (Rules recent-change-driven issues in/out)
• Does a restart fix it permanently or temporarily? (Static misconfiguration vs. accumulating resource issue)

12. Gaps & Assumptions

• The bastion/SSH scenario is explicitly unresolved in this transcript. The engineer gave hints (SSH-related, ndots, “too many files open,” and “two more factors” not disclosed) but did not provide the final root cause and fix within this walkthrough — those were reserved for a separate ~3-hour recorded video and a dedicated follow-up call not captured here. The “working theory” offered in Section 3.9 is this document’s own synthesis of the given clues, explicitly labeled as such, and should not be treated as the engineer’s confirmed answer.
• The fifth outage (GCP) was named in the session’s outage roster (alongside Slack, Cloudflare, Facebook, and Uber) but no technical details about it were covered in this transcript — likely covered in materials uploaded to the drive rather than spoken aloud in this specific call.
• Exact command syntax for several suggested diagnostics (e.g., the SSH daemon restart command, the ulimit/FD-check commands) was referenced conceptually in the discussion but not typed/shown explicitly on screen in this transcript — presented here using standard, conventional syntax rather than verbatim transcript content.
• VERDICT-7 framework details: referenced by name as the advanced material-tier troubleshooting framework but not explained in technical depth in this transcript — participants were pointed to a separate drive folder for the full model. Treat this document’s coverage of VERDICT-7 as a name-level reference only, not a working explanation.
• This walkthrough ran significantly over its original plan due to the extended pedagogical discussion in Section 3.8 and the depth of the live Q&A in Section 3.10 — as a result, several planned topics (a full walkthrough of the Foundations Track environment’s remaining layers, a full advanced material resolution) were explicitly deferred to a subsequent session rather than completed here. This document reflects that incompleteness accurately rather than inventing a resolution.
• AWS account suspension incident (cryptocurrency mining): the engineer attributed this specifically to someone on our own team (not a program participant) based on audit logs — this detail is preserved as stated, though the transcript doesn’t include independent verification of this attribution beyond the engineer’s own account.
• This document consolidates a long, interruption-heavy session (extensive live troubleshooting Q&A with many rapid-fire participant questions, a mid-call technical disconnection, and extended access-issue troubleshooting at the end) — content has been reorganized topically for clarity rather than presented in strict chronological order, consistent with the approach used for prior packages in this series.