Incident Replay
Real production incidents replayed as a story you work through. Each drill sets the scene and the stakes, then walks the investigation as a sequence of decision points — commit to what you'd check first, then reveal what the responder actually did, and why.
Four Faults, One Host
A single production Linux host goes sluggish: load average through the roof, every hostname lookup crawls, and apps start failing with 'Too many open files'. Four independent faults were injected at the same time — a CPU storm, a DNS thread storm, a resolver misconfiguration, and a file-descriptor leak — and only a disciplined, layer-by-layer sweep untangles them.
The host serves production workloads; compounding CPU, DNS, and FD exhaustion drags every process on the box toward failure.
The Eviction Cascade
A payments namespace starts shedding pods at peak — checkout restarts, scheduled jobs die mid-run — with no deployment, no config change, and every node showing Ready. The culprit is a shared node pool with no isolation between a revenue-critical service and a hungry batch workload.
18% checkout failure rate during the peak window and a 12-minute transaction-processing degradation, amplified by a client retry storm.
The Silent DNS Saturation
A payments platform starts throwing intermittent 502s and 20x latency at peak — yet every pod is Running, no pod is OOMKilled or Evicted, every node is Ready, and nothing deployed. The failure lives in the one dependency that sits in front of every service call: cluster DNS.
~3,000 transactions/minute at peak; revenue-impacting SEV-1 as checkout latency jumps from ~90 ms to over 1.8 s.
The Bastion Mirage
Thirty engineers, one bastion, one private production host — and a failure that only some people can reproduce, only some of the time. A blast-radius and OSI-layer story where the 'obvious' single cause is a mirage hiding four independent faults.
Peak reporting day; payment-analytics backend at risk of a 1-hour SLA breach if engineers can't reach prod to remediate.
The Rollout Freeze
A routine rolling update of a revenue-critical checkout service wedges: two replicas go healthy, the third sits Pending forever, and the rollout freezes. No crashes, no failing probes, no node problems — and node drains start hanging too. A pure structured-debugging exercise in a cluster that looks flawless.
Checkout runs at reduced replica count under production load, the deploy pipeline is frozen (no rollback, no hotfix), and node maintenance is blocked — a slow drift toward a larger outage.