SRE Labs (Advanced Track) — Doubt Class: Live Structured Debugging Demo (Foundations Track Resolution)
Structured educational resource covering sre labs (advanced track) — doubt class: live structured debugging demo (foundations track resolution).
Put this material to work in a story-driven Incident Replay — step into the war room and reason through it decision by decision:
Full Terminal Walkthrough of the CPU/Thread/DNS/FD Scenario + Advanced Track Planning
2. Table of Contents
- Executive Summary
- Table of Contents
- Detailed Structured Notes
- 3.1 Session Context — Delivering on the Prior Session’s Commitment
- 3.2 The Four Background Scripts (Root Cause Recap)
- 3.3 Full Structured Debugging Walkthrough (Step by Step)
- 3.4 The DNS-Specific Diagnostic Deep Dive
- 3.5 Connection-State and Kernel-Level Checks
- 3.6 System Service Health Check Methodology
- 3.7 Live Incident: A Participant Couldn’t Reproduce the Symptom
- 3.8 Advanced Track Planning for the Next Session
- 3.9 Participant Feedback — Format and Production-Architecture Expectations
- Key Concepts Table
- Architecture & Workflow Analysis
- Commands & Configurations
- Tools & Technologies
- Real-World Production Usage
- Interview Preparation (Beginner / Intermediate / Advanced)
- Exam & Certification Notes
- Cheat Sheet
- Gaps & Assumptions
3. Detailed Structured Notes
3.1 Session Context — Delivering on the Prior Session’s Commitment
At the end of the prior Week 1 war-room session, multiple experienced participants had pushed back that documentation/case-study review wasn’t as valuable as a live, terminal-based debugging demonstration, and the instructor committed to changing format going forward. This session is that commitment being fulfilled: rather than more case studies, the instructor logs directly into the simulated Foundations Track instance and walks through the entire diagnostic process live, narrating each command and its interpretation as he goes.
A participant’s framing at the very start of the session captures the intent well: rather than being told what was wrong, walk through it as if you don’t already know what’s causing it — i.e., demonstrate the discovery process itself, not just confirm a pre-known answer.
3.2 The Four Background Scripts (Root Cause Recap)
The instructor confirmed and named the four scripts that had been running in the background of the Foundations Track instance, collectively producing the confusing, intermittent symptom set from the prior session:
- CPU pressure script — a simple loop that does no real work, purely consuming CPU cycles to create artificial load.
- Thread pool exhaustion script — designed to heavily load the thread-handling side of the system, which was directly responsible for
pingandcurlintermittently failing. resolv.confmodification script — repeatedly changing the configured DNS nameservers, which caused DNS resolution (dig) to intermittently succeed or fail, and introduced added latency.- File descriptor (FD) exhaustion script — deliberately driving the system toward its FD limit, contributing to the broader resource-pressure symptom set.
These four scripts running concurrently is what produced the seemingly chaotic, hard-to-pin-down symptom pattern from the earlier session (intermittent ping/curl/dig failures with no obvious single cause).
3.3 Full Structured Debugging Walkthrough (Step by Step)
The instructor’s demonstrated sequence, narrated live on the terminal:
Step 1 — Establish the baseline symptom. Confirm exactly what’s failing: is dig failing, is ping failing, is curl failing? Document the specific combination, since (as established in the prior session) different combinations point toward different layers of the problem.
Step 2 — Rule out infrastructure-level blocking first. Before touching the OS, check whether the cloud provider’s networking layer (VPC configuration, security groups/firewall rules) is blocking traffic. This is checked first because it’s the layer furthest “outside” the machine, and ruling it out early avoids wasted effort debugging the OS for a problem that’s actually upstream.
Step 3 — Check iptables for dropped protocols. Once infrastructure-level blocking is ruled out, check the host’s own iptables rules for anything unexpectedly dropping specific ports/protocols (directly relevant here, since the earlier session’s demo environment specifically used iptables to block port 80).
Step 4 — Determine scope: single-server issue vs. broader network issue. Test connectivity to multiple different external destinations, not just one. If only one specific target is unreachable, that points toward an application/target-specific issue; if everything external is unreachable, that points toward a genuine network connectivity problem at the host or infrastructure level.
Step 5 — Check system resource health. Using top (or htop) to identify CPU hogs and any stuck/high-CPU processes; free -h-style memory checks to confirm how much memory is actually free; and uptime to check the system’s load averages (1/5/15-minute load figures) as a quick signal of overall system pressure.
Step 6 — Check DNS at both the local and upstream level. See Section 3.4 for the full DNS-specific sub-process.
Step 7 — Check open connections and connection churn. Using ss -s (socket statistics summary) to see the current count of TCP/UDP connections in various states — a high or rapidly-growing established-connection count is a signal of a retry storm or connection leak (see Section 3.5).
Step 8 — Check for kernel-level blocking. Look for processes stuck in an uninterruptible (“blocked”/“D”) state that could be choking the kernel, and review recent kernel logs for related errors (see Section 3.5).
Step 9 — Check system service health/responsiveness. Time how long core services (e.g., the SSH daemon) take to respond to a status query, using rough health thresholds to judge severity, and reload/restart services as needed (see Section 3.6).
Step 10 — Document, hypothesize, fix, verify. Write down every command run and its output as you go, form a specific hypothesis about the root cause based on the accumulated evidence, apply the fix, and re-verify that the original symptoms are actually resolved (not just superficially improved).
3.4 The DNS-Specific Diagnostic Deep Dive
This was demonstrated as its own mini-procedure within the larger walkthrough, because DNS issues have a specific, well-defined diagnostic pattern:
- Test local resolution:
dig google.comusing the system’s default (local) resolver configuration. - Test upstream resolution directly: Query a known public DNS resolver directly (e.g.,
8.8.8.8), bypassing the local resolver entirely — this isolates whether the problem is in the local resolver daemon specifically, or in DNS/network connectivity more broadly. - Interpret the result:
- If local DNS fails but the direct query to 8.8.8.8 succeeds → the problem is specifically with the local
systemd-resolveddaemon (or equivalent local resolver service), not the broader network or DNS infrastructure. - Fix: Restart the local resolver daemon (
systemd-resolvedin this case).
- If local DNS fails but the direct query to 8.8.8.8 succeeds → the problem is specifically with the local
- Check the resolver daemon’s running status as part of this process, to confirm whether it’s actually active/healthy before and after the restart.
This is exactly what happened in the actual demo: local DNS was failing, but the direct query to 8.8.8.8 worked — correctly pointing the instructor toward restarting systemd-resolved as the targeted fix, rather than a broader (and unnecessary) network investigation.
3.5 Connection-State and Kernel-Level Checks
Connection state (ss -s):
- Shows a summary of current TCP and UDP socket counts, broken down by state.
- Interpretation: if the count of established connections is unusually high (and especially if it’s climbing), this is a direct signal of a connection/retry storm — the system (or its clients) are opening far more connections than are being properly closed, which will eventually exhaust available file descriptors/ports and take the system down.
- The instructor explicitly tied this back to the retry-storm pattern discussed in the case studies from the prior session: a slow or intermittently-failing DNS resolver will cause clients to retry, and those retries manifest here as a rising connection count.
Kernel-level blocking checks:
- Check for processes stuck in an uninterruptible sleep / “blocked” (“D”) state — these are processes the kernel is unable to schedule/interrupt normally, and a buildup of them is a sign the kernel itself is choking (often due to I/O contention or a stuck driver/subsystem).
- Cross-reference with the kernel log (
journalctl, checking the most recent ~20–50 lines) for related error messages. - In this specific demo run, no blocked tasks were found — the instructor explicitly noted this as a clean result at this step, meaning the kernel itself wasn’t the bottleneck in this particular pass (a useful example of a diagnostic step correctly returning “not this” rather than assuming every step must reveal something).
3.6 System Service Health Check Methodology
A specific, quantified approach to judging whether a system service (demonstrated using the SSH daemon, sshd) is under distress:
- Run a status check against the service (e.g.,
systemctl status sshd) and time how long it takes to respond. - Interpretation thresholds (as given live):
| Response Time | Interpretation |
|---|---|
| Under ~0.5 seconds | Healthy |
| ~1–3 seconds | System under pressure |
| ~5–10 seconds, or the command hangs entirely | System is choking |
- If the response is slow/hanging: reload the systemd daemon manager itself (
systemctl daemon-reload), and/or restart the specific affected service, then re-time the same check to confirm whether the response time has actually improved. - This gives a repeatable, quantifiable way to track whether an intervention actually helped, rather than relying on a vague “seems better now” judgment.
3.7 Live Incident: A Participant Couldn’t Reproduce the Symptom
A genuinely valuable, unplanned teaching moment occurred mid-session:
- One participant, trying to follow along on the shared Foundations Track instance, reported that
dig,ping, andcurlwere all working fine for them — no reproducible issue at all, despite the instructor having just walked through diagnosing and fixing the exact same symptoms moments earlier. - The participant shared multiple screenshots showing healthy
dig/ping/curl/resolv.confoutput, genuinely confused about why they couldn’t reproduce what was being taught. - Resolution: The instructor eventually determined that someone else working on the same shared instance had already found and stopped the disruptive background scripts — silently resolving the issue for everyone sharing that VM, without communicating that they’d done so.
- The instructor’s response: rather than treating this as a distraction, he used it directly — confirming that once the scripts are stopped, symptoms disappear, and sharing the actual script files with the group so they could re-run them intentionally to recreate the scenario for further independent practice.
- This is a genuinely realistic lesson about team-based troubleshooting on shared infrastructure: one team member’s unilateral, uncommunicated action (stopping a “problem” script) can silently invalidate another team member’s ongoing diagnostic work — a coordination failure mode that happens in real incident response too, not just in a training exercise.
3.8 Advanced Track Planning for the Next Session
- The cohort selected Advanced Track assignments #3 and #4 to be demoed next (participants were offered a choice of which assignments to prioritize, rather than the instructor picking unilaterally).
- Assignment #3 involves a Kubernetes control-plane-style scenario — the instructor described creating infrastructure with control-plane elements where, for example, the API server is down or slow to respond, and demonstrating both how to construct that scenario from scratch and how to debug it.
- Plan: the instructor will build out this infrastructure first, then run a dedicated doubt-class session (timing to be confirmed — evening was preferred over morning due to timezone constraints among participants, including someone on a night shift and someone based in Europe) to walk through creating and debugging assignments #3 and #4 live, mirroring this session’s format.
- A dedicated point of contact for daytime/US-timezone support was being shared via Discord for participants who can’t attend the IST-timed live sessions.
3.9 Participant Feedback — Format and Production-Architecture Expectations
Positive confirmation of the new format: Multiple participants explicitly thanked the instructor for the terminal-first approach, with one stating directly that seeing the actual commands and reasoning step by step (rather than being told the answer) was “exactly what I was looking for” and matched their expectations for the program.
A separate, distinct concern raised by another participant: wanting more exposure to what a genuinely production-scale architecture looks like (e.g., realistic EKS node counts, real logging architecture patterns) — expressing concern that ten days into the program, they hadn’t yet seen a true production-representative architecture, and that the playground environment (a couple of small EC2 instances) doesn’t resemble real production scale.
Instructor’s response:
- Clarified that the playground’s purpose is concept practice and interaction, not full production-scale replication — it’s intentionally not meant to be a 1:1 scale replica.
- Pointed to the project call track (covering real client engagements like HealthCorp, SecureAsset, and FintechPlatform) as where genuinely realistic, full-scale production architecture will be covered — including the actual infrastructure diagrams and multi-region setups used in those real engagements.
- Specifically referenced that the war-room outage scenarios are being built on realistic multi-region infrastructure (three regions, as shown live) reflecting real organizational patterns, not just a single flat VM.
- Offered a 1:1 conversation to this participant to discuss their specific expectations further, rather than resolving the full scope of the concern in the group call — an appropriate way to handle a concern that’s more about individual expectations/pacing than a universal curriculum gap.
4. Key Concepts Table
| Concept | Explanation | Example | Why It Matters |
|---|---|---|---|
| Layered diagnostic sequence (infra → OS → DNS → connections → kernel → services) | A consistent order for triaging an ambiguous connectivity/performance issue, moving from “outside the machine” toward “inside the machine” | Checking AWS security groups before touching iptables, before touching system resources | Prevents wasted effort debugging the wrong layer; each step either confirms or rules out an entire category of cause |
| Local resolver vs. upstream DNS isolation | Querying a public DNS server directly (bypassing local resolver config) to isolate whether a DNS problem is local-daemon-specific or broader | dig against 8.8.8.8 directly vs. dig using default local config | A precise, two-query test that immediately narrows the fault domain for DNS-flavored issues |
ss -s connection summary as a retry-storm indicator | A high or rising count of established connections signals clients/system opening more connections than are being closed | Rising established-connection count correlating with an intermittent DNS resolver | Retry storms are a common amplifier of small issues into full outages (echoing the Slack case study) — this is how you’d actually spot one happening live |
| Kernel “blocked”/uninterruptible-sleep process state | Processes stuck in a state the kernel can’t normally interrupt, often due to I/O contention, can choke overall system responsiveness | Checked via process-state inspection + kernel log review | A kernel-level bottleneck can exist even when CPU/memory monitoring looks completely normal |
| Timed service health check (response-time thresholds) | Quantifying service health by timing its response to a status query against rough thresholds | systemctl status sshd timed at <0.5s (healthy) vs. 5-10s+ (choking) | Converts a vague “seems slow” judgment into a repeatable, comparable measurement — useful both for diagnosis and for confirming a fix worked |
| Shared-environment troubleshooting coordination hazard | When multiple people work on the same shared system, one person’s unannounced fix can silently invalidate someone else’s ongoing diagnosis | A participant couldn’t reproduce a symptom because someone else had already stopped the disruptive scripts | A realistic illustration of why communication discipline (announcing actions taken) matters during team-based incident response |
| Scope-of-impact test (single target vs. all targets unreachable) | Testing connectivity against multiple destinations to distinguish a target-specific issue from a general network issue | Testing reachability to several external hosts, not just one | Quickly separates “something’s wrong with this one destination/app” from “something’s wrong with my network path in general” |
5. Architecture & Workflow Analysis
5.1 Full Structured Debugging Sequence (as demonstrated)
1. Establish exact symptom combination
(what specifically fails: dig? ping? curl? all? some?)
|
v
2. Rule out infra-level blocking
(AWS VPC / Security Groups / Firewall)
|
v
3. Check iptables (host-level firewall rules)
|
v
4. Determine scope
(single target unreachable vs. ALL targets unreachable)
|
v
5. Check system resources
(top/htop for CPU hogs, free -h for memory, uptime for load avg)
|
v
6. DNS-specific sub-check (see 5.2 below)
|
v
7. Check connection state
(ss -s: is established-connection count high/rising? -> retry storm signal)
|
v
8. Check kernel-level blocking
(uninterruptible/"D"-state processes + kernel log review)
|
v
9. Check system service health
(timed status checks vs. thresholds; daemon-reload / restart as needed)
|
v
10. Document -> Hypothesize -> Fix -> RE-VERIFY
5.2 DNS Isolation Sub-Procedure
dig google.com (using LOCAL resolver config)
|
v
-------------------------------
| |
Works fine Fails / slow / intermittent
| |
DNS not the v
current issue dig @8.8.8.8 google.com (bypass local resolver)
|
-------------------------------
| |
Also fails/slow Works fine
| |
Broader network/DNS Problem is ISOLATED to the
infra issue LOCAL RESOLVER DAEMON
(investigate further (systemd-resolved)
upstream) |
v
Restart systemd-resolved
Re-test to confirm fix
5.3 Retry Storm Detection via Connection State
DNS resolver intermittently slow/failing
|
v
Clients/system automatically retry requests
|
v
More connections opened than are being closed
|
v
ss -s shows RISING established-connection count
|
v
If unaddressed: FD/port exhaustion --> system-wide failure
|
v
DETECTION POINT: monitor `ss -s` trend, not just a single snapshot
5.4 Shared-Environment Coordination Failure (What Happened Mid-Session)
Instructor demonstrates live debugging on shared Core Ops VM
|
v
------------------------------------------
| |
Participant A follows along, Participant B (unseen,
symptoms reproduce as expected uncoordinated) independently
finds + stops the disruptive
| background scripts
v |
Participant C tries to reproduce v
the SAME symptoms --> CANNOT Environment now "clean" for
(confused, shares screenshots) everyone sharing that VM
| |
-----------------------------------------------
|
v
Instructor traces the discrepancy,
identifies the uncoordinated fix,
shares the actual scripts so the
scenario can be intentionally re-run
6. Commands & Configurations
| Command / Config | Purpose | Explanation |
|---|---|---|
dig google.com | Test DNS resolution using the local/default resolver | First DNS diagnostic step |
dig @8.8.8.8 google.com (pattern; exact syntax as commonly used) | Test DNS resolution against a public resolver directly, bypassing local config | Isolates whether a DNS problem is local-daemon-specific vs. broader — the key diagnostic move that led to identifying the systemd-resolved issue in this session |
ping <host> | Basic ICMP reachability test | First-pass check; used alongside curl/nc to distinguish protocol-specific vs. general connectivity issues |
curl <host> | HTTP/TCP-level connectivity test | Complements ping — divergence between the two points at protocol/port-specific blocking |
nc <host> <port> (netcat) | Test raw TCP handshake to a specific host/port | Introduced in this session as an additional diagnostic layer: if ping+dig work but nc fails, suspect local firewall/socket exhaustion rather than DNS or general network issues |
top / htop | Identify CPU-hogging or stuck processes | Core system-resource diagnostic step |
free -h (implied — “checking memory, free side”) | Check available/used memory in human-readable form | Used to confirm whether memory pressure is a contributing factor |
uptime | Show system load averages (1/5/15-minute) | Quick, high-level signal of overall system load/pressure |
ss -s | Socket statistics summary — TCP/UDP connection counts by state | Used to detect retry storms / connection leaks via a rising established-connection count |
Process-state check for “blocked”/uninterruptible-sleep processes (pattern, e.g. via ps filtering on state D) | Identify kernel-level I/O-blocking bottlenecks | Part of the kernel-health diagnostic step |
journalctl (last ~20–50 lines) | Review recent kernel/system logs for related errors | Cross-referenced against the blocked-process check |
systemctl status sshd (timed) | Check and time the responsiveness of a specific system service | Central to the quantified service-health methodology (see Section 3.6 thresholds) |
systemctl daemon-reload | Reload the systemd daemon manager’s configuration | Used as a corrective step when a service’s status check is slow/unresponsive |
Restarting systemd-resolved (implied — “restart that” referring to the resolver daemon) | Fix a local-DNS-resolver-specific issue | The actual, confirmed fix applied in this session’s live demo once local-vs-upstream DNS isolation pointed at the local resolver daemon specifically |
7. Tools & Technologies
dig
- Purpose: DNS query and diagnostic tool.
- When to use it: Both for basic resolution testing and, critically, for the local-vs-upstream isolation technique demonstrated in this session (querying a public resolver directly to bypass local resolver configuration).
nc (netcat)
- Purpose: Low-level TCP/UDP connection testing tool.
- When to use it: Newly introduced in this session as a diagnostic layer between
ping(ICMP) andcurl(application-level HTTP) — testing whether a raw TCP handshake to a specific port succeeds, useful for isolating local firewall or socket-exhaustion issues from DNS or general network issues.
ss
- Purpose: Socket statistics tool.
- When to use it: Checking current connection counts/states — specifically used here (
ss -s) to detect retry-storm-style connection buildup.
systemd-resolved
- Purpose: The systemd-managed local DNS resolver service on many modern Linux distributions.
- When to use it / relevance: This session’s actual confirmed root cause for the local DNS failures — worth knowing as a specific, commonly-implicated service whenever local DNS resolution fails but direct queries to a public resolver succeed.
journalctl
- Purpose: Query and view systemd’s centralized logging (the systemd journal).
- When to use it: Reviewing recent kernel/system logs as part of the kernel-health diagnostic step.
systemctl
- Purpose: Control and query systemd-managed services and the systemd daemon itself.
- When to use it: Both for checking/timing individual service health (
systemctl status <service>) and for broader daemon-manager reloads (systemctl daemon-reload) when service responsiveness degrades.
8. Real-World Production Usage
- The local-vs-upstream DNS isolation technique is a genuinely standard, high-value real-world debugging move — it’s a two-command test that immediately tells you whether to focus your investigation on the local machine’s resolver configuration/daemon or escalate to a broader network/DNS-infrastructure investigation, saving significant time versus guessing.
- Timing service health checks against concrete thresholds, rather than relying on a binary “is it up” check, is a mature operational practice — it lets an engineer quantify degradation (“it’s responding, but slowly, and getting worse”) rather than only being able to say “it’s up” or “it’s down,” which is often too coarse-grained to be useful during an active incident.
- The
ss -sconnection-count check as a retry-storm early-warning signal is directly applicable to the Slack-style cascading-failure pattern discussed in the prior session — this session shows the actual, concrete command you’d run to catch that pattern happening in real time, rather than just discussing it conceptually. - The shared-troubleshooting-environment coordination failure that occurred mid-session is a completely realistic incident-response scenario, not a training artifact — in real production incidents involving multiple responders on shared systems, exactly this kind of silent, uncoordinated action (someone quietly “fixing” something without telling the team) genuinely happens and genuinely derails parallel diagnostic efforts. Recognizing and quickly tracing this kind of discrepancy (rather than assuming your own diagnostic process is broken) is itself a real skill.
- Distinguishing “playground for concept practice” from “realistic production-scale architecture” is a legitimate pedagogical distinction that real training programs have to manage — the participant’s concern (wanting to see genuine production-scale infrastructure, not just a two-VM lab) reflects a real tension in any hands-on technical training: labs need to be small enough to be tractable and reproducible, but learners also need exposure to what real scale actually looks like, which is why this program explicitly separates the “playground” track from the “project call” track (real client architectures).
9. Interview Preparation
Beginner Questions
Q1: How would you determine whether a DNS resolution failure is caused by your local machine’s resolver or by a broader network/DNS problem?
A: Run a DNS query using your system’s default (local) resolver configuration, and separately run the same query directly against a known public DNS resolver (like 8.8.8.8), bypassing your local resolver configuration entirely. If the local query fails or is slow but the direct query to the public resolver succeeds, the problem is isolated to your local resolver daemon (e.g., systemd-resolved) — restart it. If both fail, the problem is broader than your local machine.
Q2: What does a high or rising number of established connections (as shown by ss -s) typically indicate?
A: It’s a signal that connections are being opened faster than they’re being closed — often caused by a retry storm, where clients repeatedly retry failed requests (e.g., due to a slow or intermittently-failing dependency like DNS), each retry opening a new connection. Left unaddressed, this can exhaust available file descriptors or ports and take the system down entirely, even if the original triggering issue was relatively minor.
Q3: What’s the difference between what ping, curl, and nc each actually test, and why would you use all three?
A: ping tests basic ICMP-level reachability, curl tests full application-level HTTP/HTTPS connectivity, and nc (netcat) tests a raw TCP handshake to a specific host and port without any application-layer protocol involved. Using all three together lets you narrow down exactly which layer is failing: if ping works but nc fails, the problem is likely at the TCP/firewall/socket level; if ping and nc both work but curl fails, the problem is more likely at the application/HTTP layer.
Intermediate Questions
Q4: Walk through a structured approach to diagnosing an ambiguous “intermittent connectivity issue” on a Linux server, from the outside in.
A: Start outside the machine and work inward: first rule out cloud-infrastructure-level blocking (security groups, VPC/firewall rules), then check the host’s own firewall rules (iptables). Next, determine scope by testing multiple destinations — is only one target affected, or is everything external unreachable? Then check core system resources (CPU via top, memory via free, load average via uptime). Run the DNS local-vs-upstream isolation test. Check connection state (ss -s) for retry-storm signals. Check for kernel-level blocking (stuck/uninterruptible processes plus kernel logs). Finally, check the responsiveness of key system services with timed status checks, restarting/reloading as needed. Document every command and result along the way so you can form and test a specific hypothesis rather than guessing randomly.
Q5: A service status check that normally responds instantly is now taking 8 seconds to respond. What would you conclude, and what would you do next? A: Based on the thresholds demonstrated in this session (under ~0.5s = healthy, ~1–3s = under pressure, ~5–10s+ or hanging = choking), an 8-second response time indicates the system/service is significantly choking — likely under CPU, I/O, or thread-pool pressure. The next step would be to reload the systemd daemon manager and/or restart the specific affected service, then re-time the same status check to confirm whether the intervention actually improved responsiveness, rather than assuming a restart fixed it without verifying.
Q6: In a team-based incident response scenario, how would you avoid the kind of coordination failure seen in this session (where one responder’s unannounced fix invalidated another responder’s ongoing diagnosis)? A: Establish a clear communication discipline during any live incident: anyone who takes an action that changes system state (stopping a process, restarting a service, modifying a config) should announce it immediately in the shared incident channel, ideally before acting if time allows, and always immediately after if not. Maintaining a running incident timeline/log (even informally, in a shared chat) that captures who did what and when prevents exactly this kind of confusion, where one person’s fix silently changes the ground truth that everyone else is diagnosing against.
Advanced Questions
Q7: Why is it important to test DNS resolution against multiple targets/methods (local resolver vs. direct upstream query) rather than a single dig command, especially in a production incident?
A: A single dig command using default local configuration conflates two very different possible fault domains: the local resolver daemon/configuration, and the broader DNS/network infrastructure your queries ultimately depend on. If you only run the default query and it fails, you don’t yet know whether to focus your remediation effort on your own machine (fast, low-risk, narrow blast radius) or escalate to a network/DNS-infrastructure investigation (potentially much broader impact, more stakeholders, more time). The two-query isolation technique (local resolver vs. direct upstream) is a small additional cost that dramatically narrows your remediation scope and prevents wasted effort or unnecessary escalation.
Q8: How would you design monitoring/alerting to catch a retry-storm pattern (like the one discussed in this session) before it causes a full outage, rather than discovering it only during manual ss -s inspection?
A: Rather than relying on manual, point-in-time inspection during an active incident, you’d want continuous monitoring of established-connection counts (and their rate of change) as a first-class metric, with alerting thresholds tuned to your system’s normal baseline — a sudden, sustained climb in established connections (rather than just a high absolute number, which might be normal for a busy system) is the more actionable signal. Pairing this with monitoring on the specific dependency most likely to trigger retries (in this session’s case, DNS resolution latency/failure rate) lets you catch the triggering condition (e.g., DNS starting to degrade) before it fully cascades into the consequence (a connection storm), giving responders a meaningfully earlier warning window than waiting for the full cascade to manifest as a general “system is slow” symptom.
Q9: A junior engineer on your team says a system issue “just fixed itself” during a live troubleshooting session, and they can no longer reproduce a bug you were actively diagnosing together. What questions would you ask before concluding the issue is actually resolved? A: Before accepting “it fixed itself,” I’d ask: did anyone (on my team or with shared access to this system) take any action recently — restarting a service, killing a process, changing a configuration — that might explain this, even if they didn’t think it was relevant enough to mention? Is this a shared/multi-tenant environment where someone else’s unrelated action could have coincidentally resolved (or masked) the symptom? Systems don’t typically “just fix themselves” — an unexplained resolution is much more likely to indicate an uncoordinated human action (as happened in this session, where a teammate silently stopped the disruptive background scripts) or an unrelated external factor than a genuine self-resolution, and treating it as a mystery to be traced (not just a lucky break) is the more disciplined response — especially before declaring an incident closed.
10. Exam & Certification Notes
(Relevant to Linux system administration certifications — RHCSA, LPIC, CompTIA Linux+ — and the networking/troubleshooting sections of cloud/DevOps certifications.)
systemd-resolvedas the local DNS resolver on modern Linux distributions: Know that many modern distros (particularly Ubuntu-based systems, as used in this session) delegate local DNS resolution tosystemd-resolved, and that this service can independently fail or misbehave even when the underlying network and upstream DNS infrastructure are completely healthy — a frequently tested nuance distinguishing “DNS is broken” (vague) from “the local resolver daemon specifically needs attention” (precise).ssas the modern replacement fornetstat: Know thatssis the current standard tool for socket/connection inspection on Linux, generally faster and providing more detail than the oldernetstat, which is commonly deprecated on modern distributions.- Process states, specifically the uninterruptible sleep (“D”) state: A commonly tested Linux process-state concept — a process in “D” state is waiting on I/O and cannot be interrupted (not even by
SIGKILL) until the I/O operation completes, making a buildup of “D”-state processes a specific, diagnosable sign of I/O-level system distress. systemctl daemon-reloadvs. restarting an individual service: Know the distinction —daemon-reloadre-reads unit file configurations without necessarily restarting running services, whereas restarting a specific service (systemctl restart <service>) actually stops and starts that service’s running process. Exams sometimes test which is the appropriate action for a given scenario (e.g., after editing a unit file vs. after a service becomes unresponsive).- Load average (
uptime) interpretation: Know that the three load-average figures represent 1-minute, 5-minute, and 15-minute averages, and that interpreting a “high” load average requires knowing the number of CPU cores on the system (a load average that’s fine on an 8-core system may indicate severe overload on a 2-core system) — a commonly tested practical nuance.
11. Cheat Sheet
Structured Debugging Order (memorize the sequence):
- Confirm exact symptom combination
- Cloud infra (VPC/Security Groups)
- Host firewall (iptables)
- Scope test (single target vs. all targets)
- System resources (
top,free -h,uptime) - DNS isolation (local vs.
dig @8.8.8.8) - Connection state (
ss -s) - Kernel-level blocking (D-state processes +
journalctl) - Service health (timed
systemctl status) - Document → Hypothesize → Fix → Re-verify
DNS Isolation — Two Commands Tell You Everything:
dig google.com # local resolver
dig @8.8.8.8 google.com # direct upstream, bypasses local resolver
- Local fails, upstream works → restart
systemd-resolved - Both fail → broader network/DNS issue, escalate investigation
ping / nc / curl — Layered Protocol Test:
| Test | Protocol/Layer | If it fails alone |
|---|---|---|
ping | ICMP | Basic reachability blocked |
nc <host> <port> | Raw TCP handshake | Local firewall / socket exhaustion likely |
curl <host> | Application (HTTP) | Application-layer issue, network path OK |
Service Health Response-Time Thresholds:
| Response Time | Status |
|---|---|
| < 0.5s | Healthy |
| 1–3s | Under pressure |
| 5–10s+ / hangs | Choking — reload/restart |
Retry Storm Early-Warning Signal:
ss -s established-connection count rising over time (not just high in absolute terms)
Shared-Environment Discipline Rule: Always announce state-changing actions (stopping scripts, restarting services) to the team immediately — silent fixes invalidate others’ in-progress diagnosis.
12. Gaps & Assumptions
- Exact command syntax for several steps was described conceptually rather than typed verbatim in the transcript — e.g., the specific process-state filtering command for detecting “blocked”/D-state processes, and the exact
free/memory-check invocation. This document presents these using standard, conventional Linux command syntax consistent with what was described, but exact flags should be verified directly (e.g., against the instructor’s shared scripts) before being treated as verbatim transcript content. - The specific script files (CPU, thread-pool,
resolv.conf, FD-exhaustion) were shared via chat/drive during the session but their actual code content is not part of this transcript — this document describes their purpose and effect as narrated live, not their literal implementation. Refer to the actual shared script files in the drive for exact implementation details. - Advanced Track assignments #3 and #4 were only planned, not executed, within this session — the control-plane/API-server scenario described in Section 3.8 is a preview of what’s planned for a follow-up session, not something demonstrated in this transcript. Treat this document’s coverage of that scenario as forward-looking context only.
- The exact day/time of the promised Advanced Track follow-up session was left somewhat ambiguous in the live discussion (a “tomorrow if available, otherwise Thursday” framing) — this document does not assert a firm date, consistent with the actual uncertainty expressed live.
- The participant’s production-architecture concern (Section 3.9) was deferred to a 1:1 conversation not captured in this transcript — this document reports the concern and the instructor’s framing-level response accurately, but does not include the resolution of that individual conversation, since it wasn’t part of this call.
- This document consolidates a session that included a real mid-call diagnostic confusion (Section 3.7) — presented here in a clarified, reorganized form for teaching clarity, while preserving the actual sequence of confusion → investigation → resolution as it occurred, since the process itself (not just the eventual explanation) is the valuable part of that exchange.