Four Faults, One Host

The situation you’re stepping into

A single production Linux host is “acting weird.” There’s no packet, no tidy incident summary — just a box that has become hostile to work on. Everything is slow, and the symptoms don’t obviously belong to the same problem:

• The shell itself is sluggish; top takes a beat to even redraw.
• Every command that resolves a hostname (curl, apt, ping api.internal) crawls or times out.
• Application processes start logging Too many open files and refusing new connections.

uptime                                   # load average far above the core count
top -b -n1 | head -25                    # run queue deep, CPU saturated
ps -eo pid,ppid,pcpu,nlwp,comm --sort=-pcpu | head
ps --ppid <suspect_pid> | wc -l          # how many children is it spawning?

# /opt/cpu_storm.sh  — launched via: nohup /opt/cpu_storm.sh &
#!/bin/bash
while true; do
  : &          # spawn a no-op background job, forever
done

cat /etc/resolv.conf
cat /etc/nsswitch.conf | grep hosts
getent hosts api.internal      # slow / fails
dig api.internal               # watch the query time and which server answers

nameserver 127.0.0.1           # nothing is listening on a local resolver -> refused/timeout
nameserver 8.8.8.8
options timeout:1 attempts:5 rotate
nameserver 10.255.255.1        # a black-hole IP: packets go nowhere

ps -eLf | grep '[p]ython' ; ps -o nlwp -p <python_pid>   # thousands of threads
ss -s                                                    # huge UDP/socket churn

# /opt/chaos_dns_threads.py
def worker():
    while True:
        try:
            socket.gethostbyname("nonexistent.internal.local")
        except:
            pass
# 5000 daemon threads, each hammering the resolver forever

ls /proc/<pid>/fd | wc -l           # per-process open FDs, climbing
cat /proc/<pid>/limits | grep 'open files'   # its soft/hard NOFILE limit
lsof -p <pid> | wc -l
ss -tanp | wc -l                    # total sockets; many ESTABLISHED to 127.0.0.1:80

# /opt/fdstorm/fd-leak.py
sockets = []
while True:
    try:
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        s.connect(("127.0.0.1", 80))
        sockets.append(s)          # kept referenced forever -> never garbage-collected/closed
    except Exception:
        pass
    time.sleep(0.001)

Root cause

Four independent faults were injected on the same host simultaneously, each imitating a different classic outage:

1. CPU storm — /opt/cpu_storm.sh infinitely forking : & background jobs, saturating cores and bloating the process table.
2. DNS thread storm — /opt/chaos_dns_threads.py spawning 5,000 threads that hammer the resolver with lookups for a non-existent name.
3. Resolver misconfiguration — /etc/resolv.conf rewritten with a dead 127.0.0.1, a black-hole 10.255.255.1, and timeout:1 attempts:5 rotate, plus an altered nsswitch.conf, so every lookup burns seconds of dead time.
4. File-descriptor leak — /opt/fdstorm/fd-leak.py opening sockets to 127.0.0.1:80 in a tight loop and never closing them, exhausting the process’s FD limit and starving the rest of the host.

Because they overlap, no single fix makes the host “feel fixed” — which is the entire teaching point.

Containment and prevention

# Kill each injected fault (one at a time, verifying after each):
sudo pkill -f cpu_storm.sh
sudo pkill -f chaos_dns_threads.py
sudo pkill -f fd-leak.py

# Restore DNS from the backup the setup left behind, then re-check resolution:
sudo cp /etc/resolv.conf.bak /etc/resolv.conf
sudo sed -i 's/^hosts:.*/hosts: files dns/' /etc/nsswitch.conf
getent hosts api.internal      # fast again

# Verify recovery across all four axes:
uptime ; ls /proc/<any_app_pid>/fd | wc -l ; dig +short api.internal ; ss -s

Prevention: cap blast radius with systemd/cgroup resource controls (CPUQuota, TasksMax, MemoryMax), raise and monitor LimitNOFILE with alerting on per-process FD growth, protect the resolver (validate resolv.conf via config management; deploy a local caching resolver correctly rather than pointing at a dead 127.0.0.1), and alert on load average, thread count, and socket count so each of these trips a page long before it compounds with the others.

Telling this story to a recruiter

The 30-second version:

“A production Linux host degraded in four unrelated ways at once — CPU saturation, crawling DNS, and processes dying with ‘too many open files.’ Instead of chasing whichever symptom screamed loudest, I swept the box one resource axis at a time and attributed every symptom to a specific process before killing anything. It turned out to be four independent faults: a fork storm, five thousand threads hammering the resolver, a sabotaged resolver config, and a socket leak. Fixing any one of them alone would have looked like a failure; fixing all four methodically restored the host in minutes.”

The detailed telling:

Situation. A production host went hostile: the shell itself lagged, anything that resolved a hostname crawled or timed out, and applications started refusing connections with “Too many open files.” The symptoms didn’t fit any single classic outage, which is exactly what makes this kind of incident dangerous — responders fix one thing, the box still feels broken, and confidence in every subsequent fix erodes.

Task. Restore the host to service, with a hard personal rule: no killing anything until each symptom was attributed to a specific process, so the recovery would be explainable rather than lucky.

Action. I ran a fixed sweep — CPU, then threads and memory, then DNS, then file descriptors, then services — attributing as I went. CPU first: load average was far beyond core count, and the process tree showed a shell script forking no-op background jobs in an infinite loop, a near fork-bomb pegging every core. I noted it and kept going, because a fork bomb doesn’t explain DNS or descriptor failures. DNS next, and it had two separate faults: the resolver config had been rewritten to route queries through a dead localhost resolver and a black-hole IP with rotate-and-retry options, burning seconds of dead time per lookup — and separately, a Python process was running five thousand threads, each resolving a nonexistent hostname in a tight loop, flooding what remained of the resolver path. Then descriptors: per-process counts from /proc showed one process’s open-file count climbing monotonically toward its limit — a script opening a socket every millisecond and holding every one in a list, never closing them, starving the rest of the host of descriptors and ports. Four faults, four different resource axes. I killed the three offending processes one at a time, restored the resolver config from its backup, and re-verified all four axes — load, threads, resolution time, descriptor counts — after each step.

Result. Host fully recovered in minutes with a clean, attributable timeline of what each fix restored. Prevention followed the same four axes: cgroup limits so no single process can fork or consume unbounded, per-process descriptor monitoring with alerts, and the resolver config placed under configuration management so it can’t drift silently.

What this story demonstrates. Discipline under confusing, compounding symptoms; refusing to declare victory after the first fix; fluency with Linux ground truth — /proc, process trees, resolver internals — and turning a chaotic recovery into a systematic, repeatable method.

Interview deep-dive: the full case study

How the issue happened (the mechanism)

A single production host was carrying four independent faults at once, each on a different resource axis — which is what made it feel like one baffling, un-fixable outage rather than four tractable ones:

1. CPU / process-table exhaustion. A shell script ran while true; do : & done — spawning a no-op background job every iteration, forever. This is a near fork-bomb: it pegs every core and inflates the process table, so even launching top competes for a scheduler slot.
2. Resolver misconfiguration (DNS config plane). /etc/resolv.conf had been rewritten to nameserver 127.0.0.1 (nothing listening → refused), a black-hole nameserver 10.255.255.1 (packets vanish), and options timeout:1 attempts:5 rotate. With rotate, every lookup round-robins into dead servers and burns timeout × attempts seconds of dead wait. nsswitch.conf was also flipped to an odd hosts: order.
3. DNS query flood (DNS load plane). A Python process ran 5,000 threads, each in a tight loop calling gethostbyname("nonexistent.internal.local") — saturating whatever resolver path survived fault #2 and the socket layer with it.
4. File-descriptor leak. A script opened a TCP socket to 127.0.0.1:80 every millisecond and appended each one to a list so it was never closed/GC’d. The process’s open-FD count climbed monotonically toward its NOFILE limit, and once exhausted, other processes on the host started failing accept()/connect() with EMFILE/ENFILE — “Too many open files.”

The trap is that these overlap and mask each other: fix the fork storm and DNS is still broken; fix DNS and apps still can’t open files. Killing one fault makes the host feel not fixed, which erodes confidence in every subsequent action and pushes responders toward random flailing.

Impact

• Whole-host degradation: the interactive shell itself lagged, every hostname-dependent command (curl, apt, service calls) crawled or timed out, and applications refused new connections with Too many open files.
• Production workloads on the box were dragged toward failure by compounding CPU starvation, resolution latency, and descriptor exhaustion simultaneously.
• High risk of misdiagnosis: the overlapping symptoms are exactly the conditions under which teams chase ghosts and apply fixes that don’t stick.

Steps taken to resolve

1. Swept resources in a fixed order — CPU → threads/memory → DNS → file descriptors → services — attributing each symptom to a specific process before killing anything.
2. CPU: uptime (load ≫ cores), top -b -n1, ps -eo pid,ppid,pcpu,nlwp --sort=-pcpu → found the forking cpu_storm.sh.
3. DNS (two sub-faults): cat /etc/resolv.conf + /etc/nsswitch.conf and getent hosts / dig (query time, which server answers) exposed the sabotaged config; ps -eLf / ps -o nlwp + ss -s exposed the 5,000-thread flood.
4. File descriptors: ls /proc/<pid>/fd | wc -l (climbing), cat /proc/<pid>/limits, lsof -p, ss -tanp → found the leaking socket loop.
5. Remediated one axis at a time, verifying after each: pkill each offending process, restore /etc/resolv.conf from its backup and fix nsswitch.conf, then re-check load, thread count, resolution time, and FD counts before declaring recovery.

Outcomes

• Host fully recovered in minutes, with a clean, attributable timeline of which fix restored which symptom — no guesswork, no “we restarted it and it went away.”
• The incident became a reusable structured-debugging method (resource-by-resource sweep) rather than a one-off war story.

What we learned

• When a whole host is sick, sweep resources in a fixed order and attribute every symptom to a process before acting — overlapping faults masquerade as one confusing outage.
• Don’t declare victory after the first fix. If the host still feels broken, that’s evidence of a second fault, not a failed fix.
• /proc is ground truth: /proc/<pid>/fd, /proc/<pid>/limits, process trees, and resolver files tell you exactly what’s happening without guessing.
• Config-plane and load-plane faults can coexist (broken resolv.conf and a query flood) — check both, don’t stop at the first DNS explanation.

Prevention — what we changed so it won’t recur

• cgroup / systemd resource controls: CPUQuota, TasksMax (kills fork storms), and MemoryMax so no single unit can consume the host unbounded.
• File-descriptor hygiene: raise and monitor LimitNOFILE, with alerting on per-process FD growth so a leak pages long before exhaustion.
• Protect the resolver: /etc/resolv.conf and nsswitch.conf under configuration management (no silent drift), and a correctly-configured local caching resolver instead of a dead 127.0.0.1.
• Baseline host alerting: load average, thread/process count, and socket count — each of the four faults would have tripped a distinct alert before they compounded into a single confusing outage.