The Bastion Mirage

Intermittent SSH Failures Across the Bastion → Production Hop

SEV-1 Linux & Networking ~45m

The situation you’re stepping into

It’s Friday evening, 19:07 IST. You’re on the platform on-call rotation. A P1 bridge just went live and you’ve been pulled in.

The environment is deliberately simple, which is exactly what makes this hard:

One bastion host — 2 vCPU / 4 GB RAM, a public IP, and the only host allowed to accept SSH from outside. Roughly 30 engineers (a mix of data-science, DevOps, and SRE) reach production through it.
One production server — 4 vCPU / 8 GB RAM, Ubuntu 22.04 LTS, no external IP. It is reachable only by hopping through the bastion, and it hosts the payment-analytics backend.
The two live in the same VPC: bastion subnet 10.20.0.0/24, production subnet 10.20.1.0/24 (cloud region asia-south1).

🔥 Stakes

Friday is the peak reporting day. The analytics backend pulls transactions from multiple regions and feeds finance + data-science dashboards. If it stalls, fraud-detection SLAs breach and there’s a finance escalation within the hour. If engineers also can’t SSH in to fix it, remediation itself becomes impossible — a classic access-outage trap.

The pre–war-room noise

The bridge opens in chaos. The reports contradict each other:

5–6 engineers: “Timeout on SSH — I can’t even connect to prod.”
7–8 engineers: “I can connect, but it’s painfully slow. Running ls takes 30 seconds.”
The rest: “Works fine for me. Maybe they’re on bad Wi-Fi?”

One fact is consistent across everyone: every engineer can reach the bastion. The failure only appears on the next hop — bastion → prod. The Incident Charter is read aloud: IC has final say, no cowboy fixes without approval, timeline updates every 15 minutes, a scribe logs every action.

Decision Point 1

Some users fail, some are slow, some are fine — and the 'bad Wi-Fi' theory is already spreading. Where do you point the investigation: the users' laptops, the network path, or the production host?

Reason from blast radius before you touch a single command. What do all three symptom groups share, and what do they NOT share?

Commit to your answer, then reveal the responder’s move →

✓What the responder did & why

The client-laptop theory is a distractor, and chasing it is how this incident eats an hour. Blast-radius reasoning kills it fast:

Everyone can reach the bastion → the internet path and the users’ laptops are fine up to the bastion.
The failure is only on the shared bastion → prod hop, which every user funnels through identically.
A problem that is shared by the next hop but variable per connection points at the destination host (prod) and the path to it — not at 30 independent Wi-Fi connections.

So you scope the investigation to the prod host and the bastion→prod network path, and you explicitly park the “bad Wi-Fi” theory to stop the tunnel vision.

🧭 Why the variability is the clue, not the noise

Intermittent + rotating membership (“different people each time”) implies a variable, per-connection cost — latency or loss crossing a threshold — rather than a static, all-or-nothing misconfiguration. A hard outage would fail everyone; a config typo would fail the same people every time. This signature says: multiple, probabilistic faults.

Working the layers

You SSH from the bastion to prod with maximum verbosity and watch where it breaks — three times, because the failures don’t reproduce consistently:

ssh -vvv analytics@10.20.1.10

Attempt A hangs after expecting SSH2_MSG_KEX_ECDH_REPLY, then Connection timed out.
Attempt B connects, but every keystroke and ls lags for tens of seconds.
Attempt C gets to auth and returns Permission denied (publickey) — for a user whose key worked an hour ago.

Decision Point 2

Three different failure signatures from the same command against the same host. Is this one root cause presenting three ways, or three different faults? What OSI layer does each signature implicate?

Map each signature to a layer: a timeout mid-handshake, a laggy-but-working shell, and an auth rejection are not the same failure.

Commit to your answer, then reveal the responder’s move →

✓What the responder did & why

These are three independent faults, not one — which is the entire trap of this incident:

Timeout mid-handshake (Attempt A) → transport/network (L3–L4). Packets are being dropped; TCP retransmits until the handshake gives up.
Laggy shell (Attempt B) → path latency and/or host saturation (L3 delay or L7 CPU/IO starvation). The connection completes but every round-trip and every command is slow.
Auth rejection (Attempt C) → authentication (L7). The key material or its acceptance is intermittently broken.

The lesson lands here: because membership rotates, engineers average the three signatures into one vague “prod is flaky” and start guessing. Separating them by layer is what turns a mirage into a checklist.

📟 Reinforce · Interactive War Room Production SSH/Bastion Outage Debugging → The interactive War Room for this exact failure class — walk the isolate → stabilize → fix → prevent sequence hands-on. ⌨️ Reinforce · Command Reference Linux Network Diagnosis Cheat Sheet → The port-state, packet-drop, latency and routing commands you're about to run, in one reference.

Layer 3–4: confirm the loss and the latency

On the prod host you quantify the network behavior instead of eyeballing it:

# From bastion → prod: is the path lossy?
mtr -rwzbc 100 10.20.1.10        # ~20% packet loss to the destination
ping -c 50 10.20.1.10            # scattered losses + inflated RTT

# On prod: what's actually happening to inbound SSH packets?
sudo iptables -L INPUT -n -v     # a statistic/random DROP rule on dport 22
sudo tc qdisc show dev eth0      # netem qdisc injecting delay

Two smoking guns surface:

# iptables INPUT chain
DROP  tcp -- 0.0.0.0/0  0.0.0.0/0  tcp dpt:22 statistic mode random probability 0.20

# tc qdisc on eth0
qdisc netem 8001: root refcnt 2 limit 1000 delay 300.0ms 200.0ms

An iptables rule is randomly dropping ~20% of inbound port-22 packets, and a tc netem qdisc is adding 300 ms ± 200 ms of delay to everything leaving eth0. That fully explains Attempt A (drops → timeouts) and part of Attempt B (delay → lag).

Decision Point 3

The shell is laggy even on connections that DON'T time out — and part of that lag survives after you account for the 300 ms netem delay. What else on the host could be starving interactive sessions, and how do you confirm it?

Latency ≠ the only cause of a slow shell. Think host resources: CPU run-queue and I/O wait.

Commit to your answer, then reveal the responder’s move →

✓What the responder did & why

Network delay is only half the slowness. You check host load:

uptime                      # load average far above core count
top -b -n1 | head -20       # CPU pegged; high %wa (I/O wait)
vmstat 1 5                  # r column deep, high wa
ps -eo pid,pcpu,comm --sort=-pcpu | head

You find stress-ng hammering the box:

stress-ng --cpu 4 --io 2 --timeout 600s

Four CPU workers on a 4 vCPU box plus two I/O workers means every interactive session is fighting for a scheduler slot. That’s the residual lag on connections that already survived the packet loss and the netem delay.

Layer 7: the intermittent auth denials

That leaves Attempt C. A subset of users get Permission denied (publickey) — but only sometimes, and their keys demonstrably worked earlier.

Decision Point 4

A user's SSH key worked an hour ago and is now intermittently rejected, then works again. Where do you look, and what would make a valid key 'flap' like this?

Auth is deterministic per key — so if acceptance flaps, something is mutating the server-side inputs on a schedule.

Commit to your answer, then reveal the responder’s move →

✓What the responder did & why

You go to the server-side auth trail and the key file:

sudo journalctl -u ssh --since "-30min" | grep -Ei 'refused|denied|authentication'
sudo ls -la /home/analytics/.ssh/authorized_keys
sudo cat /home/analytics/.ssh/authorized_keys

The auth log shows rejections clustered on a schedule, and the authorized_keys file is shorter than it should be — some engineers’ public keys are missing. Checking scheduled tasks:

sudo crontab -l ; ls -la /etc/cron.*/ /etc/cron.d/

reveals a cron job that overwrites authorized_keys with a partial file every 10 minutes. So for a rolling subset of users, their key vanishes for a window, then reappears on the next legitimate deploy — auth “flap” by design.

📟 Reinforce · Interactive War Room Full OSI-Layer SSH Freeze Walkthrough → A line-by-line ssh -vvv verbose-log read across all seven layers — the deep-dive companion to this reveal.

Root cause

There was never a single fault. Four independent faults were injected on the production host at once, all landing on the bastion → prod hop:

Packet loss — an iptables rule randomly dropping ~20% of inbound port-22 traffic (→ connection timeouts).
Network latency — a tc netem qdisc adding 300 ms ± 200 ms of egress delay (→ laggy shells).
Host saturation — stress-ng pinning all 4 vCPUs and injecting I/O wait (→ residual slowness even without loss).
Auth corruption — a cron job periodically truncating authorized_keys (→ intermittent publickey denials).

Overlaid, these produce the exact “mirage”: failures that rotate across users and refuse to reproduce, because each connection rolls the dice against a different combination of the four.

Containment (charter order, no cowboy fixes)

With IC approval, you clear the faults one layer at a time and verify after each — never batching, so you can attribute the recovery:

# 1. Stop the packet loss
sudo iptables -D INPUT -p tcp --dport 22 -m statistic --mode random --probability 0.20 -j DROP
# 2. Remove the injected latency
sudo tc qdisc del dev eth0 root
# 3. Kill the resource storm
sudo pkill -f stress-ng
# 4. Remove the cron and restore the real key file
sudo rm -f /etc/cron.d/<offending-job>
sudo cp /home/analytics/.ssh/authorized_keys.bak /home/analytics/.ssh/authorized_keys
sudo chmod 600 /home/analytics/.ssh/authorized_keys && sudo chown analytics:analytics /home/analytics/.ssh/authorized_keys

After each step you re-run mtr, a timed ssh + ls, and an auth attempt, confirming which symptom class each fix retired. Only when all four are green do you close containment and start the RCA.

🎯 Transferable principle

A non-reproducible incident is often several reproducible ones stacked. Three moves make it tractable: (1) use blast radius to localize before touching commands — a symptom shared by the next hop but variable per connection points at the destination, not the clients; (2) sweep the OSI layers physical → logical and confirm each layer independently rather than averaging symptoms into one theory; (3) clear one fault at a time and verify, so recovery is attributable and you don’t credit the wrong fix. The “obvious single cause” is the mirage.

Telling this story to a recruiter

The 30-second version:

“On a peak reporting Friday, thirty engineers started losing SSH access to a production host that fed our payment-analytics pipeline — but only some of them, only some of the time, and the failures wouldn’t reproduce. I led the diagnosis, proved it wasn’t one fault but four independent ones stacked on the same host, cleared them one at a time with verification after each, and restored stable access before the finance SLA breached. Then I put monitoring and config-management guards in place so no single host could silently accumulate that much damage again.”

The detailed telling:

Situation. Friday evening, peak reporting window. Our production analytics host sat behind a single bastion — the only entry point for about thirty engineers. A P1 was declared because SSH to production had become unreliable: five or six people couldn’t connect at all, seven or eight could connect but every command took thirty seconds, and the rest saw nothing wrong. The backend it hosted fed fraud-detection dashboards with a one-hour SLA, so if we couldn’t get engineers onto that box, we couldn’t protect the SLA either.

Task. As the responder, my job was to restore stable, predictable SSH access — and to do it inside an incident-command structure: no unapproved fixes, timeline updates every fifteen minutes, every action documented.

Action. The room was already converging on “bad Wi-Fi” theories, so my first move was blast-radius reasoning, not commands: everyone could reach the bastion, and the failure only lived on the shared bastion-to-prod hop — which pointed at the destination host, not thirty laptops. Then I ran the hop repeatedly with verbose SSH and got three different failure signatures: mid-handshake timeouts, laggy-but-working shells, and intermittent key rejections. That told me I was probably looking at multiple faults, so I swept the layers in order instead of chasing one theory. At the network layer, mtr showed ~20% packet loss and the host’s own firewall had a rule randomly dropping a fifth of inbound SSH packets, with a traffic-shaping qdisc adding 300ms of jittery delay on top. At the host layer, load was pinned by a stress process eating all four cores. At the auth layer, the server’s authorized-keys file was being overwritten with a partial copy by a cron job every ten minutes — which explained why whose login failed kept rotating. With IC approval I removed each fault one at a time — firewall rule, qdisc, runaway process, cron — re-verifying loss, latency, and auth after every single change so the recovery was attributable.

Result. Full, stable access restored within the incident window; the analytics SLA never breached. The RCA documented four independent root causes, and the prevention items — config management ownership of firewall and resolver state, per-host alerting on load, packet loss, and auth failures — meant this class of “mirage” incident couldn’t silently rebuild itself.

What this story demonstrates. Calm scoping under leadership pressure, refusing to average contradictory symptoms into one convenient theory, disciplined layer-by-layer isolation, and change control during an incident — one fix at a time, verified, documented.

Interview deep-dive: the full case study

A longer, technical walkthrough for when an interviewer wants to go deep — mechanism, impact, resolution, outcomes, lessons, and prevention.

How the issue happened (the mechanism)

The environment had a single architectural pinch point: one bastion was the only route to a production host that had no public IP, and roughly thirty engineers all funneled through the same bastion-to-prod SSH hop. That shared hop became the amplifier — anything wrong on the prod side of it hit everyone, but hit them differently depending on timing.

What was actually wrong wasn’t one thing. Four faults had accumulated on the production host at the same time, each on a different layer of the stack:

Packet loss (L3/L4). A host firewall rule was dropping a random ~20% of inbound SSH packets (iptables ... -m statistic --mode random --probability 0.2 -j DROP). A dropped SYN or handshake packet forces TCP retransmits; enough of them and connect() times out. This is why some sessions never established.
Latency injection (L3). A traffic-control queueing discipline added 300ms ± 200ms of jittered egress delay (tc qdisc ... netem delay 300ms 200ms). Sessions that did establish paid this cost on every round trip, so interactive shells felt like wading through mud.
CPU/IO saturation (L7/host). A stress-ng --cpu 4 --io 2 process pinned all four cores and injected I/O wait, so even after the network delay, every keystroke fought the scheduler for a slice.
Auth corruption (L7). A cron job overwrote ~/.ssh/authorized_keys with a partial copy every ten minutes, so a rotating subset of engineers’ public keys vanished for a window and then reappeared.

The reason this presented as a “mirage” — non-reproducible, different victims each time — is combinatorial: every login attempt rolled the dice against a different combination of the four faults. One engineer hit packet loss and timed out; the next got through but landed in a laggy shell; a third was fine until the cron truncated their key. Because membership rotated, the room’s instinct was to average the symptoms into one vague “prod is flaky,” which is precisely the wrong mental model.

Impact

Users: ~30 engineers dependent on the hop; 5–6 fully unable to connect, 7–8 connected but effectively unusable (a ls taking 30 seconds), the rest unaffected — the uneven split is itself a diagnostic signature.
Service: the payment-analytics backend was still running, but engineers were blocked from operating or debugging it — an access outage, which is uniquely dangerous because it also blocks remediation.
Business: it was the peak reporting Friday; fraud-detection and finance dashboards depended on the backend, with a stated SLA-breach risk inside one hour and a finance escalation on the clock.
Second-order risk: if the last SSH paths had degraded fully, the host would have become unpatchable and unrecoverable — a latent path to a much larger outage.

Steps taken to resolve

Scoped before touching anything. Used blast-radius reasoning to kill the “bad Wi-Fi” theory: everyone reached the bastion, only the shared next hop failed, so the fault was destination-side, not client-side.
Separated the signatures. Ran ssh -vvv against prod repeatedly and catalogued three distinct failure modes (mid-handshake timeout, laggy shell, key rejection), which said “multiple faults” rather than “one flaky thing.”
Swept the layers, confirming each with data. mtr/ping quantified ~20% loss; sudo iptables -L INPUT -n -v exposed the random DROP rule; tc qdisc show dev eth0 exposed the netem delay; uptime/top/vmstat exposed the CPU/IO storm and named stress-ng; journalctl -u ssh plus the shrunken authorized_keys and a cron listing exposed the auth corruption.
Cleared faults one at a time, under IC approval, verifying after each — delete the iptables rule, tc qdisc del, pkill -f stress-ng, remove the cron and restore authorized_keys from backup with correct 600/ownership — re-testing loss, latency, and auth between every step so recovery was attributable.

Outcomes

Stable, predictable SSH restored within the incident window; the analytics SLA did not breach.
A clean RCA that named four independent root causes rather than a single hand-wave, with an attributable timeline of which fix retired which symptom.
The team’s mental model shifted from “prod is flaky” to “a non-reproducible incident is often several reproducible ones stacked.”

What we learned

A single shared dependency (the hop) turns any destination-side fault into a fleet-wide, variable-looking incident. Variability is a clue about per-connection cost, not noise to average away.
Symptom triage by OSI layer beats symptom averaging. Three different signatures almost always mean more than one fault.
Access paths are production systems too. Losing SSH doesn’t just degrade a service — it removes your ability to fix it, which should raise, not lower, the severity.

Prevention — what we changed so it won’t recur

Put host firewall and resolver/tc state under configuration management so ad-hoc iptables/tc/authorized_keys drift is detected and reverted automatically, not discovered during an incident.
Per-host alerting on load average, packet loss/retransmits, and SSH auth-failure rate — each of the four faults would have tripped a page long before users noticed.
Remove the single-bastion single point of failure (a second bastion / break-glass path) so an access outage can’t also block remediation.
A documented OSI-sweep runbook so the next responder isolates layer by layer instead of chasing the loudest symptom.