Production Outage: OSI-Layer Troubleshooting of a Bastion SSH Freeze

SSH Protocol Deep-Dive, Verbose Log Analysis, and the Multi-Factor Root Cause

2. Table of Contents

1. Executive Summary
2. Table of Contents
3. Detailed Structured Notes
   • 3.1 Updated Scenario & Infrastructure Recap
   • 3.2 OSI-Layer Live Troubleshooting — Layers 1–4
   • 3.3 Deep-Dive: How SSH Actually Works (Whiteboard Walkthrough)
   • 3.4 SSH Verbose Log — Full Line-by-Line Analysis
   • 3.5 Layers 5–7 and Kernel-Level Checks
   • 3.6 Synthesizing the Hypothesis
   • 3.7 Where to Look for the Fix — Configuration Files Catalog
   • 3.8 Recovering Access When SSH Itself Is Blocked
   • 3.9 Assignment & RCA Requirement
   • 3.10 Q&A — Methodology Clarifications
4. Key Concepts Table
5. Architecture & Workflow Analysis
6. Commands & Configurations
7. Tools & Technologies
8. Real-World Production Usage
9. Interview Preparation (Beginner / Intermediate / Advanced)
10. Exam & Certification Notes
11. Cheat Sheet
12. Gaps & Assumptions

3. Detailed Structured Notes

3.1 Updated Scenario & Infrastructure Recap

• Infrastructure change from the original scenario: the production server is now in a private subnet (no public IP, no internet gateway path), while the bastion is in a public subnet with a public IP — both still in the same VPC. (In the original scenario two sessions earlier, both hosts were in a private subnet.)
• Symptom intensified: SSH from bastion to production now fails roughly 9 out of 10 attempts (up from the original intermittent pattern), making it more reproducible for a live teaching demo while preserving the same underlying nature of the problem.
• Instance sizing: bastion is a smaller instance (t3.small-class, ~4 vCPU per the engineer’s description) since it only serves as a jump host; production is a larger instance (t3.xlarge-class) since it hosts an actual application.
• Access setup: a shared PEM key (infra-key.pem), user ec2-user, key-based authentication only (no password).
• A deliberately obscured detail: the engineer mentioned a separate, dedicated user account was used to seed the actual fault into the system, with shell history disabled for that user — meaning participants can’t simply read command history to shortcut the discovery process; they have to actually diagnose it.

3.2 OSI-Layer Live Troubleshooting — Layers 1–4

The engineer explicitly frames this as structured, evidence-based troubleshooting — not guesswork — deliberately climbing the OSI stack from the bottom rather than jumping to an assumed cause (DNS, iptables, etc.).

Layer 1 (Physical) — basic reachability:

• Command: ping <production-IP> (5 packets sent).
• Result: 100% packet loss — zero packets received back.
• Interpretation: the destination doesn’t appear reachable and/or ICMP isn’t working — but this alone doesn’t tell you why; it’s evidence, not a conclusion.

Layer 2 (Data Link) — ARP resolution:

• Command: arp -n <production-IP>.
• Result: no entry returned — the system was unable to resolve the destination IP to a MAC address.
• Interpretation: neighbor discovery at Layer 2 is failing — a “severe sign,” per the engineer, but still not a conclusion to act on yet; keep climbing.

Layer 3 (Network) — path tracing:

• Command: tracepath <production-IP>.
• Result: fails at hop one — no reply from the very first hop, meaning traffic isn’t even leaving the local subnet as far as this tool can observe.
• Interpretation: narrows the suspect list toward VPC routing, NACLs, or kernel-level networking — but again, not yet conclusive.

Layer 4 (Transport) — TCP handshake test:

• Command: nc <production-IP> 22 (netcat, targeting the SSH port directly).
• Result: connection succeeds — the TCP three-way handshake completes.
• Critical interpretation, explicitly highlighted as an important pivot point: this single result contradicts the pessimistic picture built up from Layers 1–3 — it directly proves that security groups are allowing SSH, NACLs are allowing SSH, and the ENI is properly attached, ruling out an entire category of suspects that the failed ping/ARP/tracepath results might otherwise have pointed toward. This is presented as the key teaching moment of the whole session: don’t let an early, dramatic-looking failure (100% packet loss) cause you to lock in a conclusion — later evidence can and does contradict earlier assumptions, and the discipline is to keep collecting evidence across layers rather than stopping at the first alarming signal.

3.3 Deep-Dive: How SSH Actually Works (Whiteboard Walkthrough)

Before continuing into the verbose log, the engineer ran a dedicated whiteboard session explaining the SSH connection sequence in detail — explicitly framed as necessary background for correctly interpreting verbose log output (rather than treating it as unreadable noise).

Full sequence, as explained:

1. (Optional) Hostname resolution — if connecting via hostname rather than IP, the client first checks /etc/hosts, then falls back to a DNS query if not found locally. Connecting via raw IP (as in this exercise) skips this step entirely — a deliberate best practice to eliminate DNS as a variable when it’s not relevant.
2. TCP three-way handshake — standard SYN → SYN-ACK → ACK between bastion and production on port 22.
3. SSH version exchange — client and server each report their OpenSSH version; a version incompatibility here can also cause connection drops.
4. Algorithm exchange / key exchange negotiation — client and server negotiate a shared key-exchange algorithm (the engineer referenced a Diffie-Hellman-family algorithm by name, imperfectly recalled live — see Gaps & Assumptions). Both sides generate temporary keys, exchange public components, and independently derive the same shared secret without ever transmitting the secret itself over the wire — this is what makes the rest of the session’s traffic (including subsequent packet captures) encrypted.
5. Host authentication — the server presents its host public key; the client checks it against its local ~/.ssh/known_hosts file. A mismatch here throws a specific “host identification has changed” style error — this step exists specifically to prevent man-in-the-middle attacks.
6. User authentication — the server asks “who are you?” and authentication proceeds through a specific ordered sequence of methods: typically starting with GSSAPI/Kerberos-style authentication (skipped if not configured — not an error), then falling through to public key authentication (the method actually used in this scenario), and finally password authentication if enabled and reached.
7. PAM (Pluggable Authentication Modules) checks — session limits, MFA requirements, and other access-control policy checks happen here.
8. Environment/profile loading — configuration under /etc/profile.d/ and related directories is loaded.
9. Shell/session spawning — sshd forks a child process to host the user’s actual login shell; once this completes successfully, the terminal session becomes usable.

Common SSH failure-symptom-to-layer mapping, as given:

3.4 SSH Verbose Log — Full Line-by-Line Analysis

With the protocol sequence now explained, the engineer re-ran the connection with ssh -v (verbose mode) and walked through the actual log output line by line, mapping each section back to the sequence from Section 3.3:

1. Client configuration loading — reads local SSH client config; no errors here.
2. Algorithm negotiation lines — references to the key-exchange algorithm; normal, expected output, not an error (a participant’s earlier concern about a “not found” style message here was clarified as benign/expected).
3. Hostname resolution skipped — confirmed no DNS lookup occurred, since the connection used a raw IP.
4. TCP connection established on port 22 — confirms the three-way handshake succeeded (consistent with the earlier netcat test) — ruling out security group, NACL, and routing issues definitively, per the engineer.
5. Private key file found and read successfully — no “bad permissions” error, meaning the key file’s own permissions are correctly configured.
6. Host key verification against known_hosts succeeds — no “host identification has failed” error.
7. Authentication method negotiation — log shows the expected order (GSSAPI attempted, not configured/not an error, falls through to public key).
8. Public key authentication succeeds — no “permission denied (publickey)” error.
9. New SSH channel/session opened, shell allocated — a login banner/prompt appears in the log, indicating the session got this far successfully.
10. ⚠️ The critical finding: the log shows no further error anywhere. Key exchange: fine. Authentication: fine. Channel/session creation: fine. And yet the terminal itself never becomes usable — it just hangs.

The engineer’s synthesized interpretation of this log: SSH itself, as a protocol exchange, is 100% healthy — there is no misconfiguration visible anywhere in the network, key-exchange, or authentication layers. The actual failure is that the login shell is not being successfully spawned/handed off after a technically-successful SSH session establishment — i.e., the problem lives specifically in the shell/child-process creation step (step 9 in Section 3.3), not anywhere earlier in the sequence. This is described as “SSH is not stuck — the login shell is blocked after SSH.”

3.5 Layers 5–7 and Kernel-Level Checks

Having isolated the likely failure point to shell-spawning, the engineer performed additional checks to look for kernel-level resource pressure that could explain a shell failing to spawn:

• top — checked for general system/kernel load; no significant load observed.
• conntrack (connection tracking table) — checked for queue buildup or a retry-storm-style pattern (explicitly referencing the Uber case study from an earlier session as the pattern this check is designed to catch). Result: no unusual entries — no evidence of a connection-tracking-table-driven kernel bottleneck.
• Layer 6 (Presentation) — briefly checked system time/identity latency (timedatectl-style check); no issues found.
• Layer 7 (Application) — folded into the kernel-level checks above in this specific walkthrough.

3.6 Synthesizing the Hypothesis

Consolidated observation set, as summarized live by the engineer:

• ping: 100% packet loss.
• arp -n: no MAC address resolved.
• tracepath: fails at hop one.
• nc (TCP handshake to port 22): succeeds.
• SSH verbose log: key exchange, host auth, user auth, and channel creation all succeed — but shell spawning hangs.
• conntrack: no queue buildup.

The engineer’s synthesized characterization (explicitly drawing a comparison to a real, large-scale incident at Telegram as a precedent for this exact failure signature): the kernel’s networking stack is “alive enough to complete a TCP handshake, but effectively dead for general data forwarding.” This reconciles the seemingly contradictory evidence: low-level protocols that depend on broad network path health (ICMP, ARP, traceroute) fail, while a narrowly-scoped TCP connection to one specific port can still complete, and the deeper shell-spawning failure reflects some further-downstream resource or configuration issue layered on top of that partially-degraded networking stack.

Explicitly stated: this is presented as a combination of 4–5 separate contributing issues, not a single root cause to be found and fixed in isolation — see Section 3.7 for the specific configuration areas identified as contributing factors.

3.7 Where to Look for the Fix — Configuration Files Catalog

Rather than declaring one definitive fix, the engineer catalogued the specific configuration locations that plausibly contribute to this failure signature, for participants to investigate directly on the (currently SSH-inaccessible — see Section 3.8) production server:

Practical remediation actions mentioned: restart the sshd daemon (systemctl restart sshd) after making configuration changes; widen the ephemeral port range; verify/adjust the UseDNS setting depending on whether reverse DNS is reliably available.

3.8 Recovering Access When SSH Itself Is Blocked

A deliberate twist built into this exercise: since SSH to the production server is exactly what’s broken, participants can’t just SSH in normally to go check the files listed in Section 3.7. The engineer outlined four alternative access paths, in order of preference:

1. AWS Systems Manager (SSM) Session Manager — the engineer’s first suggestion, but explicitly disabled/blocked in this specific exercise’s environment (SSM agent offline), forcing participants toward the remaining options.
2. EC2 Serial Console — a low-level, direct console access method that doesn’t depend on network connectivity or SSH at all.
3. Systems Manager Automation runbook to reset the EC2 user’s password — navigate to Systems Manager → Automation → run a specific runbook (tagged, per the engineer, roughly as “Reset EC2 password” or similar) — this generates a new password allowing console-based login even without working SSH/key access.
4. Detach-the-volume method (last resort) — detach the production instance’s EBS volume, attach it to another instance to directly access and modify its filesystem (including any of the config files listed in Section 3.7), then detach and reattach it back to the original instance.

Hands-On: Writing the RCA

• Assignment: using one of the four access methods above, participants must independently SSH/access into the production server, go through each of the configuration files/areas catalogued in Section 3.7, and identify the actual contributing misconfigurations themselves.
• If unable to resolve independently: post progress/findings in the team’s Platform Knowledge Base; the engineer committed to demonstrating the full live resolution in the following Tuesday’s Q&A if participants can’t get there on their own.
• RCA deliverable required: using a template already uploaded to the shared drive, participants must write a full RCA covering: the problem statement, the investigation process (what was checked, in what order, and what each check revealed), the identified contributing causes, and the implemented fix(es) — then upload the completed RCA to their GitHub.
• Supplementary materials promised: a detailed, standalone SSH-troubleshooting reference document (covering exactly which commands/files to check for which SSH symptom categories) and the Terraform code to recreate this exact multi-factor scenario independently, both to be shared via the drive.

3.10 Q&A — Methodology Clarifications

• ”Why keep going after 100% packet loss — isn’t that already the answer?” — Directly addressed: the point isn’t to fixate on the first dramatic signal, but to keep climbing the OSI ladder to gather enough evidence to form a well-supported conclusion, since (as this exact incident demonstrates) a severe-looking early signal can be contradicted or contextualized by evidence gathered at a higher layer (here, the successful TCP handshake).
• ”Shouldn’t we check the obvious/easy things first, before a full structured framework?” — The engineer agreed explicitly: check basic resource health (CPU/RAM) and other “obvious” quick checks first; only escalate to the full OSI framework if those quick checks don’t resolve or explain the issue. The framework is the fallback for genuinely ambiguous incidents, not the mandatory starting point for every incident.
• ”Is OSI the right framework for a Kubernetes-scale or monitoring-system incident?” — No — explicitly clarified that OSI is well-suited to Linux/system-level troubleshooting, but is comparatively inefficient for Kubernetes-scale incidents (where a Kubernetes-specific troubleshooting framework is faster and more appropriate) or for incidents within a monitoring system itself (where a monitoring-specific framework applies instead). The right framework depends on which layer/system the incident actually lives in.
• A participant explicitly requested (and received agreement for) a consolidated, spoken summary of everything checked and found, partway through the highly terminal-focused walkthrough — a reasonable and granted request, reflected in the engineer’s mid-session recap (captured in Section 3.6).

4. Key Concepts Table

5. Architecture & Workflow Analysis

5.1 Full OSI-Layer Investigation Sequence (as executed live)

Layer 1 (Physical): ping <prod-IP>
  -> 100% packet loss
        |
        v
Layer 2 (Data Link): arp -n <prod-IP>
  -> No MAC address resolved
        |
        v
Layer 3 (Network): tracepath <prod-IP>
  -> Fails at hop one (traffic not leaving subnet, per this tool)
        |
        v
Layer 4 (Transport): nc <prod-IP> 22
  -> SUCCEEDS  <-- pivot point: contradicts the pessimistic Layer 1-3 picture
     Rules out: security groups, NACLs, ENI attachment, routing
        |
        v
[Deep dive: SSH protocol whiteboard walkthrough -- see 5.2]
        |
        v
SSH verbose mode (ssh -v):
  -> Key exchange: OK
  -> Host authentication: OK
  -> User (public key) authentication: OK
  -> Channel/session created, shell allocated: OK
  -> HANGS: shell never becomes usable
        |
        v
Layer 5-7 + kernel checks: top, conntrack, timedatectl
  -> No load issue, no conntrack queue buildup
        |
        v
SYNTHESIS: "Networking stack alive for TCP handshake,
            dead for general data forwarding + shell spawn"
            (multi-factor: port range + DNS/UseDNS + kernel config)

5.2 SSH Protocol Sequence (Whiteboard Content)

Client (Bastion)                          Server (Production)
      |                                          |
[optional: /etc/hosts -> DNS lookup if hostname used]
      |                                          |
      |------------- TCP SYN ------------------->|
      |<---------- TCP SYN-ACK ------------------|
      |------------- TCP ACK  ------------------>|
      |         (3-way handshake complete)        |
      |                                          |
      |<-- SSH version exchange (both directions)->|
      |                                          |
      |<== Key exchange / algorithm negotiation ==>|
      |   (Diffie-Hellman-family algorithm;        |
      |    shared secret derived independently,    |
      |    NEVER sent over the wire)                |
      |                                          |
      |<-- Host authentication (known_hosts check)->|
      |                                          |
      |<== User authentication ====================>|
      |   (GSSAPI attempted -> not configured,      |
      |    falls through to public key auth)        |
      |                                          |
      |                                    [PAM checks: session
      |                                     limits, MFA, etc.]
      |                                          |
      |                              [Load /etc/profile.d/ etc.]
      |                                          |
      |                              [sshd forks child process
      |                               for user's login shell]
      |                                          |
      |<========= Shell session ready ============>|
      |        (THIS IS WHERE THIS INCIDENT
      |         ACTUALLY HANGS)

5.3 Alternate Access Recovery Paths (When SSH Itself Is Broken)

SSH to production server: BLOCKED
        |
        v
    -------------------------------------------
    |              |               |            |
  Option 1:     Option 2:      Option 3:     Option 4:
  SSM Session    EC2 Serial     Systems Mgr   Detach volume
  Manager        Console        Automation    -> attach to
  (BLOCKED in    (bypasses      runbook       another instance
   this          network/SSH   (reset EC2    -> modify files
   exercise)     entirely)      user          -> reattach
                                 password)     (last resort)

5.4 Multi-Factor Root Cause Composition

Observed Failure: SSH freezes after successful authentication
        |
        v
    -----------------------------------------------------
    |              |                |                    |
Ephemeral      UseDNS=yes      Kernel/network         [Possibly
port range     (reverse DNS    stack partially         ARP cache /
too narrow     lookup on       degraded (alive         NIC state --
(~1000)        client IP,      for TCP handshake,       not fully
               may hang if     dead for general         confirmed
               DNS degraded)   forwarding)               live]
    |              |                |                    |
    -----------------------------------------------------
                        |
                        v
              COMBINED EFFECT: intermittent,
              then near-total SSH freeze

6. Commands & Configurations

7. Tools & Technologies

ping / arp / tracepath / nc

• Purpose: The core suite of Layer 1–4 OSI diagnostic commands used throughout this walkthrough.
• When to use them: As the first structured steps in any ambiguous network/connectivity incident, executed in order from the bottom of the stack upward.

SSH verbose mode (-v / -vv / -vvv)

• Purpose: Exposes the full internal sequence of an SSH connection attempt — version exchange, key negotiation, authentication method attempts, and session/channel creation.
• When to use it: Whenever an SSH connection is failing or behaving unexpectedly and basic connectivity checks (ping, TCP handshake) don’t fully explain it — as demonstrated in this walkthrough, it was the tool that actually pinpointed the true failure point (shell spawning) after network-layer checks alone had produced an ambiguous, seemingly contradictory picture.

conntrack

• Purpose: Inspects the Linux kernel’s connection-tracking table.
• When to use it: To check for queue buildup or retry-storm-style patterns at the kernel level — directly useful for diagnosing cascading-failure-style incidents (as referenced back to the Uber case study).

AWS EC2 Serial Console

• Purpose: Direct, out-of-band console access to an EC2 instance that bypasses the network entirely.
• When to use it: When SSH (and other network-dependent access methods like SSM) are unavailable — as was the case in this exercise.

AWS Systems Manager Automation

• Purpose: Runs predefined operational runbooks against AWS resources, including credential-reset workflows.
• When to use it: As a recovery path to regain console-based access to an instance when SSH/key-based access is broken.

8. Real-World Production Usage

• This walkthrough is a genuinely strong, realistic demonstration of how contradictory-seeming diagnostic evidence actually gets resolved in real incidents — the discipline of not stopping at the first dramatic signal (100% packet loss) and instead continuing to gather evidence until a coherent picture emerges (TCP works, but general forwarding/shell-spawning doesn’t) is exactly the kind of judgment that separates effective incident response from panic-driven guesswork.
• The full SSH protocol walkthrough is a directly reusable reference for reading verbose SSH logs in any real environment — most engineers treat ssh -v output as unreadable noise; understanding the actual sequence it represents turns it into a precise diagnostic tool.
• The specific symptom-to-cause mapping (timeout → network/firewall; hang-at-login → DNS/PAM/UseDNS; permission denied → authorized_keys; slow-not-hanging → reverse DNS) is a genuinely practical triage checklist worth having memorized for any engineer who regularly operates Linux infrastructure.
• The ephemeral port range finding is a realistic, easy-to-overlook root cause category — most engineers don’t think to check net.ipv4.ip_local_port_range when debugging SSH issues, and a too-narrow range is a genuinely real, if uncommon, cause of exactly this kind of “random” connection failure pattern under concurrent load.
• Having multiple, genuinely independent out-of-band access recovery methods (SSM, Serial Console, password-reset runbook, volume detach) is standard, necessary operational practice for any production environment — an incident that breaks your primary access path is a realistic scenario, and not having a backup plan for that specific failure mode is a real operational gap many teams don’t discover until they need it.
• Choosing the right troubleshooting framework for the system in question (Linux/OSI vs. Kubernetes-specific vs. monitoring-specific) reflects mature operational judgment — applying a generically “structured” framework that’s poorly matched to the actual system can itself waste critical incident time, even when the underlying discipline (be structured, don’t guess) is sound.

9. Interview Preparation

Beginner Questions

Q1: If a TCP handshake to port 22 succeeds via netcat, but ping to the same host fails completely, what does that tell you? A: It tells you the failure isn’t a full network-path outage — since a TCP connection to a specific port was able to complete, security groups, NACLs, and basic routing to that host are working. ping uses ICMP, a different protocol than TCP, and can be independently blocked or fail even while TCP connections on specific ports succeed. This is a useful diagnostic signal to narrow the investigation, not a contradiction to be resolved by dismissing one result.

Q2: What’s the difference between an SSH connection that times out versus one that hangs after authentication succeeds? A: A timeout typically indicates the connection attempt never even reached the SSH daemon in a meaningful way — often pointing to network-path issues like firewall rules, NACLs, or security groups blocking the connection outright. A hang after successful authentication (as in this walkthrough’s scenario) means the network path and SSH’s own authentication sequence both worked correctly — the problem lies further downstream, typically in shell/session creation, PAM configuration, or reverse-DNS-related settings like UseDNS.

Q3: What are two alternative ways to access an EC2 instance if SSH access is completely broken? A: The EC2 Serial Console provides direct, out-of-band access that doesn’t depend on network connectivity or SSH at all. AWS Systems Manager’s Automation feature can run a runbook to reset the instance’s console password, allowing login through the console UI even without working SSH keys. (A third option, AWS Systems Manager Session Manager, also provides SSH-independent access, provided the SSM agent is online and functioning.)

Intermediate Questions

Q4: Walk through why a disciplined engineer would continue investigating after finding 100% packet loss on a ping test, rather than immediately concluding the network is down. A: A single failed test (even one as dramatic-looking as 100% packet loss) only tells you that specific protocol/test failed — it doesn’t tell you why, and it doesn’t rule out that other, more specific paths (like a TCP connection to one particular port) might still work. Continuing to gather evidence across multiple layers (ARP resolution, path tracing, then a direct TCP handshake test) builds a fuller picture that can either confirm or meaningfully contradict the initial signal. In this walkthrough’s case, continuing past the failed ping revealed a successful TCP handshake — evidence that materially changed the diagnosis away from “the network is fully down” toward a much more specific and different conclusion.

Q5: What is the UseDNS setting in sshd_config, and how could it cause an SSH session to hang specifically after authentication succeeds? A: When UseDNS is enabled, the SSH daemon performs a reverse DNS lookup on the connecting client’s IP address as part of completing the session — checking that the IP resolves back to a hostname, sometimes for logging or additional verification purposes. If DNS resolution is slow, degraded, or unavailable in that environment, this reverse lookup can hang or take a very long time, causing exactly the kind of “authentication succeeded, but the session never becomes usable” symptom observed in this walkthrough — even though the actual SSH authentication and key-exchange process completed without any errors.

Q6: A production incident shows a mix of failing and succeeding network-layer tests (e.g., ARP fails, but a TCP connection to a specific port succeeds). How would you characterize this kind of failure mode, and what should your next diagnostic steps be? A: This pattern is consistent with a partially-degraded kernel networking stack — one that retains enough functionality to establish a narrowly-scoped TCP connection but has broader dysfunction affecting general packet forwarding, ARP resolution, or ICMP handling. Rather than treating this as contradictory or inconclusive, the next step is to move past pure network-layer testing into protocol-specific diagnostics (like SSH verbose logging, if the affected service is SSH) to determine exactly where within that specific protocol’s sequence things are actually breaking, and separately investigate kernel-level configuration (connection-tracking tables, ephemeral port ranges, relevant sysctl parameters) that could explain a stack that’s “alive but degraded” rather than fully down.

Advanced Questions

Q7: Design a troubleshooting approach for an incident where SSH access to a critical production server is itself broken, preventing you from directly investigating the server’s own configuration. What’s your overall strategy? A: First, establish an alternative, out-of-band access path independent of the broken access method — check whether SSM Session Manager is available (fastest, least disruptive if the agent is online); if not, use the EC2 Serial Console for direct console access; if that’s insufficient for the needed changes, use a Systems Manager Automation runbook to reset console credentials; and as a last resort, detach the affected instance’s volume, attach it to a healthy instance to directly inspect/modify the filesystem, then reattach it. Once you have any form of access restored, apply the same structured, layer-by-layer diagnostic approach you would have used via SSH — checking DNS configuration, relevant kernel parameters (ephemeral port ranges, connection-tracking settings), the SSH daemon’s own configuration (particularly settings like UseDNS that specifically affect post-authentication behavior), and PAM/session-related configuration — since the underlying investigation methodology doesn’t change; only the access mechanism does. Document which access method worked and why, since a scenario where your primary access method is broken is itself worth capturing as an operational finding, separate from whatever root-caused the original SSH freeze.

Q8: How would you decide whether to apply a general troubleshooting framework like OSI versus a system-specific framework (e.g., a Kubernetes-specific troubleshooting model) to a given production incident? A: The deciding factor is which layer/system the incident actually appears to live in, based on initial symptoms. If the incident presents as fundamentally a Linux/system/networking-level problem — connectivity, authentication, kernel resource issues, as in this walkthrough’s scenario — the OSI framework’s bottom-up, layer-by-layer discipline is well-matched and efficient. If the incident instead presents primarily within a higher-level orchestration system (e.g., pods failing to schedule, a Kubernetes control-plane component behaving unexpectedly), applying pure OSI-layer network diagnostics first would be comparatively inefficient — you’d want to reach for a Kubernetes-specific troubleshooting framework instead, since it’s purpose-built to reason about that system’s specific failure modes (scheduling, control-plane health, etc.) more directly. The general principle: match the framework’s design assumptions to the system that’s actually misbehaving, rather than defaulting to one framework universally regardless of context.

Q9: This incident was deliberately engineered as a combination of multiple contributing factors (a narrow ephemeral port range, a UseDNS setting, and a partially degraded kernel networking stack) rather than a single root cause. What does this imply about how you should approach root-cause analysis and remediation for real production incidents? A: It implies that a root-cause analysis process should remain open to identifying and addressing multiple contributing factors rather than stopping as soon as a plausible cause is found — declaring victory after finding just one contributing issue (e.g., fixing the port range alone) risks leaving the underlying incident only partially resolved, since the other contributing factors (DNS/UseDNS behavior, kernel networking stack degradation) would remain unaddressed and could cause a recurrence or a related-but-distinct incident later. Practically, this means: continue the structured, layer-by-layer investigation even after finding one plausible explanation, explicitly ask whether the evidence gathered so far is fully explained by that one factor alone or whether some residual symptoms remain unexplained, and structure the resulting RCA and remediation plan to address every identified contributing factor, not just the first or most obvious one found.

10. Exam & Certification Notes

(Relevant to CompTIA Linux+/Network+, RHCSA, and the networking/troubleshooting sections of cloud certifications.)

• OSI model layer functions: A foundational, heavily-tested networking concept — know each layer’s role (Physical, Data Link, Network, Transport, Session, Presentation, Application) and which tools/protocols map to each (ICMP/ARP to lower layers, TCP/UDP to Transport, application protocols like SSH to the upper layers).
• ARP (Address Resolution Protocol): Know its specific function — resolving an IP address to a MAC address for Layer 2 communication — and that ARP failures indicate local-network-segment communication problems, distinct from routing issues further up the stack.
• TCP three-way handshake (SYN, SYN-ACK, ACK): A core, frequently tested networking fundamental — and this walkthrough is a good illustration of why understanding it matters practically, since confirming a successful handshake (via netcat) is what definitively ruled out an entire category of suspects (security groups, NACLs, routing).
• SSH UseDNS directive: A specific, sometimes-overlooked sshd_config setting — know that it controls reverse DNS lookups on connecting clients, and that disabling it is a common troubleshooting/hardening step when DNS reliability is a concern.
• Ephemeral/dynamic port range (net.ipv4.ip_local_port_range on Linux): Know that this kernel parameter controls the pool of ports available for outbound/dynamic connections, and that an insufficiently sized range can cause connection failures under concurrent load — a real, if uncommon, exam-relevant Linux kernel tuning topic.
• PAM (Pluggable Authentication Modules): Know its general role in Linux authentication policy enforcement (session limits, MFA, access control) as a layer distinct from SSH’s own key-exchange/authentication mechanics — a commonly tested distinction between “SSH-level” and “OS-level” authentication controls.

11. Cheat Sheet

OSI Troubleshooting Sequence (bottom-up):

1. Physical: ping — basic reachability
2. Data Link: arp -n — MAC resolution
3. Network: tracepath/traceroute — path tracing
4. Transport: nc <ip> <port> — TCP handshake test 5–7. Session/Presentation/Application: protocol-specific tools (e.g., ssh -v for SSH-specific issues)

SSH Symptom → Likely Cause Quick Reference:

SSH Protocol Sequence (memorize the order): (Optional DNS) → TCP handshake → version exchange → key exchange/algorithm negotiation → host auth (known_hosts) → user auth (GSSAPI → public key → password) → PAM checks → profile/environment load → shell/child-process spawn

Key Diagnostic Insight from This Session: A successful TCP handshake to a specific port can coexist with failing ping/ARP/traceroute — this means the kernel networking stack can be “alive enough for connection establishment, dead for general forwarding.” Don’t treat this as a contradiction — treat it as a specific, narrowing clue.

When SSH Itself Is Broken — Access Recovery Order:

1. SSM Session Manager (if agent online)
2. EC2 Serial Console
3. Systems Manager Automation → password-reset runbook
4. Detach volume → modify elsewhere → reattach (last resort)

Files to Check for “SSH hangs after auth succeeds”:

• /etc/ssh/sshd_config → UseDNS
• /etc/resolv.conf → DNS config
• sysctl / /etc/sysctl.d/ → ephemeral port range, conntrack buffer
• /etc/pam.d/sshd → session/PAM policy
• /etc/profile.d/ → session/environment setup

Framework Selection Rule: OSI for Linux/system-level incidents; Kubernetes-specific framework for cluster-scale incidents; monitoring-specific framework for monitoring-stack incidents. Match the framework to the system, don’t default universally.

12. Gaps & Assumptions

• This walkthrough ends without a single, confirmed, definitive root cause — consistent with the earlier session in this series that first introduced this scenario, the engineer deliberately leaves full resolution as an assignment, with a promise to demonstrate the complete fix live in a follow-up Tuesday Q&A not captured in this transcript. The “synthesis” in Section 3.6 and the contributing-factors list in Section 3.7 represent the diagnostic progress made within this walkthrough, not a confirmed final answer — treat the ephemeral port range and UseDNS findings as strong, evidence-supported leads explicitly called out by the engineer, not as a certified complete fix.
• The key-exchange algorithm name (referenced as a Diffie-Hellman-family algorithm) was recalled imperfectly and informally by the engineer live (“I guess the name is Hellman… Diffie-Hellman key exchange algorithm”) — presented here using the more complete standard terminology (Diffie-Hellman) for clarity, but worth verifying the exact algorithm/cipher suite specifics against current SSH protocol documentation if precision matters for your use case.
• Exact sysctl parameter names and file paths for the ephemeral port range and conntrack buffer settings were shown live on screen but not always fully and clearly dictated in the transcript audio — this document uses the standard, conventional Linux parameter names (net.ipv4.ip_local_port_range, /etc/sysctl.d/) consistent with what was described; verify exact paths/values against the actual system if replicating this exercise.
• The Telegram incident comparison was offered by the engineer as a precedent for this failure signature but wasn’t elaborated on in detail within this transcript — presented here as the engineer’s own analogy/reference point, not independently verified or detailed against Telegram’s own public postmortems (if any exist for this specific comparison).
• ARP cache clearing / NIC reset was mentioned as a potential remediation step but wasn’t demonstrated live or confirmed as an actual applied fix in this walkthrough — included in the configuration catalog (Section 3.7) as a plausible contributing area based on the engineer’s own framing, not as a confirmed necessary step.
• This document consolidates a long, deeply technical session that included a dedicated whiteboard segment (screen-shared separately from the main terminal) — the whiteboard content (Section 3.3) has been reconstructed from the engineer’s spoken explanation during that segment, since whiteboard drawings themselves aren’t captured in a text transcript; the sequence and terminology are preserved as accurately as the spoken description allows.