Complete Learning Package (Linux · Networking · SSH internals · Security · Structured Debugging)

2. Table of Contents

1. Executive Summary
2. Table of Contents
3. Detailed Structured Notes
   • 3.1 The Incident (problem statement & environment)
   • 3.2 The Symptom Breakdown & Why “Random” Matters
   • 3.3 Blast Radius & Impact Classification
   • 3.4 The Incident-Response Process (Isolate→Stabilize→Fix→Prevent)
   • 3.5 The Two Framing Questions + Timeline Forensics
   • 3.6 The Live Debugging Attempts (what was tried & ruled out)
   • 3.7 Why Common Fixes Failed — the “1% problem”
   • 3.8 SSH Under the Hood — full login pipeline
   • 3.9 GSSAPI, Kerberos, KDC & the Latency Mechanism
   • 3.10 PAM Session Setup & Shell Creation
   • 3.11 The Layered (OSI-style) Mental Model
   • 3.12 Dashboards, Observability & the “green dashboard” trap
   • 3.13 The Assignment
4. Key Concepts Table
5. Architecture & Workflow Analysis (diagrams)
6. Commands, Configs & Files
7. Tools & Technologies
8. Real-World Production Usage
9. Interview Preparation (Beginner / Intermediate / Advanced)
10. Exam & Certification Notes
11. Root-Cause Analysis: confirmed facts vs. hypotheses
12. Gaps, Assumptions & Things the Session Left Open
13. Cheat Sheet

3. Detailed Structured Notes

3.1 The Incident — Problem Statement & Environment

Setup (the “ideal” working state)

The trigger

On a normal Friday ~11:00 AM IST, with no known change, connectivity degraded. Critical detail: a system that was working does not break without manual intervention — so “what changed?” is the first instinct, and here the answer was “nothing,” which is exactly what makes it a 1% case.

3.2 The Symptom Breakdown & Why “Random” Matters

Three buckets, and membership rotates randomly per ~hour (no fixed set of “bad” users, no detectable pattern):

30 users (bastion --> prod)
├── ~12–13  : connect fine, smooth
├── ~5–6    : connect, but 10–15s lag per action (painfully slow)
└── ~7–8    : connection TIMES OUT (cannot reach prod from bastion)
   (all three sets reshuffle every ~hour; same user can be in any bucket over time)

Why “random/intermittent” is the key clue:

• If it were the same users always failing → an account/key/permission problem (easy to localize).
• If it were all users → a hard outage (sshd down, port closed, MaxSessions hit).
• Partial + rotating → points to a per-connection, time-variable cost (latency/timeout that sometimes crosses a threshold), not a static misconfig. This is the thread the whole session pulls on.

Important scoping facts established live:

• All 30 users can reach the bastion fine. The failure is bastion → prod only.
• No other host exhibits this; it’s specific to this one prod target.
• The application/front-end is fully healthy — services up, graphs green.

3.3 Blast Radius & Impact Classification

The engineer splits blast radius into three categories you must classify every incident against:

• Geographic impact: all 30 users in the same office / same region (e.g., ap-south-1 mentioned), so cross-region latency is not a plausible explanation — and that very fact is used to rule out geography later.
• Priority: P0 — “every single minute counts,” and management will later demand a minute-by-minute account of the response time.

3.4 The Incident-Response Process

The canonical sequence

ISOLATE  -->  STABILIZE  -->  FIX / CORRECT  -->  PREVENT (don't let it recur)

• Isolate: narrow where the problem lives (which host, which hop, which layer).
• Stabilize: get the current system to a workable state.
• Fix/Correct: apply the actual remediation.
• Prevent: config/process changes + RCA so it can’t silently return.

Core philosophy stated repeatedly: “In a production outage, see the larger context of why it’s happening, not the smaller picture of what is happening.” “Bastion can’t reach prod” is the small picture; the large picture is the full login pipeline and which layer injects the failure.

3.5 The Two Framing Questions + Timeline Forensics

Before touching anything, ask in this order:

1. “How exactly does my system work?” — You cannot debug what you don’t understand. (The session spends its second half answering this for SSH precisely because nobody could.)
2. ”Why is my system behaving like that?” — which kicks off a forensic question series:
   • Timeline / forensics: From exactly what time did you notice this? Troubleshoot around that window. A healthy system breaks only after a manual change by a developer/DevOps/SRE — so hunt the change first.
   • Blast radius: (Section 3.3).
   • Then try common fixes.

Cognitive-bias warning (raised by attendees, endorsed by instructor): “It worked before, what suddenly changed?” can misdirect you — when the honest answer is “nothing changed,” that framing makes you prematurely conclude “network is fine, port is fine, everything’s fine” and stop thinking, even though it fails for some users some of the time. Also acknowledged: sometimes it really is an underlying OS bug, a hardware fault, or an undisclosed cloud-provider change (real example cited: AWS admitted a load-balancer change only ~2 weeks later).

3.6 The Live Debugging Attempts — What Was Tried and Ruled Out

This is the heart of the “what NOT to stop at” lesson. Attendees proposed fixes; each was checked and eliminated:

The meta-lesson the engineer lands: all of the above is “throwing stones at the problem” / “trying luck.” None of it is structured debugging. In a P0, luck costs money. Hence the framework in 3.11.

3.7 Why Common Fixes Failed — the “1% Problem"

"All these solutions work for 99% of production outages. This is the 1% that isn’t in your hands or the system’s hands — where everything you try does nothing.”

The common-fix checklist is correct and necessary first, but when you hit a blank screen (nothing works, nothing changed, dashboard green), you stop guessing and switch to systematic per-layer exploration + deep protocol understanding. The session is engineered to force exactly that switch.

3.8 SSH Under the Hood — the Full Login Pipeline

This is the most reusable technical content. When a user runs ssh user@prod-server from the bastion, here is the entire sequence:

Step 0 — The connection 4-tuple

The bastion is the SSH client. The OS assigns an ephemeral source port. Example tuple:

Source IP:Port (bastion)      Dest IP:Port (prod)
10.20.0.5 : 34567     -->     10.20.0.9 : 22   (or custom port)

Step 1 — TCP three-way handshake

Bastion  -- SYN -->        Prod kernel   (if not dropped by firewall)
Bastion  <-- SYN/ACK --    Prod
Bastion  -- ACK -->        Prod
=> TCP socket ESTABLISHED  (OS + network stack agree they can talk)

Step 2 — SSH protocol handshake / negotiation

Bastion (client sshd) ---- SSH banner ----> Prod
Bastion <---- SSH banner (incl. OpenSSL config) ---- Prod sshd
Negotiate: ciphers, MACs, key-exchange algorithm
Host-key verification: bastion checks its KNOWN_HOSTS
   -> cipher/key MISMATCH or unknown host  => ABORT
   -> match                                => proceed

Step 3 — Authentication phase (system, then user)

Order as described:

1. Public-key auth:
   Bastion sends private-key SIGNATURE  -->  Prod sshd validates it
   against ~/.ssh/authorized_keys.  Mismatch => abort.
2. OS Login check (cloud VMs often enable OS Login by default).
3. sshd calls PAM  -> checks user/role/group, allow/deny, time restrictions.
4. PAM -> SSSD  -> Cloud API (GCP/AWS) to verify the user has SSH rights.
5. If authorized => continue.

Step 4 — Side effects that can inject latency (the suspect zone)

- Reverse DNS lookup: if `UseDNS yes`, sshd resolves bastion IP -> hostname (PTR).
- GSSAPI / Kerberos: if `GSSAPIAuthentication yes`, sshd attempts GSSAPI
  negotiation backed by Kerberos/KDC.

Both are global-ish services but can stall a single login if DNS is flappy or the KDC is unreachable — and that stall is per login, variable, intermittent → matches the symptom profile.

An attendee sharply objected: “GSSAPI/Kerberos/DNS are global to the environment — how does that affect a single machine intermittently?” The engineer acknowledged it as a fair challenge and deferred the reconciliation to the assignment/RCA — i.e., he did not fully close this logical gap on the call. (See Sections 11–12.)

3.9 GSSAPI, Kerberos, KDC & the Latency Mechanism

• GSSAPI = Generic Security Services API. A standard API integrated into sshd that lets SSH use an external authentication system without knowing its internals — i.e., auth beyond just password/key.
• In enterprise setups GSSAPI is typically backed by Kerberos, a network authentication protocol that proves identity without sending the password over the wire, using tickets issued by the KDC (Key Distribution Center).
• If GSSAPIAuthentication yes is set, sshd will attempt GSSAPI/Kerberos negotiation in addition to keys/passwords.

How this becomes latency (the proposed mechanism):

sshd checks allowed auth methods: publickey, password, GSSAPI...
If GSSAPI = yes:
   client sends GSSAPI init request -> prod checks Kerberos KDC keys
   If env has GSSAPI enabled but ISN'T actually using Kerberos (no/bad KDC):
       sshd WAITS for the client / a response, RE-ATTEMPTS, then TIMES OUT
If UseDNS = yes:
   sshd does a PTR (reverse) lookup of the client hostname for Kerberos mapping
   If DNS is SLOW or FLAPPING -> adds seconds to EVERY login
=> User sees a long pause after entering password / before key acceptance
=> Perceived as "terminal lagging" or "timeout"

Per the engineer: “All of this comes back to DNS. If your DNS is flappy or slow, it adds time to every user’s login because of Kerberos backed by the KDC.” This is stated as a possibility that matched their case — explicitly “I’m not saying this IS the scenario, I’m saying it CAN be.”

3.10 PAM Session Setup & Shell Creation

After successful authentication:

1. PAM session setup runs: re-checks groups/roles, applies login restrictions (config under /etc/security/..., e.g. a security config file). Cache caveat: if the PAM/SSSD cache is expired, some users hit a transient denied until it refreshes — a generic intermittent-denial cause (not claimed as the root here).
2. Shell creation: the main (parent) sshd keeps listening; it forks a child sshd dedicated to the logged-in user.
   • Verify with: ps -ef | grep sshd → you’ll see a root (parent) and a user (child) process. One parent + one child per user.
3. sshd launches the user’s default shell from /etc/passwd (bash, zsh, etc.).
4. /etc/profile + profile scripts load (this is what a profile “refresh” re-runs).
5. Prompt appears → user is “in.” Every keystroke now travels as an encrypted packet over the TCP session → sshd → executed by the shell → encrypted output sent back to the bastion.

Multi-user note (Q&A): the entire pipeline (handshake → auth → KDC ticket → PAM → shell) repeats independently for every new connection — each user gets their own parent/child shell pair and their own auth flow (no single shared token).

3.11 The Layered (OSI-style) Mental Model — the Framework

When common fixes fail, walk every configurable layer systematically. First, a generic “what does any machine need” decomposition (7 parts), explicitly likened to the OSI model:

A. The “anatomy of any system” (physical or virtual)

1. Physical layer     : CPU, RAM, NICs, ARP  (in cloud: you can't touch cables)
2. Operating System   : Linux/Windows distro
3. Runtime            : Python/Node/etc. environment (+ middleware)
4. Application        : the app itself
5. Networking         : internal + external communication
6. Data/Storage       : system + app configuration & data
7. Security           : firewall etc. — but really CROSS-CUTS every layer

B. Map classic OSI → cloud-debuggable layers

C. Consolidated cloud framework (what you sweep through)

PHYSICAL  →  OS  →  NETWORK  →  DATA  →  APPLICATION
                 (SECURITY cross-cuts ALL of them)

Procedure: start at Physical, enumerate every configuration that layer can hold in your cloud, check for misconfig or an external factor that could force a misconfig; if clean, move to the next layer; repeat. The claim: any system-related outage lives inside these layers, so a complete sweep will surface it — far faster and calmer than random guessing. (Examples mapped live: resolv.conf/ports/timeouts → Data; logs → OS; Kubernetes CNI issue → Network.)

Caveat the engineer is explicit about: this layered sweep is the second-line tool, used after the common-fix checklist, specifically for the “blank screen / nothing works” 1% — not your first move for everyday incidents.

3.12 Dashboards, Observability & the “Green Dashboard” Trap

• Claim: ~70–80% of the real outages the engineer handled showed clean Prometheus/Grafana dashboards — because typical dashboards capture system health, not user health.
• Fix: build wider dashboards — latency, connection counts, per-user error rate, 504/502 panels — and alerting (Alertmanager) on those. Most orgs lack a latency dashboard, which is precisely what would have surfaced this incident.
• SSH-specific monitoring: beyond tailing logs, you can dashboard SSH latency and connection success/failure and look for the pattern (how many of 30 drop, error rate over time).

Hands-On: Writing the RCA

The session ends as a drill, not a solution:

• Each team researches every layer, lists all configurations that layer can hold, and predicts which misconfiguration could produce this outage. (~30 hours given.)
• Goal is learning the concepts (Linux/networking/security config at each layer), not “hitting the right answer.” Instructor will then publish a detailed RCA (commands + per-layer steps) and hold a follow-up call.
• Labs are pre-built by “cloud managers”; the exact incident is not perfectly reproducible (it depends on the real 30-user network conditions) — teams simulate the framework, not the precise outage. Squads are cloud-agnostic and encouraged to cross-pollinate (GCP ↔ AWS) at the framework level.

4. Key Concepts Table

5. Architecture & Workflow Analysis

5.1 The environment & upstream flow

Local machine (engineer)
      | SSH
      v
   BASTION  (Ubuntu, 2 vCPU/4GB, custom SSH port, same VPC/subnet)
      | SSH (jump)  <-- FAILURE / LATENCY LIVES ON THIS HOP
      v
 PRODUCTION VM (Ubuntu 22, 4 vCPU/8GB, static IP)
      |  hosts: payment-analysis backend + scraper
      v
 Front-end DASHBOARD (hourly-updated, CFO-visible)

• All 30 users reach the bastion fine; only bastion → prod is affected.
• Application path is healthy; only the human SSH path degrades.

5.2 Full SSH login pipeline (where latency can hide)

ssh user@prod  (from bastion; ephemeral src port -> dst SSH port)
   │
   ├─ TCP 3-way handshake ............ Layer 4 (Network/Transport)
   ├─ SSH banner + cipher/MAC/kex .... Session/Presentation -> OS/sshd
   ├─ Host-key check (known_hosts) ... mismatch => ABORT
   ├─ AUTH:
   │     publickey (authorized_keys)
   │     -> OS Login
   │     -> PAM (roles/groups/time)
   │     -> SSSD
   │     -> Cloud API (GCP/AWS) verify SSH rights
   ├─ SIDE EFFECTS (suspect zone):
   │     reverse DNS (UseDNS yes) ........ slow/flappy => +seconds/login
   │     GSSAPI/Kerberos (KDC) ........... no/bad KDC => wait+retry+TIMEOUT
   ├─ PAM session setup (cache expiry => transient deny)
   └─ Shell: main sshd FORKS child sshd -> /etc/passwd shell -> /etc/profile
            => prompt; keystrokes = encrypted packets over TCP

5.3 The debugging decision flow

Incident
  -> Ask: How does the system work?  Why is it behaving so?
  -> Timeline forensics + Blast radius
  -> COMMON FIXES (restart sshd/sssd, reboot, CPU/RAM/disk, logs, ports, DNS, audit)
        |
        |-- solves ~99% of outages? --> DONE
        |
        +-- "blank screen", nothing works, nothing changed (the 1%)
                 |
                 v
      LAYERED SWEEP: Physical -> OS -> Network -> Data -> Application
                 (Security checked at every layer)
                 -> enumerate every config per layer
                 -> find the misconfig / external factor
                 -> RCA -> Fix -> Prevent

6. Commands, Configs & Files

The session is conceptual; exact flags weren’t all dictated. Commands below are the ones named/strongly implied. Treat unflagged items as the right tool, and confirm exact syntax for your distro.

Config knobs most relevant to the latency hypothesis:

# /etc/ssh/sshd_config
UseDNS no                 # avoid slow/flappy reverse-DNS per login
GSSAPIAuthentication no   # avoid GSSAPI/Kerberos negotiation if no working KDC

(These are the standard remediations for this symptom class — inferred, since the session withheld the official fix.)

7. Tools & Technologies

• SSH / OpenSSH (sshd): the protocol and daemon at the center of the incident. Handshake, ciphers, host keys, auth methods, UseDNS, GSSAPI.
• Bastion / jump host: controlled ingress to private VMs.
• PAM: authorization, session policy, login restrictions.
• SSSD: identity/auth caching daemon (cache expiry → intermittent denials).
• GSSAPI + Kerberos + KDC: enterprise network auth; passwordless tickets; a latency source when misconfigured.
• DNS / resolv.conf: forward + reverse (PTR) resolution; flappy DNS injects per-login delay.
• GCP (example cloud): VMs, OS Login, metadata, serial console, static IP, VPC/subnet, audit logs. Cloud-agnostic — AWS/Azure equivalents apply (CloudTrail, security groups, EBS/EC2).
• Diagnostics: ping, traceroute, tcpdump, netstat, free, top, ps, ulimit, systemctl.
• Observability: Prometheus + Grafana + Alertmanager — with the caveat to build latency/connection/per-user-error dashboards, not just system metrics.
• Config management (mentioned): done manually here; Ansible/SaltStack referenced as alternatives.

8. Real-World Production Usage

• Incident command: the Isolate→Stabilize→Fix→Prevent loop + minute-by-minute accountability mirrors real SEV/P0 handling.
• Bastion architecture & hardening: non-standard SSH port, firewall/SG rules, static IP, same-VPC placement — and the realization that the bastion hop is also a latency/availability choke point.
• Auth stack in enterprises: OS Login → PAM → SSSD → cloud API, often with GSSAPI/Kerberos; misconfigured GSSAPI/UseDNS is a classic real-world cause of “SSH is slow to log in.”
• Observability maturity: the green-dashboard trap is a genuine production pitfall; user-experience SLIs (login latency, connection success) matter as much as CPU/RAM.
• Security as cross-cutting: security controls live at every layer; “security teams can be sneaky” → always confirm whether ACL/zero-trust changes happened.
• Reproducibility limits: load/network-dependent incidents often can’t be perfectly reproduced; you simulate the framework and reason from protocol knowledge.
• Cost optimization / over-provisioning: both hosts were heavily over-provisioned (used ≤20–30%) — a cost note, and also why resources were quickly ruled out.
• Scalability: the failure is load/concurrency-sensitive (30 real users) even though resources looked idle — pointing at per-connection auth/DNS cost rather than throughput.

9. Interview Preparation

Beginner

Q1. What is a bastion/jump server and why use one? A: A hardened entry host you SSH into first, then “jump” to private servers. It centralizes and controls ingress to machines that have no public access. (Here: bastion → prod in the same VPC.)

Q2. List the steps of an SSH login at a high level. A: TCP 3-way handshake → SSH banner + cipher/MAC/key-exchange negotiation + host-key check → authentication (public key → OS Login → PAM → SSSD → cloud API) → optional reverse-DNS/GSSAPI side effects → PAM session → shell fork (parent + child sshd) → prompt.

Q3. Name five “common fix” checks for an SSH outage. A: Restart sshd/sssd; reboot; check CPU/RAM/disk (free,top); check logs; verify ports/firewall (netstat, SG); check DNS/resolv.conf; review audit logs for changes.

Q4. What’s the canonical incident-response sequence? A: Isolate → Stabilize → Fix/Correct → Prevent.

Intermediate

Q5. The affected users change every hour and the app dashboard is green. What does that tell you? A: Intermittent + rotating membership implies a variable per-connection cost (latency/timeout crossing a threshold), not a static misconfig or a hard outage. Green app dashboards suggest the failure is in the login/control path (user health), not the service — so build/inspect latency and connection metrics, not just CPU/RAM.

Q6. Why might MaxSessions and resource exhaustion be ruled out quickly here? A: MaxSessions was 999 (not a limiter), and free/top showed ≤30% usage with no zombies. A session ceiling would fail all new sessions past the cap, not a rotating subset; idle resources contradict a load-induced CPU/RAM stall.

Q7. How can DNS cause intermittent SSH login lag? A: With UseDNS yes, sshd does a reverse (PTR) lookup of the client per login. If DNS is slow or flapping, every login waits on resolution, adding seconds — perceived as lag/timeout. Disabling UseDNS removes that dependency.

Q8. What is the “green dashboard” problem? A: Standard dashboards track system health (CPU/RAM/service-up), so ~70–80% of subtle outages still look green. You need user-centric SLIs — login latency, per-user error rate, connection success — to catch them.

Advanced

Q9. Explain how GSSAPI/Kerberos can inject login latency, and a counter-argument. A: If GSSAPIAuthentication yes but the environment has no/broken Kerberos KDC, sshd attempts GSSAPI negotiation, waits for a response, re-attempts, then times out — per login. Counter-argument (raised in-session): GSSAPI/Kerberos/DNS are environment-global, so a purely global fault should hit everyone uniformly, not a rotating subset of one machine — implying the real trigger is the interaction of that global latency with per-connection timing/load thresholds. This tension was left unresolved in the session.

Q10. Walk the layered (OSI-adapted) framework for a cloud VM and why data-link folds into physical. A: Classic OSI 1–7 → cloud-debuggable Physical (NIC drivers, hypervisor, CPU/RAM, ephemeral ports, FD limits, run-queue) · OS (sshd session/cipher, logs) · Network (IP/routing/TCP/UDP/ports) · Data (resolv.conf, timeouts, authorized_keys, PAM, SSSD cache, profiles) · Application, with Security cross-cutting all. Data-link (ARP/MAC/Ethernet) is abstracted by the cloud — you can’t touch frames — so it folds into Physical. You sweep layer by layer, enumerating every config, instead of guessing.

Q11. How would you detect a single “naughty user” doing heavy transfer / port-forwarding via the bastion? A: It should appear in audit/access logs (auditd rules, SSH logs) and in per-connection metrics/tcpdump; if logs show nothing and the system seems to “change on its own,” suspect a non-user-driven cause (DNS/KDC/cache) instead.

Q12. Critique relying on “what changed?” as your first question. A: It’s usually right, but in no-change incidents it biases you toward “everything’s fine, stop looking,” and you ignore partial/intermittent failure. It can also blind you to provider-side changes (e.g., undisclosed LB change) or OS/hardware bugs. Pair it with blast-radius analysis and a protocol-level model.

10. Exam & Certification Notes

Frequently tested / emphasized

• SSH login pipeline order (TCP → SSH negotiation → auth → session → shell).
• sshd_config knobs: UseDNS, GSSAPIAuthentication, MaxSessions, AllowUsers/DenyUsers, custom Port.
• PAM vs. SSSD roles; cache-expiry intermittent denials.
• GSSAPI ↔ Kerberos ↔ KDC relationship; passwordless ticket auth.
• OSI 7 layers and their cloud-debuggable mapping.
• Incident loop: Isolate→Stabilize→Fix→Prevent.

Definitions worth memorizing

• GSSAPI (Generic Security Services API), Kerberos, KDC (Key Distribution Center), PAM, SSSD, OS Login, bastion, blast radius, ephemeral port, FD/iNode exhaustion, PTR/reverse DNS.

Likely trick points

• ”MaxSessions=999 means session limits can’t be the issue” → correct for a rotating-subset symptom.
• ”Disk looks clean, so storage is fine” → false: iNode/FD exhaustion blocks new files/connections while disk usage looks low.
• ”Dashboards are green, so there’s no incident” → false (system vs. user health).
• ”Reboot/reSSH fixed nothing, so it’s the network” → premature; sweep layers.
• Data-link layer in cloud → abstracted; fold into physical.

Memorization anchors

• Auth chain: publickey → OS Login → PAM → SSSD → Cloud API.
• Latency suspects: UseDNS (flappy reverse DNS) + GSSAPI/Kerberos (no KDC) timeouts.
• Cloud layers: Physical · OS · Network · Data · Application (+ Security everywhere).

11. Root-Cause Analysis: Confirmed Facts vs. Hypotheses

Confirmed in-session (ruled OUT as the cause): recent change/audit (nothing changed), CPU/RAM/zombies, MaxSessions (999), session timeout config, ports/firewall/SG/VPC, packet loss/routing (at time of check), systemctl services, authorized_keys, SSH port-forwarding, security-team ACL change, geography, GCP metadata os.guest.attributes, iNodes/FD limits (“smart check” but not it), spinning a clone VM (works fine → not image-level).

The leading HYPOTHESIS (heavily telegraphed, NOT confirmed):

Per-login latency injected by UseDNS-driven flappy/slow reverse DNS and/or GSSAPI/Kerberos negotiation against a missing/unreachable KDC, which intermittently pushes some logins past the timeout threshold — explaining the rotating “fine / laggy / timeout” buckets. The engineer said explicitly: “I’m not saying this IS the scenario; I’m saying it CAN be,” and an attendee’s objection (global services vs. single-host intermittency) was left unresolved on the call.

Status of the official answer: Withheld. The engineer promised a detailed written RCA (commands + per-layer steps) after teams complete a ~30-hour research assignment. So no confirmed fix exists in this transcript.

What a competent RCA would therefore still need to nail down (and this walkthrough did not):

• Whether GSSAPIAuthentication/UseDNS were actually yes on this prod host.
• Why a global DNS/KDC issue would manifest as a rotating per-user pattern (load/timing threshold? per-connection retry jitter? KDC rate-limiting?).
• Logs/timestamps correlating slow logins with DNS/KDC round-trip times.

12. Gaps, Assumptions & Things the Session Left Open

Inherent gaps (by design):

• No confirmed root cause or fix — it’s a teaching drill; the RCA comes later. A reader cannot leave with “the answer,” only with the method + the strongest hypothesis.
• The global-service vs. single-host-intermittency logical gap was acknowledged but not closed.

Transcription/clarity issues:

• ASR garbles: “triple SD” = sshd; “SSSD/SSS demon” = sshd/sssd (used interchangeably/ambiguously); “triple 9” = 999; “iodes” = iNodes; “u limit” = ulimit; “carros/carbos/kerros” = Kerberos; “GCS/GSSI” = GSSAPI; “result.config/resolve.com” = /etc/resolv.conf; “one-on-one (1101)” = a custom SSH port; “AP south one” = ap-south-1 (AWS region name used loosely in a GCP example).
• Exact command flags were mostly not dictated — I listed the right tools/files; verify syntax per distro.
• Prod storage stated as 50 GB then corrected to 10 GB.

Assumptions I made (flagged):

• I treated the GSSAPI/UseDNS latency story as the intended hypothesis (it’s the only mechanism the engineer developed in depth and tied to “we had the same case”) — but presented it as a hypothesis, not the answer, consistent with the source.
• The two sshd_config remediation lines in §6 are the standard fixes for this symptom class, inferred — not stated in the session.

Missing context / promised-but-not-delivered:

• The written RCA (per-layer commands), the SSH architecture doc, an exhaustive concept list per layer, and lab/reproduction instructions (built by “cloud managers”) — all promised for later, none present here.
• Roles (“teams,” “cloud managers,” “BAU vs. implementation teams”) are org-specific and only loosely defined.

Honest assessment (sparring-partner mode): The transferable value here is genuinely strong and well worth internalizing: the incident-response loop, the complete SSH login pipeline, the GSSAPI/Kerberos/DNS latency mechanism, the FD/iNode and SSSD-cache gotchas, and the layered sweep as a fallback for the 1%. That’s a solid mental toolkit. But be clear-eyed about three weaknesses: (1) it withholds the answer, so don’t expect closure; (2) the central hypothesis has an unresolved logical hole (global service → rotating single-host failures) that the engineer waved past — a rigorous engineer should be unsatisfied until logs prove the DNS/KDC timing story; and (3) some confident generalizations (“70–80% of outages look green,” “any system issue lives in these layers”) are useful heuristics, not measured facts — treat them as framing, not data. Use the method; demand the evidence the session skipped.

13. Cheat Sheet

Incident loop: Isolate → Stabilize → Fix → Prevent. See the why, not just the what.

First questions: How does it work? → Why is it misbehaving? → Timeline (what changed? — healthy systems break only on change) → Blast radius (system / user / team; cost / reputation).

Symptom → meaning: intermittent + rotating users = variable per-connection cost (latency/timeout), not static config; app green, users blocked = control/login-path problem → check latency, not CPU.

Common-fix checklist (do first, solves ~99%): systemctl restart sshd|sssd · reboot · free/top (CPU/RAM/zombies) · logs · netstat/firewall/SG/ports · ping/traceroute/tcpdump · /etc/resolv.conf · cloud audit (CloudTrail) · ulimit+iNodes · authorized_keys · serial console.

SSH login pipeline: TCP 3-way → SSH banner+cipher/MAC/kex+known_hosts → publickey → OS Login → PAM → SSSD → Cloud API → (reverse DNS / GSSAPI-Kerberos) → PAM session → parent forks child sshd → /etc/passwd shell → /etc/profile → prompt (keystrokes = encrypted TCP).

Latency suspects (this symptom class):

/etc/ssh/sshd_config:
  UseDNS no                # kill slow/flappy reverse-DNS per login
  GSSAPIAuthentication no  # kill GSSAPI/Kerberos stalls if no working KDC

The 1% framework (when nothing works): sweep layers PHYSICAL → OS → NETWORK → DATA → APPLICATION (Security at every layer). Cloud folds data-link into physical; enumerate every config per layer; find the misconfig/external factor.

Observability: build latency / connection / per-user-error / 504-502 dashboards + Alertmanager — system-only dashboards stay green during ~70–80% of subtle outages.

Gotchas: disk “clean” but iNode/FD exhausted blocks new conns · MaxSessions caps fail all extra sessions (not a rotating subset) · PAM/SSSD cache expiry → transient denials · provider may make undisclosed changes · sometimes it’s an OS/hardware bug.

Remember: this drill teaches the method and the strongest hypothesis — the confirmed root cause was deliberately withheld for the follow-up RCA. Learn the sweep; demand the logs.

14. Gap-Fill — What the Session Left Unfinished, Completed Here

What this section is and isn’t. Everything above is faithful to the transcript. This section fills the gaps the engineer explicitly left open (assignment fodder, promised RCA material, the unresolved logical hole in the hypothesis) using established Linux/networking/SSH engineering knowledge. It is clearly labelled as gap-fill, not source material. Read it as the notes a senior SRE would add after the session.

GAP 1 — Resolving the Logical Hole: How a “Global” DNS/GSSAPI Issue Produces a Rotating Subset of Failures on One Host

The sharpest attendee question — “if GSSAPI/Kerberos/DNS are environment-global, why does only one machine and only some users fail?” — was left unanswered. Here is the engineering resolution.

The mechanism is not that DNS/KDC are globally broken. It’s that they are intermittently slow, and a per-login timeout threshold converts variable latency into binary pass/fail.

Walk through it:

Each login attempt has a server-side timeout budget
(ConnectTimeout / LoginGraceTime in sshd_config, typically 30–120 s)

DNS PTR lookup (if UseDNS=yes):
  - normally resolves in < 5 ms (local resolver cache)
  - if the cache EXPIRES or the upstream resolver is slow: 500 ms – 3 s per attempt
  - if the resolver FLAPS (answers sometimes, silently drops others): 5–30 s per attempt

GSSAPI negotiation (if GSSAPIAuthentication=yes, no working KDC):
  - sshd sends GSSAPI init token; KDC/client doesn't respond
  - sshd waits ConnectTimeout / a fixed internal retry (typically 5 s default)
  - retries 2–3 times before giving up and falling back to publickey
  - total added latency: 5–30 s depending on retry count and timeout config

COMBINED cost per login = DNS_lookup_time + GSSAPI_timeout * retries

Why does this produce three symptom buckets, rotating per hour?

Why does it rotate? The DNS resolver cache has a TTL. When the TTL expires, the next login from that source IP triggers a live DNS query. If the upstream resolver is flapping — answering some queries and silently dropping others — the outcome is probabilistic per query, not per user. A user who logged in 30 minutes ago (cache warm) is “fine.” One connecting now (cache cold, live query, dropped packet) times out. An hour later the sets have shuffled because different users’ resolver caches have expired. This is why the same user can be in all three buckets over the course of a day.

Why only prod, not bastion or other hosts? The PTR lookup resolves the client’s hostname — i.e., the bastion’s IP. If prod’s resolver entry for the bastion’s IP has a short TTL and prod’s local resolver is the one misbehaving (network issue within the VPC resolver path, or a recently rotated/misconfigured nameserver), then only prod’s SSHD is affected when it does the lookup. The bastion doesn’t try to resolve itself, and other hosts may not be using the same SSHD config (e.g., they might already have UseDNS no).

This also explains why the app looks healthy: HTTP/HTTPS traffic to the backend doesn’t go through SSHD’s PTR resolution or GSSAPI negotiation — those are only in the SSH control path. Services, Prometheus scrapes, and application requests are entirely unaffected.

GAP 2 — Per-Layer Config Enumeration (the Assigned Practice exercise, Done)

The engineer told teams to enumerate every SSH-relevant configuration on each layer. Here it is.

Layer 1: Physical (in cloud = kernel/VM resource limits)

What can be misconfigured here that affects SSH sessions:

Layer 2: OS (sshd daemon configuration)

This is the richest layer for this incident. Full relevant sshd_config parameters:

# /etc/ssh/sshd_config — parameters relevant to this incident

# --- Auth method selection ---
PubkeyAuthentication yes           # must be yes; check authorized_keys format
PasswordAuthentication no          # good hardening; confirm it matches intent
AuthenticationMethods publickey    # explicit; avoids GSSAPI being tried as fallback

# --- The two prime suspects ---
UseDNS no              # DISABLE: prevents reverse-PTR lookup per login (removes DNS latency)
GSSAPIAuthentication no # DISABLE: if no Kerberos KDC in this env, prevents wait+retry+timeout

# --- Session + user limits ---
MaxSessions 999        # (already confirmed set here)
MaxAuthTries 3         # reduce brute-force surface; too low causes legitimate failures
LoginGraceTime 60      # seconds sshd waits for auth; if DNS/GSSAPI takes >60s, login fails
ClientAliveInterval 15 # keepalive ping interval (seconds) to detect dead clients
ClientAliveCountMax 3  # disconnect after 3 missed keepalives (3×15=45s idle timeout)

# --- Connection limits ---
MaxStartups 10:30:100  # rate-limit: start:rate%:max unauthenticated connections
                       # syntax: block at 100 unauth conns; start dropping at 10 with 30% chance
                       # if many users connect simultaneously, hits this gate early

# --- Port + access control ---
Port 1101              # custom port (confirmed; must match firewall rule)
AllowUsers user1 user2 # whitelist (checked and "looked good")
DenyUsers ...          # blacklist

MaxStartups — the overlooked candidate. When 30 users log in simultaneously (e.g., all arrive at 11 AM), there may be a burst of unauthenticated connections before auth completes. With the default MaxStartups 10:30:100, sshd randomly drops new connections once 10 are in-flight (30% drop rate, rising to 100% at 100 unauth). This would produce exactly the “some succeed, some lag, some get timed out” pattern, including the randomness — because the drop is probabilistic. This is a strong secondary hypothesis the session did not name.

# How to verify MaxStartups as the cause:
# On prod, check current value:
sudo sshd -T | grep maxstartups

# If you see concurrent logins spike:
ss -tnp | grep :1101 | grep -c SYN_RECV   # count half-open connections

Layer 3: Network (VPC, routing, TCP stack)

Layer 4: Data (files, identity, caching)

Layer 5: Application (sshd as the application; OS Login agent)

GAP 3 — The Exact Diagnostic Procedure (Per-Layer Commands, in Order)

This is what the engineer’s promised RCA document would have contained. Run this top-to-bottom on the prod host.

# ============================================================
# PHASE 0: ESTABLISH BASELINE (run on the BASTION first)
# ============================================================
time ssh -v -p 1101 user@<prod_ip> exit 2>&1 | grep -E 'debug1|Authenticated|Elapsed'
# -v verbose output shows exactly which auth stage takes time.
# Run 5 times in quick succession; compare timings.
# Fast some times, slow others = intermittent (confirms variable latency, not hard block).

# ============================================================
# PHASE 1: PHYSICAL LAYER — resource limits
# ============================================================
# FD limits
ulimit -n                              # per-process (should be >= 65535)
cat /proc/sys/fs/file-max              # system-wide max
cat /proc/sys/fs/file-nr               # (used, free-minus-zero, max)
lsof 2>/dev/null | wc -l              # total open FDs right now

# iNodes
df -i                                  # look for 100% on any filesystem

# Ephemeral ports
cat /proc/sys/net/ipv4/ip_local_port_range   # should be wide (e.g., 1024–65535)

# conntrack
cat /proc/sys/net/netfilter/nf_conntrack_max 2>/dev/null
conntrack -C 2>/dev/null              # current count; if near max, that's the issue

# Load / steal
uptime                                 # load avg vs. vCPU count
vmstat 1 5                             # look for 'st' steal column > 5%
top -bn1 | head -5

# TCP backlog
sysctl net.core.somaxconn
sysctl net.ipv4.tcp_max_syn_backlog
ss -s                                  # connection summary (established, syn-recv, etc.)
ss -tnp | grep :1101                   # current SSH connections on the custom port

# ============================================================
# PHASE 2: OS LAYER — sshd configuration
# ============================================================
sudo sshd -T | grep -E 'usedns|gssapi|maxsessions|maxstartups|logingracetime|maxauthtries|clientalive'
# usedns no            <-- desired
# gssapiauthentication no  <-- desired
# maxstartups 10:30:100    <-- default; if you see many concurrent SYN_RECV conns this is the gate
# logingracetime 60s       <-- if DNS+GSSAPI take >60s, login always fails

# Check sshd is actually using the config you think:
sudo sshd -t          # test config validity; watch for warnings

# Active sshd processes (parent + per-user child):
ps -ef | grep sshd    # if many zombie children, FDs and PID slots are leaking

# ============================================================
# PHASE 3: NETWORK LAYER — DNS is the prime suspect
# ============================================================
cat /etc/resolv.conf
resolvectl status 2>/dev/null || systemd-resolve --status 2>/dev/null

# Run 20 reverse lookups against each nameserver; look for variance in timing:
BASTION_IP="10.20.0.5"   # replace with actual bastion IP
for i in $(seq 1 20); do
  { time dig +short -x $BASTION_IP; } 2>&1 | grep -E 'real|^[0-9]'
done
# A flapping resolver will show some entries taking < 5 ms and some taking 5000 ms (timeout).

# MTU check (rule out fragmentation on SSH key-exchange):
ping -M do -s 1472 <prod_ip>

# Routing:
ip route show
traceroute -T -p 1101 <prod_ip>    # TCP traceroute to custom port

# ============================================================
# PHASE 4: DATA LAYER — auth files, logs, SSSD cache
# ============================================================
# authorized_keys sanity:
stat /root/.ssh/authorized_keys    # mode must be 600, owner root
# For each user:
# stat /home/<user>/.ssh/authorized_keys   # mode 600, owner <user>

# known_hosts on BASTION (not prod):
# ssh-keygen -F <prod_ip>    # does bastion have a (possibly stale) entry?

# THE SMOKING GUN — timing in auth.log:
# On prod:
sudo grep "sshd" /var/log/auth.log | tail -200 \
  | awk '/Connection from/{t=$1" "$2" "$3; ip=$NF} /Accepted/{print t" -> "ip" accepted at "$1" "$2" "$3}'
# Large time gaps between "Connection from" and "Accepted" = DNS/GSSAPI latency.

# SSSD cache and connectivity:
sudo sssctl user-checks <username>      # end-to-end auth test for a specific user
sudo sssctl cache-expire -E            # flush SSSD cache (if cache is the culprit)
cat /var/log/sssd/sssd.log | grep -E 'Backend|offline|timeout'

# PAM:
sudo cat /etc/pam.d/sshd              # review module stack
sudo authselect current               # (RHEL/CentOS) what profile is active

# /etc/security/access.conf:
cat /etc/security/access.conf        # confirm users are in allow rules

# ============================================================
# PHASE 5: APPLICATION LAYER — OS Login, sshd as app
# ============================================================
# GCP OS Login:
systemctl status google-oslogin 2>/dev/null
sudo sshd -T | grep authorizedkeyscommand   # if non-empty, OS Login is active and calls GCP API

# GCP SSSD → API latency:
sudo tail -f /var/log/sssd/sssd.log &   # watch for "Backend is offline" during a failed login

# GSSAPI — test by disabling temporarily (NON-DESTRUCTIVE TEST):
# Add to /etc/ssh/sshd_config:  GSSAPIAuthentication no
# sudo systemctl reload sshd
# Then test a login: if lag disappears -> GSSAPI was the cause.
# Add UseDNS no as well and test again.

# RESTORE if testing:
# Remove/comment the test lines; sudo systemctl reload sshd

# ============================================================
# PHASE 6: CROSS-CUTTING — security / audit
# ============================================================
# Audit log for auth events:
sudo ausearch -k sshd 2>/dev/null | tail -50
sudo tail -f /var/log/audit/audit.log | grep sshd

# Check for any firewall/iptables rules that rate-limit:
sudo iptables -L -n -v | grep -E 'limit|REJECT|DROP'
sudo ip6tables -L -n -v | grep -E 'limit|REJECT|DROP'
# (GCP also has VPC-level stateful firewall; check console too)

GAP 4 — The MaxStartups Hypothesis (Named and Developed)

The session never mentioned MaxStartups. It is, alongside UseDNS/GSSAPI, the most plausible single-config explanation for the exact three-bucket rotating pattern described.

How it works:

MaxStartups in sshd_config controls how many unauthenticated connections sshd will accept simultaneously, using a three-part start:rate:full syntax:

MaxStartups 10:30:100
            │   │   └── refuse ALL new connections above 100 pending-auth
            │   └────── 30% chance of refusing once count exceeds 10
            └────────── start applying probabilistic drop at 10 pending-auth

When 30 users try to SSH at 11 AM:

1. The first 10 connect and reach the auth stage.
2. Users 11+ are randomly refused (30% chance each), experiencing an immediate connection refused / timeout — not an auth failure, a TCP-level refusal.
3. As some of the first 10 complete auth quickly, slots open and waiting users get through.
4. This is random (30% drop rate, not deterministic), produces exactly the three buckets (in / laggy due to retry / timed out), and the victim set rotates as connection slots are held and released.

Why it fits even better than GSSAPI/DNS alone:

• GSSAPI/DNS explains individual login lag, but not why the set rotates. MaxStartups explains the rotation directly: whoever happens to be in the connection queue when slots are full gets dropped, regardless of user identity.
• The two mechanisms can compound: GSSAPI/DNS adds 10–20 s to auth completion, keeping connections in the unauthenticated state longer, filling the MaxStartups queue faster.

Verify:

# Current value:
sudo sshd -T | grep maxstartups
# Count half-open SSH connections at peak time:
ss -tnp | grep ':1101' | grep 'SYN_RECV' | wc -l
# Temporarily raise (for diagnosis only):
# MaxStartups 100:30:200 in sshd_config; reload; test

GAP 5 — The Full PAM Stack Explained

The session named PAM but didn’t explain the module stack. For the SSH auth path on a Ubuntu/GCP VM:

# /etc/pam.d/sshd (typical Ubuntu 22 with OS Login / SSSD)

@include common-auth          # pam_unix.so (local /etc/shadow check) +
                              # pam_sssd.so (SSSD/cloud identity check)
@include common-account       # account validity (expiry, access.conf)
@include common-session       # session setup (limits, environment)
@include common-password      # (only for password changes; SSH key auth skips this)

# GCP-specific additions (with OS Login):
auth       [success=done ...] pam_google_oslogin.so
session    optional           pam_google_oslogin_admin.so

PAM execution flow for an SSH key login:

User connects (public key already accepted by sshd)
   |
   ├─ pam_unix.so (account) → checks /etc/passwd, /etc/shadow (not expired?)
   ├─ pam_sssd.so (account) → calls SSSD → calls cloud directory API
   │       if SSSD cache is EXPIRED:
   │         SSSD makes a live call to GCP/Cloud Identity API
   │         if API is slow → waits up to sssd.conf 'timeout' (default 5s)
   │         if API is unreachable → falls back to cached value (if within negcache_timeout)
   ├─ pam_access.so → checks /etc/security/access.conf (allow/deny rules)
   ├─ pam_limits.so → applies /etc/security/limits.conf (FD limits, etc.)
   └─ pam_env.so + pam_motd.so → sets environment, prints motd
Session established.

The SSSD negative cache (negcache_timeout): if a user was recently denied, SSSD caches that negative result for negcache_timeout seconds (default 15 s). A user who hit a transient API failure gets a cached “denied” for the next 15 s even if the API recovers — explaining why retrying immediately doesn’t help, but retrying a minute later does.

GAP 6 — SSSD Configuration Deep-Dive (the cache that matters)

# /etc/sssd/sssd.conf (relevant parameters)

[sssd]
services = nss, pam
domains = example.com    # or your cloud domain

[pam]
offline_credentials_expiration = 1  # allow cached creds for 1 day if backend offline

[domain/example.com]
id_provider = ldap         # or 'google' for GCP OS Login via SSSD
auth_provider = ldap

# Cache lifetimes:
entry_cache_timeout = 5400      # how long a positive result (user exists) is cached (90 min)
entry_cache_failure_timeout = 10 # how long a negative result (user not found) is cached (10 s)
# ↑ If this is 300 (5 min), a transient auth failure keeps the user "denied" for 5 minutes.

# Backend timeout:
timeout = 5                     # seconds to wait for backend response before marking offline

ldap_search_timeout = 6         # LDAP query timeout
ldap_network_timeout = 3        # TCP connection timeout to LDAP/backend

The intermittency link: if entry_cache_timeout = 5400, most logins hit the cache (fast). When the cache expires (every 90 min), the next login for that user triggers a live backend call. If the backend (GCP directory API, LDAP) is slow at that moment → that one login is slow. Other users whose cache hasn’t expired are fast. This produces a rotating pattern roughly correlated with SSSD cache TTLs — independent of DNS or GSSAPI.

GAP 7 — SSH Verbose Debugging (-vvv) — What Each Line Means

The most powerful diagnostic tool the session never mentioned explicitly:

# On the BASTION, run a verbose login (will print every step):
ssh -vvv -p 1101 user@<prod_ip> 2>&1 | tee /tmp/ssh_debug.txt

Key output lines and what they tell you:

debug1: Connecting to <prod_ip> [10.20.0.9] port 1101.
debug1: Connection established.                         ← TCP handshake done

debug1: SSH2_MSG_KEXINIT sent                          ← cipher/kex negotiation start
debug1: SSH2_MSG_KEXINIT received                      ← banner exchanged
debug2: kex: server->client cipher: aes128-ctr ...     ← cipher agreed

debug1: Server host key: <fingerprint>                 ← host-key check
debug1: Host '10.20.0.9' is known and matches ...      ← known_hosts match

debug1: Authentications that can continue: publickey   ← server's allowed methods
debug1: Trying private key: /home/user/.ssh/id_rsa

# ← IF THIS LINE IS MISSING OR DELAYED, GSSAPI IS BEING ATTEMPTED:
debug1: Offering public key: ...
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply

# ← TIME GAP HERE = DNS lookup + GSSAPI negotiation time:
debug1: Server accepts key: ...
debug1: Authentication succeeded (publickey).

debug2: PTY allocation request accepted on channel 0
debug1: Entering interactive session.                  ← shell created

If GSSAPI is being tried, you will see lines like:

debug3: Trying GSSAPI methods...
debug1: Miscellaneous failure (see text)
debug1: No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_...)

followed by a delay before falling back to publickey. That delay is the latency.

If DNS is the problem, the delay appears before the “Server host key” line — because sshd does the PTR lookup before completing the handshake.

GAP 8 — What a Production RCA Document Should Contain (Template)

The engineer promised to write one. Here is the standard structure:

# RCA — Production Incident: SSH Latency/Timeout, Bastion→Prod
# Date: <date>  | Severity: P0  | Duration: <HH:MM>

## 1. Incident Summary
One paragraph: what broke, when, for how long, who was affected, business impact.

## 2. Timeline
| Time (IST) | Event |
|---|---|
| 11:00 | Issue reported by DS team lead |
| 11:05 | War room opened; BAU team engaged |
| 11:10 | Initial triage: CPU/RAM/logs checked — no issues found |
| 11:25 | sssd/sshd restarted — no improvement |
| 11:35 | VM rebooted — no improvement |
| ... | ... |
| HH:MM | Root cause identified |
| HH:MM | Fix applied; login tested for all 30 users |
| HH:MM | Incident closed; monitoring confirmed stable |

## 3. Root Cause
One clear sentence: what specific misconfiguration/failure caused the incident.
[e.g. "GSSAPIAuthentication=yes in /etc/ssh/sshd_config, with no active Kerberos KDC
reachable from the prod VM, caused sshd to attempt and time out GSSAPI negotiation
on every login, adding 15–30s latency. Combined with UseDNS=yes and intermittent
PTR lookup failures from the VPC resolver, some logins exceeded LoginGraceTime and
timed out entirely. Affected users rotated as resolver cache TTLs expired."]

## 4. Contributing Factors
Bullet list of things that made it worse or masked it:
- No per-login latency dashboard (incident would have been visible immediately)
- GSSAPI/UseDNS default values not reviewed at VM creation time
- MaxStartups at default (compounding effect during simultaneous logins)

## 5. Impact
- N users affected for X hours
- Business: Y dashboard updates missed; CFO visibility lost for Z hours
- No customer/external impact

## 6. Resolution
Exact commands run to fix:
```bash
# /etc/ssh/sshd_config
GSSAPIAuthentication no
UseDNS no
# sudo systemctl reload sshd
# Verified: time ssh -v ... showed \< 2s per login for all 30 users

7. Prevention / Action Items

8. Lessons Learned

• ”App looks healthy” ≠ “users are healthy.” Build user-path SLIs.
• sshd default values (GSSAPIAuthentication, UseDNS) are inherited from OpenSSH defaults and are not always appropriate for cloud-VM environments without Kerberos.
• Structured per-layer debugging (Physical→OS→Network→Data→App) reaches root cause faster than random checklist execution.

---

### GAP 9 — Hardening & Prevention Checklist (What Should Have Been in Place)

Things that, if already configured, would have either prevented this or made diagnosis instant:

**`sshd_config` baseline for a cloud VM without Kerberos:**

GSSAPIAuthentication no # disable unless you have a working KDC UseDNS no # disable unless you need hostname-based access control MaxStartups 50:30:200 # raise from default to handle burst logins LoginGraceTime 30 # tight; forces fast detection of auth stalls ClientAliveInterval 10 # keepalive every 10s to detect dead sessions quickly ClientAliveCountMax 3 # drop dead session after 30s MaxAuthTries 3 # reduce brute-force surface

**Observability additions (the dashboards that would have caught this in seconds):**

| Panel | Data source | Query idea |
|---|---|---|
| SSH login latency P50/P99 | Prometheus node-exporter + custom script or `sshguard` | `time ssh ... exit` measured every 30s from bastion; histogram |
| Active SSH sessions | `ss -tnp \| grep sshd \| wc -l` via node-exporter text collector | Line chart; alert on > X |
| SSSD backend status | SSSD provides metrics; or parse `/var/log/sssd/sssd.log` | Alert on "Backend is offline" |
| DNS resolution latency | `dig` probe via Blackbox exporter | Alert on PTR lookup time > 100ms |
| FD usage | `node_filefd_allocated` (node-exporter default) | Alert on > 80% of system max |
| conntrack usage | `node_nf_conntrack_entries` | Alert on > 80% of `nf_conntrack_max` |

**Ansible role snippet (infrastructure-as-code prevention):**
```yaml
# roles/hardened_sshd/tasks/main.yml
- name: Harden sshd_config
  lineinfile:
    path: /etc/ssh/sshd_config
    regexp: "{{ item.regexp }}"
    line: "{{ item.line }}"
    validate: 'sshd -t -f %s'
  loop:
    - { regexp: '^#?GSSAPIAuthentication', line: 'GSSAPIAuthentication no' }
    - { regexp: '^#?UseDNS',               line: 'UseDNS no' }
    - { regexp: '^#?MaxStartups',          line: 'MaxStartups 50:30:200' }
    - { regexp: '^#?LoginGraceTime',       line: 'LoginGraceTime 30' }
    - { regexp: '^#?ClientAliveInterval',  line: 'ClientAliveInterval 10' }
    - { regexp: '^#?ClientAliveCountMax',  line: 'ClientAliveCountMax 3' }
  notify: Reload sshd

- name: Verify sshd config is valid
  command: sshd -t
  changed_when: false

GAP 10 — Closing the Loop: What the “Official” Answer Most Likely Is

The session never revealed it, and engineering honesty requires flagging it as inference. That said, given everything the engineer telegraphed and the engineering analysis above, the most defensible answer is a compound root cause, not a single one:

Primary: GSSAPIAuthentication yes (default) + no working Kerberos KDC reachable from the prod VM → sshd attempts GSSAPI negotiation on every login, waits the GSSAPI timeout, retries, then falls back to publickey. Added cost per login: 10–30 seconds.

Secondary / amplifying: UseDNS yes (default) → sshd does a PTR lookup of the bastion’s IP on every login. The VPC’s internal DNS resolver intermittently drops or delays responses (flapping nameserver, expired TTL). Added cost per login: 0–15 seconds (intermittent).

Compounding: MaxStartups at its default of 10:30:100 → when all 30 users are slow to authenticate (because of GSSAPI + DNS), more connections pile up in the unauthenticated state, pushing the count past 10 and triggering the probabilistic drop. This converts some “slow logins” into “timed-out logins,” and which users get dropped is random — producing the rotating three-bucket pattern.

Fix (two lines, instant effect):

# /etc/ssh/sshd_config
GSSAPIAuthentication no
UseDNS no
# sudo systemctl reload sshd   ← no restart needed; reload is enough

Raise MaxStartups as a second step; tune LoginGraceTime downward to 30 s to surface these stalls faster in future.

Why SSSD cache / PAM expiry is a supporting (not primary) factor: it explains some of the occasional-denial aspect for specific users whose cache entries expired and had to be live-resolved — but it doesn’t explain the rotating-among-all-30 pattern as cleanly as GSSAPI+DNS+MaxStartups does.