Complete Learning Package (SSH · OSI Troubleshooting · Linux Networking · Verbose Logs · Production Debugging)

2. Table of Contents

1. Executive Summary
2. Table of Contents
3. Detailed Structured Notes
   • 3.1 Infrastructure Setup (Replicated on AWS)
   • 3.2 The Symptom Revisited (Modified Problem Statement)
   • 3.3 The Troubleshooting Philosophy — Structured vs. Guesswork
   • 3.4 OSI Troubleshooting Framework — Applied Layer by Layer
   • 3.5 Layer 1 (Physical) — Ping / ICMP Reachability
   • 3.6 Layer 2 (Data-Link) — ARP Resolution
   • 3.7 Layer 3 (Network) — Tracepath / Routing
   • 3.8 Layer 4 (Transport) — Netcat / TCP Handshake
   • 3.9 Layer 5–6 (Session/Presentation) — SSH Verbose Log (-vvv)
   • 3.10 SSH Verbose Log — Line-by-Line Interpretation
   • 3.11 The Critical Finding: Shell Allocated But Frozen
   • 3.12 Layer 7 (Application) — Kernel and Config Files
   • 3.13 Observed Evidence Summary
   • 3.14 Hypothesis and Contributing Factors
   • 3.15 Common SSH Failure Patterns and Where to Look
   • 3.16 Configuration Files to Inspect on the Production Server
   • 3.17 Four Ways to Access EC2 When SSH Is Broken
   • 3.18 Assignment
   • 3.19 OSI Framework — When to Use It and When Not To
   • 3.20 Week 2 Content Overview (Kubernetes Control Plane)
   • 3.21 Battle-Ops Chaos Simulation — How It Works (Ravi)
4. Key Concepts Table
5. Architecture & Workflow Analysis
6. Commands Reference (All Commands Executed Live)
7. Tools & Technologies
8. Real-World Production Usage
9. Interview Preparation (Beginner / Intermediate / Advanced)
10. Exam & Certification Notes
11. Cheat Sheet — SSH Troubleshooting Quick Reference
12. Gaps, Assumptions & Things the Session Left Open
13. Gap-Fill — What the Session Left Unfinished, Completed Here

3. Detailed Structured Notes

3.1 Infrastructure Setup (Replicated on AWS)

Key infrastructure files:

• Bastion ~/.ssh/ — contains prod-key (private key) and prod-key.pub (public key)
• Prod /root/.ssh/authorized_keys — contains the bastion’s public key

3.2 The Symptom Revisited (Modified Problem Statement)

Same scenario as Week 1 but with deliberate modifications to add more contributing factors and a higher failure rate:

Live demonstration:

# On the bastion:
ssh -i prod-key ec2-user@<prod-private-ip>
# → terminal hangs indefinitely
# → eventually times out
# → on rare occasions, connects but is extremely slow (10–15s per keystroke)

Differences from Week 1:

1. Network topology: bastion in public subnet, prod in private subnet (more production-realistic).
2. Failure rate raised from ~70% failure (Week 1) to ~90% failure (Week 2).
3. Multiple contributing factors deliberately embedded (4–5 compounding issues vs. the single/dual from Week 1).

3.3 The Troubleshooting Philosophy — Structured vs. Guesswork

The engineer opened with this principle:

“In a production outage, don’t do guesswork. Don’t assume it’s a CPU issue and check CPU. Don’t assume it’s DNS and check DNS. Start from a structured troubleshooting framework — OSI — and climb the ladder layer by layer. Each layer adds evidence that builds your hypothesis.”

The two-phase approach:

1. Quick/obvious fixes first (5 minutes): Check resources (CPU/RAM), common config files (DNS, SSH config), restart daemons. If fixed → done. These solve ~99% of SSH issues.
2. Structured OSI traversal (if obvious fixes fail): Systematic, evidence-building, bottom-up. Never skip layers or jump ahead.

Critical caveat on OSI scope:

• OSI is ideal for Linux system troubleshooting.
• For Kubernetes outages, use the V7 (Verdict-7) Kubernetes troubleshooting framework instead — OSI still works but takes far longer on a distributed, multi-component system.
• For monitoring system failures, use a monitoring-specific framework.

3.4 OSI Troubleshooting Framework — Applied Layer by Layer

The classic OSI model mapped to what you actually debug in a Linux/cloud environment:

Traversal rule: Start at Layer 1. Gather evidence at each layer. Do not conclude at any single layer — climb all the way up before forming a hypothesis. Each layer adds to the picture.

3.5 Layer 1 (Physical) — Ping / ICMP Reachability

Purpose: Verify the destination host is reachable at the most basic level.

Command executed:

# Send 5 ICMP packets from bastion to prod:
ping -c 5 <prod-private-ip>

Observed result:

5 packets transmitted, 0 received, 100% packet loss

Interpretation:

• Destination is not reachable at ICMP level.
• Network path is broken somewhere.
• But: this alone tells us nothing about where the break is. Do not conclude at this layer.

What to check via ping:

• Packet loss %: 0% = healthy; any loss = signal of degradation; 100% = total ICMP block or routing failure.
• RTT latency: spikes indicate congestion or routing issues.
• Whether ICMP is even allowed (security groups block ICMP by default in AWS if not explicitly allowed — but in this case, prod is in the same VPC/subnet, so ICMP between private IPs should be allowed).

3.6 Layer 2 (Data-Link) — ARP Resolution

Purpose: Verify the system can resolve IP → MAC address. ARP (Address Resolution Protocol) is the layer-2 mechanism by which a machine finds its neighbor’s hardware address. If ARP fails, the machine cannot send frames to the target, even if the IP is correct.

Command executed:

arp -n | grep <prod-private-ip>
# or
arp -n <prod-private-ip>

Observed result:

(no output / no entry)

Interpretation:

• ARP returned no MAC address → Layer 2 neighbor discovery is broken.
• The production server is not responding at the data-link layer.
• This is a severe signal — even more fundamental than ping failure.
• Combined with Layer 1 failure: both L1 and L2 are failing → “the destination cannot be discovered at all.”

Note on ARP cache: An ARP entry that was previously cached may appear temporarily even after connectivity is lost. The absence of any entry (no stale cache) here confirms the machine has never successfully resolved this IP in recent memory, or the cache was cleared.

3.7 Layer 3 (Network) — Tracepath / Routing

Purpose: Find where packets stop travelling — i.e., at which hop do they drop. This isolates whether the issue is inside the subnet (routing table, NACL, kernel forwarding) or outside it (upstream routing, internet gateway, etc.).

Command executed:

tracepath <prod-private-ip>
# Alternative: traceroute <prod-private-ip>

Observed result:

1: localhost          (hop 1) — no reply
2:                   (hop 2) — no reply
3:                   (hop 3) — no reply
...
(all hops: no reply)

Interpretation:

• Traffic is not leaving the subnet — it stops at hop 1 (the local gateway or routing entry).
• This narrows the issue to one of:
  • VPC routing table misconfiguration (local route missing or corrupt)
  • NACL (Network ACL) blocking traffic between subnets
  • Kernel-level forwarding issue (ip_forward disabled, conntrack table full, iptables rules)
  • NIC / driver issue at the OS level
• Does not implicate security groups yet (security groups are stateful and checked at the ENI, not the routing level).

3.8 Layer 4 (Transport) — Netcat / TCP Handshake

Purpose: Test whether TCP can establish a connection to the destination port (22). This is the key differentiator — if TCP succeeds while ICMP fails, it proves that security groups, NACLs, and routing are not blocking SSH specifically.

Command executed:

nc -zv <prod-private-ip> 22
# or
netcat -z <prod-private-ip> 22

Observed result:

Connection to <prod-private-ip> 22 port [tcp/ssh] succeeded!

Interpretation — this is the most important finding so far:

• TCP 3-way handshake succeeded on port 22.
• This proves (eliminates from suspicion):
  • ✅ Security groups are not blocking SSH traffic
  • ✅ NACLs are not blocking SSH traffic
  • ✅ ENI (Elastic Network Interface) is attached and functional
  • ✅ The routing table does have a path to the destination (at least at TCP level)
• But ping (ICMP) and ARP still fail — this creates a paradox: TCP works, ICMP does not.
• This pattern points to: ICMP is blocked at firewall/iptables/security-group level while TCP on port 22 is allowed. Or the kernel’s networking stack is partially degraded (alive for established TCP sessions, dead for ICMP/ARP broadcasts).

The TCP success narrows the hypothesis significantly:

“The networking stack is alive enough for TCP handshakes but dead for data forwarding of ICMP/ARP. This is a classic pattern of a partially degraded kernel networking stack.”

3.9 Layer 5–6 (Session/Presentation) — SSH Verbose Log

Purpose: Watch every step of the SSH negotiation in real time to find exactly where the connection freezes.

Command executed:

ssh -vvv -i prod-key ec2-user@<prod-private-ip>
# -v = verbose; -vv = more verbose; -vvv = maximum verbosity

What verbose SSH output shows (in order):

1. Loading client SSH configuration files
2. OpenSSH client version
3. IP/hostname resolution (DNS lookup if hostname used; skipped if IP used directly)
4. TCP connection establishment
5. SSH version negotiation (client version ↔ server version)
6. Key exchange algorithm negotiation (Diffie-Hellman / ECDH)
7. Host key verification (against ~/.ssh/known_hosts)
8. Authentication method selection (GSSAPI → public key → password)
9. Private key loading and server match against authorized_keys
10. Session channel creation
11. Shell allocation
12. Login banner + prompt

3.10 SSH Verbose Log — Line-by-Line Interpretation

This section maps what was observed in the live session to what each log line means.

Phase 1 — Client configuration loading

debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line N: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config.d/...

Meaning: SSH client is loading its configuration. Lines like “match not found” are normal — it’s checking Host blocks in the config for a matching entry.

Phase 2 — No DNS lookup (using IP directly)

When using an IP address (ssh -i key ec2-user@10.0.2.204), no DNS lookup occurs. If you had used a hostname, you’d see:

debug1: Resolving <hostname> to <IP>

The engineer noted: “Using IP is best practice for internal SSH — independent of DNS.”

Phase 3 — TCP connection

debug1: Connecting to 10.0.2.204 [10.0.2.204] port 22.
debug1: Connection established.

Meaning: TCP 3-way handshake completed successfully. Confirms no security group / NACL issue.

Phase 4 — Version negotiation

debug1: Remote protocol version 2.0, remote software version OpenSSH_X.X
debug1: match: OpenSSH_X.X ...

Meaning: Both sides have announced their OpenSSH version. A version incompatibility here would cause Protocol major versions differ and abort. No error here = OK.

Phase 5 — Diffie-Hellman key exchange

debug2: kex: server->client cipher: aes128-ctr mac: ...
debug2: kex: client->server cipher: aes128-ctr mac: ...
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY

Meaning: Client and server are negotiating encryption algorithms (the Diffie-Hellman / ECDH key exchange the engineer referenced as “Hellman”). Both sides generate ephemeral keys, share public components, and derive a shared secret without it ever traversing the wire. This protects against packet capture.

Phase 6 — Host key verification

debug1: Server host key: ecdsa-sha2-nistp256 SHA256:...
debug1: Host '10.0.2.204' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:1

Meaning: Client checked ~/.ssh/known_hosts and found a matching entry — the server’s identity is verified. If the host key had changed (e.g., instance rebuilt with same IP), you’d see: WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!

Phase 7 — Authentication method negotiation (GSSAPI → Public Key)

debug3: Trying to reverse map address 10.0.2.204.
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: no credentials for the GSSAPI mechanism 'Kerberos v5' (krb5)
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey

Meaning: SSH tries authentication methods in order:

1. GSSAPI/Kerberos first — fails because this environment has no Kerberos KDC. The “no credentials” message is not an error — it’s expected when GSSAPI is not configured. SSH just moves to the next method.
2. Public key next — this is our configured method.

Key insight: This is the same GSSAPI latency mechanism from Week 1. If GSSAPIAuthentication yes is in sshd_config, SSH tries Kerberos first, waits for a response, and only then falls back to publickey. This adds latency. Fix: GSSAPIAuthentication no.

Phase 8 — Public key authentication

debug1: Offering public key: /root/.ssh/prod-key RSA ...
debug1: Authentications that can continue: publickey
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: /root/.ssh/prod-key RSA ...
debug1: Authentication succeeded (publickey).

Meaning: Client proved possession of the private key; server validated the corresponding public key in authorized_keys. No error here = auth is fully working. If this line had been Permission denied (publickey), it would indicate a key mismatch or wrong authorized_keys content.

Phase 9 — Shell allocation (where it freezes)

debug1: channel 0: new [client-session]
debug2: channel 0: send open
debug1: Requesting pty-req
debug1: Requesting shell
debug1: channel 0: open confirm, rwindow 0 rmax 32768

Meaning: The SSH session channel opened. A PTY (pseudo-terminal) and shell were requested. The server confirmed channel open. At this point, the connection freezes.

Critical finding: “SSH authentication is working correctly but it freezes when trying to create an interactive session / child process. The SSH is not stuck — but the login shell is blocked after authentication.”

Phase 10 — Login banner (appears briefly then freezes)

debug1: Found target ...
/usr/bin/lesspipe.sh: ...

Meaning: The shell started (login banner appeared), but the full interactive session never initialized. The freeze happens after the shell fork — in the execution of login scripts (/etc/profile, /etc/profile.d/*, PAM session setup, etc.).

3.11 The Critical Finding: Shell Allocated But Frozen

What this means technically:

The SSHD process flow is:

SSHD (parent, listens for connections)
  → TCP + SSH handshake → auth → success
  → SSHD forks a child process for this session
  → child process calls /bin/bash (or configured shell) via /etc/passwd
  → /etc/profile + /etc/profile.d/*.sh scripts execute
  → PAM session setup runs
  → Interactive prompt appears

In this outage: The freeze occurs between “shell allocated” and “interactive prompt appears.” The child process was forked, but the session initialization is hanging.

Possible causes (all must be investigated):

1. A script in /etc/profile.d/ that blocks (DNS lookup, remote call, infinite loop, file lock).
2. PAM session module making a slow external call (SSSD → cloud API → timeout).
3. Ephemeral port exhaustion — the child process needs to open a new socket for the PTY or a local service; if no ports available, it blocks.
4. Conntrack table full — new connection tracking entries cannot be created.
5. /etc/resolv.conf misconfiguration causing reverse DNS lookup timeout on the connecting client’s IP (if UseDNS yes in sshd_config).
6. A GSSAPI timeout before falling back to public key (adds latency; combined with other factors crosses the threshold).

3.12 Layer 7 (Application) — Kernel and Config Files

The engineer walked through several configuration locations on the production server, checking for contributing factors:

3.12.1 DNS Configuration — /etc/resolv.conf

cat /etc/resolv.conf

What to look for:

• nameserver entries: AWS internal DNS (169.254.169.253) should be present.
• Multiple name servers: if first is slow/broken, resolver tries the next, adding retry delay per query.
• options timeout:N attempts:M: if timeout=5 and attempts=3, a broken DNS = 15s delay.
• On a local Linux system you’d see 3 nameservers: system IP, internet gateway, destination.

3.12.2 Ephemeral Port Range — /etc/sysctl.conf.d/ or sysctl

# Check the current ephemeral port range:
cat /proc/sys/net/ipv4/ip_local_port_range
# or
sysctl net.ipv4.ip_local_port_range

Observed finding in this walkthrough:

port range: 1000 only (extremely narrow)

A range of only 1,000 ports means that if there are more than ~1,000 concurrent outbound connections or new sockets being created, the system runs out of ephemeral ports. Any new process needing a port (including new SSH session child processes) will block until a port becomes available.

Healthy range: 32768–60999 (28,231 ports). Some tuned systems use 1024–65535.

This is a confirmed contributing factor for the random SSH freezes.

3.12.3 Connection Tracking — conntrack

conntrack -C   # count current entries
cat /proc/sys/net/netfilter/nf_conntrack_max   # maximum entries

Observed finding: No entries / conntrack not full (ruled out in this walkthrough). However, a full conntrack table causes new TCP connections to be silently dropped — a situation similar to a retry storm (referenced to Uber’s architecture incident).

3.12.4 Kernel Load — top / system load

top -bn1 | head -5

Checked for: zombie processes, CPU steal, run-queue depth. Not identified as an issue in this walkthrough.

3.12.5 SSHD Configuration — /etc/ssh/sshd_config

cat /etc/ssh/sshd_config

Key settings visible in the session:

• Port 22 (standard SSH port, confirmed)
• UseDNS yes — this is a contributing factor. With UseDNS yes, SSHD performs a reverse PTR lookup of the connecting client’s IP. If DNS is slow or flapping, this adds seconds to every login.
• Cipher exchange, key exchange algorithm settings visible.
• AllowUsers / DenyUsers settings.
• SSH port forwarding settings.
• MaxSessions, MaxAuthTries visible.

Fix to apply: UseDNS no in /etc/ssh/sshd_config + systemctl restart sshd.

3.12.6 PAM Configuration — /etc/pam.d/sshd

cat /etc/pam.d/sshd

PAM modules run after SSH authentication succeeds. A misconfigured or slow PAM module can block the shell initialization. Specifically:

• pam_sssd.so calling a slow or offline identity backend.
• pam_access.so with strict access control lists.
• pam_limits.so enforcing session limits (max sessions exceeded → new session blocked).

3.12.7 Profile and Shell Init — /etc/profile.d/

ls /etc/profile.d/
cat /etc/profile.d/<any-script>.sh

Every .sh file in this directory executes for every new login session. A script that:

• Makes a DNS lookup.
• Calls an external service.
• Has a blocking wait.
• Has a bug causing it to hang.

…will freeze the login session exactly as observed.

3.12.8 Identity Latency Check (Layer 6 — Presentation)

time id

Purpose: Measures how long it takes to resolve the current user’s identity. If SSSD or LDAP is slow, id will hang — and since PAM / profile scripts call id implicitly, this adds to login latency.

Observed in session: No anomaly (not flagged as a confirmed issue, but included in the diagnostic checklist).

3.13 Observed Evidence Summary

3.14 Hypothesis and Contributing Factors

Combined conclusion from all observed evidence:

“The networking stack is alive enough for TCP handshakes but dead for ICMP data forwarding. The SSH protocol is completely healthy — auth, key exchange, host verification all pass. The freeze happens in shell initialization due to a combination of factors.”

Contributing factors identified (4–5 compound issues):

Important instructor note: “This is not a single issue. It is 4–5 contributing issues working together. That’s why guesswork fails — fixing only one doesn’t resolve the outage. You need to find and fix all of them.”

3.15 Common SSH Failure Patterns and Where to Look

The engineer gave this diagnostic map (the most reusable content from the whiteboard section):

3.16 Configuration Files to Inspect on the Production Server

The complete list of files the engineer identified as relevant to this outage:

3.17 Four Ways to Access EC2 When SSH Is Broken

When SSH is broken and you need to access the production server to investigate, the engineer listed four methods in order:

Best practice order: Try SSM first (zero-downtime) → serial console → automation runbook → EBS detach (last resort, requires instance stop).

Hands-On Exercise

Assignment 1 — SSH into the production server using one of the four alternative methods, then:

1. Go through every configuration file listed in Section 3.16.
2. Identify which settings are misconfigured or contributing to the outage.
3. Note your findings per file.
4. Post in the team channel: “I found the issue: <description>.”

Assignment 2 — Fix the issues: After identifying all contributing factors, apply fixes:

• UseDNS no and GSSAPIAuthentication no in /etc/ssh/sshd_config + systemctl restart sshd.
• Fix ephemeral port range: echo "1024 65535" > /proc/sys/net/ipv4/ip_local_port_range (temporary) or persist via /etc/sysctl.conf.
• Fix any blocking /etc/profile.d/ scripts.
• Fix any PAM session limits.
• Fix ARP/ICMP issues if identifiable.

Assignment 3 — Write an RCA: Use the RCA template uploaded to Google Drive. Structure:

• Problem statement
• Timeline of troubleshooting (what was checked at each step)
• Root cause(s) identified
• Solution applied
• Prevention / how to avoid recurrence
• Upload to your GitHub directory.

Instructor note: The solution will be shown live on Tuesday’s Q&A if not solved by participants.

3.19 OSI Framework — When to Use It and When Not To

Use OSI when:

• Linux system troubleshooting (network connectivity, service unreachability).
• Classic infrastructure components (EC2, bare metal, VMs).
• You have no clue where the issue is.
• The issue involves network stack behavior (ping, SSH, HTTP connectivity).

Do NOT use OSI when (use a more targeted framework):

• Kubernetes outages: Use the V7 (Verdict-7) framework — covers API server, etcd, scheduler, kubelet, kube-proxy, CNI, and workload layers. OSI works but takes far longer in a distributed multi-component environment.
• Monitoring system failures (Prometheus/Grafana): Use a monitoring-specific investigation path.
• Application performance issues: Use APM (distributed tracing), not OSI.
• You already know the root cause: Apply the known fix directly; don’t waste P0 time on a full OSI sweep.

When gut feeling is valid: If you own the infrastructure and have seen the same failure before, apply your known fix first. Only if that fails → structured framework.

3.20 Week 2 Content Overview (Kubernetes Control Plane)

The engineer briefly walked through what Week 2 covers (30-question assessment pushed to participants via Discord):

Core-ops Week 2 topics:

• API server: Acts as the front door to Kubernetes (all kubectl commands go through it). How it works; what happens when it’s down.
• etcd internals and failures: etcd is the distributed key-value store backing all cluster state. etcd failure = complete cluster state loss. How to back up, restore, and recover etcd.
• Node internals: kubelet, container runtime, CNI plugin.
• kube-proxy: Networking component managing iptables/IPVS rules for service routing. Critical for understanding why Service to Pod routing works or fails.
• Version upgrades and version skew: How to upgrade control plane components without breaking the cluster; acceptable version differences between components.
• Control plane failure scenarios: API server down, etcd down, kubelet stopped.

Uploaded resources in Week 2 drive directory:

• Module documentation (per day)
• Assignments (per day) with assignment-starter setup instructions
• Interview playbook (how to structure answers to K8s scenario questions in interviews)
• OSI troubleshooting framework reference card
• V7 (Verdict-7) Kubernetes troubleshooting framework reference card
• RCA template
• Recorded videos: specific control plane failure scenarios (API server down, etcd failure, kubelet issues)

Week 2 practical setup:

# Step 1: Create the cluster (from assignment-starter doc)
# Initialize control plane
sudo kubeadm init --pod-network-cidr=10.244.0.0/16

# Step 2: Configure kubectl
mkdir -p $HOME/.kube
sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config

# Step 3: Install CNI (e.g., Flannel)
kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

# Step 4: Join worker nodes
sudo kubeadm join <control-plane-ip>:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>

3.21 Battle-Ops Chaos Simulation — How It Works (Ravi)

The second presenter (Ravi) briefly demonstrated how the battle-ops chaos agents work in the training cluster. The key concept: in a lab environment, you have to simulate load and failures that would happen naturally in production.

Chaos agents run under the system-monitoring namespace as DaemonSets.

Agent 1 — Kubelet chaos injection:

# DaemonSet that sends stop signals to kubelet:
command:
  - /bin/sh
  - -c
  - |
    while true; do
      sleep $CHAOS_INTERVAL
      kill -SIGSTOP $(pidof kubelet)
      sleep $PAUSE_DURATION
      kill -SIGCONT $(pidof kubelet)
    done

What it simulates: kubelet becoming unresponsive (as would happen with high memory pressure, kernel panic recovery, or resource exhaustion). Nodes appear NotReady intermittently.

Agent 2 — Random pod crash:

# DaemonSet that randomly selects a pod and kills its container:
command:
  - /bin/sh
  - -c
  - |
    while true; do
      sleep $INTERVAL
      TARGET_POD=$(kubectl get pods -A -o name | shuf -n 1)
      kubectl exec $TARGET_POD -- kill -9 1  # kill PID 1 in the container
    done

What it simulates: Container crashes (OOMKill, application crash, or process killed externally). Tests whether your application handles container restarts correctly.

Agent 3 — Network load: Injects network traffic to simulate bandwidth saturation or network pressure.

Agent 4 — Memory pressure: Allocates memory to fill available RAM, testing OOM behavior and eviction policies.

Agent 5 — CPU pressure / disk I/O pressure: Stress-tests CPU and disk I/O to observe scheduler behavior, resource quota enforcement, and pod eviction.

How to investigate when chaos strikes:

1. Check Kubernetes events: kubectl get events -A --sort-by='.lastTimestamp' — events show the first trigger.
2. Check pod logs: kubectl logs <pod> --previous — logs from before the crash.
3. Check node conditions: kubectl describe node <node> — look for MemoryPressure, DiskPressure, PIDPressure.
4. Find the agent: look for unusual DaemonSet pods in system-monitoring namespace → understand what the agent is doing → stop it.

Resolution example (for pod crash agent): Stop the crash-agent DaemonSet → pods stop crashing → RCA: DaemonSet in system-monitoring was sending SIGKILL to container PID 1.

4. Key Concepts Table

5. Architecture & Workflow Analysis

5.1 Network Topology (Week 2)

Internet
    │
Internet Gateway (IGW)
    │
  VPC
  ├── Public Subnet
  │     └── Bastion Server (T3.small)
  │           ├── Public IP:  <bastion-public-ip>
  │           ├── Private IP: 10.0.0.106
  │           └── SSH entry: ssh -i sre-labs.pem ec2-user@<public-ip>
  │
  └── Private Subnet
        └── Production Server (T3.xlarge)
              ├── Private IP: 10.0.2.204 (NO public IP)
              ├── No internet gateway
              └── SSH target: ssh -i prod-key ec2-user@10.0.2.204
                  (from bastion only)

5.2 OSI Debugging Ladder (Applied)

SYMPTOM: SSH hangs/times out from bastion → prod

L7 Application    check sshd_config, PAM, profile.d, sysctl ports
    ↑
L6 Presentation   check time id (identity latency), cipher negotiation
    ↑
L5 Session        ssh -vvv (verbose): auth OK, shell allocated, FREEZE HERE
    ↑
L4 Transport      nc -zv 10.0.2.204 22 → CONNECTION SUCCEEDED ✓
    ↑
L3 Network        tracepath 10.0.2.204 → stops at hop 1 (no reply) ✗
    ↑
L2 Data-link      arp -n | grep 10.0.2.204 → no entry (no MAC) ✗
    ↑
L1 Physical       ping -c 5 10.0.2.204 → 100% packet loss ✗

KEY CONTRADICTION: L4 (TCP) works ✓ but L1+L2+L3 fail ✗
→ Classic partial networking stack failure
→ "Alive for TCP, dead for ICMP/ARP"

5.3 SSH Login Pipeline with Freeze Points Annotated

Bastion (SSH client)                     Production (SSHD)
    │                                           │
    ├──[TCP SYN]─────────────────────────────►│
    │◄─[TCP SYN-ACK]──────────────────────────┤   L4: WORKS ✓
    ├──[TCP ACK]─────────────────────────────►│
    │                                           │
    ├──[SSH banner]──────────────────────────►│
    │◄─[SSH banner + version]─────────────────┤   L5: WORKS ✓
    │                                           │
    ├──[GSSAPI init (times out)]──────────────►│   LATENCY HERE ← GSSAPIAuthentication yes
    │◄─[GSSAPI failed]────────────────────────┤
    │                                           │
    ├──[publickey offer: prod-key]────────────►│
    │◄─[publickey accepted]───────────────────┤   Auth: WORKS ✓
    │                                           │
    ├──[request PTY + shell]─────────────────►│
    │◄─[channel open confirmed]───────────────┤
    │                                           │
    │                 sshd forks child process  │
    │                 child calls /bin/bash     │
    │                 PAM session setup         │   ← SSSD slow?
    │                 /etc/profile.d/ scripts   │   ← blocking script?
    │                 ephemeral port needed     │   ← port range exhausted?
    │                 UseDNS=yes → PTR lookup  │   ← DNS slow?
    │                                           │
    │                 [FREEZE - never returns]  │   ← OUTAGE POINT
    │◄──[...silence...]─────────────────────────┤
    │
    [timeout]

5.4 Four EC2 Access Methods When SSH Fails

SSH broken → Try in order:

1. SSM Session Manager
   aws ssm start-session --target <instance-id>
   (needs: SSM agent running + IAM role)
   [In this session: SSM agent offline → BLOCKED]

2. EC2 Serial Console
   AWS Console → EC2 → Connect → EC2 Serial Console
   (needs: enabled in account + local user with password set)

3. AWS Systems Manager Automation
   SSM → Automation → Run → AWSSupport-ResetAccess
   Sets a new password → use with serial console

4. EBS Detach → Modify → Reattach [LAST RESORT]
   Stop instance → detach root volume → attach to rescue instance
   → mount → fix files → detach → reattach to original → start

6. Commands Reference (All Commands Executed Live)

OSI Layer Diagnostics (from bastion → prod)

# L1: Ping (ICMP reachability)
ping -c 5 <prod-private-ip>
# Expected healthy: 0% packet loss, RTT < 1ms (same VPC/subnet)
# Observed: 100% packet loss

# L2: ARP resolution
arp -n | grep <prod-private-ip>
arp -n <prod-private-ip>
# Expected healthy: <prod-private-ip>  ether  <mac-address>  C  <interface>
# Observed: (no output)

# L3: Tracepath (routing hops)
tracepath <prod-private-ip>
traceroute -T -p 22 <prod-private-ip>   # TCP traceroute alternative
# Expected healthy: hop 1 = local gateway, hop 2 = prod (or direct)
# Observed: all hops = no reply

# L4: TCP handshake (netcat)
nc -zv <prod-private-ip> 22
netcat -z <prod-private-ip> 22
# Expected healthy: "Connection to <ip> 22 port [tcp/ssh] succeeded!"
# Observed: SUCCEEDED ← TCP works

# L5-6: SSH verbose log
ssh -vvv -i prod-key ec2-user@<prod-private-ip>
# Read output line by line; watch for where it freezes after "shell allocated"

# L7: Identity resolution latency
time id
# Expected healthy: < 100ms
# Slow = SSSD/PAM making remote call

Config File Inspection

# DNS configuration
cat /etc/resolv.conf

# Ephemeral port range
cat /proc/sys/net/ipv4/ip_local_port_range
sysctl net.ipv4.ip_local_port_range
# Healthy: "32768 60999" or wider
# Fix (temporary): echo "1024 65535" > /proc/sys/net/ipv4/ip_local_port_range
# Fix (permanent): add to /etc/sysctl.conf: net.ipv4.ip_local_port_range = 1024 65535

# Conntrack (connection tracking table)
conntrack -C                                    # current entry count
cat /proc/sys/net/netfilter/nf_conntrack_max    # max entries
# If current ≈ max → table full → silent TCP drops

# Kernel system call and port settings
ls /etc/sysctl.d/
cat /etc/sysctl.d/*.conf

# SSHD configuration
cat /etc/ssh/sshd_config
# Key settings to check:
#   UseDNS no                      ← fix if yes
#   GSSAPIAuthentication no        ← fix if yes
#   MaxSessions 10                 ← increase if too low
#   AllowUsers ec2-user admin      ← verify your user is listed

# PAM SSH configuration
cat /etc/pam.d/sshd
ls /etc/pam.d/

# Profile and login scripts
ls /etc/profile.d/
cat /etc/profile.d/*.sh           # look for blocking operations

# Security limits (FD limits, session limits)
cat /etc/security/limits.conf

# ARP cache management
# Clear ARP cache (remove stale entries):
ip -s -s neigh flush all
arp -d <prod-private-ip>

# NIC reset (if link-layer issue suspected)
ip link set <interface> down
ip link set <interface> up

Fixes to Apply

# Fix 1: Disable reverse DNS lookup in sshd (primary fix for SSH lag)
sudo sed -i 's/UseDNS yes/UseDNS no/' /etc/ssh/sshd_config
# Or manually edit /etc/ssh/sshd_config: UseDNS no

# Fix 2: Disable GSSAPI (if no Kerberos KDC)
sudo sed -i 's/GSSAPIAuthentication yes/GSSAPIAuthentication no/' /etc/ssh/sshd_config

# Fix 3: Expand ephemeral port range
sudo sysctl -w net.ipv4.ip_local_port_range="1024 65535"
echo "net.ipv4.ip_local_port_range = 1024 65535" | sudo tee -a /etc/sysctl.conf

# Apply all sysctl changes permanently:
sudo sysctl -p

# Restart SSHD to apply config changes (must be done via non-SSH access):
sudo systemctl restart sshd
# Check it's running:
sudo systemctl status sshd

# Fix 4: Clear ARP cache
sudo ip -s -s neigh flush all

# Fix 5: Reset NIC
sudo ip link set eth0 down && sudo ip link set eth0 up

SSM Access (Alternative to SSH)

# Check SSM agent status (on the instance):
sudo systemctl status amazon-ssm-agent

# Install/restart SSM agent:
sudo systemctl enable amazon-ssm-agent
sudo systemctl start amazon-ssm-agent

# Start session from local machine:
aws ssm start-session \
  --target <instance-id> \
  --region <region>

RCA Template Sections (from Google Drive template)

# RCA Document Structure:
1. Incident title + date + severity
2. Timeline:
   - When reported
   - When investigation started
   - What was checked at each time point
   - When root cause identified
   - When fix applied
   - When incident closed
3. Root cause(s): specific, factual
4. Contributing factors: list all
5. Solution: exact commands applied
6. Prevention: what changes prevent recurrence
7. Action items: owner + deadline
8. Lessons learned

7. Tools & Technologies

8. Real-World Production Usage

The TCP-works-but-ICMP-fails paradox is not uncommon:

• AWS security groups block ICMP by default unless an inbound rule explicitly allows it.
• In production, ICMP is often blocked by security policy for security reasons — so ping always fails between instances even when they’re healthy. This is why netcat/nc is more reliable than ping for connectivity testing in AWS.
• The implication: never use “ping works” as the primary success criterion in AWS.

Compound faults are the most dangerous class of production outage:

• Any single factor here might cause occasional SSH lag on its own.
• All five together make the system nearly unusable.
• In a real incident, you have a P0 clock ticking. Fixing one factor and seeing “it works now” is a false resolution — the next deployment or traffic burst will surface the remaining factors.
• Real RCA practice: Fix all identified factors in one maintenance window, not one at a time.

The Telegram reference: The engineer mentioned Telegram experienced a similar “networking stack alive for TCP but dead for data forwarding” incident at scale. The underlying mechanism (partial kernel networking degradation) affects any large-scale Linux infrastructure, not just lab environments.

SSM as a production best practice: Many mature AWS organizations mandate SSM as the primary EC2 access method and disable SSH entirely:

• No port 22 open to the internet.
• No key pairs to manage.
• Session recording in S3 for audit.
• IAM controls access (no key distribution).
• Works even if the network is misconfigured (uses HTTPS to AWS endpoints, not inbound TCP/22).

9. Interview Preparation

Beginner

Q1. Walk through the OSI model layers from L1 to L7 in the context of debugging an SSH connection failure. A: L1 Physical: ping to check ICMP reachability and packet loss. L2 Data-Link: arp -n to check MAC address resolution. L3 Network: tracepath to find where packets stop. L4 Transport: nc -zv <ip> 22 to test TCP handshake. L5–6 Session/Presentation: ssh -vvv verbose log to watch every negotiation step. L7 Application: inspect sshd_config, PAM, /etc/profile.d/, sysctl (port range, conntrack).

Q2. SSH times out connecting to an EC2 instance. Ping also fails. But nc -zv <ip> 22 succeeds. What does this tell you? A: TCP handshake on port 22 works → security groups, NACLs, and routing are not blocking SSH. Ping failure is a separate issue — ICMP may be blocked by security group rules (common in AWS). The problem is likely higher in the stack: key mismatch, GSSAPI timeout, reverse DNS, PAM session limits, or ephemeral port exhaustion — not a network connectivity issue.

Q3. What is ARP and why does an arp -n check matter during SSH troubleshooting? A: ARP (Address Resolution Protocol) maps IP addresses to MAC addresses at Layer 2. Without a MAC address, the OS cannot build Ethernet frames to send to the target, even if the IP is correct. If arp -n returns no entry for the destination, it means Layer 2 neighbor discovery is failing — possibly a NIC issue, ARP cache problem, or the destination is not on the expected subnet.

Intermediate

Q4. You run ssh -vvv and see “Authentication succeeded (publickey)” followed by “channel 0 opened, shell allocated” — and then the terminal freezes. What do you investigate? A: The freeze is happening in shell initialization, after SSH auth. Investigate: (1) /etc/profile.d/ scripts for any blocking operations. (2) PAM session setup — cat /etc/pam.d/sshd; SSSD or external auth making slow calls. (3) Ephemeral port range — cat /proc/sys/net/ipv4/ip_local_port_range; if only 1,000 ports, new child processes block waiting for a port. (4) UseDNS yes in sshd_config — reverse DNS adding latency. (5) Conntrack table full — conntrack -C vs nf_conntrack_max.

Q5. SSH is hanging at login (after entering password/accepting key), but it eventually connects after 30 seconds. What are the likely causes? A: The classic causes for login lag (vs full timeout) are: (1) Reverse DNS lookup — UseDNS yes + slow PTR resolution; fix: UseDNS no. (2) GSSAPI/Kerberos timeout — SSH tries GSSAPI first, waits for KDC response, retries, then falls back to publickey; fix: GSSAPIAuthentication no. (3) SSSD/PAM slow external call — SSSD calling cloud identity API with high latency; check SSSD logs. (4) /etc/profile.d/ script with remote call. These are the same root causes as a full freeze, just below the timeout threshold.

Q6. Four ways to access an EC2 instance when SSH is broken. List them in order of preference and explain the tradeoff. A: (1) SSM Session Manager — best; no network dependencies, audit-logged, IAM-controlled; requires SSM agent running. (2) EC2 Serial Console — console-level access; needs pre-set local password and must be enabled in account settings; doesn’t require network. (3) SSM Automation Runbook (AWSSupport-ResetAccess) — resets OS password; combine with serial console; slow but non-disruptive. (4) EBS detach/reattach — most powerful (can fix any OS-level config); requires stopping the instance; highest disruption.

Advanced

Q7. Explain the “networking stack alive for TCP but dead for ICMP/ARP” scenario. What causes it and what does it indicate? A: It indicates a partial degradation of the Linux kernel networking stack. The TCP/IP stack has different code paths for: (a) ICMP processing (handled at raw socket / kernel level) and (b) ARP (handled at the link layer, requires NIC driver). TCP (especially when an existing flow is established or the SYN/SYN-ACK succeeds) may work because the connection-tracking state machine has already set up the entry. ICMP and ARP, which are lower-level and require the full link layer to be functional, fail. This can be caused by: kernel memory pressure causing link-layer buffer exhaustion, NIC driver fault, conntrack table overflow, iptables rules specifically blocking ICMP while allowing established TCP, or corruption in the kernel’s networking data structures. In AWS, it often indicates the EC2 instance’s ENI or its underlying hypervisor networking is degraded.

Q8. An incident involves a compound fault with 5 contributing factors. You fix one (UseDNS), verify SSH works, and close the incident. Three days later, the same symptom returns. What went wrong and how do you prevent it? A: Fixing only one of five contributing factors resolved the dominant bottleneck temporarily — the others remained. Under slightly different load conditions (more concurrent SSH sessions, slightly slower DNS at that moment), the remaining factors became sufficient to reproduce the symptom. Prevention: complete the full RCA before closing the incident; fix all identified factors in the same maintenance window; test all fixes together. Post-incident: add monitoring for each identified factor (port range utilization, conntrack table usage, DNS PTR latency, SSSD backend health) so future degradation is caught before it causes a full outage.

Q9. You’re in a P0 incident. SSH to production is broken. SSM agent is offline. The instance has no serial console password set. The application is serving traffic via a different path (not SSH-dependent). What do you do? A: Since the application is still serving traffic, you have some breathing room. Options: (1) Use SSM Automation Runbook to reset the local OS password → then use serial console. (2) If the EBS volume can be detached safely (instance can be stopped), stop the instance, attach root volume to a rescue instance, mount it, fix /etc/ssh/sshd_config (UseDNS no, GSSAPIAuthentication no), resize port range in /etc/sysctl.conf, unmount, reattach to original, start → SSH works. (3) If stopping the instance is unacceptable: snapshot the root volume → create a new instance from that snapshot with the fixes applied → cut over DNS/load balancer to the new instance. In parallel: post-incident, mandate SSM agent installation and serial console password setup in your AMI baking process so you’re never locked out again.

10. Exam & Certification Notes

Linux networking (LFCS, CKA, CKS):

• ping, traceroute, netstat, nc/netcat, arp — all appear in Linux certification practical exams.
• sysctl parameters: net.ipv4.ip_local_port_range, net.ipv4.ip_forward, net.netfilter.nf_conntrack_max — common exam topics.
• sshd_config options: UseDNS, GSSAPIAuthentication, AllowUsers, MaxSessions, PasswordAuthentication, PubkeyAuthentication — all testable.
• PAM configuration: understanding module stack order (required/sufficient/optional) — CKS and LFCS.

CKA (Kubernetes):

• Diagnosing node NotReady — maps to Week 2 content (kubelet failure scenario).
• etcd backup and restore — explicitly on CKA exam.
• kubectl drain, cordon, uncordon — from last session.
• Control plane component troubleshooting.

Potential trick questions:

• “Ping fails → therefore there’s a network problem” → Not necessarily. AWS security groups block ICMP by default. Always test with nc on port 22 or 443, not just ping.
• ”SSH auth succeeds → SSH is working” → Not complete. Authentication success and session initialization are separate. Auth can succeed while shell initialization freezes.
• ”ARP failure means the host is down” → Not necessarily. ARP cache can be expired or NIC degraded while the host’s TCP stack still accepts connections.
• ”Fixing UseDNS stops the SSH hang” → May be one of several contributing factors. Verify all factors before declaring the incident resolved.

11. Cheat Sheet — SSH Troubleshooting Quick Reference

When SSH hangs or times out, run this in order:

STEP 1: Obvious quick checks (5 minutes)
├── ping -c 5 <target>           Check reachability
├── cat /etc/resolv.conf         Check DNS config
├── systemctl status sshd         Check sshd running
└── cat /proc/sys/net/ipv4/ip_local_port_range  Check port range

STEP 2: OSI ladder (if quick checks don't find it)
├── L1: ping -c 5 <ip>            ICMP reachability
├── L2: arp -n | grep <ip>        ARP / MAC resolution
├── L3: tracepath <ip>            Routing hops
├── L4: nc -zv <ip> 22            TCP handshake
├── L5: ssh -vvv -i key user@ip   Verbose SSH (read EVERY line)
└── L7: (see config files below)

STEP 3: SSH verbose log interpretation
├── "Connection established"         → TCP OK (SG/NACL fine)
├── "GSSAPI no credentials"          → Normal; not an error
├── "Authentication succeeded"       → Auth OK
├── "Shell allocated, then silence"  → Freeze in shell init
└── "Permission denied"              → Check authorized_keys (mode 600)

STEP 4: Config files to check
├── /etc/ssh/sshd_config        → UseDNS no | GSSAPIAuthentication no
├── /etc/resolv.conf            → Nameserver IPs; timeout/attempts
├── /proc/sys/net/ipv4/ip_local_port_range → Should be ~28,000 ports wide
├── /etc/pam.d/sshd             → No slow modules
├── /etc/profile.d/             → No blocking scripts
└── conntrack -C vs nf_conntrack_max → Table not full

FIXES
├── UseDNS no in sshd_config + systemctl restart sshd
├── GSSAPIAuthentication no in sshd_config
├── sysctl -w net.ipv4.ip_local_port_range="1024 65535"
├── ip -s -s neigh flush all  (clear ARP cache)
└── ip link set eth0 down && ip link set eth0 up  (reset NIC)

IF SSH IS BROKEN AND YOU'RE LOCKED OUT:
1. SSM Session Manager:  aws ssm start-session --target <instance-id>
2. EC2 Serial Console:   AWS Console → EC2 → Connect → Serial Console
3. SSM Automation:       SSM → Automation → AWSSupport-ResetAccess
4. EBS Detach/Reattach:  Stop instance → attach volume to rescue → fix → reattach

Common SSH failure symptoms → likely cause:

12. Gaps, Assumptions & Things the Session Left Open

Promised for later:

• Detailed SSH troubleshooting PDF with all commands + where to check — promised by instructor for EOD or next morning.
• Terraform files to replicate the exact outage environment — to be uploaded so participants can rebuild and re-experience.
• Tuesday Q&A — instructor will show the solution live if participants haven’t solved it.
• Complete command reference sheet — instructor said he’d share by EOD.
• RCA solution — not shown in this walkthrough; assignment to participants.

Session did not reveal the root cause: The engineer deliberately left the root cause for participants to discover. Five contributing factors were identified but not confirmed; the assignment is to find them via the alternative access methods.

Transcription artifacts:

• “Knackl / Knackal” = NACL (Network Access Control List)
• “SSD / SSSD” used interchangeably = context-dependent; in PAM context = SSSD (System Security Services Daemon); in sshd context = sshd
• ”Cont uh trac” / “contract” = conntrack (connection tracking)
• “Sis ctl / Sys ctl” = sysctl
• ”Profile uh D” = /etc/profile.d/
• “ARP cashes” = ARP cache
• ”ENI” = Elastic Network Interface (AWS)
• “Helman” = Diffie-Hellman key exchange algorithm
• ”Cubectl” = kubectl
• ”Cubl uh lit” = kubelet
• ”V7 / Verdict 7” = V7 Kubernetes troubleshooting framework (distinct from OSI)

Assumptions made:

• The “Telegram” incident referenced is consistent with documented Telegram infrastructure incidents involving kernel networking stack degradation. Treated as an illustrative real-world reference, not verified.
• ip_local_port_range = only 1000 is described as intentionally set by the lab to simulate port exhaustion. In production, this would be a serious misconfiguration.
• The five contributing factors are inferred from the engineer’s descriptions; the exact fifth factor was not clearly named in the transcript.

13. Gap-Fill — What the Session Left Unfinished, Completed Here

Filled from Linux networking and SSH engineering knowledge. Clearly labelled as gap-fill.

GAP 1 — The Five Contributing Factors (Identified and Completed)

The engineer said there are “4–5 issues” but never enumerated them completely. Based on everything observed in the session, here is the complete compound fault list with the evidence that implicates each:

GAP 2 — Conntrack: What Happens When It’s Full

The session checked conntrack and found it not full — but didn’t explain the failure mode. This is important for completeness.

When the conntrack table is full:

nf_conntrack_max = 65536 (default)
current entries   = 65536

New connection attempt:
  → kernel tries to create conntrack entry
  → table full
  → kernel silently DROPS the SYN packet
  → from the client's perspective: connection timeout
  → ICMP "host unreachable" is NOT sent
  → looks exactly like: ping fails, TCP times out

Why this is dangerous: It’s completely invisible from the outside. The server is running, the application is up, but no new connections can be established. No error is logged at the application level.

Detection and fix:

# Detect:
conntrack -C                                      # current count
cat /proc/sys/net/netfilter/nf_conntrack_max     # max

# If current ≈ max: table is full

# Fix (temporary — increase max):
sysctl -w net.netfilter.nf_conntrack_max=131072

# Fix (permanent):
echo "net.netfilter.nf_conntrack_max = 131072" >> /etc/sysctl.conf
sysctl -p

# Fix (structural — flush stale entries):
conntrack -F    # flush all tracked connections (CAUTION: drops all established connections briefly)

GAP 3 — Complete /etc/profile.d/ Blocking Script Diagnosis

The session identified this as a likely cause but didn’t show how to diagnose it specifically. Here is the procedure:

# Step 1: List all scripts in profile.d
ls -la /etc/profile.d/

# Step 2: Run each script individually with time measurement
for script in /etc/profile.d/*.sh; do
  echo -n "Testing $script: "
  timeout 5 bash -x "$script" 2>&1 | tail -3
  echo "  (exit: $?)"
done

# Step 3: If a script hangs (timeout 5 catches it):
# Identify which script → read its content
cat /etc/profile.d/<offending-script>.sh

# Common blocking patterns to look for:
# - curl / wget to an external URL
# - nslookup / dig (DNS lookups)
# - aws metadata calls (curl http://169.254.169.254/...)
# - SSSD/LDAP lookups
# - File locking (flock without timeout)
# - Infinite loops (while true; do ...; done without exit)

# Step 4: Fix
# Option A: Remove the script
sudo rm /etc/profile.d/<offending-script>.sh

# Option B: Add a timeout to the blocking call inside the script
# Before: curl http://slow-service/config
# After:  timeout 2 curl http://slow-service/config || true

GAP 4 — V7 Kubernetes Troubleshooting Framework (Referenced but Not Explained)

The engineer mentioned “Verdict-7 / V7 framework” for Kubernetes as an alternative to OSI. Here is what it covers:

V7 KUBERNETES TROUBLESHOOTING FRAMEWORK
(Start from the workload and work downward toward infrastructure)

Layer 1: WORKLOAD (Pod/Deployment/StatefulSet)
  → Is the pod Running? CrashLoopBackoff? Pending?
  → kubectl describe pod <pod> → events, conditions
  → kubectl logs <pod> --previous

Layer 2: SCHEDULING (kube-scheduler)
  → Why is the pod Pending?
  → Node selector / affinity mismatch?
  → Resource requests exceeding available capacity?
  → kubectl describe pod → "Events: FailedScheduling"

Layer 3: RUNTIME (kubelet + container runtime)
  → Is kubelet running? (systemctl status kubelet)
  → Is the container image pullable? (ImagePullBackOff)
  → Is the container starting? (container runtime logs)
  → journalctl -u kubelet -f

Layer 4: NETWORKING (CNI + kube-proxy)
  → Can pods reach each other? (kubectl exec → ping/curl)
  → Is the Service resolving? (kubectl exec → nslookup <service>)
  → Are iptables rules correct? (iptables -L -t nat)
  → Is CoreDNS running? (kubectl get pods -n kube-system)

Layer 5: CONTROL PLANE (API server + etcd + scheduler + controller-manager)
  → Is kube-apiserver responding? (kubectl get nodes)
  → Is etcd healthy? (etcdctl endpoint health)
  → Are control plane pods running? (kubectl get pods -n kube-system)
  → sudo journalctl -u kube-apiserver

Layer 6: NODE (OS + kernel)
  → Node Ready/NotReady? (kubectl get nodes)
  → kubelet started? Running? (systemctl status kubelet)
  → Disk pressure? Memory pressure? PID pressure?
  → OOM kills? (dmesg | grep -i oom)

Layer 7: INFRASTRUCTURE (cloud provider / hardware)
  → EC2 instance reachable? (ping, nc)
  → VPC/SG/NACL allowing traffic?
  → EBS volumes attached and healthy?
  → Cloud provider status page

V7 vs OSI for Kubernetes:

• OSI is bottom-up (infrastructure first). V7 is top-down (workload first) — aligns with how Kubernetes issues usually manifest (pod fails → investigate why).
• V7 is faster for K8s because it follows the natural failure visibility order.
• OSI is still useful for node-level or VPC-level networking issues.

GAP 5 — Complete NIC Reset and ARP Cache Fix Procedure

The session mentioned these fixes briefly without showing the commands. Here is the complete procedure for resetting the network interface and clearing ARP to resolve L1/L2 failures:

# Step 1: Identify the network interface name
ip link show
# or
ifconfig -a
# Common names: eth0, ens3, ens5 (AWS Nitro instances), enX0

# Step 2: Check interface state and statistics
ip -s link show eth0
# Look for: RX errors, TX errors, dropped packets — high values = hardware/driver issue
ethtool eth0  # NIC driver info, link status

# Step 3: Clear stale ARP entries
# Clear all:
sudo ip -s -s neigh flush all
# Clear specific IP:
sudo arp -d <prod-private-ip>
# Verify cleared:
arp -n | grep <prod-private-ip>   # should return nothing

# Step 4: Reset the NIC (brings link down then up; reloads driver state)
sudo ip link set eth0 down
sleep 2
sudo ip link set eth0 up
# Wait a few seconds for link to come back up
ip link show eth0  # should show "state UP"

# Step 5: Re-verify ARP after reset
ping -c 1 <prod-private-ip>
arp -n | grep <prod-private-ip>   # should now show a MAC address if L2 is restored

# Step 6: If NIC reset doesn't help (driver-level issue)
# Reload NIC driver module (kernel module):
lsmod | grep ixgbe   # or ena, virtio_net — whichever your NIC uses
sudo modprobe -r ena && sudo modprobe ena   # example for AWS ENA driver

# On AWS EC2: NIC hardware issue usually requires stopping/starting the instance
# (not just reboot — stop+start migrates to new hypervisor/physical host)

GAP 6 — SSM Session Manager Full Setup (Prerequisite Checklist)

The session showed SSM was blocked because the agent was offline. Here is the complete setup to ensure SSM always works as a fallback:

On the EC2 instance (via AMI or user data):

# Install SSM agent (Amazon Linux 2 / Amazon Linux 2023 — usually pre-installed):
sudo yum install -y amazon-ssm-agent
sudo systemctl enable amazon-ssm-agent
sudo systemctl start amazon-ssm-agent
sudo systemctl status amazon-ssm-agent  # should show: active (running)

# For Ubuntu:
sudo snap install amazon-ssm-agent --classic
sudo systemctl enable snap.amazon-ssm-agent.amazon-ssm-agent.service
sudo systemctl start snap.amazon-ssm-agent.amazon-ssm-agent.service

IAM requirements (EC2 instance must have this role):

{
  "PolicyName": "SSMAccess",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssm:UpdateInstanceInformation",
        "ssm:GetMessages",
        "ssm:SendReply",
        "ssmmessages:CreateControlChannel",
        "ssmmessages:CreateDataChannel",
        "ssmmessages:OpenControlChannel",
        "ssmmessages:OpenDataChannel"
      ],
      "Resource": "*"
    }
  ]
}

Attach AmazonSSMManagedInstanceCore managed policy (includes all the above) to the EC2 instance role.

Network requirement: The instance must be able to reach SSM endpoints via HTTPS (port 443) either through:

• Internet gateway (public subnet), or
• VPC Interface Endpoint for SSM: com.amazonaws.<region>.ssm + ssmmessages + ec2messages

For private-subnet instances without NAT Gateway:

# Create VPC endpoints for SSM:
aws ec2 create-vpc-endpoint \
  --vpc-id <vpc-id> \
  --vpc-endpoint-type Interface \
  --service-name com.amazonaws.<region>.ssm \
  --subnet-ids <subnet-id> \
  --security-group-ids <sg-id>

# Repeat for:
# com.amazonaws.<region>.ssmmessages
# com.amazonaws.<region>.ec2messages

Production hardening: Disable SSH entirely (AllowUsers set to empty, PasswordAuthentication no, PubkeyAuthentication no) and use SSM exclusively for management access. This eliminates the entire SSH attack surface.