Production SSH/Bastion Outage Debugging

1. Scenario Situation

During normal operations, a development team reports that connection jumps from the Bastion (jump) server to the core production database VM are failing intermittently:

• ~12 users connect instantly and run commands smoothly.
• ~6 users can log in, but suffer a painful 10-15 second lag per keystroke or command.
• ~8 users experience complete connection timeouts when trying to log in.

Crucially, the specific users affected rotate randomly throughout the day. System diagnostics show:

• CPU, RAM, and Disk space are under 25% utilization on both servers.
• The target application database is running normally.
• The SSH service has been hardened to use a custom port (1101) instead of the standard port 22.

2. Investigation Phase

Step 2.1: Network Isolation

To confirm basic network layer connectivity between the bastion and production host, execute a ICMP ping check from the bastion terminal:

$ ping -c 5 prod-vm-ip

Output:

5 packets transmitted, 5 received, 0% packet loss, time 4004ms
rtt min/avg/max/mdev = 0.420/0.450/0.482/0.024 ms

The sub-millisecond round-trip time (RTT) rules out physical network routing delays.

Step 2.2: Port Checking

Check if the custom SSH port 1101 is accepting connections:

$ telnet prod-vm-ip 1101

Output:

Trying prod-vm-ip...
Connected to prod-vm-ip.
Escape character is '^]'.
SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1

The TCP 3-way handshake completes instantly, indicating the firewall and security groups are correctly configured.

Step 2.3: Diagnostic Verbose SSH Connection

To pinpoint where the login process stalls, trigger a connection from the bastion in verbose mode:

$ ssh -vvv -i bastion-key.pem -p 1101 ec2-user@prod-vm-ip

Review the debug traces. The connection initiates quickly but hangs for exactly 10 seconds at this line:

debug1: Next authentication method: publickey
... [Hangs here for 10 seconds] ...
debug1: Authentication succeeded (publickey).

The connection successfully authenticates using public key files, but only after a significant, repeatable delay.

Step 2.4: Reviewing Server Auth Logs

Inspect the authentication log files on the target production VM:

$ sudo tail -f /var/log/auth.log

(On RHEL/CentOS systems, use /var/log/secure)

Output:

sshd[9482]: Connection from 10.0.1.52 port 49202 on 10.0.1.18 port 1101
sshd[9482]: debug1: PAM: establishing credentials
sshd[9482]: Accepted publickey for ec2-user from 10.0.1.52 port 49202 ssh2: RSA SHA256:...

The timestamps show a delay of exactly 10 seconds between the initial connection request and the PAM credential step.

3. Root Cause Analysis

The intermittent latency and timeouts are caused by the combination of two default settings in /etc/ssh/sshd_config on the target production server:

3.1 Reverse DNS Lookup (UseDNS)

By default, OpenSSH has UseDNS yes enabled. When a client initiates a connection, the SSH daemon:

1. Performs a reverse DNS lookup on the incoming client IP (the bastion’s internal IP) to locate its hostname.
2. Performs a forward DNS lookup on that hostname to verify it maps back to the client IP.

If the VPC’s internal DNS resolver is flappy or experiences packet drops, the DNS queries will hang. The SSH daemon blocks the login handshake until the DNS queries time out (typically 5, 10, or 15 seconds).

If the resolver fails completely, the connection times out entirely. If the resolver answers quickly (due to cached records), the login is instant.

3.2 GSSAPI Authentication (GSSAPIAuthentication)

GSSAPIAuthentication yes is enabled by default. This forces the server to attempt Kerberos ticket authentication and lookup Key Distribution Centers (KDCs) that do not exist in this VPC layout. This adds a secondary timeout delay.

4. Mitigation and Resolution

To fix the login delays and timeouts, disable DNS lookups and GSSAPI authentication:

1. Edit the SSH configuration file on the production VM:

   $ sudo nano /etc/ssh/sshd_config
   

2. Locate or append the following configuration parameters:

   UseDNS no
   GSSAPIAuthentication no
   

3. Test the configuration for syntax errors:

   $ sudo sshd -t
   

4. Reload the SSH daemon to apply changes:

   $ sudo systemctl reload sshd
   

Once updated, all SSH connection handshakes complete in under 150ms.