Network Diagnostics: Layer-by-Layer Triage

Network incidents reward discipline over cleverness. Test one layer at a time, in order, and each result eliminates a whole category of causes. Skip around and you will spend an hour proving things you had already disproved.

Topic 1: The Triage Ladder

Work upward. Each rung assumes the one below it passed.

The result that confuses people:

ping fails but nc -zv host 443 succeeds. This is not a contradiction. ping uses ICMP, and a great many networks — most cloud security groups by default — drop ICMP while permitting TCP. A failed ping proves nothing on its own. Treat it as one data point, never as a verdict.

The reverse is more informative: ping succeeds but the port does not connect. The host is reachable and routing is fine, so the problem is the service or a port-level filter.

Topic 2: Interfaces and Routing

The ip suite replaced ifconfig and route years ago; the old commands are often not even installed on modern images.

ip -br addr              # brief: interface, state, addresses
ip -br link              # layer 2 state and MAC addresses
ip route                 # the routing table
ip route get 10.0.4.19   # which route WILL be used for this destination
ip neigh                 # ARP cache -- layer 2 neighbours

ip route get is the one to remember. Rather than making you interpret the whole table, it tells you the exact route, source address, and interface the kernel would choose for one destination.

$ ip route get 10.0.4.19
10.0.4.19 via 10.0.0.1 dev eth0 src 10.0.1.18 uid 1000

Read it as: to reach that address, go via gateway 10.0.0.1, out of eth0, using 10.0.1.18 as the source. On a multi-homed host, an unexpected interface here explains asymmetric routing immediately.

Topic 3: Sockets and Connection States

ss replaced netstat and is dramatically faster on busy hosts, because it reads kernel structures directly rather than parsing /proc/net/tcp line by line.

ss -tlnp                     # TCP, listening, numeric, with process
ss -tunap                    # TCP and UDP, all states, numeric, with process
ss -s                        # summary counts by state
ss -tn state established     # filter by state
ss -tn dst 10.0.4.19         # everything talking to one peer
ss -ti                       # per-socket TCP internals: rtt, cwnd, retransmits

The listening-address trap:

State   Recv-Q  Send-Q  Local Address:Port
LISTEN  0       128     127.0.0.1:8080
LISTEN  0       128     0.0.0.0:9090

The first socket is bound to loopback only — it works from the host and refuses every remote connection. The second accepts on all interfaces. “The service is running and the port is open” is a claim that has to name the address, not just the port.

Connection states worth recognising:

CLOSE-WAIT deserves emphasis: the remote end hung up and your code never closed its descriptor. It will not clear on its own, it leaks descriptors, and no kernel tuning fixes it. That is a code fix.

Queues tell you where the backlog is:

• Recv-Q on a LISTEN socket — completed connections the application has not accepted yet. Nonzero means the app is too slow to pick up work.
• Send-Q on an ESTAB socket — bytes written but unacknowledged. Sustained values suggest the network or the receiver is the bottleneck.

netstat, and the translation table:

netstat ships in the net-tools package and is frequently absent from modern images, but it is what most existing runbooks say. The translations:

lsof answers the same question from the file side, and is often the quickest way to resolve “address already in use”:

lsof -i :80          # who owns port 80
lsof -i -nP | grep LISTEN

Try it yourself: Run ss -s for a state census, then ss -tan state close-wait | wc -l. On a healthy host that number is near zero.

Topic 3b: Proving Reachability with curl

curl tests the application layer end to end, and the shape of its failure tells you which layer actually broke. Work through an application that cannot reach its database:

# 1. Is our own service even answering?
curl -I -s myapplication:5000
# HTTP/1.0 500 INTERNAL SERVER ERROR      <- app is up, but unhappy

# 2. Can we reach the database directly?
curl -I -s database:27017
# curl: (6) Could not resolve host: database

# 3. Is it DNS in general, or just this name?
curl -I -s https://example.com
# HTTP/1.1 200 OK                          <- internet and DNS both fine

Step 3 is the one people skip, and it is the one that narrows the fault: general name resolution works, so the problem is specific to the database record — a missing service entry, the wrong search domain, or a resolver that does not know about internal names.

Useful flags: -I for headers only, -s to silence the progress meter, -v for the full handshake, -o /dev/null -w '%{time_total}\n' to time a request, and --resolve name:port:IP to test a specific backend while bypassing DNS entirely.

When the firewall is the answer:

If packets leave and nothing returns, check the local rules before escalating to the network team:

sudo iptables -S            # rules in a readable, replayable form
sudo nft list ruleset       # nftables, on newer systems
sudo ufw status verbose     # Ubuntu's front-end

-P INPUT DROP
-P OUTPUT DROP
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A OUTPUT -o eth0 -p udp -m udp --dport 53 -j ACCEPT

Read the policies (-P) first: this host drops everything by default and only permits inbound SSH and outbound DNS. Any other traffic times out — which looks exactly like a dead remote host from inside the application.

Topic 4: DNS — Where Intermittent Latency Hides

DNS causes more “random” slowness than anything else, because failures are usually partial: one resolver in a list is unhealthy, so a fraction of lookups pay a timeout.

The actual resolution path:

1. The application calls getaddrinfo().
2. glibc consults /etc/nsswitch.conf to decide the source order — typically files dns.
3. files means /etc/hosts. A match here returns instantly and no DNS query is ever sent.
4. dns means query the servers in /etc/resolv.conf, in order, applying search domains.
5. On systems running systemd-resolved, /etc/resolv.conf points at 127.0.0.53, a local stub that forwards to the real upstreams.

  app -> getaddrinfo() -> nsswitch.conf -> /etc/hosts        (hit: done)
                                        -> resolv.conf -> stub 127.0.0.53
                                                       -> upstream resolvers

The tools, and their critical difference:

dig +short example.com              # query DNS directly
dig @8.8.8.8 example.com            # bypass local config, ask a specific server
dig +trace example.com              # follow delegation from the root
getent hosts example.com            # go through nsswitch -- what the APP sees
resolvectl status                   # systemd-resolved view, per-link servers
resolvectl statistics               # cache hits, misses, failed transactions

dig talks to DNS. getent hosts goes through the full nsswitch path including /etc/hosts. When dig and getent disagree, the answer is nearly always an /etc/hosts entry — and that discrepancy is the fastest way to find one.

The two settings behind most DNS latency:

• timeout and attempts in /etc/resolv.conf. Defaults are 5 seconds and 2 attempts. If the first listed resolver is dead, every lookup that reaches it stalls 5 seconds before failing over. Users describe this as “sometimes it takes ages” — the exact fingerprint of one bad resolver in a list.
• ndots and search. With ndots:5 (the Kubernetes default), any name containing fewer than five dots is first tried against each search domain in turn. api.example.com can therefore generate four failed queries before the correct one. A trailing dot — api.example.com. — marks the name absolute and skips the search list entirely.

Reverse DNS: the SSH connection

sshd with UseDNS yes performs a reverse lookup on every connecting client’s IP before authenticating. If reverse resolution is slow or unanswered, every login stalls for the resolver timeout — while ping, telnet to the port, and the TCP handshake all look perfectly healthy, because they are. This is precisely the failure reproduced in the SSH war room, and the reason ssh -vvv shows a repeatable pause at the publickey step.

dig -x 10.0.1.18 +short          # does reverse resolution answer, and how fast?
grep -E 'UseDNS|GSSAPIAuth' /etc/ssh/sshd_config

Common mistake: Testing DNS once, getting a fast answer, and ruling it out. Intermittent resolver failures need repetition to surface: for i in $(seq 20); do time dig +short internal.example.com; done and look at the outliers, not the median.

Topic 5: Capturing Packets

When every layer looks fine and the behaviour still makes no sense, look at the wire.

sudo tcpdump -i any -nn port 443 -c 100          # 100 packets on 443, no name lookups
sudo tcpdump -i eth0 -nn host 10.0.4.19          # one peer
sudo tcpdump -i any -nn 'tcp[tcpflags] & (tcp-syn) != 0'   # SYNs only
sudo tcpdump -i any -nn port 53 -A               # DNS queries, with payload
sudo tcpdump -i any -w capture.pcap port 443     # write for Wireshark

Always pass -nn. Without it, tcpdump resolves addresses and ports to names — which issues DNS queries of its own, and will happily hang your capture during a DNS incident.

Reading a handshake:

10:14:02.113 IP 10.0.1.18.51234 > 10.0.4.19.443: Flags [S], seq 1829
10:14:02.115 IP 10.0.4.19.443 > 10.0.1.18.51234: Flags [S.], seq 4471, ack 1830
10:14:02.115 IP 10.0.1.18.51234 > 10.0.4.19.443: Flags [.], ack 4472

That last pattern is the most valuable thing a capture gives you: definitive proof of which side of the boundary the latency lives on.

Try it yourself: Run sudo tcpdump -i any -nn -c 20 port 53 in one terminal and dig example.com in another. Watch the query leave and the response arrive, and note the elapsed time between them.