The 60-Second Triage & the USE Method

You have been handed a host and told it is slow. There is a bridge call running and nobody has data. This lesson is the first sixty seconds — a fixed sequence that narrows four possible bottlenecks to one, with evidence you can quote.

Topic 1: Load Average, Properly Understood

uptime gives three numbers: the 1, 5, and 15-minute load averages. Almost everyone reads them wrong.

$ uptime
 14:22:07 up 41 days,  3:11,  2 users,  load average: 8.42, 6.15, 3.90

What Linux actually counts:

On most Unixes, load average counts processes wanting CPU. Linux additionally counts processes in uninterruptible sleep — state D. So Linux load average is:

runnable processes (R) + processes blocked on I/O (D)

This is why a host with idle CPUs can show a load of 40: every one of those processes is parked on storage, not competing for a core.

Reading the three numbers:

Compare load against core count, and compare the three figures to each other for direction.

nproc                          # how many cores
cat /proc/loadavg              # raw values, plus running/total processes

• Load 8 on 16 cores: roughly half loaded. Fine.
• Load 8 on 4 cores: twice oversubscribed. Something is queueing.
• 8.42, 6.15, 3.90 — rising steeply. The event is happening now.
• 3.90, 6.15, 8.42 — falling. The worst has passed; you may be looking at the aftermath.

The same numbers, two machines:

Take load average: 0.58, 1.13, 2.46 and read it against core count. On a 1-core box:

On a 2-core box, the identical output means something else:

The number on its own is meaningless. nproc is not an optional second step, and the direction of travel across the three windows tells you whether you have arrived during the event or after it.

Common mistake: Treating high load as “CPU is the problem” and adding cores. If the load is composed of D-state processes, more CPU changes nothing — the queue is on the disk. Always confirm the composition:

ps -eo stat,comm | awk '$1 ~ /^R/ {r++} $1 ~ /^D/ {d++} END {print "running:", r+0, " blocked-io:", d+0}'

That one line tells you which of the two populations is producing your load number, and it is the single most useful thing to run before saying the word “CPU” out loud.

Topic 2: Utilisation, Saturation, Errors — the USE Method

Brendan Gregg’s USE method is a checklist that stops you fixating on one metric. For every resource, ask three questions:

• Utilisation — what fraction of the time is it busy?
• Saturation — how much work is queued and waiting?
• Errors — is it failing outright?

Utilisation alone lies. A resource at 100% utilisation with no queue is fine — it is being used fully, which is what you bought it for. A resource at 60% utilisation with a deep queue is in trouble. Saturation is what users feel.

ip -s link show eth0     # RX/TX errors, drops, overruns per interface
nstat -az | grep -i retrans   # TCP retransmission counters

Try it yourself: Pick the resource that looks busiest on your machine and fill in all three columns for it. If you cannot produce a saturation number, you do not yet know whether it is a problem.

Topic 3: The 60-Second Checklist

Run these in order. Each one either implicates a resource or clears it.

uptime                    # 1. load trend: rising or falling?
dmesg -T | tail -30       # 2. OOM kills, I/O errors, link flaps
vmstat 1 5                # 3. run queue, swap traffic, io wait
mpstat -P ALL 1 3         # 4. per-core: is one core pinned?
pidstat -u 1 3            # 5. which processes are burning CPU
iostat -xz 1 3            # 6. per-device await and queue depth
free -h                   # 7. available memory, not free memory
ss -s                     # 8. socket census, TIME-WAIT / CLOSE-WAIT counts
ps -eo stat,comm | awk '$1 ~ /^[DZ]/'   # 9. anything stuck or unreaped
top -b -n1 | head -20     # 10. the overall picture, sorted

Reading vmstat 1 — the densest single view:

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 9  3      0 210344  91232 115088    0    0   842  1204 4211 8822 71 14  4 11  0

st is worth calling out. On a cloud instance, sustained steal time means a noisy neighbour or an over-committed hypervisor. No amount of tuning inside your VM will fix it — the fix is a different instance or instance type, and that is a useful thing to be able to prove.

Topic 4: Discriminating Between the Four Bottlenecks

The whole point of the checklist is to land in exactly one of these columns.

The case everyone gets wrong:

High load, high iowait, low CPU utilisation. It looks like the machine is overloaded, and the instinct is to scale up compute. It is a storage problem. The processes are asleep in D, contributing to load without using a cycle, and the queue is on the device. Confirm with iostat -xz 1 and read await — then go and look at what changed on the storage layer.

Common mistake: Stopping at the first abnormal number. Memory pressure causes swapping, which causes disk I/O, which causes iowait, which raises load. All four resources look unhealthy. The discipline is to find the first thing to move — check dmesg timestamps and metric history, and treat the earliest deviation as the candidate cause.

Topic 5: From Symptom to Root Cause

Numbers make a bottleneck visible. They do not explain why it appeared. Three questions close that gap:

1. What changed? Deployments, config pushes, feature flags, certificate rotations, cron jobs, traffic shifts. Correlate the start time from your metrics against the change log before theorising.

   journalctl --since '2 hours ago' | grep -iE 'started|stopped|reload|failed'
   ls -lt /etc | head            # recently modified configuration
   uptime -s                     # did the host reboot?
   

2. Is it one host or many? A single unhealthy host in a fleet is a host problem — replace it and investigate offline. The same symptom fleet-wide is a change, a dependency, or a traffic pattern, and replacing hosts will not help.
3. Does the timeline fit? If the resource curve started climbing twenty minutes before the deploy, the deploy is not the cause, however tempting the coincidence.

On a container host, add one more pass:

The node-level checklist attributes load to processes, but on a Kubernetes node those processes belong to containers with their own limits. Two extra commands map one onto the other:

docker stats --no-stream          # CPU, memory, net and block I/O per container
kubectl top pods -A --sort-by=memory
systemd-cgtop                     # live CPU/memory by cgroup, including k8s slices

systemd-cgtop is the useful one on a node you cannot run kubectl from: it sorts the cgroup tree by resource use, so the noisy pod surfaces without you needing to know which container is which.

Remember what lesson 3 established — a container hitting its own cgroup limit produces an OOM kill while the node has memory to spare. The node checklist will look clean. Always check both boundaries before concluding the host is fine.

Writing it down as you go:

Keep a running note during the incident: timestamp, command run, result, conclusion. It costs nothing while you work and it becomes the postmortem timeline. It also prevents the most expensive mistake on a long call — re-running the same check twice because nobody wrote down the answer the first time.

14:22  uptime           load 8.42 rising, 4 cores            -> oversubscribed
14:23  vmstat 1         b=3, wa=11, r=1                      -> NOT cpu; io
14:24  iostat -xz 1     nvme0n1 await 240ms, aqu-sz 14       -> disk saturated
14:25  ps D-state       5 procs blocked, all postgres        -> db storage
14:27  dmesg -T         "EXT4-fs error" at 14:19             -> filesystem fault

Five lines, five minutes, and a root cause with a timestamp that precedes the symptom. That is the whole objective.

Try it yourself: Run the checklist on a healthy host first and record what normal looks like. Baselines are what make an abnormal number obvious at 3am, and the worst time to discover you have never seen the healthy values is during the incident.