Linux Performance Triage Cheat Sheet

The 60-second triage

Run these in order. They narrow a vague “the server is slow” to a subsystem before you touch anything.

uptime                  # load average — trend, not a percentage
dmesg -T | tail -30     # OOM kills, disk errors, link flaps
vmstat 1 5              # r and b columns, plus si/so
mpstat -P ALL 1 3       # per-core: is one core pinned?
pidstat 1 3             # which process is actually burning CPU
iostat -xz 1 3          # per-device await and %util
free -m                 # available, not free
sar -n DEV 1 3          # interface throughput and errors
top -b -n1 | head -20

Load average is not CPU usage

On Linux, load counts processes in both R (running) and D (uninterruptible sleep, usually disk I/O). A load of 20 on 8 cores can mean a saturated CPU or a stalled disk — they need opposite fixes.

# Split the two: r = runnable, b = blocked on I/O
vmstat 1 5

# Which processes are stuck in D state
ps -eo state,pid,comm | awk '$1 ~ /D/'

High b with low r is an I/O problem. High r with low b is CPU.

Memory: read available, not free

free -m

free looks alarmingly low on a healthy machine because Linux uses spare RAM for page cache and reclaims it on demand. available is the number that matters — it estimates what a new process could actually get.

# Top memory consumers by RSS
ps -eo pid,rss,comm --sort=-rss | head -10

# Was anything OOM-killed?
dmesg -T | grep -i -E "out of memory|killed process"
journalctl -k | grep -i oom

Swap activity matters more than swap usage. Steady si/so in vmstat means active thrashing; a full-but-idle swap is usually harmless.

Disk

df -h                   # space
df -i                   # inodes — a full inode table looks like a full disk
du -xh --max-depth=1 / 2>/dev/null | sort -rh | head
iostat -xz 1 3          # await = latency, %util = saturation

When df says full but du disagrees, a deleted file is still held open by a running process. The space returns only when the process closes it or restarts.

sudo lsof +L1           # open files with zero directory links

Process state

ps -eo pid,ppid,state,pcpu,pmem,etime,comm --sort=-pcpu | head -15

State	Meaning
R	Running or runnable
S	Interruptible sleep — normal idle
D	Uninterruptible sleep — blocked on I/O, cannot be killed
Z	Zombie — finished, parent has not reaped it
T	Stopped

A process in D will not respond to SIGKILL. Find what it is waiting on:

sudo cat /proc/PID/stack
sudo cat /proc/PID/wchan; echo

What is this process doing?

sudo strace -p PID -f -T -tt          # system calls, with timings
sudo strace -c -p PID                 # summary: where time goes
sudo lsof -p PID                      # open files and sockets
cat /proc/PID/limits                  # fd limits — a silent, common ceiling
ls /proc/PID/fd | wc -l               # fds currently open

strace pauses the target on every syscall. It is safe for diagnosis but will slow a hot process noticeably — do not leave it attached in production.

Files and connections

sudo lsof -i :PORT                    # who owns a port
sudo lsof -u USER                     # everything a user holds open
sudo lsof /path/to/file               # who is holding this file

# System-wide fd usage vs limit
cat /proc/sys/fs/file-nr

Systemd services

systemctl status SERVICE
systemctl list-units --failed
journalctl -u SERVICE -n 100 --no-pager
journalctl -u SERVICE --since "10 minutes ago" -f
journalctl -p err -b                  # errors this boot

A service in a crash loop that stops retrying has hit StartLimitBurst. It stays dead until reset:

systemctl reset-failed SERVICE

Metrics that mislead

Looks bad	Actually
free memory near zero	Page cache. Read available
High load average	Includes disk-blocked processes, not just CPU
100% %util on SSD/NVMe	Meaningless on parallel devices — use await
High CPU in top	Sums across cores; 400% on 8 cores is 50%
Full swap	Only matters if si/so show active paging