Filesystems, Inodes & Disk Full Triage

“Disk is full” is one of the most common production pages, and one of the most commonly mishandled. The trap is that the obvious tool often reports the wrong thing, and deleting the obvious file frequently frees nothing at all.

Topic 1: Inodes — What a File Actually Is

A filename is not a file. On a Unix filesystem, the file is an inode: a metadata record holding size, permissions, owner, timestamps, link count, and pointers to the data blocks. A directory is simply a table mapping names to inode numbers.

Two hard links are two names for one inode — delete either and the data survives, because the link count is still above zero. A symlink is its own inode holding a path string, so it resolves by name and can dangle.

Consequences that matter operationally:

Hard links are extra directory entries pointing at the same inode. The data survives until the link count reaches zero. ln target name creates one; they cannot cross filesystems.
Symbolic links are their own tiny inode containing a path string. They can dangle, and they can cross filesystems. ln -s target name.
Deleting is unlinking. rm removes a directory entry and decrements the link count. The data is freed only when the count hits zero and no process still holds it open. That second condition is the whole of Topic 3.

Proving it in four commands:

touch soft hard
ln    hard hard-link      # hard link
ln -s soft soft-link      # symbolic link
ls -i                     # -i prints the inode number

19924730 hard    19924730 hard-link    19924727 soft    19924856 soft-link
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
identical inode -- same file            different inodes -- a pointer by name

The numbers settle the argument: hard and hard-link are the same file with two names. soft-link is a separate object whose contents happen to be the string soft.

Reading the inode itself:

stat app.log

  File: app.log
  Size: 11          Blocks: 8          IO Block: 4096   regular file
Device: 802h/2050d  Inode: 19147527    Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/    app)   Gid: ( 1000/    app)
Access: 2026-08-04 18:42:09.111527150 +0000
Modify: 2026-08-04 18:42:12.703534639 +0000
Change: 2026-08-04 18:42:12.743534723 +0000

Links: 1 is the field that decides whether rm frees space. The three timestamps are distinct and get confused constantly:

Timestamp	Changes when	Common use
Access (atime)	The file is read. Often disabled with `noatime` for performance.	Rarely trustworthy.
Modify (mtime)	The contents change.	What `ls -l` shows and `find -mtime` matches.
Change (ctime)	The inode changes — contents, permissions, owner, or link count.	Detecting a `chmod` or `chown` nobody admits to.

find /path -inum 19147527      # every name pointing at one inode
tree /etc -L 1 --inodes        # inode numbers alongside a tree view
find /etc -newer /etc/passwd   # files changed more recently than a reference

Try it yourself: Run the four-command sequence above in a scratch directory and read the inode numbers. Then rm hard and confirm with cat hard-link that the data is still there.

Topic 2: Inode Exhaustion

Inodes are allocated when the filesystem is created and their number is fixed. A partition can therefore be 4% full by bytes and completely unable to create a file.

The symptom is the confusing one: No space left on device while df -h shows plenty of room.

df -h    # space
df -i    # inodes -- ALWAYS check both

Filesystem      Inodes  IUsed  IFree IUse% Mounted on
/dev/nvme0n1p1  655360 655360      0  100% /var

Millions of tiny files cause this: unrotated session files, per-request temp files, a cache directory nobody pruned, or a mail spool. Finding them:

# Directories holding the most entries, under /var
for d in /var/*/; do echo "$(find "$d" -xdev 2>/dev/null | wc -l) $d"; done | sort -rn | head

The -xdev flag keeps find on a single filesystem, so it does not wander into other mounts and give you a misleading count.

Common mistake: Seeing No space left on device and immediately deleting large files. If the problem is inodes, removing one 2 GB file frees one inode out of 655,360 and changes nothing.

Topic 3: When df and du Disagree

This is the classic, and it comes up in interviews as often as in incidents.

du walks the directory tree and adds up the files it can see.
df asks the filesystem how many blocks are allocated.

They diverge when a file has been unlinked but is still open by a running process. The directory entry is gone, so du cannot see it. The inode’s link count is zero but its open count is not, so the filesystem keeps the blocks allocated and df still counts them.

The same filesystem, 35 GB apart. du cannot see a file with no name; the filesystem still counts its blocks because a descriptor keeps the inode alive.

How this happens in practice:

Someone finds a huge log and runs rm /var/log/app.log while the application still has it open for writing. Disk usage does not drop. The application keeps writing into an inode with no name, and the free space keeps falling with no visible file to blame.

Finding the culprits:

lsof +L1                              # every open file whose link count is below 1
lsof -nP | grep '(deleted)'           # the same, by annotation
ls -l /proc/<PID>/fd | grep deleted   # per-process view

The two fixes:

Correct: make the process release the descriptor — reload or restart it. systemctl reload nginx is enough for daemons that reopen their logs on SIGHUP.
Emergency, buys time without a restart: truncate the file through the descriptor the process still holds.
```
: > /proc/<PID>/fd/3        # replace 3 with the offending fd number
```
This zeroes the file in place while the process keeps writing to it — space is reclaimed immediately and nothing crashes.

Preventing the recurrence:

Never rm an active log. Truncate it, or fix log rotation. logrotate with copytruncate handles applications that cannot reopen their own files.

truncate -s 0 /var/log/app.log     # correct way to empty a live log

Try it yourself: In a scratch directory, run tail -f test.log in one terminal, rm test.log in another, then compare du -sh . with df -h . and find the descriptor via lsof +L1.

Topic 4: Finding Space Fast

An ordered routine for a disk-full page:

# 1. Which filesystem, and is it space or inodes?
df -h; df -i

# 2. Biggest directories on THAT filesystem only
du -xh /var --max-depth=2 2>/dev/null | sort -rh | head -20

# 3. Biggest individual files
find /var -xdev -type f -size +500M -exec ls -lh {} \; 2>/dev/null

# 4. Deleted-but-open files holding space hostage
lsof +L1 2>/dev/null | head

# 5. Recently grown files -- what changed today?
find /var -xdev -type f -mmin -60 -size +100M 2>/dev/null

-xdev appears throughout deliberately: without it, du and find descend into other mounted filesystems and attribute their contents to the one you are investigating.

The usual suspects:

/var/log — unrotated or newly verbose logs. Check journalctl --disk-usage too.
/var/lib/docker — dangling images, stopped containers, unused volumes. docker system df breaks it down.
/tmp — files a process wrote and never removed.
Core dumps — a repeatedly crashing service can write gigabytes per crash.

du -sh /var/log/*      # -s summarises each entry, -h in human units

1.8M  /var/log/anaconda
384K  /var/log/audit
4.0K  /var/log/boot.log
64K   /var/log/messages

Act at 80%, not at 100%:

The practical threshold is 80% used. Past that you are into the window where a log burst, a core dump, or a package upgrade tips the filesystem over while nobody is watching — and a full root filesystem takes down things that have nothing to do with disk: sessions cannot write, databases go read-only, and systemd itself starts failing to write the journal.

Filesystems also reserve a slice for root (5% by default on ext4), which is why df can report 100% while root can still write. That reserve is what leaves you a shell to fix things with, not headroom to spend.

# Anything above 80%, as a one-liner for a check script
df -hP | awk 'NR>1 && int($5) >= 80 {print $5, $6}'

tune2fs -l /dev/nvme0n1p1 | grep -i 'reserved block'   # ext4 reserve

Topic 5: I/O Performance

Space is one failure mode; speed is the other. When processes pile up in D state, you need I/O metrics.

iostat -xz 1          # extended per-device stats, skipping idle devices

Column	Meaning	How to read it
`r/s`, `w/s`	Reads and writes completed per second.	Raw throughput in operations.
`rkB/s`, `wkB/s`	Kilobytes per second.	Compare against the device’s rated bandwidth.
`await`	Average time per request, queue time included, in ms.	The number users feel. Single-digit ms for SSD, 10–20 for spinning disks.
`aqu-sz`	Average queue depth.	Consistently above 1 means requests are waiting.
`%util`	Percentage of time the device had at least one request in flight.	Not a saturation measure on modern devices.

Why `%util` misleads:

On a single spinning disk, 100% util genuinely meant saturated. On an SSD or a RAID array servicing many requests in parallel, a device can show 100% util while running far below capacity — it merely means it was never completely idle. Judge saturation from await and aqu-sz instead.

Tying I/O back to a process:

sudo iotop -oPa           # only processes actually doing I/O, accumulated
pidstat -d 1              # per-process read/write throughput
cat /proc/<PID>/io        # lifetime byte counters for one process

Common mistake: Reporting “disk at 100% utilisation” as the root cause of an incident. On NVMe that is frequently normal. Quote await and queue depth, and compare them against the same host’s numbers when it was healthy.

Try it yourself: Run iostat -xz 1 5 while copying a large file, and watch await and aqu-sz move together as the queue builds.