Storage Optimization: LVM, RAID & Filesystem Tuning

The previous storage lesson was about a disk that is already full. This one is about not getting there: layouts that can grow without downtime, redundancy that matches what you are actually afraid of, and the handful of mount options that change real performance.

Topic 1: Seeing the Whole Stack

Before changing anything, map what exists. Storage is layered, and each layer has its own tool.

lsblk -f                    # tree: device → partition → LVM → filesystem → mountpoint
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT,ROTA,SCHED
blkid                       # UUIDs and filesystem types
findmnt                     # mounts as a tree, with options
df -hT                      # usage WITH filesystem type
sudo parted -l              # partition tables

NAME              FSTYPE      SIZE MOUNTPOINT
nvme0n1                       500G
├─nvme0n1p1       vfat        512M /boot/efi
└─nvme0n1p2       LVM2_member 499G
  ├─vg_app-lv_root ext4       100G /
  └─vg_app-lv_data xfs        300G /var/lib/data

ROTA is worth adding to the columns: 1 means a rotating disk, 0 means solid state. It decides which I/O scheduler and which mount options make sense, and on a virtualised host it is not always what you assume.

Topic 2: LVM — Storage That Can Grow

Partitions are fixed at creation and painful to resize. LVM inserts an abstraction so that filesystems no longer map to physical layout.

Three layers:

1. PV (Physical Volume) — a disk or partition handed to LVM.
2. VG (Volume Group) — one pool of extents built from PVs.
3. LV (Logical Volume) — a slice of that pool, which you put a filesystem on.

sudo pvs && sudo vgs && sudo lvs      # the three-command summary
sudo vgdisplay vg_app | grep -E 'Free|Alloc'

Building it:

sudo pvcreate /dev/nvme1n1
sudo vgcreate vg_app /dev/nvme1n1
sudo lvcreate -L 300G -n lv_data vg_app
sudo mkfs.xfs /dev/vg_app/lv_data
sudo mkdir -p /var/lib/data
sudo mount /dev/vg_app/lv_data /var/lib/data

Growing it online — the reason LVM exists:

# 1. more space into the pool (only if the VG is out of free extents)
sudo pvcreate /dev/nvme2n1
sudo vgextend vg_app /dev/nvme2n1

# 2. grow the logical volume
sudo lvextend -L +100G /dev/vg_app/lv_data
sudo lvextend -l +100%FREE /dev/vg_app/lv_data   # or take everything left

# 3. grow the FILESYSTEM to match -- the step people forget
sudo resize2fs /dev/vg_app/lv_data     # ext4
sudo xfs_growfs /var/lib/data          # xfs -- takes the MOUNTPOINT, not the device

Both filesystems grow while mounted and in use. Step 3 is separate because the LV and the filesystem are different objects: after step 2 the block device is bigger and df still shows the old size, which looks like the resize failed.

# lvextend can do both in one go
sudo lvextend -r -L +100G /dev/vg_app/lv_data     # -r resizes the filesystem too

Common mistake: Extending the LV and declaring victory. lvs shows the new size, df -h shows the old one, and the disk fills anyway. Always verify with df -h after, not lvs.

Snapshots:

sudo lvcreate -L 20G -s -n lv_data_snap /dev/vg_app/lv_data
sudo mount -o ro /dev/vg_app/lv_data_snap /mnt/snap    # consistent backup source
sudo lvremove /dev/vg_app/lv_data_snap

A snapshot is copy-on-write and only stores changes, so it needs far less space than the origin — but if it fills, it is invalidated and dropped. Size it for the write volume during the backup window, and never leave one lying around: every write to the origin also writes to the snapshot, which costs real performance.

Shrinking:

ext4 can shrink, but only unmounted, and it is genuinely risky. XFS cannot shrink at all — ever. Plan LV sizes to grow, because the reverse operation may not exist.

Topic 3: RAID — Redundancy, Not Backup

cat /proc/mdstat                          # array status and rebuild progress
sudo mdadm --detail /dev/md0
sudo mdadm --create /dev/md0 --level=10 --raid-devices=4 /dev/nvme{1,2,3,4}n1
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

The rebuild window:

When a disk in RAID 5 fails, the array reads every block on every remaining disk to reconstruct it. On modern large disks that takes many hours under full load — and a second failure during it loses everything. This is why RAID 6 or RAID 10 is preferred at scale, not RAID 5.

Watch for silent degradation:

cat /proc/mdstat        # [UU_U] -- an underscore is a MISSING disk

An array runs perfectly well degraded, with no user-visible symptom, right up until the second failure. If nothing alerts on mdstat, you find out at the worst possible moment.

Three things RAID is not: a backup (it faithfully replicates rm -rf), protection against corruption (except with checksumming filesystems like ZFS/Btrfs), or a substitute for testing your restores.

Topic 4: Choosing and Tuning a Filesystem

For a server the honest answer is usually: XFS for data volumes, ext4 for root, and Btrfs/ZFS only when you specifically want their snapshot and integrity features.

Mount options that matter:

# /etc/fstab
UUID=9555d31f-...  /var/lib/data  xfs  defaults,noatime  0 2
UUID=dca7dd33-...  /              ext4 defaults,errors=remount-ro 0 1
UUID=D790-E415     /boot/efi      vfat defaults,nofail   0 1

The last two fstab columns are dump (leave 0) and fsck order: 1 for root, 2 for others, 0 to skip. A missing nofail on a detached cloud volume is a classic cause of a host that will not finish booting.

sudo mount -o remount,noatime /var/lib/data   # apply without a reboot
sudo findmnt -o TARGET,SOURCE,FSTYPE,OPTIONS  # what is ACTUALLY mounted with what
systemctl status fstrim.timer                 # batch TRIM on a schedule

Always validate fstab before rebooting. A syntax error here means the host boots into emergency mode:

sudo findmnt --verify --verbose
sudo mount -a           # mounts everything in fstab -- if this errors, so will boot

The I/O scheduler:

cat /sys/block/nvme0n1/queue/scheduler        # [none] mq-deadline kyber
echo mq-deadline | sudo tee /sys/block/sda/queue/scheduler

• NVMe/SSD — none. The device parallelises better than the kernel can reorder.
• Spinning disk — mq-deadline or bfq, where seek ordering genuinely helps.

Persist it with a udev rule, not /etc/rc.local:

# /etc/udev/rules.d/60-scheduler.rules
ACTION=="add|change", KERNEL=="nvme[0-9]n[0-9]", ATTR{queue/scheduler}="none"

Topic 5: Capacity Planning

Optimisation is mostly about not being surprised.

# Growth rate: same command, a week apart, is the whole method
df -hP | awk 'NR>1 {print $6, $5}' | tee -a /var/log/capacity.log

# What is growing right now
sudo du -xh --max-depth=2 /var 2>/dev/null | sort -rh | head -20
find /var -xdev -type f -mtime -1 -size +100M 2>/dev/null

# Inodes as well as bytes -- both fill
df -i

Three habits that prevent most disk incidents:

1. Separate volumes for volatile data. /var/log and /var/lib/docker on their own LVs means a runaway log fills its own volume and the root filesystem survives. This one decision converts a host-down into a service-degraded.
2. Alert on trend, not threshold. “Will cross 90% within 48 hours at the current rate” gives you a working day. “Is at 90%” gives you an evening.
3. Leave the volume group with free extents. Unallocated space in the VG is the cheapest insurance there is — growing an LV takes seconds, ordering a disk does not.

Try it yourself: Run lsblk -f and findmnt -o TARGET,FSTYPE,OPTIONS and check whether any busy filesystem is missing noatime, and whether any non-root mount is missing nofail.