Kernel Diagnostics: dmesg, sysctl & Modules

When an application log says nothing and the metrics say “slow”, the kernel usually knows exactly what happened. It has been writing it down the whole time in a buffer most people never read.

Topic 1: The Kernel Ring Buffer

The kernel writes its messages into a fixed-size in-memory ring buffer. Fixed size means it wraps: old messages are overwritten, so a long-uptime host may have lost the evidence of an event from last month.

dmesg -T                       # -T = human-readable timestamps. ALWAYS use it.
dmesg -T --level=err,crit,alert,emerg
dmesg -T -w                    # follow, like tail -f
dmesg -T | grep -iE 'error|fail|timeout|reset'
journalctl -k                  # the same stream, via the journal -- survives reboots
journalctl -k -b -1            # kernel messages from the PREVIOUS boot

Raw dmesg prints seconds since boot ([ 1234.567890]), which is useless for correlating against an incident timeline. -T converts to wall-clock. On systemd hosts prefer journalctl -k, because the journal persists across reboots and the ring buffer does not.

The messages that matter:

Common mistake: Reading dmesg without -T, seeing [248591.331], and quietly giving up on correlating it with the 14:22 incident. One flag turns the buffer into a usable timeline.

Topic 2: sysctl — Tuning a Running Kernel

sysctl reads and writes kernel parameters exposed under /proc/sys/. The dotted name maps directly to a path: net.ipv4.ip_forward is /proc/sys/net/ipv4/ip_forward.

sysctl -a                        # every parameter (there are thousands)
sysctl vm.swappiness             # read one
sudo sysctl -w vm.swappiness=10  # set it NOW -- lost on reboot
sysctl net.ipv4 | head           # a whole subtree

Making it survive a reboot:

# /etc/sysctl.d/99-tuning.conf
vm.swappiness = 10
net.core.somaxconn = 4096
net.ipv4.tcp_max_syn_backlog = 8192
fs.file-max = 2097152

sudo sysctl --system     # load every file in /etc/sysctl.d/, in order
sysctl vm.swappiness     # verify it actually took

That verification step is not optional. Files load in lexical order, so a later file can override yours, and a typo in a parameter name fails silently at boot.

Parameters worth knowing:

The container caveat:

Some sysctls are namespaced and can be set per-container; most are not. A container cannot change vm.swappiness for the host — that is a node-level setting, applied by a DaemonSet or the node image. Kubernetes only allows a documented “safe” subset in a pod spec unless the kubelet is configured otherwise.

Try it yourself: Run sysctl net.core.somaxconn and compare it with the backlog your web server requests in its config. If the server asks for more than the kernel permits, it is silently capped.

Topic 3: Kernel Modules

Most drivers are modules — loadable objects the kernel pulls in on demand rather than compiling in permanently.

lsmod                        # loaded modules, size, and use count
modinfo nvme                 # what a module is, its parameters and dependencies
sudo modprobe br_netfilter   # load, resolving dependencies
sudo modprobe -r br_netfilter# unload (fails if the use count is above zero)

Module          Size  Used by
nvme           49152  3
overlay       147456  18
br_netfilter   32768  0

The Used by column is the reason modprobe -r refuses: unloading a module something depends on would take the dependants with it.

Persisting and blocking:

# Load at boot
echo 'br_netfilter' | sudo tee /etc/modules-load.d/k8s.conf

# Parameters for a module
echo 'options nvme_core io_timeout=255' | sudo tee /etc/modprobe.d/nvme.conf

# Prevent one from loading at all
echo 'blacklist nouveau' | sudo tee /etc/modprobe.d/blacklist-nouveau.conf
sudo update-initramfs -u        # blacklists must reach the initramfs too

That last line catches people: blacklisting a driver that the initramfs loads has no effect until the initramfs is rebuilt.

Where this bites in practice:

Container networking needs br_netfilter and overlay. Kubernetes will not pass its preflight checks without them, and the failure message points at networking rather than at a missing module:

lsmod | grep -E 'br_netfilter|overlay'
sysctl net.bridge.bridge-nf-call-iptables    # only exists once br_netfilter is loaded

Topic 4: /proc and /sys — the Kernel’s Two Interfaces

Both are virtual filesystems generated on read. They divide roughly by age and purpose:

• /proc — process information plus assorted kernel state. /proc/<PID>/* per process, /proc/meminfo, /proc/cpuinfo, /proc/loadavg, /proc/sys/ for sysctl.
• /sys — the modern, structured device and driver model. Hardware, block devices, network interfaces, cgroups, kernel features.

# Hardware inventory without installing anything
lscpu                                    # CPU topology, flags, virtualisation
lsblk -f                                 # block devices with filesystems and UUIDs
lspci -nnk                               # PCI devices and the driver bound to each
lsusb
dmidecode -t system -t memory            # firmware's view: model, serials, DIMMs

# Useful /sys reads
cat /sys/block/nvme0n1/queue/scheduler   # active I/O scheduler
cat /sys/class/net/eth0/speed            # link speed in Mb/s
cat /sys/class/net/eth0/statistics/rx_dropped
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

The I/O scheduler is a real tuning lever: none/noop is usually right for NVMe (the device reorders better than the kernel can), while mq-deadline suits spinning disks.

Errors the application never sees:

ip -s link show eth0        # interface-level drops, errors, overruns
ethtool -S eth0 | grep -iE 'err|drop|discard'
sudo smartctl -a /dev/nvme0n1 | grep -iE 'error|wear|temperature'

ip -s link reporting rising dropped or overrun counters is a hardware or driver problem. No amount of application tuning fixes packets the NIC discarded before anything in userspace saw them.

Topic 5: Panics, Hangs and Hardware

Kernel panic:

The kernel has hit an unrecoverable state and stopped. Nothing is written to disk, because the subsystem that would write it is part of what stopped — which is why panics are so often invisible after the fact.

journalctl -k -b -1 | tail -50      # last kernel messages before the restart
sudo kdumpctl status                # is crash dumping configured?
ls /var/crash/                      # captured dumps, if kdump is enabled
sysctl kernel.panic                 # seconds before auto-reboot; 0 = hang forever

Setting kernel.panic = 10 is standard on cloud fleets: a panicked node reboots itself and rejoins rather than sitting dead until somebody notices.

Soft and hard lockups:

watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [java:4471]

A CPU spent 20+ seconds in the kernel without yielding. Causes are a driver bug, a spinlock deadlock, or — most commonly on a VM — extreme host contention. Cross-check st (steal time) in vmstat before blaming the guest.

Machine check exceptions:

journalctl -k | grep -iE 'mce|hardware error|EDAC'
sudo mcelog --client 2>/dev/null

Correctable ECC errors mean a DIMM is starting to fail. Uncorrectable errors mean it has. Either way the action is the same: drain the host and replace the hardware — this is not a software problem and it will not be fixed by a reboot.

Try it yourself: Run journalctl -k -p warning -b on any host. Anything there is something the kernel thought worth mentioning that never reached an application log.

Common mistake: Concluding “nothing in the logs” from application logs and journalctl -u myservice. The kernel log is a separate stream, and OOM kills, storage errors, and dropped packets appear only there.