Memory, Page Cache & the OOM Killer

More production alerts are misconfigured around memory than any other resource, because the number most people watch — free memory — is the one that means the least on Linux. This lesson fixes that.

Topic 1: Virtual Memory in One Diagram

No process ever touches physical RAM directly. Each one sees a private virtual address space, and the CPU’s memory management unit translates virtual addresses to physical ones using page tables the kernel maintains.

Memory is managed in fixed-size chunks called pages, almost always 4 KB on x86-64.

  Process A virtual pages        Physical RAM frames
  +------------------+           +------------------+
  | 0x0000 code      | -------->  | frame 812        |
  | 0x1000 heap      | -------->  | frame 44         |
  | 0x2000 heap      | --------\  +------------------+
  | 0x3000 stack     | ----\    \-> (not yet mapped)
  +------------------+      \       +------------------+
                             \----> | frame 1907       |
                                    +------------------+

Two consequences fall out of this design, and both matter operationally:

• Allocated does not mean resident. A process can reserve a huge address range and touch almost none of it. Pages are backed by real RAM only on first write.
• Shared pages are counted many times. Fifty processes running the same binary share one copy of its code, but naive per-process accounting bills each of them for it.

Topic 2: RSS vs VSZ vs PSS

A JVM with a 512 MB heap routinely shows several gigabytes of VSZ. Nothing is wrong. It reserved address space, which is free.

# RSS per process, biggest first
ps -eo pid,rss,comm --sort=-rss | head

# PSS for one process, in kB (Linux only)
grep '^Pss:' /proc/<PID>/smaps | awk '{sum += $2} END {print sum " kB"}'

Common mistake: Summing RSS across all processes and comparing it to total RAM. Because shared pages are counted once per process, the total can exceed physical memory on a perfectly healthy machine.

Topic 3: Reading free -h Correctly

              total        used        free      shared  buff/cache   available
Mem:           15Gi        4.2Gi       312Mi       180Mi        11Gi        10Gi
Swap:         2.0Gi        0.0Ki       2.0Gi

At a glance this looks like a machine about to fall over: 312 MB free out of 15 GB. It is in fact perfectly healthy.

What each column means:

• total — physical RAM the kernel can use.
• used — genuinely allocated to processes (anonymous memory). This one is real.
• free — completely untouched. Low free memory is the goal, not a problem. Unused RAM is wasted RAM.
• buff/cache — the page cache: file contents the kernel is holding because reading from RAM beats reading from disk. It is reclaimable on demand.
• available — the kernel’s own estimate of how much a new process could get without swapping, counting reclaimable cache. This is the number to alert on.

Why the page cache fills:

Every file read populates the cache. A host that has been up for a week and served files will show most of its RAM in buff/cache, because the kernel would rather hold the data than re-read the disk. When a process needs memory, the kernel evicts cache instantly.

Going underneath free — /proc/meminfo:

free is a formatter over /proc/meminfo. When you need the fields it does not summarise:

grep -E 'MemTotal|MemAvailable|Dirty|Writeback|Slab|Committed_AS' /proc/meminfo

Watching it live with top:

top is more useful than most people get from it because the interactive keys are undocumented on screen:

The c key matters more than it sounds: on a box running fifteen java processes, the truncated name tells you nothing and the full command line names the service immediately.

Try it yourself: Run free -h, then cat /some/large/file > /dev/null, then free -h again. Watch buff/cache grow and free shrink — with available barely moving.

Common mistake: Alerting on free < 10%. It fires constantly on healthy hosts, everyone learns to ignore memory alerts, and the real event gets missed. Alert on available instead.

Topic 4: Swap and Swappiness

Swap is disk space the kernel uses to park memory pages it does not think are needed soon, freeing RAM for active work.

• vm.swappiness (0–100, default 60) tunes the tradeoff between evicting page cache and swapping out anonymous pages. Lower means “prefer dropping cache”.
• Swap usage is not itself a problem. Idle pages sitting in swap cost nothing.
• Sustained swap traffic is the problem. Check si/so (swap in / swap out) in vmstat 1. Nonzero and continuous means the working set does not fit in RAM, and every access is now paying a disk round trip.

vmstat 1 5
# procs -----------memory---------- ---swap-- -----io----
#  r  b   swpd   free   buff  cache   si   so    bi    bo
#  2  0      0 320104  91232 1150884    0    0    12    38
#           ^^^^ constant nonzero si/so here = thrashing

Swap in containers:

Kubernetes historically required swap to be disabled entirely, because swap breaks the accounting the scheduler relies on: a container could exceed its memory limit and keep running slowly instead of being terminated predictably. Newer versions support it behind a feature gate, but the default assumption on cluster nodes remains swap-off.

Topic 5: The OOM Killer

When the kernel genuinely cannot satisfy an allocation and has nothing left to reclaim, it invokes the out-of-memory killer: it picks a process and terminates it with SIGKILL to save the system.

How the victim is chosen:

Every process has an oom_score, derived mainly from how much memory it is using, adjusted by oom_score_adj (a tunable from -1000 to +1000). Roughly: the biggest consumer dies, unless an operator has biased the score.

# What the kernel currently thinks of a process
cat /proc/<PID>/oom_score
cat /proc/<PID>/oom_score_adj

# Protect a critical process (needs root); -1000 disables OOM kill entirely
echo -500 > /proc/<PID>/oom_score_adj

Finding the evidence after the fact:

The OOM killer always leaves a record in the kernel log. This is the first thing to check when a process vanished with no application-level error:

dmesg -T | grep -i -E 'killed process|out of memory'
journalctl -k --since '1 hour ago' | grep -i oom

A typical line names the victim, its RSS, and the cgroup it belonged to.

Cgroup OOM vs system OOM — the distinction that matters:

• System OOM — the whole host ran out. Everything on the box is at risk.
• Cgroup OOM — one container hit its own limit. The kernel kills inside that cgroup only; the rest of the node is fine.

In Kubernetes a cgroup OOM surfaces as a container with exit code 137 and reason OOMKilled. That is a limit that is too low or a leak in the app — not a node problem. Node-level memory pressure produces something different: eviction, where the kubelet proactively removes whole pods by QoS class before the kernel has to act.

Reading the container’s own limit:

Under cgroup v2 the numbers a container is actually judged against live in its cgroup, not in free:

# Inside the container (cgroup v2)
cat /sys/fs/cgroup/memory.max       # the limit -- "max" means unlimited
cat /sys/fs/cgroup/memory.current   # what it is using right now
cat /sys/fs/cgroup/memory.events    # how many times it hit the ceiling

# cgroup v1 paths, still common on older nodes
cat /sys/fs/cgroup/memory/memory.limit_in_bytes
cat /sys/fs/cgroup/memory/memory.usage_in_bytes

The oom_kill counter in memory.events is the honest record: it increments every time the kernel killed something in that cgroup, even if the container restarted so fast nobody saw it. A pod that “seems fine” with a climbing oom_kill count is silently losing work.

Try it yourself: On a non-production box, run dmesg -T | grep -i 'killed process'. If anything appears, identify the victim and whether the trigger was a cgroup limit or the whole host.

Common mistake: Treating OOMKilled as “the node needs more memory”. If it was a cgroup OOM, adding node RAM changes nothing — the container will hit the same limit at the same point. Read the kernel log and find out which boundary was crossed before sizing anything.