Processes, PIDs & the Fork/Exec Lifecycle

Every outage you will ever debug comes down to a process doing something it should not: consuming too much, waiting on something that never returns, or exiting when it should have stayed up. Before you can diagnose any of that, you need an exact model of what a process is.

Topic 1: What a Process Actually Is

A program is a file on disk. A process is that program loaded into memory and running, with the kernel tracking its state. The same program can be running as fifty separate processes at once, each with its own memory and its own identity.

The kernel gives every process a PID (Process ID) — a unique number — and records its PPID (Parent Process ID), the process that started it.

What the kernel tracks per process:

• Address space — the memory the process can see: its code, its heap, its stack.
• File descriptor table — every open file, socket, and pipe (covered in the syscalls lesson).
• Credentials — the UID and GID it runs as, which decide what it is allowed to touch.
• State — what the scheduler should do with it right now.
• Exit status — set when it terminates, held until the parent collects it.

The process tree:

Processes are not a flat list. Every process except one has a parent, forming a tree rooted at PID 1 — on modern distributions, systemd. When you run a command in bash, bash becomes its parent.

PID 1  systemd
 └─ PID 842   sshd
     └─ PID 1503  bash          <- your login shell
         └─ PID 2011  ./deploy.sh
             └─ PID 2012  kubectl apply -f app.yaml

Try it yourself: Run ps -ef --forest | head -40 to see the tree on your machine. Find your own shell in it and trace the chain back to PID 1.

Common mistake: Assuming a PID identifies a program permanently. PIDs are recycled — when a process exits, its number goes back into the pool. Scripts that store a PID and later kill it without checking may kill an entirely unrelated process that inherited the number.

Topic 2: Fork and Exec — How Processes Are Born

Linux does not have a single “run this program” call. It has two, used in sequence, and understanding the split explains a lot of otherwise strange behaviour.

1. fork() — the kernel makes a near-identical copy of the calling process. Both continue from the same line. The only difference is the return value: the parent receives the child’s PID, and the child receives 0.
2. execve() — replaces the current process image with a new program. The PID stays the same, but the code, heap, and stack are thrown away and rebuilt from the new binary.

Why the two-step design matters:

Between the fork and the exec, the child is still a copy of the shell — and that gap is where the shell sets everything up. Redirecting output with >, setting an environment variable for one command, changing the working directory: all of it happens in the child, after the fork, before the new program takes over. That is why FOO=bar ./script.sh sets FOO for that one command and nothing else.

Copy-on-write:

Copying an entire address space on every fork would be ruinously slow. Instead the kernel marks the pages shared and read-only, and only makes a real copy when one side writes. A fork of a 4 GB process therefore costs almost nothing if the child immediately calls exec.

Try it yourself: Run echo $$ to print your shell’s PID. Then run bash -c 'echo $$' — a different number, because that is a forked child. Now run exec sleep 5 in a throwaway terminal and watch the shell itself be replaced (the terminal closes when sleep finishes).

Topic 3: The Five Process States

ps reports state as a single letter in the STAT column. Reading it correctly is the difference between “the box is busy” and “the box is stuck on storage”.

The D state deserves special attention:

A process in D is inside a kernel call that deliberately cannot be interrupted, because interrupting it would leave kernel data structures inconsistent. kill -9 will not touch it. The process leaves D when the underlying I/O completes — or never, if the storage backing it is gone.

Seeing several processes in D on the same host almost always means the storage layer is the problem: a hung NFS mount, a detached network volume, a failing disk. Chasing the process is wasted effort; the fix is one layer down.

Common mistake: Escalating kill -9 when a process will not die, without checking its state first. If ps -o stat shows D, no signal will help, and repeating the command just adds noise to the incident channel.

Topic 4: Reading ps and top Under Pressure

The two ps syntaxes:

ps accepts both BSD-style flags (no dash) and UNIX-style (with dash). Both are common in real runbooks:

• ps aux — BSD style. Every process, with user and resource columns.
• ps -ef — UNIX style. Every process, with full command lines and PPID.
• ps -o pid,ppid,stat,rss,etime,comm — pick exactly the columns you want. The most useful form during an incident.

What the letters in aux actually mean:

People type ps aux by muscle memory without knowing it is three separate flags:

Drop the x and daemons disappear from the output, which is exactly how people conclude a service “is not running” when it is.

Equivalent full output in UNIX style is ps -ef, where -e is every process and -f is the full-format listing that includes PPID and the complete command line.

Columns worth knowing:

• %CPU — share of one CPU over the process’s lifetime, so a process that was busy an hour ago still shows high. top gives you the live figure instead.
• RSS — resident memory in kilobytes: physical RAM actually held.
• ETIME — how long the process has been alive. Invaluable for spotting the one worker that never got restarted.
• STAT — the state letter above, sometimes with suffixes: s (session leader), + (foreground), l (multi-threaded).

Sorting to find the culprit:

# Top 10 memory consumers, largest first
ps -eo pid,ppid,rss,etime,comm --sort=-rss | head -11

# Anything not in a normal state
ps -eo pid,stat,comm | awk '$2 ~ /^[DZT]/'

The top-consumers pipeline:

ps has no notion of “show me the worst offenders”, so pipe it through awk and sort. This is the single most-used one-liner on a struggling host:

# PID, %MEM and command for the 20 heaviest processes
echo "[PID]  [MEM]  [COMMAND]" && ps aux | awk '{print $2, $4, $11}' | sort -k2rn | head -20

[PID] [MEM] [COMMAND]
4578 19.8 /usr/lib/jvm/java-8-openjdk-amd64/bin/java
23490 17.9 /usr/bin/java
27389 10.0 /usr/bin/mongod
23952 7.1 uwsgi

Two JVMs holding 38% of RAM between them is the sort of thing that never shows up in an application dashboard but explains the whole incident. sort -k2rn sorts on the second field, reverse, numerically.

htop, when you have it:

htop is top with colour, mouse support and a per-core meter. It is not installed by default; sudo apt-get install htop or sudo dnf install htop. Two keys make it worth the install:

• F5 — tree view. Shows the parent/child structure, which turns “forty python processes” into “one supervisor with forty workers”.
• F6 — sort by any column.

Try it yourself: Run ps -eo pid,stat,comm | awk '{print $2}' | sort | uniq -c | sort -rn. This gives you a census of process states on the host — a healthy machine is overwhelmingly S.

Topic 5: Priority — nice and renice

Not every process deserves equal CPU. Linux exposes this as a niceness value from -20 (most favourable scheduling) to +19 (least). The name is literal: a high nice value means the process is being nice to everything else by yielding.

nice -n 10 ./batch-report.sh        # start a job at low priority
renice -n 5 -p 2011                 # re-prioritise a running process
renice -n 5 -u backupuser           # everything owned by a user
ps -eo pid,ni,comm --sort=ni | head # who is running at what niceness

Only root can assign a negative niceness. An unprivileged user can lower their own priority but never raise it — otherwise every process would claim to be the important one.

When this actually helps:

A nightly batch job, a backup, or a log-compaction task competing with request-serving traffic. Renicing the batch work to +10 costs it very little wall-clock time on an idle box and keeps it out of the way when the box is busy.

When it does not help:

Niceness only arbitrates CPU. If the contention is disk or memory, renicing changes nothing — you need ionice for I/O priority, or a cgroup limit for memory. Reaching for renice on an I/O-bound host is a common wasted move.

Topic 6: Containers Are Just Processes

Nothing about a container escapes what you have just learned. A container is an ordinary Linux process with two kernel features wrapped around it.

• Namespaces — control what the process can see. A PID namespace gives the container its own PID numbering, so its main process believes it is PID 1. Separate mount, network, user, and UTS namespaces do the same for filesystems, interfaces, users, and hostname.
• Cgroups (control groups) — control what the process can use. CPU shares, memory limits, I/O weight, and PID counts are all enforced here.

# From the HOST, a container's processes are just processes
ps -ef | grep nginx

# Which namespaces does a process belong to?
ls -l /proc/<PID>/ns

# What cgroup is it in? (cgroup v2)
cat /proc/<PID>/cgroup

# Live resource use per container
docker stats --no-stream

This is why the same PID shows different numbers inside and outside the container, and why kubectl top and free -h can disagree — free reports the node, while the container’s limit lives in its cgroup.

Common mistake: Debugging a container by trusting only what is visible inside it. The container’s view is deliberately partial. When a container is being killed for memory, the answer is in the host’s kernel log and the cgroup limit, neither of which the container can see.

Topic 7: Zombies and Orphans

These two terms get confused constantly, and they mean opposite things.

Zombie (state Z):

The child has already exited. It holds no memory and uses no CPU. All that remains is an entry in the process table holding its exit status, waiting for the parent to call wait() and collect it. A zombie is not a resource problem in itself — but thousands of them mean the parent is buggy and never reaping, and the process table is finite.

You cannot kill a zombie. It is already dead. Restart or fix the parent.

Orphan:

The parent exited first, leaving the child running. The kernel immediately re-parents the orphan to PID 1, which is written to reap its children correctly. Orphans are harmless and normal — this is exactly how daemons detach.

Zombie:  child exited  ->  parent still alive but not reaping  ->  entry stuck
Orphan:  parent exited ->  child re-parented to PID 1          ->  cleaned up fine

Why this matters in containers:

Inside a container, your application is often PID 1 — and most applications were never written to reap adopted children. Long-running containers that spawn subprocesses can therefore accumulate zombies until the PID table fills. This is why container runtimes offer an init shim (--init, or shareProcessNamespace patterns in Kubernetes): it puts a proper reaping PID 1 underneath your app.

Try it yourself: Count zombies on any host with ps -eo stat | grep -c '^Z'. On a healthy machine this is 0. If it is climbing, find the parent with ps -eo pid,ppid,stat,comm | awk '$3 ~ /^Z/' and investigate that PPID.

Common mistake: Filing a ticket to “clean up the zombie processes”. There is nothing to clean up on the child side — the only real fix is in the parent’s code or a proper init process.