System Calls, File Descriptors & /proc

When logs stop short and metrics show nothing, there is one question left: what is this process asking the kernel to do right now? System call tracing answers it directly, and it is the tool that ends arguments.

Topic 1: The User/Kernel Boundary

Application code cannot open a file, send a packet, or allocate memory by itself. Those operations require privileged access to hardware, so the process asks the kernel to perform them on its behalf through a system call.

A syscall is a controlled trap into kernel mode: the process puts a syscall number and arguments in registers, executes a special instruction, and the CPU switches privilege level and jumps to the kernel’s handler.

The syscalls worth recognising:

Common mistake: Reading a blocked epoll_wait or accept as a hang. Those are exactly what an idle, healthy server looks like while waiting for work.

Topic 2: File Descriptors

A file descriptor is a small non-negative integer indexing into the kernel’s per-process table of open resources. Linux’s design treats almost everything as a file, so the same integer abstraction covers regular files, sockets, pipes, devices, and terminals.

Three are opened for you and are the same everywhere:

Shell redirection is nothing more than rewiring these numbers before exec:

command > out.log          # point fd 1 at a file
command 2> err.log         # point fd 2 at a file
command > all.log 2>&1     # fd 1 to file, then fd 2 to wherever fd 1 goes
command 2>&1 > all.log     # WRONG: fd 2 copies the terminal, then fd 1 moves

The order in that last line matters and catches people out constantly. 2>&1 copies the current destination of fd 1 — so it must come after the redirect, not before.

Inspecting descriptors on a live process:

ls -l /proc/<PID>/fd        # symlinks to every open resource
ls /proc/<PID>/fd | wc -l   # how many are open right now
cat /proc/<PID>/limits | grep 'open files'   # the ceiling
lsof -p <PID>               # the same thing, annotated

Descriptor exhaustion:

Every process has a limit (RLIMIT_NOFILE). A service that opens sockets or files without closing them climbs toward it, and on arrival every new connection fails with EMFILE: too many open files. The service is up, the port is listening, and nothing works — a failure mode that looks bewildering until you count descriptors.

# Which processes hold the most descriptors
for p in /proc/[0-9]*; do
  echo "$(ls $p/fd 2>/dev/null | wc -l) $(basename $p) $(cat $p/comm 2>/dev/null)"
done | sort -rn | head

Raising the ceiling — ulimit and LimitNOFILE:

The limit has two halves: a soft limit the process actually gets, and a hard limit the soft one may be raised to. An unprivileged user can raise soft up to hard, but only root raises hard.

ulimit -n            # current soft limit for this shell
ulimit -Hn           # the hard ceiling
ulimit -n 65535      # raise soft for this shell and its children only
ulimit -a            # every limit at once

ulimit in your shell does not affect a service systemd started. That needs the unit:

[Service]
LimitNOFILE=65535

For login sessions the equivalent lives in /etc/security/limits.conf. Setting one and expecting the other to change is a standard afternoon lost — always confirm on the running process with cat /proc/<PID>/limits rather than trusting the config you edited.

lsof, the other lens:

lsof answers the reverse question: not “what does this process hold” but “who is holding this thing”.

lsof -p 2011              # everything one process has open
lsof /var/log/app.log     # which processes hold this file
lsof -i :80               # who is listening on or connected to port 80
lsof -i -nP               # every network connection, numeric
lsof -u appuser           # everything a user has open
lsof +D /var/lib/data     # everything open under a directory tree

lsof -i :80 is the fastest answer to “address already in use”, and lsof /path is how you find the process pinning a filesystem you are trying to unmount.

Try it yourself: Pick any long-running service PID and compare ls /proc/<PID>/fd | wc -l against the soft limit in /proc/<PID>/limits. A ratio climbing over time is a leak.

Topic 3: The /proc Filesystem

/proc is not on disk. It is a virtual filesystem the kernel generates on read, exposing its own state as ordinary files. Every diagnostic tool you use is reading these files underneath.

Per-process, under /proc/<PID>/:

System-wide:

• /proc/meminfo — the source free reads.
• /proc/loadavg — the source uptime reads.
• /proc/cpuinfo — per-core CPU details.
• /proc/net/tcp — every TCP socket, in hex. ss renders this for humans.
• /proc/mounts — currently mounted filesystems.

# Read a process's environment safely (NULs become newlines)
tr '\0' '\n' < /proc/<PID>/environ

# What binary is this really running?
ls -l /proc/<PID>/exe

That last command is worth remembering during a security review: exe resolves to the real binary even if it has been deleted from disk, which is exactly what a process running a since-removed file looks like.

Common mistake: Running cat /proc/<PID>/environ and getting one unreadable line. The entries are NUL-separated; pipe through tr '\0' '\n'.

Topic 4: strace — Watching the Boundary

strace intercepts and prints every syscall a process makes. It is the most direct answer to “what is it actually doing”.

strace ./my-script.sh              # trace a command from the start
strace -p 2011                     # attach to a running process
strace -f -p 2011                  # follow forks and threads too
strace -e trace=openat,read -p 2011  # only the syscalls you care about
strace -c -p 2011                  # no per-call output; a summary table on exit
strace -T -p 2011                  # show time spent inside each call
strace -o trace.log -f -p 2011     # write to a file rather than the terminal

The three patterns you are looking for:

1. Blocked on one call. The trace prints a single line and stops. Whatever is in that line is your answer — a read on a socket means it is waiting on a peer; a futex means lock contention.
2. A tight loop of failures. The same call repeating with the same errno. ENOENT on a config path means it is looking somewhere you did not expect.
3. Volume. strace -c summarises counts and time. A process making a hundred thousand stat calls a second is not hung, it is thrashing on path resolution.

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 82.14    1.204221          12    100352     99981 stat
 11.02    0.161554          21      7692           read
  4.33    0.063511           8      7690           write

That table names the problem in one line: nearly every stat is failing, so the process is searching a path that does not exist.

The warnings that matter:

• strace is slow. It stops the process at every syscall. Overhead of 10–100x is normal. Do not attach it to a latency-sensitive service at peak and walk away.
• It needs permission. Either root, or the same UID plus a permissive kernel.yama.ptrace_scope.
• In containers, it needs SYS_PTRACE. Without that capability the attach fails.
• Prefer -c or a narrow -e trace= filter in production, and detach as soon as you have your answer.

Try it yourself: Run strace -c ls -R /usr/share > /dev/null and read the summary. Note which syscall dominates and how the count scales with the size of the tree.

Common mistake: Attaching strace without -f to a threaded or forking service, then concluding it is idle. All the work is happening in children the trace never followed.

Topic 5: Putting It Together on a Hung Process

A repeatable sequence when something is stuck and the logs have gone quiet:

1. Check the state. ps -o pid,stat,wchan,comm -p <PID>. If it is D, this is storage — go to the disk layer, and note that no signal will kill it.
2. Ask where in the kernel. sudo cat /proc/<PID>/stack names the kernel function it is parked in.
3. Ask what it wanted. strace -p <PID> shows the syscall. One line is often the whole diagnosis.
4. Check what it holds. ls -l /proc/<PID>/fd — the socket or file in question is usually right there.
5. Check the ceiling. Descriptor count against /proc/<PID>/limits, in case the real story is exhaustion.

This ladder takes about ninety seconds and replaces a great deal of speculation.