Signals, Graceful Shutdown & Job Control

Every deploy, every pod eviction, and every rolling restart is a signal being delivered to a process. If you know what each signal means and which ones can be caught, you can explain exactly why a service dropped connections during a release — and how to stop it happening again.

Topic 1: What a Signal Is

A signal is a small asynchronous notification the kernel delivers to a process. It carries no payload — only a number. The process can, for most signals, install a handler: a function that runs when the signal arrives, interrupting whatever the process was doing.

Every signal has one of three dispositions:

1. Default — the kernel’s built-in action, usually terminate.
2. Ignored — the process explicitly asks for it to be discarded.
3. Caught — the process runs its own handler instead.

Two signals can never be caught, blocked, or ignored: SIGKILL (9) and SIGSTOP (19). The kernel enforces them directly so that an administrator always has a way out.

Topic 2: The Signal Table

The reload convention:

SIGHUP originally meant “the modem hung up”. Since daemons have no terminal to lose, the convention became: use it to re-read configuration without dropping connections. nginx -s reload, for instance, is a SIGHUP under the hood. This is a convention, not a rule — always check the daemon’s documentation.

Try it yourself: Run kill -l to list every signal your system supports, with numbers.

Topic 3: Graceful Shutdown, and Why SIGKILL Costs You Data

This is the single most useful thing in this lesson.

What SIGTERM buys you:

When a well-written service receives SIGTERM, it should:

1. Stop accepting new work (close the listening socket, fail readiness checks).
2. Finish the requests already in flight.
3. Flush buffers to disk, commit or roll back open transactions.
4. Deregister from service discovery and close downstream connections cleanly.
5. Exit with status 0.

What SIGKILL does:

The kernel removes the process. Immediately. No handler runs, no buffer is flushed, no connection is closed politely. In-flight requests become client-side timeouts. Buffered writes that had not reached disk are gone.

The escalation pattern everything uses:

Send SIGTERM, wait a grace period, then send SIGKILL if the process is still alive. You will see the same pattern everywhere:

• systemd — TimeoutStopSec, default 90 seconds.
• Docker — docker stop waits 10 seconds by default.
• Kubernetes — terminationGracePeriodSeconds, default 30 seconds.

Reading exit codes from signals:

When a shell reports the exit status of a signalled process, it uses 128 + signal number:

• 137 = 128 + 9 = killed by SIGKILL. In Kubernetes this usually means the OOM killer or a grace period that expired.
• 143 = 128 + 15 = terminated by SIGTERM. A normal, orderly shutdown.

Common mistake: Setting a grace period shorter than the application’s slowest request. The pod then gets SIGKILLed mid-request on every single deploy, and the resulting error spike gets blamed on the new code rather than the shutdown configuration.

Topic 4: Sending Signals

kill -TERM 2011        # by name, one PID
kill -15 2011          # identical, by number
kill 2011              # SIGTERM is the default

pkill -f 'deploy.sh'   # by command-line pattern
pkill -u www-data      # every process owned by a user
killall nginx          # every process with that exact name

Choosing between them:

• kill — precise. You have the PID and you mean that process.
• pkill -f — matches against the full command line. Powerful and dangerous: always run the same pattern through pgrep -af first to see what it would hit.
• killall — matches the process name exactly. Behaves differently on non-Linux Unixes, where it has historically meant “kill everything”. Prefer pkill.

Finding the right PID first:

Signalling the wrong process is the expensive mistake, so the lookup deserves as much care as the kill:

pgrep -af nginx           # PIDs plus full command lines -- ALWAYS run this first
pgrep -u www-data -l      # by owning user
pidof nginx               # bare PIDs, script-friendly
systemctl show -p MainPID nginx    # the PID systemd considers authoritative
ps -eo pid,ppid,etime,comm --sort=etime | head   # oldest processes first

systemctl show -p MainPID is the safest of these for a managed service: it returns the one process systemd will supervise, rather than every worker and helper that matches a name.

Permission to signal:

You can only signal a process you own, unless you are root. Attempting otherwise returns Operation not permitted. This is why runbook steps that kill a daemon need sudo — and why a stray sudo kill -9 is worth pausing over, since with root there is nothing left to stop you taking out PID 1’s children.

Try it yourself: Start sleep 600 &, then run pgrep -af sleep to see what a pattern would match before you act on it. Only then run pkill -f 'sleep 600'.

Common mistake: Running pkill -f java on a host running several JVMs. The pattern matches all of them and you take down services nobody asked you to touch. Confirm with pgrep first, every time.

Topic 5: Traps — Catching Signals in Scripts

In a shell script, trap installs a handler. This is how you make automation clean up after itself.

#!/bin/bash
set -euo pipefail

WORKDIR=$(mktemp -d)

cleanup() {
  echo "Removing scratch directory ${WORKDIR}"
  rm -rf "${WORKDIR}"
}

# EXIT fires on any exit path; INT and TERM cover interruption
trap cleanup EXIT INT TERM

echo "Working in ${WORKDIR}"
sleep 30

Trapping EXIT is the highest-value line here: it runs whether the script finished, errored out under set -e, or was interrupted. One handler covers every path.

Try it yourself: Run the script above, press Ctrl+C partway through, and confirm the scratch directory was removed anyway.

Topic 6: Job Control and Surviving Disconnection

Foreground and background:

• command & — start in the background; the shell returns immediately.
• Ctrl+Z — suspend the foreground job (sends SIGTSTP, state becomes T).
• jobs — list this shell’s jobs with their job numbers.
• bg %1 — resume job 1 in the background.
• fg %1 — bring job 1 back to the foreground.

The disconnection problem:

When your SSH session drops, the kernel sends SIGHUP to the shell’s job group. Anything still attached dies with it — which is how people lose four-hour migrations to a flaky connection.

Try it yourself: Start sleep 600, press Ctrl+Z, run jobs to see it stopped, then bg to resume it in the background, then disown it. Confirm with ps -o pid,ppid,stat -p <PID> that it survives after you exit the shell — its parent becomes PID 1.

Common mistake: Reaching for nohup on a long migration and then having no way to see progress or interact when it needs a decision. tmux costs one extra command and lets you re-attach from anywhere.