Background Jobs, Signals & Parallel Execution

Shell scripts are process managers. The moment one launches background work — a parallel deploy, a batch of health checks — it inherits responsibility for cleaning up after itself, and most scripts do not.

Topic 1: Background Jobs and wait

long_task &                 # run in background
pid=$!                      # $! holds the PID of the LAST background job
echo "started as ${pid}"
wait "$pid"                 # block until it finishes
echo "exit status: $?"      # wait returns the JOB's status

$! must be captured immediately — the next & overwrites it.

Collecting several jobs properly:

declare -A pids
for host in web-01 web-02 db-01; do
    deploy_to "$host" &
    pids[$!]=$host          # map PID → what it was doing
done

failed=0
for pid in "${!pids[@]}"; do
    if wait "$pid"; then
        echo "✓ ${pids[$pid]}"
    else
        echo "✗ ${pids[$pid]} (exit $?)" >&2
        (( failed++ )) || true
    fi
done
(( failed == 0 )) || exit 1

A bare wait with no argument waits for everything and returns 0 regardless of what failed — which is how parallel deploy scripts report success while one host failed. wait <pid> per job is the only way to get individual statuses.

Bash 4.3+ adds wait -n, which returns as soon as any job finishes — the basis of a worker pool:

max_parallel=4
for item in "${items[@]}"; do
    while (( $(jobs -rp | wc -l) >= max_parallel )); do
        wait -n
    done
    process "$item" &
done
wait

When xargs is the better answer:

printf '%s\n' "${hosts[@]}" | xargs -P8 -I{} ssh {} 'uptime'
find . -name '*.log' -print0 | xargs -0 -P4 gzip

xargs -P gives you a bounded worker pool with no bookkeeping. Use it whenever the work is uniform; keep the bash loop when you need per-job identity in the output. GNU parallel goes further, with per-job output buffering that stops interleaved lines.

Topic 2: Signals a Script Must Handle

The one that matters operationally is SIGTERM. Every orchestrator sends it first and waits a grace period before escalating to SIGKILL — Docker gives 10 seconds, Kubernetes 30, systemd 90. A script that ignores SIGTERM gets killed mid-write on every single deploy.

trap 'cleanup; exit 143' TERM      # 143 = 128 + 15
trap 'cleanup; exit 130' INT       # 130 = 128 + 2
trap cleanup EXIT

The 128 + signal convention is what the shell itself reports, so matching it keeps your script consistent with everything else.

Topic 3: Cleaning Up Children

The default behaviour is bad: interrupt a script with background jobs and the children keep running, re-parented to PID 1. They hold locks, keep writing to files, and nobody knows they exist.

#!/usr/bin/env bash
set -Eeuo pipefail

cleanup() {
    local code=$?
    # Kill every job still running in THIS shell's job table
    local running
    running=$(jobs -rp)
    if [[ -n $running ]]; then
        echo "terminating $(wc -w <<< "$running") child processes" >&2
        # shellcheck disable=SC2086
        kill $running 2>/dev/null || true
        sleep 2
        kill -9 $running 2>/dev/null || true       # escalate for anything left
    fi
    rm -rf "${WORKDIR:-}"
    return "$code"
}

trap cleanup EXIT
trap 'exit 130' INT
trap 'exit 143' TERM

WORKDIR=$(mktemp -d)

jobs -rp lists the PIDs of running jobs. The pattern is: SIGTERM, brief grace, then SIGKILL — exactly what the orchestrators do to you.

Killing a whole process group:

Children that spawn their own children need the group, not the process:

set -m                       # enable job control so children get their own group
long_task &
child_pid=$!
kill -TERM -"$child_pid"     # negative PID = the whole process GROUP

The leading - on the PID is what makes this a group signal. Without it, a ssh that spawned a remote command leaves the remote side running.

timeout for anything that can hang:

timeout 30 curl -s https://api.example.com          # exit 124 if it times out
timeout --kill-after=5 30 ./slow-task               # TERM at 30s, KILL at 35s

Exit code 124 specifically means “timed out”, so callers can distinguish a hang from a failure. Wrapping every network call in timeout is the cheapest defence against a script that hangs forever in cron.

Topic 4: Signals Are Deferred, Not Immediate

Bash does not run a trap in the middle of a foreign command. If the script is inside sleep 300 when SIGINT arrives, the trap fires after sleep returns.

trap 'echo interrupted' INT
sleep 300        # Ctrl+C here: sleep dies immediately, THEN the trap runs

For a long wait that must stay responsive, background it and wait:

trap 'echo interrupted; exit 130' INT
sleep 300 &
wait $!          # wait IS interruptible -- the trap fires immediately

wait is one of the few builtins that a signal interrupts directly. This is the idiom for any script that must respond promptly to a shutdown request while idle.

Ignoring and restoring:

trap '' INT                  # IGNORE -- Ctrl+C does nothing
critical_section
trap - INT                   # restore the DEFAULT

Use trap '' sparingly and briefly. A script that cannot be interrupted during a five-minute operation is a script people will kill -9, which skips your cleanup entirely.

Topic 5: Surviving Disconnection

nohup ./long-task &          # ignore SIGHUP, output to nohup.out
./long-task & disown         # remove from the job table after the fact
setsid ./long-task           # new session — fully detached
tmux new -d -s work './long-task'    # re-attachable, the usual right answer

For anything that should genuinely outlive your session, hand it to the init system rather than detaching it yourself:

systemd-run --unit=migration --collect ./migrate.sh
journalctl -u migration -f

That gives you logging, restart policy, and status — none of which nohup provides.

Topic 6: Locking — One Instance at a Time

The cron job that runs every five minutes and occasionally takes six is a standard production incident: two copies run concurrently and corrupt each other’s state.

#!/usr/bin/env bash
set -Eeuo pipefail

readonly LOCKFILE=/var/lock/myjob.lock
exec 200>"$LOCKFILE"
if ! flock -n 200; then
    echo "another instance is running; exiting" >&2
    exit 0            # exit 0 -- this is expected, not a failure
fi

# ... work ...

flock takes a lock on a file descriptor. The lock is held by the kernel for as long as fd 200 is open, and it is released automatically when the process exits — including on kill -9, a crash, or a power loss. That is the crucial advantage over a PID file, which survives a crash and blocks every subsequent run until someone deletes it by hand.

Options: -n fails immediately rather than blocking; -w 30 waits up to 30 seconds.

You can also wrap the whole script from crontab, with no code change:

*/5 * * * * /usr/bin/flock -n /var/lock/myjob.lock /usr/local/bin/myjob.sh

Common mistake: Exiting non-zero when the lock is held. Cron mails every non-zero exit, so a job that legitimately skips a run because the previous one is still going will page you every five minutes. Exit 0 and log it.

Topic 7: Safe Temporary Files

tmpfile=$(mktemp)                       # /tmp/tmp.a8Xk2p — unpredictable name
tmpdir=$(mktemp -d)                     # a directory
tmpfile=$(mktemp -t myapp.XXXXXX)       # with a recognisable prefix
trap 'rm -rf "$tmpdir" "$tmpfile"' EXIT

Never construct a temp path by hand. /tmp/myapp.$$ is predictable — the PID space is small and reused — which makes it a symlink-attack target: an attacker pre-creates /tmp/myapp.1234 as a symlink to /etc/passwd and your script writes through it. mktemp creates the file atomically with mode 600 and an unguessable name.

Pair it with the EXIT trap from the error-handling lesson so the file is removed on every exit path.

Try it yourself: Write a script that starts three sleep jobs of different lengths, collects each status with wait <pid>, and cleans them all up on Ctrl+C. Interrupt it and confirm with pgrep sleep that nothing survived.

Common mistake: trap cleanup EXIT INT TERM. On Ctrl+C this runs cleanup twice — once for INT, once for EXIT. Route signals to exit with the right code and let EXIT do the work exactly once.