Cron, systemd Timers & Safe Scheduling

“It works when I run it manually” is the most common bug report for scheduled jobs, and it always has the same handful of causes. Cron does not give you your shell — it gives you a nearly empty one.

Topic 1: Crontab Syntax

┌───────────── minute        (0-59)
│ ┌─────────── hour          (0-23)
│ │ ┌───────── day of month  (1-31)
│ │ │ ┌─────── month         (1-12)
│ │ │ │ ┌───── day of week   (0-7, both 0 and 7 = Sunday)
│ │ │ │ │
* * * * *  /usr/local/bin/backup.sh

*/5 * * * *      every 5 minutes
0 * * * *        hourly, on the hour
30 2 * * *       02:30 daily
0 3 * * 0        03:00 on Sunday
0 9 1 * *        09:00 on the 1st of the month
0 2 * * 1-5      02:30 weekdays only
@reboot          once at boot
@daily           midnight (= 0 0 * * *)

crontab -e         # edit YOUR crontab (validates on save)
crontab -l         # list
crontab -r         # remove ALL of it -- no confirmation, easy to hit by accident
sudo crontab -u www-data -l    # another user's

System-wide locations behave differently: /etc/crontab and files in /etc/cron.d/ take an extra field — the user to run as:

30 2 * * * backupuser /usr/local/bin/backup.sh

Omitting it in /etc/cron.d/ is a silent failure; cron parses the username as the command.

Day-of-month and day-of-week are OR’d, not AND’d. 0 0 13 * 5 runs on the 13th and every Friday, not only Friday the 13th. This surprises everyone once.

Topic 2: Why It Breaks Under Cron

Three of those cause nearly every failure:

1. PATH. docker, kubectl, aws, and anything from Homebrew or /usr/local/bin are not found.

# In the script -- best
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# Or in the crontab, above the entries
PATH=/usr/local/bin:/usr/bin:/bin

2. /bin/sh, not bash. A crontab line runs under sh unless told otherwise, so bash syntax in the crontab line itself fails. The script’s own shebang still governs the script.

SHELL=/bin/bash

3. Nothing is sourced. Any variable you set in .bashrc — a token, a KUBECONFIG, an AWS_PROFILE — does not exist. Set them explicitly in the script.

The % character:

In a crontab, an unescaped % is turned into a newline and everything after the first one becomes stdin. This mangles every date format string:

# BROKEN -- stops at the first %
0 2 * * * /bin/echo "run at $(date +%Y-%m-%d)" >> /var/log/x.log

# Escaped
0 2 * * * /bin/echo "run at $(date +\%Y-\%m-\%d)" >> /var/log/x.log

This is a strong argument for putting all logic in a script and keeping the crontab line trivial.

Reproducing cron’s environment:

env -i /bin/sh -c '/usr/local/bin/myscript.sh'     # near-empty environment

Or capture the real thing once:

* * * * * env > /tmp/cron-env.txt

diff <(env | sort) <(sort /tmp/cron-env.txt)

Topic 3: The Four Defences

Every scheduled job needs all four. Together they are about fifteen lines.

#!/usr/bin/env bash
set -Eeuo pipefail

# 1. EXPLICIT ENVIRONMENT
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
export HOME=${HOME:-/root}
cd -- "$(dirname -- "${BASH_SOURCE[0]}")" || exit 1

# 2. LOGGING -- cron mail is not a log
readonly LOG=/var/log/myjob/$(date +%Y%m%d).log
mkdir -p "$(dirname "$LOG")"
exec >> "$LOG" 2>&1
echo "=== $(date -Iseconds) starting (pid $$) ==="

# 3. LOCKING -- overlapping runs corrupt state
exec 200>/var/lock/myjob.lock
flock -n 200 || { echo "already running; skipping"; exit 0; }

# 4. TIMEOUT -- never hang until the next run
readonly MAX_RUNTIME=3600
( sleep "$MAX_RUNTIME"; echo "TIMEOUT after ${MAX_RUNTIME}s"; kill -TERM $$ ) &
readonly WATCHDOG=$!
trap 'kill "$WATCHDOG" 2>/dev/null || true' EXIT

# ... the actual work ...

echo "=== $(date -Iseconds) finished ok ==="

Notes on each:

Locking with flock beats a PID file because the kernel releases the lock when the process dies — including on kill -9 or a power cut. A PID file survives a crash and blocks every future run until a human removes it. Exit 0 when the lock is held: a skipped run is expected behaviour, and a non-zero exit makes cron mail you every five minutes.

You can also apply it from the crontab with no code change:

*/5 * * * * /usr/bin/flock -n /var/lock/myjob.lock /usr/local/bin/myjob.sh

Timeouts matter because the default failure mode of a network call is to hang. Simpler than the watchdog above, wrap the whole thing:

0 2 * * * /usr/bin/timeout 3600 /usr/local/bin/myjob.sh

Jitter on a fleet: five hundred hosts running 0 3 * * * hit the backend simultaneously.

sleep $(( RANDOM % 300 ))       # spread over 5 minutes

Topic 4: Output, Mail and Silence

Cron mails any output — stdout or stderr — to the crontab owner. Two failure modes follow:

• Noisy job → mail every run → everyone filters it → the real failure is filtered too.
• >/dev/null 2>&1 on the crontab line → silence, including errors → failures are invisible for months.

The correct shape is: log everything to a file, and let only genuine failures produce output.

MAILTO=ops@example.com
0 2 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1 || echo "backup FAILED, see /var/log/backup.log"

Better still, have the script decide — silent on success, loud on failure, having already written the detail to its log.

Checking it actually ran:

grep CRON /var/log/syslog | tail          # Debian/Ubuntu
journalctl -u cron --since today          # systemd
journalctl -u crond --since today         # RHEL family

Cron logs that it started a job. It does not log the outcome — that is your script’s responsibility.

Dead-man switches:

The failure cron cannot report is the job that never ran at all — the host was down, the crontab was wiped, the disk was full. A heartbeat catches it:

# At the end of a successful run
curl -fsS --max-time 10 "https://hc-ping.com/${HEALTHCHECK_UUID}" >/dev/null || true

The monitoring service alerts when the ping stops arriving, which is the only way to detect absence.

Topic 5: systemd Timers

On a systemd host, timers are the better choice for anything that matters. Two files instead of one line, and you get logging, dependencies, and manual runs.

# /etc/systemd/system/backup.service
[Unit]
Description=Nightly backup
After=network-online.target

[Service]
Type=oneshot
User=backup
Environment=PATH=/usr/local/bin:/usr/bin:/bin
ExecStart=/usr/local/bin/backup.sh
TimeoutStartSec=3600

# /etc/systemd/system/backup.timer
[Unit]
Description=Run backup nightly

[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true
RandomizedDelaySec=600

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now backup.timer
systemctl list-timers backup.timer         # next and last run
journalctl -u backup.service -n 50         # the output, indexed
sudo systemctl start backup.service        # run it NOW, by hand

That last row is worth noting: systemd will not start a service that is already running, so a timer gives you locking for free.

Try it yourself: Schedule a script for one minute from now that dumps env, pwd, and id to a file. Compare with the same script run from your shell and count the differences.

Common mistake: Testing a cron job by running it in your terminal and declaring it works. Test it the way cron will run it: env -i /bin/sh -c /path/to/script.sh. Everything that breaks there breaks at 3am.