systemd Units, Restart Policies & journalctl

On any modern Linux host, systemd decides what runs, what restarts, and what gets logged. Most “the service did not come back” incidents are a misunderstanding of three directives in a unit file.

Topic 1: Units and Their Types

systemd manages units — objects with a name, a type suffix, and a dependency graph. PID 1 is systemd itself, and everything else descends from it.

Where unit files live, in precedence order:

1. /etc/systemd/system/ — your overrides and custom units. Highest precedence.
2. /run/systemd/system/ — runtime, transient units. Lost on reboot.
3. /lib/systemd/system/ — units shipped by installed packages. Never edit these; a package update overwrites them.

To adjust a vendor unit safely, use systemctl edit nginx — it creates a drop-in at /etc/systemd/system/nginx.service.d/override.conf containing only your changes, which survives upgrades.

Topic 2: Anatomy of a Unit File

[Unit]
Description=Checkout API service
Documentation=https://internal.example/runbooks/checkout
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=appuser
Group=appuser
WorkingDirectory=/opt/checkout
Environment=NODE_ENV=production
EnvironmentFile=-/etc/checkout/env
ExecStart=/usr/bin/node /opt/checkout/server.js
ExecReload=/bin/kill -HUP $MAINPID

Restart=on-failure
RestartSec=5s
TimeoutStopSec=30s

# Hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/var/lib/checkout

[Install]
WantedBy=multi-user.target

The directives that decide behaviour:

• Type= — how systemd decides the service has finished starting.
  • simple (default) — the process ExecStart launches is the service. Considered started immediately.
  • exec — like simple, but waits until execve succeeds, so a bad binary path fails at start rather than silently.
  • forking — the process forks and the parent exits. Requires PIDFile=. Used by older daemons.
  • oneshot — runs to completion; pair with RemainAfterExit=yes for setup tasks.
  • notify — the service tells systemd when it is ready via sd_notify. The most accurate, if the app supports it.
• After= vs Requires= — After is ordering only; Requires is a hard dependency that fails your unit if the other fails. Ordering without a dependency is After alone, and that distinction catches people out.
• EnvironmentFile=-/path — the leading - means “continue if the file is missing” rather than failing to start.
• $MAINPID — expands to the service’s main PID, which is how ExecReload sends a signal to the right process.

The [Install] section:

This section is only read by systemctl enable. WantedBy=multi-user.target is what creates the symlink that starts your service at boot. A unit with no [Install] section can be started manually but will never come up on its own — a common reason a service does not return after a reboot.

Try it yourself: Run systemctl cat sshd to see a real, complete unit file, then systemctl show sshd | head -40 to see every resolved property including defaults.

Topic 3: Restart Policies — and the Rate Limit That Surprises Everyone

The rate limit:

This is the part that turns a small incident into a long one. systemd will refuse to keep restarting a unit that fails too quickly, too often:

• StartLimitBurst=5 — allowed starts…
• StartLimitIntervalSec=10s — …within this window.

Exceed both and the unit enters failed state with start request repeated too quickly, and systemd stops trying. Your crash-looping service goes permanently down, and the reason is in systemctl status, not in the application log.

systemctl reset-failed checkout    # clear the counter
systemctl start checkout           # then start again

To make a legitimately flaky service keep retrying, widen the window and back off:

[Unit]
StartLimitBurst=10
StartLimitIntervalSec=120s

[Service]
Restart=on-failure
RestartSec=10s

Common mistake: Setting Restart=always with RestartSec=0 and no rate-limit tuning. The service restarts in a tight loop, trips the default limit within seconds, and stays dead — the opposite of what the operator intended.

Shutdown behaviour:

KillSignal= defaults to SIGTERM, and TimeoutStopSec= (90s by default) is how long systemd waits before escalating to SIGKILL. If your service needs longer to drain, raise it — and if it never exits cleanly, that grace period is being spent on every single restart.

Topic 4: systemctl in Practice

systemctl status checkout          # state, main PID, recent log lines
systemctl start|stop|restart checkout
systemctl reload checkout          # runs ExecReload, no downtime
systemctl enable --now checkout    # enable at boot and start immediately
systemctl disable checkout
systemctl is-active checkout       # script-friendly: prints active/inactive
systemctl is-enabled checkout

systemctl list-units --failed      # everything currently broken
systemctl list-timers              # scheduled units and next run
systemctl daemon-reload            # after ANY unit file edit

daemon-reload is the one people forget. systemd caches unit files; edit one without reloading and your change simply does not exist yet.

Reading systemctl status output:

● checkout.service - Checkout API service
     Loaded: loaded (/etc/systemd/system/checkout.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Tue 2026-08-04 14:22:07 UTC; 3min ago
    Process: 4471 ExecStart=/usr/bin/node /opt/checkout/server.js (code=exited, status=1/FAILURE)
   Main PID: 4471 (code=exited, status=1/FAILURE)

Three fields carry the diagnosis: Loaded tells you whether it starts at boot, Active gives the state and how long, and status= gives the exit code the application actually returned.

Topic 5: journalctl — Finding the First Failure

The journal is a structured, indexed binary log. Filters compose, and using them properly is the difference between reading 40 lines and scrolling 40,000.

journalctl -u checkout                     # one unit
journalctl -u checkout -f                  # follow live
journalctl -u checkout -n 100              # last 100 lines
journalctl -u checkout --since '30 min ago'
journalctl -u checkout --since '2026-08-04 14:00' --until '14:30'
journalctl -u checkout -p err              # priority err and worse
journalctl -u checkout -o json-pretty      # full structured fields
journalctl -k                              # kernel ring buffer (dmesg)
journalctl -b                              # this boot only
journalctl -b -1                           # the previous boot
journalctl --disk-usage                    # how much space the journal holds

Priority levels, for -p:

emerg (0), alert (1), crit (2), err (3), warning (4), notice (5), info (6), debug (7). Passing -p err includes everything of that severity and above.

The technique that matters:

During an incident, resist following the tail. Find the first error instead — the tail is usually full of downstream consequences.

# When did it last transition to failed?
systemctl show checkout -p ActiveEnterTimestamp -p InactiveEnterTimestamp

# Then read a window starting slightly before that, across ALL units
journalctl --since '14:21:30' --until '14:23:00'

Dropping -u at that point is deliberate: the cause is often in a different unit — the database, the mount, the network target — a few seconds earlier.

Persistence:

If /var/log/journal/ does not exist, the journal is memory-only and vanishes on reboot, which is a painful discovery after a crash. Make it persistent:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

Try it yourself: Run journalctl -b -p err on any host to see every error since boot. Then pick the earliest timestamp and re-run without -p around that moment to see what led up to it.

Common mistake: Reporting “no errors in the logs” after checking only journalctl -u myservice. If the unit failed to start at all, the reason is frequently logged by systemd or the kernel, not by your service — which never got far enough to log anything.

Topic 6: Timers, and the SysV Scripts You Will Still Meet

systemd timers instead of cron:

A timer unit activates a service unit on a schedule. It costs one extra file compared with a crontab line, and buys you the journal, dependency ordering, and the ability to run the job manually with systemctl start.

# /etc/systemd/system/cleanup.timer
[Unit]
Description=Nightly cleanup

[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true
RandomizedDelaySec=300

[Install]
WantedBy=timers.target

systemctl enable --now cleanup.timer
systemctl list-timers --all      # next and last run for every timer
journalctl -u cleanup.service    # the output, kept and indexed

Two directives earn their place: Persistent=true runs a missed job once the machine comes back (a laptop or a spot instance that was asleep at 02:30), and RandomizedDelaySec spreads a fleet out so five hundred hosts do not hit the same backend at the same second.

/etc/init.d and SysV:

Before systemd, services were shell scripts in /etc/init.d/, invoked directly:

/etc/init.d/networking stop
/etc/init.d/nginx restart

You will still find these on older estates and inside some container images. On a systemd host, most are shimmed — systemctl generates a unit for them automatically — so prefer systemctl and let the shim handle it. The one thing to know is that a SysV script has no supervision: it starts a process and forgets it. Nothing restarts it, nothing tracks it, and status is whatever the script author chose to implement.

When the unit is fine but the service still cannot work:

If a service starts and then fails on a file it clearly has permission to read, check whether SELinux is intervening before you re-read the unit file:

sestatus                                    # is SELinux enforcing?
sudo ausearch -m avc -ts recent             # recent denials
sudo grep -i denied /var/log/audit/audit.log | tail

The systemd hardening directives have the same effect by design. ProtectSystem=strict makes the whole filesystem read-only to that unit except paths named in ReadWritePaths=; PrivateTmp=true gives it its own /tmp, so a file it wrote is not the file another process sees. Both produce permission errors that look nothing like permission errors on disk.

Try it yourself: Run systemctl list-timers on any host. Compare NEXT and LAST for a unit, and then read that unit’s output with journalctl -u <name>.service — this is the visibility cron never gave you.