A host that will not come back after a reboot is one of the most stressful incidents there is, because the usual tools are on the other side of the problem. Knowing the stages — and which one you are stuck in — turns panic into a short checklist.
Topic 1: The Five Stages
1. Firmware (UEFI or BIOS)
Power-on self-test, hardware initialisation, then find something bootable. UEFI reads a boot entry pointing at an EFI executable on the EFI System Partition; legacy BIOS reads the first 512 bytes of a disk.
[ -d /sys/firmware/efi ] && echo "UEFI" || echo "legacy BIOS"
efibootmgr -v # UEFI boot entries, in order
Nothing here reaches any log file. If the machine never leaves this stage you need console access — a serial console, IPMI/iDRAC, or the cloud provider’s console screenshot.
2. Bootloader (GRUB2)
Presents a menu, loads the selected kernel and its initramfs into memory, and passes the kernel command line.
cat /proc/cmdline # the command line THIS boot used
sudo grubby --info=ALL # RHEL family: entries and their args
cat /boot/grub/grub.cfg | grep -E '^menuentry' # Debian family
/proc/cmdline is worth knowing: it is the definitive record of what the running kernel was told, including any nomodeset, systemd.unit=, or memory limits somebody added months ago and forgot.
3. Kernel
Decompresses itself, initialises the CPU, memory and core subsystems, mounts the initramfs as a temporary root, and starts probing hardware.
4. initramfs
A small in-memory root filesystem carrying just enough drivers and tooling to reach the real root filesystem — unlocking LUKS volumes, assembling RAID arrays, activating LVM, or bringing up networking for an NFS root. It then pivots to the real root and hands over.
This stage is the usual suspect when a host boots fine until somebody changes storage. A new disk layout that the initramfs has no driver for produces a hang with a bare (initramfs) prompt.
sudo update-initramfs -u -k all # Debian: rebuild after storage/driver changes
sudo dracut -f --regenerate-all # Red Hat: same idea
lsinitramfs /boot/initrd.img-$(uname -r) | grep -i nvme # is the driver in there?
5. systemd (PID 1)
Reads its unit graph and activates everything needed to reach default.target.
systemctl get-default # usually multi-user.target on a server
systemctl set-default multi-user.target
Topic 2: Targets — the Replacement for Runlevels
SysV had numbered runlevels. systemd has named targets, which are just units that group other units. The mapping is worth knowing because older runbooks still say “boot to runlevel 3”.
| Runlevel | Target | Meaning |
|---|---|---|
| 0 | poweroff.target | Halt |
| 1 | rescue.target | Single user, minimal services, root shell |
| 3 | multi-user.target | Full multi-user, no GUI — the server default |
| 5 | graphical.target | Multi-user plus display manager |
| 6 | reboot.target | Restart |
| — | emergency.target | Even more minimal than rescue: root filesystem mounted read-only, almost nothing started |
systemctl list-units --type=target # what is active now
systemctl isolate rescue.target # switch NOW -- stops services not in that target
systemctl list-dependencies multi-user.target
isolate is a live change, not a reboot setting. Running it on a production host stops every service that is not part of the target you named — which is occasionally exactly what you want and more often a self-inflicted outage.
Booting into rescue deliberately:
When a host will not finish booting, interrupt GRUB, press e on the entry, and append to the linux line:
| Append | Effect |
|---|---|
systemd.unit=rescue.target | Single-user with most filesystems mounted |
systemd.unit=emergency.target | Bare minimum, root read-only |
init=/bin/bash | Skip systemd entirely — the last resort |
systemd.log_level=debug | Verbose boot logging |
nomodeset | Skip KMS graphics init, for a black screen after boot |
Ctrl+X boots with the modification, for that boot only. To make the root filesystem writable in emergency mode:
mount -o remount,rw /
Topic 3: Diagnosing a Slow Boot
systemd ships its own profiler, and it answers a question people usually guess at.
systemd-analyze # total, split into firmware/loader/kernel/userspace
systemd-analyze blame # every unit, slowest first
systemd-analyze critical-chain # only what actually delayed the boot
systemd-analyze critical-chain nginx.service
systemd-analyze plot > boot.svg # a full timeline you can open in a browser
Startup finished in 4.512s (firmware) + 3.108s (loader) + 1.204s (kernel) + 18.442s (userspace) = 27.267s
That split alone directs the investigation: 18 seconds in userspace is a unit problem, while 4.5 seconds of firmware is a hardware or BIOS setting nothing in Linux can fix.
blame versus critical-chain — the distinction that matters:
blame sorts every unit by how long it took. critical-chain shows only the units that blocked the boot, with the time at which each became active. A unit can take 20 seconds and cost you nothing if the boot was not waiting on it.
Optimising the top of blame is the classic wasted afternoon. Always cross-check against critical-chain first.
The usual culprits:
NetworkManager-wait-online.serviceorsystemd-networkd-wait-online— blocks until an interface has an address. On a host with an interface that never comes up, this waits out its full timeout, every boot.- Filesystem mounts for devices that no longer exist. An
/etc/fstabentry for a removed disk stalls boot for 90 seconds by default; addnofailto any non-essential mount. - Cloud-init fetching metadata over a network that is not ready.
journalctl -b -u systemd-networkd-wait-online
sudo systemctl disable NetworkManager-wait-online.service # only if nothing needs it
Topic 4: Reading the Boot Afterwards
journalctl -b # this boot, everything
journalctl -b -1 # the PREVIOUS boot -- what you want after a crash
journalctl --list-boots # every boot the journal still holds
journalctl -b -p err # errors only, this boot
journalctl -k -b # kernel messages only
systemctl --failed # units that gave up
journalctl -b -1 is the command for “the host rebooted and I do not know why”. It only works if the journal is persistent — if /var/log/journal/ does not exist, the previous boot’s logs went with it. Make it persistent before you need it:
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald
Was it a crash or a clean reboot?
last -x reboot | head # reboot and shutdown records
last -x shutdown | head
uptime -s # when this boot started
journalctl -b -1 -n 50 # the last 50 lines BEFORE the reboot
A clean shutdown leaves an orderly sequence of units stopping. A crash, a kernel panic, or a hypervisor-level kill leaves the log ending mid-sentence — and that abrupt ending is itself the diagnosis.
Try it yourself: Run journalctl --list-boots. If only one boot is listed, your journal is not persistent — fix that now rather than after the next unexplained restart.
Common mistake: Rebooting a host to “clear” a problem before capturing anything. If the journal is not persistent, the reboot destroys the only record of the cause, and the second occurrence is no more diagnosable than the first.