Packages, Repositories & Reproducible Hosts

Half of “it works on my machine” is a version difference nobody recorded. Package managers are the audit trail for what is actually on a host — and the tool most likely to change production without anybody calling it a deploy.

Topic 1: The Two Families

Nearly every server distribution descends from one of two packaging lineages. The concepts are identical; only the commands differ.

Both formats are archives with metadata attached: the files to install, where they go, which other packages are required, and scripts to run before and after. The difference is convention, not capability.

Identify what you are on before you type anything:

cat /etc/os-release        # the portable answer -- works everywhere
hostnamectl                # includes kernel and virtualisation
uname -r                   # kernel version only

Never guess from the hostname. A box called prod-ubuntu-04 is not evidence of anything.

Low-level versus high-level:

dpkg and rpm install a single file you already have. They do not resolve dependencies — they fail and tell you what is missing. apt and dnf sit on top, talk to repositories, and work out the dependency graph for you. Reach for the low-level tool only for a .deb/.rpm you downloaded deliberately.

sudo dpkg -i google-chrome-stable_current_amd64.deb   # may fail on deps...
sudo apt-get install -f                                # ...then this fixes them
sudo apt install ./google-chrome-stable_current_amd64.deb  # or just do this

Topic 2: The Commands You Actually Need

Debian family:

sudo apt update                 # refresh the package INDEX (downloads nothing else)
sudo apt upgrade                # upgrade installed packages
sudo apt full-upgrade           # ...allowing removals to resolve conflicts
sudo apt install htop
sudo apt remove htop            # remove binaries, KEEP config
sudo apt purge htop             # remove config too
sudo apt autoremove             # drop orphaned dependencies

apt list --installed
apt list --upgradable
apt policy nginx                # which versions are available, from where
apt-cache depends nginx
dpkg -l | grep nginx            # installed packages matching a name
dpkg -S /usr/bin/ss             # WHICH PACKAGE owns this file
dpkg -L iproute2                # every file a package installed

Red Hat family:

sudo dnf check-update
sudo dnf upgrade
sudo dnf install htop
sudo dnf remove htop
sudo dnf autoremove

dnf list installed
dnf info nginx
dnf repoquery --requires nginx
rpm -qa | grep nginx
rpm -qf /usr/bin/ss             # which package owns this file
rpm -ql iproute                 # every file a package installed
rpm -qi nginx                   # full metadata including install date

The two you will use in anger are the ownership queries — dpkg -S and rpm -qf. When a config file appears in a place nobody expected, or you need to know whether a binary came from a package or somebody’s scp, that is the command that answers it.

Common mistake: Running apt update and believing something was upgraded. update refreshes the index only. upgrade is what changes the system. The two words are close enough that people report “I updated the box” having changed nothing at all — or having changed everything.

Topic 3: What an Upgrade Actually Is

apt upgrade on a production host is a deployment. It can restart services, replace configuration, and change kernel behaviour. Treat it with the same ceremony as a code release.

Know what will change before it changes:

apt list --upgradable                       # what is pending
sudo apt upgrade --dry-run                  # simulate, change nothing
sudo apt-get -s dist-upgrade                # -s = simulate

sudo dnf check-update
sudo dnf upgrade --assumeno                 # show the transaction, then decline

Read the history afterwards — or during an incident:

This is the single most valuable thing in this lesson. “What changed?” is the first question of any triage, and the package manager keeps a precise, timestamped answer.

# Debian
grep -E ' (install|upgrade) ' /var/log/dpkg.log | tail -30
zcat -f /var/log/apt/history.log* | grep -A3 'Start-Date'

# Red Hat
sudo dnf history                 # numbered transactions with dates
sudo dnf history info 42         # everything one transaction touched
sudo dnf history undo 42         # roll a transaction back

dnf history undo has no clean Debian equivalent, which is worth knowing before you need it.

Pinning and holding:

sudo apt-mark hold nginx         # never upgrade this
sudo apt-mark unhold nginx
apt-mark showhold

# Red Hat: in /etc/dnf/dnf.conf
exclude=kernel* nginx

Holds are how you stop an unattended upgrade taking your database version with it. They are also how a host quietly drifts three years behind on security patches, so record them somewhere a human reads.

Unattended upgrades:

Most cloud images ship with automatic security updates enabled. That is usually right for security patches and usually wrong for everything else.

cat /etc/apt/apt.conf.d/50unattended-upgrades      # what is allowed through
systemctl status unattended-upgrades
journalctl -u unattended-upgrades --since '7 days ago'

If a service restarted at 06:00 with no deploy, this is the first place to look.

Topic 4: Repositories and Trust

A repository is a server holding packages plus a signed index. The signature is what stops an attacker serving you a modified package.

# Debian: modern signed-by form
curl -fsSL https://example.com/key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/example.gpg
echo "deb [signed-by=/usr/share/keyrings/example.gpg] https://example.com/apt stable main" \
  | sudo tee /etc/apt/sources.list.d/example.list

apt policy                       # every configured repo and its priority

# Red Hat
sudo dnf config-manager --add-repo https://example.com/example.repo
dnf repolist
rpm -qa gpg-pubkey*              # trusted signing keys

Two rules worth holding to:

1. Never disable signature verification. --allow-unauthenticated and gpgcheck=0 turn a supply chain into an open door. If verification fails, the correct response is to find out why.
2. Prefer one repository per file in sources.list.d/ or yum.repos.d/. Editing the main list makes it far harder to see what a host trusts.

The other formats:

• snap / flatpak — bundle their own dependencies and update independently of apt. Convenient on desktops, an extra unaudited update channel on a server. snap list, snap refresh --list.
• Language package managers (pip, npm, gem) install outside the system database entirely. dpkg -S will never find them, which is exactly why a pip install into system Python is worth avoiding — use a virtualenv.
• Containers side-step the question by shipping the whole userland. The packages are still there; they are just pinned inside an image.

Try it yourself: Run dpkg -S $(which ss) or rpm -qf $(which ss). Then run the history command for your family and find the most recent thing that changed on the host. If anything landed outside a change window, that is worth asking about.

Common mistake: Treating package upgrades as maintenance rather than change. A minor version bump to a library restarts the services linked against it. If your change log has no entry for it, your incident timeline will have a gap exactly where the cause is.