Hardening: SSH, Firewalls, SELinux & Capabilities

Hardening is not a checklist you run once. It is a set of decisions about what a host is allowed to do, and every one of them can also cause an outage — which is why an SRE needs to read them as fluently as they write them.

Topic 1: SSH — the Front Door

SSH is how you get in, which makes it the thing most worth locking down and the easiest to lock yourself out of.

Read the effective config, not the file:

sudo sshd -T                    # every setting, INCLUDING defaults
sudo sshd -T | grep -E 'permitrootlogin|passwordauthentication|port|allowusers'
sudo sshd -t                    # validate syntax before reloading

sshd -T matters because /etc/ssh/sshd_config shows only what was overridden. A setting you cannot find in the file is still in effect at its default, and on many distributions a sshd_config.d/*.conf drop-in overrides what you are reading.

The settings that carry weight:

# /etc/ssh/sshd_config
Port 22
PermitRootLogin no                 # or prohibit-password for automation
PasswordAuthentication no          # keys only -- the single biggest win
PubkeyAuthentication yes
PermitEmptyPasswords no
MaxAuthTries 3
LoginGraceTime 30
AllowUsers deploy ops
X11Forwarding no
ClientAliveInterval 300
ClientAliveCountMax 2

UseDNS no                          # see below
GSSAPIAuthentication no

UseDNS, the operational trap:

With UseDNS yes, sshd performs a reverse lookup of every connecting client before authenticating. If reverse DNS is slow or unanswered, every login stalls for the resolver timeout — while ping, the TCP handshake, and the port check all look perfectly healthy. This is exactly the failure in the SSH outage war room, and it is why ssh -vvv shows a repeatable pause at the publickey step.

GSSAPIAuthentication yes causes the same class of stall via Kerberos lookups. Both default to off on modern builds, but older images and hardening templates still turn them on.

The rule that prevents lockouts:

Never close your session after changing sshd config. Reload, then open a second connection and confirm it works before letting go of the first.

sudo sshd -t && sudo systemctl reload sshd
# now, from ANOTHER terminal:
ssh -v deploy@host

reload re-reads config without dropping existing connections; restart also leaves them alone in modern sshd, but reload is the safer habit.

Key hygiene:

ssh-keygen -t ed25519 -C 'deploy@ci'     # ed25519 over RSA: shorter, faster, safer
chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys
ssh-keygen -lf ~/.ssh/authorized_keys    # fingerprint every authorised key

SSH refuses keys with loose permissions — a “permission denied (publickey)” with a correct key is very often a ~/.ssh that is group-writable. Restrict what a key may do at the point of authorisation:

# ~/.ssh/authorized_keys
from="10.0.0.0/8",no-agent-forwarding,no-port-forwarding,command="/usr/local/bin/backup.sh" ssh-ed25519 AAAA...

Topic 2: Firewalls — Close Everything, Open What You Need

First, know what is exposed:

ss -tlnp                       # every listening TCP socket and its process
ss -ulnp                       # UDP
sudo lsof -i -nP | grep LISTEN

Read the address, not just the port. 127.0.0.1:5432 is reachable only from the host; 0.0.0.0:5432 is reachable from anywhere the network allows. A database bound to 0.0.0.0 behind a permissive security group is the most common real-world exposure there is.

The three front-ends:

# ufw -- Ubuntu, simplest
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow from 10.0.0.0/8 to any port 22 proto tcp
sudo ufw enable && sudo ufw status verbose

# firewalld -- RHEL family, zone-based
sudo firewall-cmd --state
sudo firewall-cmd --list-all
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload

# nftables / iptables -- what both of the above write
sudo nft list ruleset
sudo iptables -S

Whatever the front-end, read the policies first:

-P INPUT DROP
-P FORWARD DROP
-P OUTPUT DROP
-A INPUT -p tcp --dport 22 -j ACCEPT
-A OUTPUT -o eth0 -p udp --dport 53 -j ACCEPT

Default-drop with two exceptions. Anything else times out — which from inside the application is indistinguishable from a dead remote host, and is why a firewall rule should be on the checklist whenever a connection hangs rather than refuses.

Rule ordering:

Rules are evaluated top to bottom, first match wins. A permissive rule above a restrictive one makes the restrictive one dead code. When a rule “does not work”, check what is above it before you check the rule itself.

Common mistake: Adding a firewall rule and forgetting --permanent (firewalld) or netfilter-persistent save (iptables). It works perfectly until the next reboot, and then the outage arrives with no recent change to blame.

Topic 3: SELinux and AppArmor

Ordinary permissions are discretionary — the owner decides. SELinux and AppArmor add a mandatory layer the owner cannot override: a policy states what each process may do, and the kernel enforces it even for root.

SELinux (RHEL family):

sestatus                          # enabled? enforcing or permissive?
getenforce
sudo setenforce 0                 # permissive TEMPORARILY -- for diagnosis only
ls -Z /var/www/html               # security context of files
ps -eZ | grep nginx               # context of a process

SELinux status:      enabled
Current mode:        enforcing
Loaded policy name:  targeted

Every file and process carries a context like system_u:object_r:httpd_sys_content_t:s0. Policy allows httpd_t to read httpd_sys_content_t — so a file moved into /var/www from elsewhere keeps its old label and the web server gets permission denied on a file that looks perfectly readable in ls -l.

sudo ausearch -m avc -ts recent           # recent denials
sudo grep denied /var/log/audit/audit.log | tail
sudo restorecon -Rv /var/www/html         # relabel to the policy default -- usually the fix
sudo semanage fcontext -a -t httpd_sys_content_t '/srv/web(/.*)?'
sudo setsebool -P httpd_can_network_connect on    # common boolean for reverse proxies

AppArmor (Debian/Ubuntu):

Path-based rather than label-based, and simpler to read.

sudo aa-status
sudo aa-complain /etc/apparmor.d/usr.sbin.nginx    # log but do not block
sudo aa-enforce /etc/apparmor.d/usr.sbin.nginx
sudo journalctl -k | grep -i apparmor

The diagnostic move:

When something fails only on the hardened host, set permissive briefly. If the problem disappears, mandatory access control is the cause and you now know to fix the label or the profile.

Never leave it disabled as the fix. setenforce 0 is a diagnostic step with a follow-up, not a resolution — and SELINUX=disabled in /etc/selinux/config requires a relabel to re-enable, so it is a decision that is expensive to reverse.

Topic 4: Capabilities — Root Without Root

Historically a process either had UID 0 and could do everything, or did not and could do nothing privileged. Capabilities split root’s powers into ~40 individually grantable pieces.

getcap /usr/bin/ping                         # what a binary carries
sudo setcap cap_net_bind_service=+ep /usr/local/bin/myserver
getpcaps <PID>                               # what a running process holds
capsh --print                                # this shell's capability set

This is how a web server binds port 443 without running as root — and it is strictly better than setuid, because setuid grants everything while a capability grants one thing.

In systemd:

[Service]
User=appuser
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/var/lib/myapp
ProtectHome=true
ProtectKernelTunables=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

NoNewPrivileges=true is the one to internalise: it prevents the process or any child from ever gaining privilege, which neuters setuid binaries inside the service entirely.

In containers:

docker run --cap-drop=ALL --cap-add=NET_BIND_SERVICE myimage

securityContext:
  runAsNonRoot: true
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop: ["ALL"]
    add: ["NET_BIND_SERVICE"]

Drop everything, then add back only what fails. That order is the whole method — starting from “what should I remove” never converges.

Topic 5: A Short Audit

Ten minutes, any host:

# Who can log in, and how
sudo sshd -T | grep -E 'permitrootlogin|passwordauthentication|allowusers'
awk -F: '$3>=1000 && $7 !~ /nologin|false/ {print $1, $7}' /etc/passwd
sudo awk -F: '$2 == "" {print $1}' /etc/shadow          # passwordless accounts

# What is exposed
ss -tlnp
sudo iptables -S | head -20

# What can escalate
sudo find / -xdev -perm -4000 -type f -exec ls -l {} \; 2>/dev/null
getent group sudo docker wheel                          # membership IS privilege
sudo grep -rE 'NOPASSWD|ALL=\(ALL\)' /etc/sudoers /etc/sudoers.d/

# What is enforcing
sestatus 2>/dev/null || sudo aa-status

# What changed
sudo last -x | head -20
sudo lastb | head -10                                    # FAILED logins
grep -E ' (install|upgrade) ' /var/log/dpkg.log | tail

Two of these deserve a second look. Membership of docker or wheel is equivalent to root — a “read-only” account in the docker group is not read-only. And lastb (failed logins) is the fastest evidence of whether anybody is actively trying the front door.

Try it yourself: Run the setuid audit on any host and confirm you recognise every binary. Then run sudo grep -r NOPASSWD /etc/sudoers.d/ and check each entry names a specific command rather than ALL.

Common mistake: Hardening a host without a rollback path or a second session open. Every control here can lock you out — an SSH change, a firewall default-deny, an SELinux relabel. Change one thing, verify from a new connection, then move on.