Project 2: Service Health Monitor & Alerter

Anyone can write a script that checks disk space. The difference between that and a monitor people keep enabled is entirely in the alerting discipline: not alerting on a blip, not alerting twice, and saying when it is over.

The Requirements

Step 1: Core Engine

#!/usr/bin/env bash
#
# monitor.sh — run health checks, alert with damping and deduplication.
#
set -Eeuo pipefail

readonly SCRIPT_NAME=${0##*/}
readonly SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)
readonly HOSTNAME_FQDN=$(hostname -f 2>/dev/null || hostname)

readonly STATE_DIR="${MONITOR_STATE_DIR:-/var/lib/monitor}"
readonly FAIL_THRESHOLD="${MONITOR_FAIL_THRESHOLD:-3}"   # consecutive failures before alerting
readonly COOLDOWN="${MONITOR_COOLDOWN:-3600}"            # seconds between repeat alerts
readonly DRY_RUN="${MONITOR_DRY_RUN:-0}"

mkdir -p "$STATE_DIR"

log()  { printf '%s [%-5s] %s\n' "$(date -Iseconds)" "$1" "${*:2}" >&2; }
info() { log INFO "$@"; }
warn() { log WARN "$@"; }

# Sanitise a check name into something safe for a filename
_key() { printf '%s' "${1//[^a-zA-Z0-9_-]/_}"; }

The _key helper matters: check names like disk:/var/lib become filenames, and an unsanitised / creates a directory traversal rather than a state file.

Step 2: State — Damping and Deduplication

# Track consecutive failures; return 0 when the threshold is newly reached
record_failure() {
    local name; name=$(_key "$1")
    local f="${STATE_DIR}/fail-${name}"
    local count
    count=$(( $(cat "$f" 2>/dev/null || echo 0) + 1 ))
    printf '%s\n' "$count" > "$f"
    (( count >= FAIL_THRESHOLD ))
}

# Clear the counter; return 0 if this is a RECOVERY from an alerted state
record_success() {
    local name; name=$(_key "$1")
    local f="${STATE_DIR}/fail-${name}"
    local a="${STATE_DIR}/alerted-${name}"
    rm -f "$f"
    if [[ -f $a ]]; then
        rm -f "$a"
        return 0        # was alerting, now healthy → recovery
    fi
    return 1
}

# Cooldown: return 0 only if we are allowed to alert now
should_alert() {
    local name; name=$(_key "$1")
    local a="${STATE_DIR}/alerted-${name}"
    if [[ -f $a ]]; then
        local age=$(( $(date +%s) - $(stat -c %Y "$a" 2>/dev/null || stat -f %m "$a") ))
        (( age < COOLDOWN )) && return 1
    fi
    touch "$a"
    return 0
}

Three files per check, and the whole alerting policy falls out of them: fail-* counts consecutive failures, alerted-* exists while an alert is outstanding and its mtime is the cooldown clock.

Step 3: Notification

notify() {
    local severity=$1 title=$2 body=$3

    if (( DRY_RUN )); then
        info "DRY-RUN alert [${severity}] ${title}: ${body}"
        return 0
    fi

    local colour
    case $severity in
        critical) colour="#dc2626" ;;
        warning)  colour="#f59e0b" ;;
        resolved) colour="#10b981" ;;
        *)        colour="#6b7280" ;;
    esac

    if [[ -n ${SLACK_WEBHOOK_URL:-} ]]; then
        local payload
        payload=$(jq -n \
            --arg c "$colour" \
            --arg t "[${severity^^}] ${HOSTNAME_FQDN}: ${title}" \
            --arg b "$body" \
            '{attachments:[{color:$c, title:$t, text:$b, ts:(now|floor)}]}')

        curl -fsS --max-time 10 -X POST \
             -H 'Content-Type: application/json' \
             -d "$payload" "$SLACK_WEBHOOK_URL" >/dev/null \
            || warn "slack delivery failed"
    fi

    if [[ -n ${ALERT_EMAIL:-} ]] && command -v mail >/dev/null; then
        printf '%s\n' "$body" | mail -s "[${severity}] ${HOSTNAME_FQDN}: ${title}" "$ALERT_EMAIL" \
            || warn "email delivery failed"
    fi
}

Every delivery path ends in || warn. Under set -e an unguarded curl failure would abort the whole run — so the check that found a genuine problem would exit before reporting anything else.

Step 4: Checks as Plugins

Each check is a function returning 0 for healthy, non-zero for failing, and printing its evidence to stdout.

check_disk() {
    local threshold="${DISK_THRESHOLD:-85}"
    local failed=0
    while read -r pct mount; do
        if (( pct >= threshold )); then
            local top
            top=$(du -xh --max-depth=2 "$mount" 2>/dev/null | sort -rh | head -3 |
                  awk '{printf "%s (%s); ", $2, $1}')
            echo "${mount} at ${pct}% [threshold ${threshold}%] — largest: ${top}"
            failed=1
        fi
    done < <(df -hP -x tmpfs -x devtmpfs -x overlay |
             awk 'NR>1 {gsub(/%/,"",$5); print $5, $6}')
    return "$failed"
}

check_memory() {
    local threshold="${MEM_THRESHOLD:-90}"
    local total avail used_pct
    read -r total avail < <(free -m | awk '/^Mem:/ {print $2, $7}')
    used_pct=$(( (total - avail) * 100 / total ))
    (( used_pct < threshold )) && return 0

    local top
    top=$(ps -eo comm,rss --sort=-rss | awk 'NR>1 && NR<=4 {printf "%s(%dMB) ", $1, $2/1024}')
    echo "memory ${used_pct}% used [threshold ${threshold}%] — top: ${top}"
    return 1
}

check_service() {
    local svc=$1
    systemctl is-active --quiet "$svc" && return 0
    local state logs
    state=$(systemctl is-active "$svc" 2>&1 || true)
    logs=$(journalctl -u "$svc" -n 3 --no-pager -o cat 2>/dev/null | tr '\n' '; ')
    echo "service ${svc} is ${state} — recent: ${logs}"
    return 1
}

check_http() {
    local url=$1 expect="${2:-200}"
    local code time_total
    read -r code time_total < <(
        curl -sS -o /dev/null -w '%{http_code} %{time_total}' \
             --max-time 10 "$url" 2>/dev/null || echo "000 0"
    )
    [[ $code == "$expect" ]] && return 0
    echo "${url} returned ${code} (expected ${expect}) after ${time_total}s"
    return 1
}

check_cert_expiry() {
    local host=$1 days="${CERT_WARN_DAYS:-14}"
    local end_date end_epoch remaining
    end_date=$(echo | openssl s_client -servername "$host" -connect "${host}:443" 2>/dev/null |
               openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2) || return 0
    [[ -n $end_date ]] || return 0
    end_epoch=$(date -d "$end_date" +%s 2>/dev/null || date -j -f '%b %d %T %Y %Z' "$end_date" +%s)
    remaining=$(( (end_epoch - $(date +%s)) / 86400 ))
    (( remaining > days )) && return 0
    echo "TLS certificate for ${host} expires in ${remaining} days"
    return 1
}

Every message names the resource, the measurement, the threshold, and the top contributors. That is the difference between an alert someone acts on and one they acknowledge and forget.

Step 5: The Runner

run_check() {
    local name=$1; shift
    local output rc=0

    output=$("$@" 2>&1) || rc=$?

    if (( rc == 0 )); then
        if record_success "$name"; then
            notify resolved "${name} recovered" "${name} is healthy again."
            info "RECOVERED ${name}"
        else
            info "OK ${name}"
        fi
        return 0
    fi

    if record_failure "$name"; then
        if should_alert "$name"; then
            notify critical "$name" "$output"
            warn "ALERTED ${name}: ${output}"
        else
            info "SUPPRESSED ${name} (cooldown active)"
        fi
    else
        local n; n=$(cat "${STATE_DIR}/fail-$(_key "$name")")
        info "FAILING ${name} (${n}/${FAIL_THRESHOLD}) — damping"
    fi
    return 1
}

main() {
    exec 200>"/var/lock/${SCRIPT_NAME%.sh}.lock"
    flock -n 200 || { info "already running; skipping"; exit 0; }

    local failures=0

    run_check "disk"          check_disk                                  || (( failures++ )) || true
    run_check "memory"        check_memory                                || (( failures++ )) || true
    run_check "svc-nginx"     check_service nginx                         || (( failures++ )) || true
    run_check "svc-postgres"  check_service postgresql                    || (( failures++ )) || true
    run_check "http-app"      check_http "http://localhost:8080/health"   || (( failures++ )) || true
    run_check "cert-www"      check_cert_expiry "www.example.com"         || (( failures++ )) || true

    # Dead-man switch: ping only on a completed run
    if [[ -n ${HEALTHCHECK_URL:-} ]]; then
        curl -fsS --max-time 10 "$HEALTHCHECK_URL" >/dev/null || warn "heartbeat failed"
    fi

    info "run complete: ${failures} checks failing"
    return 0        # ALWAYS 0 — cron mail is not the alerting channel
}

if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
    main "$@"
fi

Two design decisions worth defending:

return 0 regardless of failures. Alerts go through notify. If the script also exits non-zero, cron mails you a second copy of everything, and people start filtering cron mail — including the message telling them the monitor itself broke.

The heartbeat only fires at the end. A dead-man switch that pings at the start cannot detect a run that hangs halfway.

Step 6: Deploy It

# /etc/systemd/system/monitor.service
[Unit]
Description=Health monitor
After=network-online.target

[Service]
Type=oneshot
EnvironmentFile=/etc/monitor/env        # secrets live here, mode 0600
ExecStart=/usr/local/bin/monitor.sh
TimeoutStartSec=120

# /etc/systemd/system/monitor.timer
[Unit]
Description=Run health monitor every 5 minutes

[Timer]
OnBootSec=2min
OnUnitActiveSec=5min
RandomizedDelaySec=30

[Install]
WantedBy=timers.target

sudo install -m 0600 /dev/stdin /etc/monitor/env <<'EOF'
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...
HEALTHCHECK_URL=https://hc-ping.com/...
DISK_THRESHOLD=85
EOF
sudo systemctl enable --now monitor.timer
journalctl -u monitor.service -f

OnUnitActiveSec rather than OnCalendar means the next run is scheduled 5 minutes after the last one finished, which cannot overlap. EnvironmentFile with mode 0600 keeps the webhook out of the unit file and out of ps.

Verifying Your Work

# 1. Damping — one failure must be silent
MONITOR_DRY_RUN=1 DISK_THRESHOLD=0 ./monitor.sh    # "FAILING disk (1/3)"
MONITOR_DRY_RUN=1 DISK_THRESHOLD=0 ./monitor.sh    # "FAILING disk (2/3)"
MONITOR_DRY_RUN=1 DISK_THRESHOLD=0 ./monitor.sh    # "ALERTED disk"

# 2. Deduplication — the 4th run must be suppressed
MONITOR_DRY_RUN=1 DISK_THRESHOLD=0 ./monitor.sh    # "SUPPRESSED disk"

# 3. Recovery
MONITOR_DRY_RUN=1 ./monitor.sh                     # "RECOVERED disk"

# 4. Locking
./monitor.sh & ./monitor.sh; wait                  # second: "already running"

# 5. State inspection
ls -la /var/lib/monitor/

Extensions worth building: emit Prometheus textfile-collector metrics alongside the alerts so you get graphs for free; add a --check NAME flag to run one check; add severity tiers so disk-at-95% pages while disk-at-85% only posts to Slack; write the check functions into separate files under checks.d/ and source them in a loop, so adding a check is adding a file.

The lesson to take away: the checks are the easy half. Everything that makes this usable — damping, dedup, recovery, exit-zero — exists to protect the humans receiving the alerts. An alerting channel people mute is worse than no monitoring at all, because it creates the belief that something is watching.