WorksOnMyCluster

Log Analysis, Monitoring & Alerting

Turn raw logs into a ranked answer, decide what genuinely deserves a page, and deliver alerts that carry enough context to act on.

advanced 17 min lesson hands-on task included

Monitoring scripts fail in one of two directions: they say nothing when something is wrong, or they say so much that everyone mutes them. Both are the same defect — an alert that does not carry a decision.

Topic 1: Extracting Signal From a Log

The pipeline from the awk lesson, applied to real questions:

# Error rate over the last hour, by minute
awk -v since="$(date -d '1 hour ago' '+%H:%M')" '
    $3 >= since && /ERROR/ { split($3, t, ":"); count[t[1]":"t[2]]++ }
    END { for (m in count) print m, count[m] }
' /var/log/app.log | sort

# Top endpoints returning 5xx
awk '$9 ~ /^5/ { count[$7]++ } END { for (p in count) print count[p], p }' access.log |
    sort -rn | head -10

# Slowest requests (nginx $request_time in the last field)
awk '{ print $NF, $7 }' access.log | sort -rn | head -20

# Collapse IDs so paths group into endpoints
awk '{ p=$7; gsub(/[0-9]+/, "N", p); count[p]++ } END { for (k in count) print count[k], k }' access.log |
    sort -rn | head

Comparing against a baseline:

A count alone is meaningless. “412 errors” matters only against what is normal.

current=$(grep -c ERROR /var/log/app.log)
baseline_file=/var/lib/monitoring/error_baseline

if [[ -r $baseline_file ]]; then
    baseline=$(< "$baseline_file")
    # Alert if more than double the baseline AND above an absolute floor
    if (( current > baseline * 2 && current > 50 )); then
        alert "error count ${current} vs baseline ${baseline}"
    fi
fi
printf '%s\n' "$current" > "$baseline_file"

The absolute floor is what stops 1 error becoming 3 and paging someone at 3am for a 200% increase.

Finding the first occurrence, not the last:

# When did this error start?
grep -m1 'connection refused' /var/log/app.log

# What happened in the 30 seconds before?
grep -B50 -m1 'connection refused' /var/log/app.log

Tailing shows consequences. -m1 with -B shows causes.

Topic 2: A Health Check Worth Running

#!/usr/bin/env bash
set -Eeuo pipefail

readonly HOSTNAME=$(hostname -f)
readonly DISK_THRESHOLD=${DISK_THRESHOLD:-85}
readonly MEM_THRESHOLD=${MEM_THRESHOLD:-90}

declare -a problems=()

check_disk() {
    while read -r pct mount; do
        (( pct >= DISK_THRESHOLD )) || continue
        local top
        top=$(du -xh --max-depth=2 "$mount" 2>/dev/null | sort -rh | head -3 | tr '\n' '; ')
        problems+=("DISK ${mount} at ${pct}% (threshold ${DISK_THRESHOLD}%) — largest: ${top}")
    done < <(df -hP -x tmpfs -x devtmpfs | awk 'NR>1 {gsub(/%/,"",$5); print $5, $6}')
}

check_memory() {
    local total avail pct
    read -r total avail < <(free -m | awk '/^Mem:/ {print $2, $7}')
    pct=$(( (total - avail) * 100 / total ))
    (( pct >= MEM_THRESHOLD )) || return 0
    local top
    top=$(ps -eo comm,rss --sort=-rss | awk 'NR>1 && NR<=4 {printf "%s(%dMB) ", $1, $2/1024}')
    problems+=("MEMORY at ${pct}% available-based (threshold ${MEM_THRESHOLD}%) — top: ${top}")
}

check_services() {
    local svc
    for svc in nginx postgresql; do
        systemctl is-active --quiet "$svc" && continue
        problems+=("SERVICE ${svc} is ${_:-inactive} — last logs: $(journalctl -u "$svc" -n3 --no-pager -o cat | tr '\n' '; ')")
    done
}

check_disk
check_memory
check_services

(( ${#problems[@]} == 0 )) && exit 0

printf '%s\n' "${problems[@]}"
exit 1

The design choice that matters: every problem string carries its own evidence. “Disk at 91%” makes someone log in. “Disk at 91% — largest: 42G /var/lib/docker” tells them what to do before they log in.

Note free -m field 7 (available) rather than used — the memory lesson’s point, applied.

Topic 3: Alert Delivery

Slack:

notify_slack() {
    local severity=$1 message=$2
    local colour
    case $severity in
        critical) colour="#dc2626" ;;
        warning)  colour="#f59e0b" ;;
        *)        colour="#10b981" ;;
    esac

    local payload
    payload=$(jq -n \
        --arg colour "$colour" \
        --arg title "${severity^^} on $(hostname -f)" \
        --arg text "$message" \
        '{attachments: [{color: $colour, title: $title, text: $text, ts: now|floor}]}')

    curl -fsS --max-time 10 \
        -X POST -H 'Content-Type: application/json' \
        -d "$payload" \
        "${SLACK_WEBHOOK_URL:?SLACK_WEBHOOK_URL is required}" >/dev/null
}

Email:

printf '%s\n' "${problems[@]}" |
    mail -s "[${severity}] $(hostname -s): ${#problems[@]} issues" ops@example.com

mail needs a configured MTA, which is often absent on a container host — check before relying on it.

PagerDuty / Alertmanager:

curl -fsS --max-time 10 -X POST \
    -H 'Content-Type: application/json' \
    -d "$(jq -n --arg key "$PD_ROUTING_KEY" --arg summary "$message" \
        '{routing_key: $key, event_action: "trigger",
          payload: {summary: $summary, severity: "error", source: "'"$(hostname -f)"'"}}')" \
    https://events.pagerduty.com/v2/enqueue

Never let the notifier take down the script:

notify_slack critical "$message" || echo "WARN: slack delivery failed" >&2

Under set -e, a failed webhook aborts the run — so the check that found a real problem exits before doing anything about it. Alerting is best-effort by definition.

Topic 4: Rate Limiting and Deduplication

An alert that repeats every five minutes gets muted, and then the real one is missed too.

readonly STATE_DIR=/var/lib/monitoring
readonly ALERT_WINDOW=3600          # one hour

should_alert() {
    local key=$1
    local marker="${STATE_DIR}/alert-${key//\//_}"
    mkdir -p "$STATE_DIR"

    if [[ -f $marker ]]; then
        local age=$(( $(date +%s) - $(stat -c %Y "$marker") ))
        (( age < ALERT_WINDOW )) && return 1        # too soon
    fi
    touch "$marker"
    return 0
}

if should_alert "disk-/var"; then
    notify_slack critical "disk /var at ${pct}%"
fi

${key//\//_} replaces slashes so a mount point makes a valid filename — parameter expansion from the quoting lesson.

Recovery notifications:

An alert with no “resolved” message leaves people checking manually.

if (( pct >= DISK_THRESHOLD )); then
    should_alert "disk-${mount}" && notify_slack critical "disk ${mount} at ${pct}%"
elif [[ -f "${STATE_DIR}/alert-${mount//\//_}" ]]; then
    rm -f "${STATE_DIR}/alert-${mount//\//_}"
    notify_slack info "RESOLVED: disk ${mount} back to ${pct}%"
fi

Flap damping:

Require N consecutive failures before alerting, so a single blip stays quiet:

readonly FAIL_COUNT_FILE="${STATE_DIR}/fails-${check_name}"
if check_failed; then
    count=$(( $(cat "$FAIL_COUNT_FILE" 2>/dev/null || echo 0) + 1 ))
    echo "$count" > "$FAIL_COUNT_FILE"
    (( count >= 3 )) && notify_slack critical "${check_name} failed ${count}x"
else
    rm -f "$FAIL_COUNT_FILE"
fi

Topic 5: Log Rotation for Your Own Scripts

A monitoring script that logs forever eventually fills the disk it is monitoring.

# /etc/logrotate.d/myapp-monitoring
/var/log/monitoring/*.log {
    daily
    rotate 14
    size 50M
    compress
    delaycompress
    missingok
    notifempty
    create 0640 monitoring monitoring
}

sudo logrotate -d /etc/logrotate.d/myapp-monitoring    # dry run
cat /var/lib/logrotate/status                          # when it last ran

Self-managed rotation, when logrotate is not available:

readonly LOG=/var/log/monitoring/check.log
readonly MAX_BYTES=$(( 10 * 1024 * 1024 ))

rotate_if_needed() {
    [[ -f $LOG ]] || return 0
    local size
    size=$(stat -c %s "$LOG")
    (( size < MAX_BYTES )) && return 0
    mv "$LOG" "${LOG}.1"
    gzip -f "${LOG}.1"
    : > "$LOG"
}

Try it yourself: Write a disk check that alerts with the top three space consumers included, add the one-hour rate limiter, and run it twice in a row. The second run should be silent.

Common mistake: Alerting on a threshold rather than a trend. “Disk at 85%” fires on a host that has been at 84% for two years. “Disk will reach 100% in under 48 hours at the current rate” is the alert that gives you a working day to respond — and it needs only two samples and a subtraction.