Log Analysis & Rotation Under Pressure

During an incident you are handed a file with two million lines and three minutes to find the pattern. The skill is not knowing more commands — it is knowing the five-stage pipeline that turns any log into a ranked table.

Topic 1: Where Logs Come From

That last-but-one row is the one that surprises people: an application that opens /var/log/myapp.log itself bypasses journald entirely. journalctl -u myapp shows nothing, and the conclusion “there are no logs” is wrong.

# Which files is this service actually writing to?
sudo lsof -p $(systemctl show -p MainPID --value myapp) | grep -E 'REG.*log'

Topic 2: The Five-Stage Pipeline

Almost every log question resolves into the same shape. Learn the shape and the commands fill themselves in.

  SELECT      →   EXTRACT    →   NORMALISE  →   COUNT           →   RANK
  grep/awk        awk/cut        sed/tr         sort | uniq -c      sort -rn
  narrow to       pull the       collapse       group identical     biggest
  what matters    field          variants       lines               first

A worked example — which IPs are failing to authenticate:

grep 'Failed password' /var/log/auth.log \
  | awk '{print $(NF-3)}' \
  | sort | uniq -c | sort -rn | head -10

   4821 203.0.113.44
    118 198.51.100.9
      3 192.0.2.17

Four thousand attempts from one address against a hundred from another is not the same story, and no amount of reading the file top-to-bottom would have made that obvious.

Finding the busiest minute:

awk '{print $1, $2, substr($3,1,5)}' /var/log/syslog \
  | sort | uniq -c | sort -rn | head -5

Truncating the timestamp to HH:MM buckets every line into minutes. The same trick with substr($3,1,2) buckets by hour. This is how you find the moment a burst started without any graphing tool.

Topic 3: The Tools, With the Flags That Matter

grep — select

grep -i error app.log            # case-insensitive
grep -v healthcheck app.log      # INVERT -- drop the noise
grep -c 'ERROR' app.log          # count matches, do not print them
grep -n 'ERROR' app.log          # with line numbers
grep -A5 -B5 'panic' app.log     # 5 lines of context After and Before
grep -E 'ERROR|FATAL' app.log    # extended regex alternation
grep -o 'user=[a-z]*' app.log    # print ONLY the matching part
grep -r 'api_key' /etc/          # recursive
grep -F 'literal.string' app.log # fixed string -- no regex, much faster

-v earns its place first. Most logs are 95% routine; dropping health checks and successful requests often makes the problem visible with no further work.

awk — extract and compute

awk splits each line on whitespace and numbers the fields from $1. $0 is the whole line, NF is the number of fields, so $NF is the last field and $(NF-1) the one before it.

awk '{print $1, $9}' access.log                 # client IP and status code
awk '$9 == 500' access.log                      # only 500s
awk '$9 >= 500 {c++} END {print c+0}' access.log  # count them
awk -F: '{print $1}' /etc/passwd                # -F sets the separator
awk '{sum += $10} END {print sum/NR}' access.log  # mean response size
awk '$4 > "[05/Aug/2026:14:20" && $4 < "[05/Aug/2026:14:30"' access.log  # time window

The END block runs once after the last line — that is where totals and averages go. NR is the record number, so at END it holds the line count.

sed — normalise

sed 's/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/IP/g' app.log   # mask IPs
sed 's/[0-9]\+/N/g' app.log      # collapse all numbers -- groups templated messages
sed -n '100,200p' app.log        # print a line range
sed '/^$/d' app.log              # delete blank lines

That second one is the trick worth remembering. Replacing every number with N turns user 4471 timed out after 30s and user 9902 timed out after 45s into the same string, so uniq -c can count message types rather than unique messages.

The rest of the pipeline

cut -d' ' -f1,7 access.log       # fixed delimiter, faster than awk for simple splits
sort -k2 -rn                     # sort on field 2, reverse, numeric
uniq -c                          # count adjacent duplicates -- REQUIRES sorted input
tr -s ' '                        # squeeze repeated spaces
wc -l                            # count lines
tail -f app.log | grep -i error  # follow and filter live

Common mistake: uniq -c without sorting first. uniq only collapses adjacent duplicates, so unsorted input produces a count of 1 for nearly everything and the result looks plausible while being wrong.

Topic 4: Searching the Journal Instead

If the logs are in journald, filter there rather than piping text — it is indexed, so the filters are far faster on a busy host.

journalctl -u nginx --since '14:00' --until '14:30'
journalctl -p err -b                       # errors, this boot
journalctl _PID=4471                       # by PID
journalctl _UID=1000                       # by user
journalctl -g 'timed out'                  # grep, with the index behind it
journalctl -u nginx -o json | jq -r '.MESSAGE'   # structured output
journalctl -f -u nginx                     # follow
journalctl --since '1 hour ago' -o short-precise # microsecond timestamps

-o json is the door to structured analysis: every message carries fields like _PID, _SYSTEMD_UNIT, _HOSTNAME, and PRIORITY that plain text has thrown away.

# Count messages per unit in the last hour
journalctl --since '1 hour ago' -o json \
  | jq -r '._SYSTEMD_UNIT // "kernel"' \
  | sort | uniq -c | sort -rn | head

Topic 5: Rotation — Keeping Logs From Filling the Disk

An unrotated log grows until the filesystem is full, at which point everything on the host fails at once. Rotation is what stops that, and it is misconfigured constantly.

logrotate, for file-based logs:

# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    daily
    rotate 14
    size 100M
    compress
    delaycompress
    missingok
    notifempty
    create 0640 myapp myapp
    sharedscripts
    postrotate
        systemctl reload myapp > /dev/null 2>&1 || true
    endscript
}

sudo logrotate -d /etc/logrotate.d/myapp    # -d = debug, changes nothing
sudo logrotate -f /etc/logrotate.d/myapp    # force a rotation now
cat /var/lib/logrotate/status               # when each file last rotated

The two failure modes:

1. Rotation without a reopen. logrotate renames the file, but the application still holds a descriptor to the old inode and keeps writing into it. The new file stays at zero bytes and disk usage never drops — exactly the deleted-but-open situation from the storage lesson. The fix is postrotate signalling the app, or copytruncate if it cannot reopen.
2. copytruncate losing lines. Between the copy and the truncate there is a window in which writes are lost. It is the fallback, not the default.

journald retention:

# /etc/systemd/journald.conf
Storage=persistent
SystemMaxUse=2G
MaxRetentionSec=1month

journalctl --disk-usage
sudo journalctl --vacuum-size=500M      # free space NOW
sudo journalctl --vacuum-time=7d

journalctl --vacuum-size is a genuinely useful emergency command: it reclaims space immediately without touching anything else on a full root filesystem.

Try it yourself: Run sudo logrotate -d /etc/logrotate.conf 2>&1 | head -40. It prints exactly what it would do for every configured log, which is the fastest way to find a file nobody is rotating.

Common mistake: Assuming a log is rotated because a config file exists for it. Check /var/lib/logrotate/status for the last rotation date — a config with a typo’d path is silently doing nothing, and you find out when the disk fills.