awk & sed in Depth

awk and sed are separate programming languages that happen to live in your shell. Most “I need Python for this” log tasks are four lines of awk, and knowing which of the two fits saves a great deal of pipeline plumbing.

Topic 1: awk’s Model — Records and Fields

awk reads input one record (line) at a time, splits it into fields on whitespace, and runs your program against each. That model is the whole language.

awk 'pattern { action }' file

Both parts are optional: a bare pattern prints matching lines, a bare action runs on every line.

awk '{print $1, $NF}' access.log          # first and last field
awk -F: '{print $1}' /etc/passwd          # -F sets FS
awk -F'\t' '{print $3}' data.tsv
awk 'NR > 1' data.csv                     # skip the header
awk 'NF' file                             # print non-empty lines (NF is truthy)
awk 'END {print NR}' file                 # line count, like wc -l

$NF and $(NF-1) are worth internalising — they let you address fields from the end, which is how you handle logs where the leading columns vary.

Topic 2: The Three-Phase Lifecycle

awk '
BEGIN { print "starting"; FS=":" }     # once, BEFORE any input
      { count++ }                       # once per record
END   { print "saw", count, "lines" }   # once, AFTER all input
'

BEGIN runs before the first line is read — it is where you set FS, print headers, and initialise. END runs after the last, and it is where aggregation gets reported. An awk program with only an END block reads the whole file and prints one summary.

Aggregation in one pass:

This is awk’s real value. Where a shell loop forks per line, awk does it in a single process:

# Top 5 client IPs by request count
awk '{ count[$1]++ } END { for (ip in count) print count[ip], ip }' access.log |
  sort -rn | head -5

# Total and average bytes served
awk '{ bytes += $10 } END { printf "total %.2f MB, avg %d B\n", bytes/1048576, bytes/NR }' access.log

# Requests per status code
awk '{ code[$9]++ } END { for (c in code) printf "%-6s %d\n", c, code[c] }' access.log

# Sum by group -- bytes per IP
awk '{ sum[$1] += $10 } END { for (ip in sum) print ip, sum[ip] }' access.log

awk’s arrays are associative by default and spring into existence on first use, which is why count[$1]++ needs no declaration.

Filtering by field:

awk '$9 >= 500'                              # numeric comparison
awk '$9 ~ /^5/'                              # regex match on a field
awk '$9 !~ /^2/'                             # negated
awk '$7 == "/api/checkout" && $9 == 500'     # combined conditions
awk 'NR >= 100 && NR <= 200'                 # a line range
awk '/START/,/END/'                          # a RANGE pattern, inclusive

Time windows in logs:

awk '$4 >= "[05/Aug/2026:14:20" && $4 <= "[05/Aug/2026:14:30"' access.log

String comparison works because the timestamp format sorts lexically. That is the fastest way to slice a log to an incident window without a date parser.

Topic 3: awk as a Real Language

awk '
BEGIN { OFS="\t"; print "IP", "HITS", "PCT" }
{
    hits[$1]++
    total++
}
END {
    for (ip in hits) {
        pct = (hits[ip] / total) * 100
        if (pct > 1.0)
            printf "%s\t%d\t%.1f%%\n", ip, hits[ip], pct
    }
}
' access.log | sort -k2 -rn

It has if/else, for, while, functions, and a useful string library:

# Extract a query parameter from a URL field
awk '{ split($7, parts, "?"); print parts[1] }' access.log

# Normalise then count -- collapse numeric IDs so paths group together
awk '{ p=$7; gsub(/[0-9]+/, "N", p); count[p]++ } END { for (k in count) print count[k], k }' access.log |
  sort -rn | head

That second one is the templating trick: /api/user/4471/orders and /api/user/9902/orders both become /api/user/N/orders, so the count reflects endpoints rather than individual requests.

Passing shell variables in:

threshold=500
awk -v limit="$threshold" '$10 > limit' access.log        # -v, the correct way

Never interpolate the shell variable into the awk program text — quoting breaks and it is an injection risk if the value is not yours. -v is what it exists for.

Topic 4: sed — the Stream Editor

sed applies editing commands to each line. In practice 90% of use is s, the substitution.

s/pattern/replacement/flags

sed 's/foo/bar/'          file    # FIRST occurrence on each line
sed 's/foo/bar/g'         file    # every occurrence
sed 's/foo/bar/gi'        file    # ...case-insensitively
sed 's/foo/bar/2'         file    # only the 2nd occurrence
sed -i 's/foo/bar/g'      file    # edit the file IN PLACE
sed -i.bak 's/foo/bar/g'  file    # in place, keeping a .bak

-E for sane regex:

Without -E, sed uses POSIX basic regular expressions where +, ?, |, ( and { must all be backslash-escaped. With -E they behave as you expect:

sed  's/[0-9]\{1,3\}\.[0-9]\{1,3\}/X/g'     # BRE -- unreadable
sed -E 's/[0-9]{1,3}\.[0-9]{1,3}/X/g'       # ERE -- same thing

Use -E always. It has been in GNU sed forever and POSIX-standardised since 2024.

Any delimiter works:

sed 's|/var/log|/mnt/log|g' file           # | avoids escaping every /
sed 's#old/path#new/path#g' file

Capture groups:

echo "2026-08-05" | sed -E 's/([0-9]{4})-([0-9]{2})-([0-9]{2})/\3\/\2\/\1/'
# 05/08/2026

\1–\9 in the replacement refer to parenthesised groups. & inserts the entire match:

sed -E 's/[0-9]+/[&]/g' file        # wrap every number in brackets

Beyond substitution:

sed -n '5p'            file    # print line 5 (-n suppresses default printing)
sed -n '10,20p'        file    # a range
sed -n '/START/,/END/p' file   # between two patterns
sed '5d'               file    # delete line 5
sed '/^#/d'            file    # delete comments
sed '/^$/d'            file    # delete blank lines
sed '$d'               file    # delete the last line
sed '2i\new line'      file    # insert before line 2
sed '2a\new line'      file    # append after line 2
sed -n '$='            file    # print the line count

The in-place portability trap:

sed -i    's/a/b/' f     # GNU (Linux)
sed -i '' 's/a/b/' f     # BSD (macOS) -- REQUIRES an explicit empty suffix

sed -i is not portable between them. A script that works on your Mac corrupts files on Linux and vice versa. In portable scripts, write to a temp file and move it:

tmp=$(mktemp) && sed 's/a/b/' f > "$tmp" && mv "$tmp" f

Topic 5: Choosing Between Them

Rule of thumb: if you are addressing columns, it is awk. If you are rewriting text, it is sed. If you find yourself piping sed into awk into sed, the whole thing is usually one awk program.

# Three processes
grep ERROR app.log | sed 's/.*user=//' | awk '{print $1}' | sort | uniq -c

# One
awk '/ERROR/ { sub(/.*user=/, ""); count[$1]++ } END { for (u in count) print count[u], u }' app.log |
  sort -rn

Try it yourself: Take a pipeline you already use with three or more stages and rewrite it as a single awk program. Time both with time on a large file.

Common mistake: Using awk '{print $2}' on output whose column positions vary — ls -l, ps aux with long commands, or any log with optional fields. Anchor on a delimiter with -F, or match a pattern with match()/gsub() instead of trusting position.