The Text Processing Toolkit

Analyzing telemetry streams, reporting metrics, and automating configurations require a solid grasp of stream filters and editing tools.

Topic 15: Text-Processing Tools: grep, cut, sort, uniq

These are Linux’s built-in “find and reshape text” tools — the backbone of log analysis and monitoring scripts.

• grep searches for patterns. It compiled regex lines using highly optimized search algorithms.
• cut slices out vertical fields: cut -d',' -f2 file.csv cuts the second field using a comma separator.
• sort orders lines. Use -n for numerical sorting (which parses numeric values instead of ASCII sorting) and -k to target specific columns.
• uniq removes consecutive duplicate lines. Always combine with sort first because uniq only detects duplicates that are adjacent to each other.

Topic 16: awk Basics

awk is not just a tool — it is a complete, Turing-complete pattern scanning and processing language. It operates by reading input streams line-by-line, splitting lines into fields based on the Field Separator (FS), and evaluating patterns.

The Awk Lifecycle (BEGIN, Pattern-Action, END):

• BEGIN { ... } — Executes once before any input file lines are read. Useful for initializing headers or default variable keys.
• pattern { action } — Evaluates and executes once for each line in the input file. If pattern is omitted, the action executes for all lines.
• END { ... } — Executes once after all input file lines are exhausted. Useful for reporting aggregates, averages, or final counts.

# Calculate average response time from Nginx log
# Column $10 is response time
awk '
  BEGIN { sum=0; count=0 }
  $10 > 0 { sum+=$10; count++ }
  END { if (count > 0) print "Average Response Time: " sum/count "s" }
' access.log

Built-in Variables:

• NR — Number of Records (current line index, 1-indexed).
• NF — Number of Fields in the current line.
• FS — Input Field Separator (default is whitespace).
• OFS — Output Field Separator (default is space).

Topic 17: sed Basics

sed (“stream editor”) is a non-interactive text editor that reads streams line-by-line, performs substitutions, and outputs the result.

The Substitution Command syntax:

sed 's/regex/replacement/flags' file

• s — Substitute.
• g — Global flag (substitute all matches in the line, not just the first).
• I — Case-insensitive matching flag.
• -i — In-place edit flag. Edits the file on disk directly. Under the hood, sed creates a temporary file in the directory, writes the modified stream, and renames it over the original file once complete.

Common mistake: Using -i before you are confident in your regex pattern. Always run without -i first to print results to stdout and verify substitutions are correct.

Topic 18: Pipes & Redirection

These manage the standard stream file descriptor tables of spawned system processes.

File Descriptor Redirections:

Every process has a file descriptor table. Redirections change where these descriptors point:

• > (redirect stdout) — Closes file descriptor 1 and opens the target file in write mode.
• >> (append stdout) — Closes file descriptor 1 and opens the target file in append mode.
• 2> (redirect stderr) — Redirects file descriptor 2.
• 2>&1 (merge streams) — Directs file descriptor 2 to write to the same destination as file descriptor 1.
• &> — Shorthand for redirecting both standard output and standard error.

# Redirect all outputs and errors to a log file silently
./deploy.sh > system.log 2>&1

Topic 19: Command Substitution

$(command) runs a command and drops its output directly into a variable or another command.

Performance Cost (Fork-Exec):

When the shell evaluates $(command), it must fork a child process, exec the target binary inside the child shell, pipe the standard output back to the parent shell, block execution until the child terminates, and read the buffered characters into memory.

• Optimization: Minimize command substitutions inside tight loops (like while reads with thousands of iterations) to avoid high process scheduling overhead.

Topic 4: find — Selecting Files, Not Text

grep filters lines; find filters files. It is the front of most real pipelines.

find /var/log -name '*.log'                 # by name (quote it — the SHELL must not expand it)
find /var/log -iname '*.LOG'                # case-insensitive
find . -type f                              # files only; -type d for directories, -l for symlinks
find . -maxdepth 2 -mindepth 1              # bound the depth
find /var -xdev -size +100M                 # -xdev stays on one filesystem
find . -mtime -1                            # modified in the last 24h
find . -mmin -60                            # ...last 60 minutes
find . -newer reference.txt                 # newer than a reference file
find . -user deploy -perm -u+w
find . -name '*.tmp' -delete                # built-in delete — no xargs needed

Acting on results:

find . -name '*.log' -exec gzip {} +        # BATCHES args — one gzip for many files
find . -name '*.log' -exec gzip {} \;       # one gzip PER file — much slower
find . -name '*.log' -print0 | xargs -0 -P4 gzip   # parallel

+ versus \; is the difference between one process and ten thousand. Use + unless the command genuinely takes one file at a time.

-print0 paired with xargs -0 splits on NUL, which is the only byte a filename cannot contain — the safe pairing for names with spaces or newlines.

Combining tests:

find . \( -name '*.log' -o -name '*.txt' \) -mtime +30      # OR, escaped parens
find . -name '*.log' ! -name 'debug*'                      # negation
find . -type f -size +1M -mtime -7 -exec ls -lh {} +       # implicit AND

Topic 5: Composing a Pipeline

The filters combine into a small number of recurring shapes.

# Frequency table — the single most useful pipeline in operations
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10

# Unique values only
cut -d, -f3 data.csv | sort -u

# Lines in A but not in B
comm -23 <(sort a.txt) <(sort b.txt)

# Count matches per file
grep -c ERROR *.log

# Two columns, tab-separated, ready for `read`
awk -F: '{print $1 "\t" $7}' /etc/passwd

# Deduplicate while preserving original order
awk '!seen[$0]++' input.txt

That last one is worth memorising: seen[$0]++ is 0 (falsy) the first time a line appears and non-zero after, so !seen[$0]++ prints only first occurrences — with no sort, and in the original order.

sort flags that matter:

sort -n            # numeric — otherwise "10" sorts before "9"
sort -rn           # numeric, descending
sort -k2 -t,       # by field 2, comma-delimited
sort -u            # unique (equivalent to sort | uniq, one process)
sort -h            # human-readable sizes: 2K < 1M < 3G
sort -V            # version numbers: 1.9 < 1.10

sort -h is what makes du -h | sort -rh work, and -V is the only correct way to order release tags.

uniq requires sorted input. It only collapses adjacent duplicates, so unsorted input silently produces counts of 1 for nearly everything.

tr for the small stuff:

tr 'a-z' 'A-Z' < file          # case conversion
tr -d '\r' < file              # strip CRLF line endings
tr -s ' '                      # squeeze repeated spaces into one
tr ':' '\n' <<< "$PATH"        # split a delimited string onto lines

Topic 6: Performance

The order of your filters decides the runtime. Reduce the data as early as possible.

# Slow: sorts everything, then discards most of it
sort huge.log | grep ERROR | head

# Fast: grep first, sort only what survived
grep ERROR huge.log | sort | head

Other habits worth having:

export LC_ALL=C        # byte comparison instead of locale-aware collation

On a large sort or grep, setting LC_ALL=C can be several times faster — it skips Unicode collation entirely. Safe whenever you are handling ASCII data.

Try it yourself: Take an access log and produce the top 10 client IPs by request count, then rewrite it putting grep before sort and time both with time.

Common mistake: Piping uniq -c without sorting first. The output looks plausible — a long list of counts of 1 — and is wrong.