Reading Files & Streams Safely

while read line is the most-copied and most-broken loop in shell scripting. Each of its three defects is invisible with tidy input and appears the moment real data arrives.

Topic 1: The Correct Idiom, and Why Each Piece Exists

while IFS= read -r line; do
    printf '%s\n' "$line"
done < "$input_file"

Four defences, three characters of syntax:

Demonstrating each failure:

printf '  indented\nback\\slash\nno-newline' > test.txt

while read line; do echo "[$line]"; done < test.txt
# [indented]        ← leading spaces GONE
# [backslash]       ← backslash GONE
#                   ← last line MISSING entirely

The third failure is the nastiest. read returns non-zero at EOF, so a final line with no trailing newline is read into the variable and then discarded when the loop condition fails. Files produced by printf without a trailing \n, or truncated mid-write, hit this.

while IFS= read -r line || [[ -n "$line" ]]; do
    echo "[$line]"
done < test.txt
# [  indented]
# [back\slash]
# [no-newline]      ← all three intact

The || [[ -n "$line" ]] clause runs the body one more time when read failed but still put something in the variable. Include it whenever the input may not end in a newline.

Topic 2: Splitting Fields While Reading

read takes multiple variable names and splits the line across them on IFS:

while IFS=: read -r user _ uid gid _ home shell; do
    printf '%-12s %-6s %s\n' "$user" "$uid" "$shell"
done < /etc/passwd

Two conventions worth adopting:

• Set IFS on the read line itself. That scopes the change to that one command — no save-and-restore dance.
• Use _ for fields you do not want. It is a normal variable by convention, and it makes the field positions self-documenting.

The last variable absorbs the remainder. If a line has more fields than you named, everything left over lands in the final variable, separators included:

while read -r severity message; do      # message gets the whole rest of the line
    [[ $severity == ERROR ]] && echo "$message"
done < app.log

That is usually what you want for log parsing, and a surprise if you were expecting the extra fields to be dropped.

Topic 3: The Subshell Trap

count=0
cat access.log | while read -r line; do
    (( count++ )) || true
done
echo "$count"          # 0

Every stage of a pipeline runs in a subshell, so the loop increments a copy. Use redirection or process substitution instead:

# From a file -- simplest
while read -r line; do (( count++ )) || true; done < access.log

# From a command -- process substitution, no subshell
while read -r line; do (( count++ )) || true; done < <(journalctl -u nginx)

# From a variable -- here-string
while read -r line; do (( count++ )) || true; done <<< "$captured_output"

This is also why cat file | is redundant: < file is one fewer process and avoids the trap. ShellCheck flags it as SC2002, the “useless use of cat”.

Topic 4: Filenames Are Not Text

A POSIX filename may contain any byte except NUL and / — including spaces, newlines, quotes, and leading dashes. Any loop that splits on whitespace is wrong for the general case.

# WRONG in three separate ways
for f in $(ls *.log); do ... done
for f in $(find . -name '*.log'); do ... done

The null-delimited idiom:

while IFS= read -r -d '' file; do
    process "$file"
done < <(find /var/log -name '*.log' -print0)

-print0 terminates each name with NUL, -d '' tells read to split on NUL. Since NUL cannot appear in a filename, this is unambiguous for every possible name.

Into an array, on bash 4.4+:

mapfile -d '' files < <(find /var/log -name '*.log' -print0)
echo "${#files[@]} files"

Or avoid the loop entirely:

find /var/log -name '*.log' -exec gzip {} +      # + batches; \; runs one per file
find /var/log -name '*.log' -print0 | xargs -0 -P4 gzip

-exec ... + is usually fastest and needs no pipeline at all. xargs -0 -P4 adds parallelism when the work is CPU-bound.

When a glob suffices, use a glob:

shopt -s nullglob
for f in /var/log/*.log; do
    process "$f"
done

Globs never split and never mis-parse. nullglob stops the loop running once with a literal *.log when nothing matches.

Topic 5: Reading Structured Input

CSV, when it is genuinely simple:

while IFS=, read -r name env port; do
    [[ $name == \#* ]] && continue        # skip comments
    [[ -z $name ]] && continue            # skip blanks
    echo "$name → $env:$port"
done < hosts.csv

This works only for CSV with no quoted fields and no embedded commas. Real CSV has both, and shell has no parser for it — that is the point at which the Google style guide’s advice applies and you reach for Python.

JSON — always jq, never regex:

jq -r '.items[] | "\(.name)\t\(.status)"' response.json |
while IFS=$'\t' read -r name status; do
    echo "$name is $status"
done

Emitting tab-separated output from jq and splitting on tab is the clean handoff: jq handles the parsing, shell handles the iteration.

Skipping a header:

{
    read -r _                      # consume the header line
    while IFS=, read -r a b c; do
        echo "$a"
    done
} < data.csv

Grouping with { } gives both reads the same stdin, so the first read eats the header and the loop starts on real data.

Topic 6: Reading From Multiple Sources

A loop that reads a file while also running a command that reads stdin will find its input consumed:

while IFS= read -r host; do
    ssh "$host" 'uptime'          # ssh reads stdin -- eats the rest of your list
done < hosts.txt

The first iteration works, then ssh drains hosts.txt and the loop ends. Two fixes:

while IFS= read -r host; do
    ssh -n "$host" 'uptime'       # -n redirects ssh's stdin from /dev/null
done < hosts.txt

# General form: give the loop its own descriptor
while IFS= read -r -u 3 host; do
    ssh "$host" 'uptime'
done 3< hosts.txt

The second is the general answer for any stdin-consuming command — ffmpeg, mysql, and anything interactive have the same behaviour.

Try it yourself: Write a loop over a hosts file that runs ssh without -n and watch it process exactly one host. Then add -n and confirm it processes all of them.

Common mistake: Reaching for for line in $(cat file) because it reads more naturally. It splits on every space, not every newline, so a log line becomes a dozen iterations. while IFS= read -r is the only form that iterates over lines.